US6055330A - Methods and apparatus for performing digital image and video segmentation and compression using 3-D depth information - Google Patents

Methods and apparatus for performing digital image and video segmentation and compression using 3-D depth information Download PDF

Info

Publication number
US6055330A
US6055330A US08723467 US72346796A US6055330A US 6055330 A US6055330 A US 6055330A US 08723467 US08723467 US 08723467 US 72346796 A US72346796 A US 72346796A US 6055330 A US6055330 A US 6055330A
Authority
US
Grant status
Grant
Patent type
Prior art keywords
object
video
signal
depth
information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
US08723467
Inventor
Alexandros Eleftheriadis
Dimitris Anastassiou
Shif-Fu Chang
Shree Nayar
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
COLUMBIA UNIVERSITY TRUSTEES OF IN CITY OF NEW YORK
Columbia University of New York
Original Assignee
Columbia University of New York
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Grant date

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00Image coding, e.g. from bit-mapped to non bit-mapped
    • G06T9/007Transform coding, e.g. discrete cosine transform
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/124Quantisation
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/136Incoming video signal characteristics or properties
    • H04N19/14Coding unit complexity, e.g. amount of activity or edge presence estimation
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/146Data rate or code amount at the encoder output
    • H04N19/149Data rate or code amount at the encoder output by estimating the code amount by means of a model, e.g. mathematical model or statistical model
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/146Data rate or code amount at the encoder output
    • H04N19/15Data rate or code amount at the encoder output by monitoring actual compressed data size at the memory before deciding storage at the transmission buffer
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/146Data rate or code amount at the encoder output
    • H04N19/152Data rate or code amount at the encoder output by measuring the fullness of the transmission buffer
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/176Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/597Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/20Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video object coding

Abstract

Apparatus and methods for identifying one or more separate objects within depth information which corresponds to a fleid or a frame of video information are disclosed. In a preferred embodiment, an appratus includes an object map generation circuit for receiving depth information and for converting depth information into an object map to associate each pixel within the frame of video information with one of one or more regions of varying perceptual importance is disclosed. This preferred apparatus also includes a region masking circuit for masking the object map to generate one or more depth region masks indicative of pixels within the frame which substantially correspond to preselected regions of depth, and a video object selection circuit for identifying one or more separate objects within each of the one or more preselected regions indicated by each of the one or more region masks, such that each object associated with each depth region is identified as a separate object.

Description

BACKGROUND OF THE INVENTION

I. Field of the Invention

The present invention relates to techniques for digital image and video segmentation and compression, and more specifically, digital image and video compression techniques which make use of three dimensional shape information as part of the video segmentation and compression processes.

II. Description of the Related Art

In recent years, numerous techniques for digital image and video compression have been introduced. Current image/video compression standards such as JPEG, H.261, MPEG-1, MPEG-2, H.263, which do not have the inherent capability to encode semantically different visual objects separately, treat content as a two or three dimensional (2-D space plus time) array of pixels on which redundancy reduction techniques are applied. In such standard techniques, the Discrete Cosine Transform ("DCT") is utilized in order to transform 8×8 blocks of pixel data into the DCT domain where quantization is more readily performed. Run-length encoding and entropy coding (i.e., Huffman coding) are applied to the quantized bitstream to produce a compressed bitstream which has a significantly reduced bit rate than the original uncompressed source signal. The process is assisted by additional side information, in the form of motion vectors, which are used to construct frame or field-based predictions from neighboring frames or fields by taking into account the inter-frame or inter-field motion that is typically present. As of the date of preparation of this patent document, numerous personal and commercial applications, such as satellite television, digital-video-disks ("DVDs"), and computer video adapters utilize one or more of the above-listed techniques in order to enhance the video capability of the application. Numerous additional applications are contemplated, especially in the case of MPEG-2.

Other more recently developed image/video compression techniques, such as the MPEG-4 standardization effort by the ISO/IEC JTC1/SC29/WG11 group, posses the inherent capability to encode semantically different visual objects separately. MPEG-4 utilizes an object-based structure to provide for the independent coding of objects of the same frame or sequence and the capability to incorporate synthetic audio and graphics objects. A complete description of the MPEG-4 compression technique, including the MPEG-4 System Description Language (MSDL), is contained in ISO document ISO/IEC JTC1/SC29/WG11 N1277 (July 1996), the disclosure of which is incorporated by reference herein. While most current video compression techniques are frame or field-based, MPEG-4 provides a flexible and extensible compression technique which is not limited to field or frame based compression. Thus, with the advent of frame or field-based compression techniques such as MPEG-2, and object-based compression techniques such as MPEG-4, there has been a revolution in the art of video compression during the early and mid 1990's.

Concurrently with this video compression revolution, there have also been great strides in the art of video capture. In particular, optical sensors that are capable of delivering depth information for a scene in real time, i.e., a "depth camera" are now feasible. Such a device is capable of producing a regular video signal in digital form at 30 or 25 frames per second (e.g., a NTSC or PAL signal), and also producing at the same frame rate an estimate of the distance of the pixels of the captured image from a fixed point or plane, such as the focal center of the camera. Such distance or three-dimensional shape information is also delivered by the sensor in digital form. One such sensor has been described in both active configuration, where a special illumination pattern is required, and in a passive configuration in Shree Nayer et al., "Real Time Focus Range Sensor," Proceedings Int'l Conf. Computer Vision pp. 995-1001 (IEEE 1995), the disclosure of which is incorporated by reference herein.

There have been several attempts to make use of three dimensional shape information as part of the video compression process. For example, J. J. D. van Schalkwyk et al., "Low Bitrate Video Coding with Depth Compensation," IEEE Proceedings: Vision, Image and Signal Processing, Vol. 141, No. 3, pp. 149-53 (1994), discloses a video compression technique which makes use of three-dimensional depth information generated by a depth-sensing algorithm in order to separate moving objects from static background. In the disclosed technique, a stereo algorithm is used to extract depth information from the scene and to locate the edges of objects within the scene. The form of the object is taken on a sub-block basis as the area covering the object as a whole. Global motion vectors, which represent the motion or displacement of the object as a whole from one frame to another, are generated by correlating the object's position vectors of the previous and present frames. During the prediction process, the global motion vectors are used to displace objects in a previous frame of data in order to generate a "globally compensated frame" of data that may be used as a first-order prediction of the present frame. The globally compensated frame replaces the past frame of data in a standard field or frame-based compression process in order to generate a more accurate representation of the scene.

In M. A. H. Venter et al., "Stereo Imaging in Low Bitrate Video Coding," COMSIG 1989--Proceedings South Africa Conference [of] Communication Signal Processing, pp. 115-118 (IEEE Jun. 23, 1989), two video compression techniques which make use of three dimensional depth information retrieved by a stereo imaging camera are disclosed. In the first technique, Venter et al. disclose the use of depth information to generate an "object motion vector" as a check on the accuracy of motion vectors which are generated in a normal coding algorithm, i.e., if a generated motion vector substantially differs from the object motion vector, it is assumed to be incorrect and is therefore replaced by the object motion vector. In the second technique, the reference proposes that depth information can be used to create a three-dimensional model of a moving object in a scene, e.g., the head and shoulders of a person, which can be reoriented and used for image prediction by projecting the three-dimensional model onto a two-dimensional image plane.

In Bernd Girod, "Image Sequence Coding Using 3D Scene Models," Proceedings of SPIE--The International Society for Optical Engineering, Vol. 2308, pp. 1576-1591 (SPIE 1994), two video compression techniques which make use of three dimensional depth information retrieved by a depth-sensing camera are also disclosed. In the first technique, Girod discloses the "implicit" use of depth information to generate a matrix which represents the translational and rotational movement of a rigid body, that is used during block matching as a constraint on the motion vector field to yield more accurate motion compensation. In the second technique, depth information is explicitly used to generate a model of a moving object, e.g., a head, which is transmitted to a receiver along with preselected facial motion parameters (e.g., mouth opening, head rotation, etc.) in order to effectuate facial animation.

The above-mentioned prior art techniques fail to adequately bridge the gap between current field or frame based video compression techniques and three-dimensional video retrieval techniques, because in each of the prior art techniques, three-dimensional shape information is used only in a tangential manner, e.g., in order to generate a first-order prediction of a frame of video data or as a check on the accuracy of motion vectors, rather than in a direct manner. Moreover, where the prior art techniques discuss the use of three-dimensional shape information in the context of object based compression, the do so only to create a three-dimensional model of a moving object, rather than in a direct manner to assist in the compression process. Thus, there exists a need for a technique which directly utilizes three-dimensional shape information in the video compression process, both in the case of field or frame based compression techniques and in the case of object-based compression techniques.

SUMMARY OF THE INVENTION

An object of the present invention is to provide a technique which directly utilizes shape information in the video compression process, both in the case of field or frame based compression techniques and in the case of object-based compression techniques.

A further object of the present invention is to provide a technique which utilizes depth information generated by a depth-sensing camera in a field or frame based video compression process by enabling selective bit-allocation during the compression process.

An additional object of the present invention is to provide a technique which utilizes depth information in an object based video compression technique to accurately represent the object during compression.

A still further object of the present invention is to provide a technique that utilizes depth information to assign perceptual significance to different objects in a frame according to the objects' positions in space.

Yet another object of the present invention is to provide a modified MPEG-2 encoder which utilizes three-dimensional shape information by quantizing objects with low perceptual importance at a high granularity, and objects with high perceptual importance at a low granularity.

Yet a further object of the present invention is to provide a modified MPEG-4 encoder which utilizers three-dimensional shape information as a robust means to perform segmentation of content into individual objects.

In order to meet these and other objects which will become apparent with reference to further disclosure set forth below, the present invention provides an object segmentation circuit for receiving depth information which corresponds to a frame of video information and for identifying one or more separate objects within the frame of video information, and includes an object map generation circuit for converting the depth information into an object map in order to associate each pixel within the frame of video information with one of one or more regions of varying perceptual importance within the frame. The object segmentation circuit also includes a region masking circuit for masking the object map to generate one or more depth region masks that are indicative of pixels within the frame which substantially correspond to preselected regions of depth, and a video object selection circuit for identifying one or more separate objects within each of the one or more preselected regions indicated by each of the one or more region masks, so that each object associated with each depth region is identified as a separate object.

Preferably, the object map generation circuit includes a histogram generation circuit for computing a histogram of the depth information in order to provide the number of pixels which have a predetermined depth value for a range of predetermined values, a first logic circuit for setting all values in the histogram which are below a predetermined threshold value to zero in order to generate a clipped histogram, a second logic circuit for scanning the clipped histogram to find boundaries of n regions with n different threshold depth values, and a variable step quantization circuit for quantizing the depth information based on the n different threshold values in order to generate the object map.

The histogram generation circuit itself may include a buffer for receiving and temporarily storing the depth information, a memory for receiving the depth information from the buffer as memory addresses and for storing histogram values as memory addresses, and a logic circuit for reading a histogram value from the memory at an address location, updating the histogram value, and providing the updated histogram value to the memory at the address location.

In one preferred embodiment, the present invention provides an apparatus for encoding fields or frames of video information in the form of a two dimensional array of pixels, using a depth component of each of those pixels to enhancing encoding, and includes an encoder for receiving frames or fields of video information and generating a compressed video signal from the received frames or fields of video information, where the encoder includes a multi-mode quantizer for quantizing data which corresponds to a portion, such as a macroblock, of the fields or frames of video information. A depth sensing camera, capable of generating in real-time both frames or fields of video information and depth information which corresponds to the video information, is preferably included in such apparatus.

This embodiment advantageously includes an object map generation circuit as well as a rate controller which provides a signal, responsive to the object map generated by the object map generation circuit, to the multi-mode quantizer to select a quantization mode such that the selected quantization mode is reflective of the perceptual importance of the regions indicated by the object map. This embodiment may be assisted by a macroblock labeling circuit, connected between the object map generation circuit and the rate controller, for assigning a current macroblock of video data to one of the regions of varying perceptual importance and for providing a signal indicative of said assigned region to the rate controller.

A video buffer having a preselected storage capacity may be coupled to the encoder and to the rate controller in order to receive and temporarily store the generated compressed video signal, and to provide a signal indicative of an overflow condition to the rate controller, so that the signal provided by said rate controller to the multi-mode quantizer is also responsive to the overflow signal.

In one especially preferred arrangement, the encoder generates an output signal when compressed video data is output to a video buffer, and the macroblock labeling circuit generates a signal indicative of a target bit rate associated with an assigned region. This arrangement beneficially includes a clock signal generating circuit for providing a clock signal to the buffer so that the buffer outputs a predetermined amount of compressed video signal in response to the clock signal, and a counter for counting the number of bits that are in a virtual buffer associated with the video buffer by adding to the count in response to the encoder output signal and subtracting from the count in response to the clock signal and the target bit rate signal in order to determine the occupancy of the virtual buffer. The counter also provides a virtual buffer occupancy signal indicative of the count to the rate controller so that the signal provided by the rate controller to the multi-mode quantizer is also responsive to the virtual buffer occupancy signal.

In order to avoid sudden changes in compression granularity, the rate controller may advantageously include a buffer size logic circuit for generating a buffer size modulation signal whenever the assigned region is different from an immediately proceeding assigned region, so that the signal provided by the rate controller to the multi-mode quantizer is responsive to the buffer size modulation signal and the selected quantization mode is reflective of the perceptual importance of the regions indicated by the virtual buffer capacity as modulated by the buffer size modulation signal and as constrained by the storage capacity of the video buffer.

In an alternative preferred arrangement, the present invention provides an apparatus for performing object-based encoding of video information using a depth component of the video information to enhance encoding, an includes an object segmentation circuit and an object-based encoder. The encoder receives the frame of video information which corresponds to the received depth information and one or more object identification signals generated by the object segmentation circuit, and generates a compressed video signal representing a portion of the video information which substantially corresponds to the one or more objects identified by the one or more object identification signals.

The present invention also provides a method for identifying one or more separate objects within depth information which corresponds to a frame of video information. The method should include the steps of receiving the depth information; converting the received depth information into an object map in order to associate each pixel within the frame of video information with one of one or more regions of varying perceptual importance within the frame; masking the object map to generate one or more depth region masks indicative of pixels within the frame which substantially correspond to preselected regions of depth; and identifying one or more separate objects within each of the one or more preselected regions indicated by the one or more region masks.

The converting step preferably includes computing a histogram of the received depth information to thereby provide the number of pixels which have a predetermined depth value for a range of predetermined values; setting all values in the histogram which are below a predetermined threshold value to zero to generate a clipped histogram; scanning the clipped histogram to find boundaries of n regions with n different threshold depth values; and quantizing the depth information based on the n different threshold values.

Favorably, the identifying step can include scanning a preselected depth region mask until a pixel with a nonmasked value is found; searching neighboring pixels within the mask to find any other neighboring pixels with nonmasked values; repeating the searching step until no neighboring pixels have a nonmasked value to identify all neighboring found pixels with nonmasked values as a video object plane which corresponds to an object within the depth region; masking the object from the depth region mask; repeating the above until all pixels within the depth region mask are masked to thereby identify one or more video object planes within the received frame of video information; selecting one of the one or more video object planes, and one of one or more video object planes associated with an immediately preceding frame of video information; comparing the selected video object plane and the selected previous frame video object plane to determine a depth difference therebetween; repeating the comparing step after selecting a different one of the one or more previous frame video object planes unless all of the one or more previous frame video object planes have been selected; and assigning the selected video object plane to a video object which corresponds to one of the one or more previous frame video object planes for which a depth difference therebetween is minimized as compared to all of the determined depth differences.

The accompanying drawings, which are incorporated and constitute part of this disclosure, illustrate a preferred embodiment of the invention and serve to explain the principles of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of a depth-sensing camera;

FIG. 2 is a system diagram of a typical MPEG-2 type encoder;

FIG. 3 is system diagram showing the conversion of a variable bit rate encoder into a constant bit rate encoder;

FIG. 4 is a system diagram of a constant bit rate encoder in accordance with the invention;

FIG. 5 shows an object map generation circuit useful in the FIG. 4 encoder;

FIG. 6 shows a histogram generation circuit useful in the FIG. 5 object map generation circuit;

FIG. 7 shows a histogram clipping circuit useful in the FIG. 5 object map generation circuit;

FIG. 8 shows a boundary searching circuit useful in the FIG. 5 object map generation circuit;

FIG. 9 shows a quantization circuit useful in the FIG. 5 object map generation circuit;

FIG. 10 is a system diagram of a constant bit rate encoder in accordance with a preferred aspect of the invention;

FIG. 11 shows a macroblock labeling circuit useful in the FIG. 10 encoder;

FIG. 12 shows a target bitrate generation circuit useful in the FIG. 10 encoder;

FIG. 13 shows a buffer size modulation signal generation circuit useful in the FIG. 10 encoder;

FIG. 14 is a schematic diagram of video objects and video object planes within a frame of video information;

FIG. 15 is a system diagram of an object-based encoder in accordance with the invention;

FIG. 16 is a system diagram of a typical MPEG-4 Encoder;

FIG. 17 shows a video object segmentation circuit useful in the FIG. 15 encoder;

FIG. 18 is a flow chart of a method of encoding video information including the preferred steps for performing rate control in accordance with an aspect of the invention; and

FIGS. 19a and 19b are flow charts of a method of encoding video information including the preferred steps for performing video object separation in accordance with an aspect of the invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring to FIG. 1, there is shown an operating model of a depth sensing camera 100. The camera 100 views a natural three-dimensional scene 110 which includes objects 111 and 112 at different distances or depths from the camera. The camera 100 produces two digital outputs, one 101 consisting of a standard digital video or image signal V, and a second 102 consisting of depth information D for the pixels contained in the normal video signal 101. The camera 100 may be constructed in accordance with the above discussed Nayar et al. article, or may be any other commercially available camera which produces both a standard digital video output and a depth information output.

The signal 101 preferably contains eight bits per pixel per component, in YUV 4:2:0 chroma format, although other color spaces or chroma formats could be used as well. The signal 102 preferably contains a 16-bit linear PCM representation for the depth information, although other formats, including floating point or logarithmic PCM can be used, providing different tradeoffs between complexity and accuracy.

In FIG. 1, the camera 100 will generally output a higher D value for pixels in object 111 compared to the values produced for pixels in object 112, as the object 111 is positioned farther away from the camera compared to object 112. The camera's output for a frame k can then be mathematically described by the two signals:

v.sub.k (i,j)ε{0,1,.sub.--,255}0≦i≦N-10≦j≦M-1(1)

and

d.sub.k (i,j)ε(0,∞)0≦i≦N-10≦j≦M-1(2)

where N and M are the width and height of a frame respectively. Equation (1) ignores the chrominance components for simplicity of notation; as the different components are subjected to substantially identical processing steps (with the exception of motion estimation) so there is no need to individually identify them.

In FIG. 2 there is shown a system 200 for a standard field or frame based video compression technique such as H.261, MPEG-1, MPEG-2, or H.263. A normal uncompressed video signal is input to macroblock separator/adder 240 and to motion estimation circuit 210. A discrete consign transform ("DCT") 250 is applied to macroblock data from 240. To achieve greater compression efficiency, DCT coefficients are then quantized 251, Zig-Zag run-length encoded 252 and variable length coded 253. Quantized DCT coefficients are also inverse quantized 255 and inverse discrete consign transformed 256 to provide past frame video data to the motion estimation circuit 210 concurrently with present frame video data. The motion estimation circuit 210 uses the past and present frame video data, which may be stored in frame store 220, to generate motion vectors which are encoded 211, 212 and multiplexed 260 with the compressed DCT coefficients. In the case of MPEG-2, a complete description of the encoder architecture is provided in U.S. Pat. No. Re. 35,093, the disclosure of which is incorporated by reference herein.

In all cases, the encoder 200 has the flexibility of selecting a quantization mode in the quantizer 251 that is used in the compression of a particular block of video data. The selection of a fine quantization mode will produce an accurate bitstream representation for the block subject to compression, while a cowers mode will produce a more granular representation of the block. This decision affects a 2×2 array of blocks of the luminance signal, and a 2×8 array of corresponding luminance blocks, i.e., a macroblock.

The flexibility to select quantization mode at a fine scale is the means through which the encoder can effect rate control, thus producing constant bit rate ("CBR") bitstreams even though entropy (variable length) coding is used in the video compression process. Since each macroblock of data includes a header in addition to PCT coefficients for the relevant luminance and chrominance blocks, the use of a different quantizer step size can be signaled in the compressed bitstream in the headers associated with each macroblock.

Referring to FIG. 3, rate control may be performed by placing a buffer 320 at the output 310 of the variable bit rate (VBR) encoder 200. In turn, the buffer 320 outputs compressed data at a constant rate which is dependant on the bandwidth of the channel which is accepting data from the buffer. The buffer's occupancy Bmax and other, possibly signal dependent, parameters are then taken into account by a rate controller 340 in order to decide the quantizer step size in quantizer 251 to be used for subsequent macroblocks so that buffer overflow or underflow does not occur.

The buffer regulation function f(.) which is calculated by the rate controller 340 can take various forms. The most simple is a "linear" relationship with the buffer occupancy B. Assuming that the quantizer factor Q can take values in the set {1, 2, . . . , 31}, then this relationship takes the form: ##EQU1## More sophisticated functions are also well-known in the art, such as RM-8 rate control utilized in H.261, or TMN-5 rate control utilized in H.263.

Although the above is directed to a constant bit rate encoder, rate control is also usable in a purely VBR encoder to provide higher quality for some image areas of perceptual importance (e.g., a face), and less for areas that have smaller significance (e.g., background areas). In the remainder to this specification, the term rate control is used without discriminating whether or not a CBR or VBR encoder is used. In a VBR encoder, the buffer could be eliminated and the rate control function would depend only on parameters other than buffer occupancy. Alternatively, the buffer output rate can be made into a function of time, and may be known a priori or provided dynamically from the network.

In accordance with one embodiment of the present invention, depth information provided by the camera 100 is utilized by the rate control circuit 340 to partition the input frames into several different areas that are associated with different objects. These objects are assigned perceptual significance according to their position in space as well as their size. This information is in turn used in an encoder so that more bits are spent in the more significant objects, thus resulting in higher visual quality, and less bits are spent in the least significant objects such as the background.

With reference to FIG. 4, such an embodiment of the present invention is now described. A depth-sensing camera 100 provides consecutive frames of video data via cable 101 to CBR encoder 400, and for each frame of video data, an array of depth values via cable 102 to an object segmentation circuit 500. Each value in the array corresponds to the depth (or a measure thereof) of the corresponding pixel in the video signal. Object Segmentation circuit 500 converts the depth signal into an object map (O), i.e., an array of the same size in which each pixel is associated with a particular object (or the background), as more fully explained below, and provides the object map to rate controller 440 of the CBR. As described below, rate controller 440 provides a quantization mode selection signal to VBR encoder 200 to select a quantization step size depending both on the perceptual importance of the object from which data is being encoded, as well as on the current occupancy of the buffer 420.

Referring to FIG. 5, an object segmentation circuit 500 which is suitable for use in the FIG. 4 system is now described. The circuit includes a histogram generating circuit 510, a histogram clipping circuit 520, a boundary computing circuit 530, and a variable step quantizer 550.

The histogram generating circuit 510 is shown in further detail in FIG. 6. The array of depth values, or depth map, generated by camera 100 are fed into a 16-bit wide buffer 512 via bus 511. The depth value for each pixel in the array is then input via bus 513 as the address of a 16×64K RAM 514 that contains the histogram values. An adder 516 reads the current value for the indicated RAM address via data line 515, adds the value 1, and stores the revised value back in the RAM 514 (the clocking details for the read/write stages are not shown). This creates a function which gives the number of depth map pixels of a certain depth value for each depth value that is present within the depth map.

FIG. 7 shows the architure of histogram clipping circuit 520. The clip circuit 520 accesses histogram RAM 514 to clip histogram entries at a threshold value T so that histogram values which are less than T are set to 0, while higher values remain unchanged. The threshold value should be experimentally determined depending on the object distances which are in the scene viewed by camera 100. Each address of RAM 514 is accessed by the generation of an address by address generator 527, which is responsive to a system clock signal 526 (clock not shown). The stored histogram value at that address is fed to a comparator 522 via line 521 and is compared with the threshold value T stored in register 523 and accessed via line 524. If the value is higher than T, it is returned to the RAM unchanged. If the value is equal to or below T, a zero is output and stored in the RAM at the same position.

The details of boundary computing circuit 5303 are shown in FIG. 8. The circuit uses the thresholded histogram to identify depth points that designate ares between objects in the scene viewed by camera 100. The circuit assumes that these points correspond to the mid-points of areas of all zeros in the thresholded histogram.

Data in RAM 514 is again addressed by the address generator 527 and sent to comparator 533 via line 531, where it is compared with the value 0. If the data is a zero, a logical 1 is output by comparator 533; if the data is not a zero, a logical 0 is output. The output value is then stored in register A 534, while the previous value stored in register A is shifted to Register B 535.

When Register A contains a 1 and register B contains a 0, a zero value of a region in the histogram has been identified. The values are combined via inverter 537 and AND gate 538 to generate a write-enable signal for register T1 540, which will hold the depth value for the beginning of the current zero-value region in the histogram. When Register A contains a 0 and Register B contains a 1, the last 0 value of the region has been identified. In such case, inverter 537 and AND gate 539 generate a write-enable signal for register T2 541, which indicates that the circuit can proceed and compute the boundary value. The boundary value is computed by adder 542, which adds the contents of T1 and T2, and shifter 543, which divides the result by two. The boundary value is stored in a Boundary Value RAM 551 (FIG. 9). The write enable signal for T2 also acts as a write enable signal for a boundary value RAM, which is described below.

In this manner, the clipped histogram is scanned to find contiguous regions where the clipped histogram has the value 0. For the i-th such region, the threshold ti is set to the middle point of the region. At the end of this process there are n different thresholds, which will separate n objects plus the assumed background.

Referring to FIG. 9, variable step quantizer 550 is shown in detail. Depth values 501 from a depth frame buffer (not shown) are fed into comparator 553 and are compared with corresponding boundary values retrieved from Boundary Value RAM 551 by address generator 552. If the depth value is less than the boundary value, the comparator outputs a 1; otherwise a 0 is output. The comparator output, a sign bit, is used as a write enable input for quantized value register 554. The register 554 is loaded with the address of the first boundary value for which the current depth value is smaller, i.e., the desired quantized value which shall be used as an object identification. The content of register 554 is then loaded into an object map RAM 1101 (FIG. 11) to associate the depth value with a particular object.

In order to prohibit further writes of the quantized value register 554 during the current depth value cycle, the sign bit output of comparator 553 also acts as a reset signal for address generator 552. Upon being reset, the address generator will generate the next address in RAM 551 for comparison with a corresponding depth value 501 in comparator 553.

In this manner, the quantizer 550 quantizes the original depth pixels map using a variable step size quantizer with the decision levels ti identified by the boundary computation circuit 530 and stored in Boundary Value RAM 551. This process generates an "object map," i.e., an association of each pixel with a particular object, which are labeled 0 through n, 0 corresponding to the nearest object and n to the farthest (background). The use of 16-bit values to represent object map entries allows for up to 65,536 different objects to be present in the same image. Of course, more can be accommodated with longer word sizes (e.g., 24 bits or more).

The object map is provided to the rate control circuit 440 so that quantization decisions for the current frame can be made. While the above process assumes that pixels at the same distance from the camera 100 belong to the same object, more sophisticated techniques can be utilized, such as by taking into account the spatial separation between picture areas that have the same distance (depth) from the camera.

Further, while the simple technique described herein provides a good tradeoff between accuracy and complexity, the depth map is simply an image itself, and therefore numerous classical image segmentation techniques are applicable to the present invention. Some of those techniques are discussed in R. C. Gonzalez et al., "Digital Image Processing" (Addison Wesley 1993); B. K. P. Horn, "Robot Vision" (MIT Press 1986); and W. K. Pratt, "Digital Image Processing" (Wiley Interscience 1991). Those skilled in the art may utilize such alternative techniques to perform object segmentation on the depth map.

FIG. 10 is a system diagram of a constant bit rate encoder in accordance with a preferred aspect of the invention. The object map generated by object segmentation circuit 500 is received by macroblock labeling circuit 1100. Since the encoder 200 splits each frame of video information received from camera 100 into macroblocks and quantizes DCT coefficients on a macroblock basis, it is desirable to assign each macroblock of video data to a specific object, or in the case of the simple segmentation technique described above, to a region which contains one or more objects at the same depth from the camera 100. Thus, even if the macroblock includes pixels from more than one object or region, it will be assigned to one object or region, e.g., to the object or region which is nearest to the camera 100 and hence the most perceptually important. Once such an assignment has been made by macroblock labeling circuit 1100, the rate controller 1040 can select an appropriate quantizer step size for the entire current macroblock.

Referring to FIG. 11, the macroblock labeling circuit 1100 includes object map RAM 1101 which contains an object identifications for each pixel in the macroblock (a 16×16 array of pixels). Because the object map is constructed by assigning the lowest object identification to the nearest object, the circuit simply must search for the lowest object identification for the macroblock, i.e., the smallest number within the 16×16 array of numbers held in RAM 1101.

To accomplish this, minimum value register 1120 is set to its maximum value (sixteen 1's for sixteen bit words) at the beginning of each cycle. Comparator 1110 compares each pixel identification, retrieved from object map RAM 1101 when addresses are generated by address generator 1102, with the value stored in minimum value register 1120. If the current value is smaller than the value held in register 1120, comparator 1110 outputs a logical 1 which acts as a write enable signal to register 1120 so that the current value is stored in register 1120. If the current value is equal to or larger than the value stored in comparator 1110, no write enable signal is generated. At the end of the cycle, the minimum value register will contain the value of the lowest object identification in the macroblock. This is output to rate controller 1040 as the macroblock label 1130.

For VBR coding, the macroblock labels can be directly used for rate control by associating particular quantizer step sizes with each object. Typically, the nearest object is assigned a finer quantizer (more bits, hence higher quality), while the background (farthest object) is assigned a coarser quantizer (less bits and less quality). In this case, quantizer selection is simply a lookup operation into a table which indexes the possible object identifications generated by macroblock labeling circuit 1100. The encoder can also employ techniques to "smooth out" quantizer differences at object boundaries by gradually changing the quantization step while entering or exiting an object. An exemplary smoothing process is outlined below in connection with the CBR encoder.

For CBR coding, the rate controller 1040 must additionally regulate quantizer selection so that the output buffer 1020 neither overflows nor underflows, as explained above. Since the total number of bits per second which may by output is now fixed, object sizes become important. A technique for performing area-selective rate control when the object locations are known has been described in Eleftheriadis et al., "Automatic Face Location Detection for Model-Assisted Rate Control in H.261-Comparable Coding of Video," 7 Signal Processing: Image Communication 435-55 (1995), the disclosure of which is incorporated by reference herein.

In accordance with that technique, each object is associated with a particular target average bit rate Ri, i=1, . . . , n-1, except for the background (object n). In order to maintain the given total average rate R necessary to prevent buffer overflow, the background rate is determined according to the formula: ##EQU2## where αi is the proportion (from 0.0 to 1.0) of the pixels in the frame that belong to object i. By definition, ##EQU3## While it is possible that Rn is negative, this simply has the effect of assigning as coarse quantization as possible to the background, and may result in less average bits per second per object than the target bit rates Ri indicate.

FIG. 12 shows a target bitrate generation circuit useful in the FIG. 10 encoder. The hardware of the target bitrate generation circuit 1200 may be included in the macroblock labeling circuit 1100 by coupling minimum value register 1120 (FIG. 11) to Object Bit Rate RAM 1210 so that the macroblock labels 1130 are sent to RAM 1210. RAM 1210 contains the individual desired target bit rates Ri. The size of RAM 1210 will depend on the number of bits available for the macroblock labels, so that for example, if 8-bit macroblock labels are used, RAM 1210 will require 28 memory locations (all of the memory locations will be needed only if there are a large number of objects in the scene 110 at different depths from camera 100).

To avoid floating point calculations, the bit rates are indicated as integer values that are subsequently scaled. Each address of RAM 1210 corresponds to a particular object label, so that the macroblock labels 1130 are directly used as memory addressed for RAM 1210. Upon being accessed by a macroblock label 1130, RAM 1210 will output 1220 the corresponding target bit rate Ri.

Referring again to FIG. 10, the rates Ri are used in a buffer regulation process that uses a technique of buffer rate and buffer size modulation. In order to realize this technique, a "virtual buffer" having an occupancy which is determined by a virtual buffer counter 1030 is added to the circuit in addition to physical buffer 1020. While the physical buffer 1020 continues to function to output compressed video data and to track overflow (so that a macroblock skip flag 1021 can be sent to encoder 200 if necessary to avoid buffer overflow), the virtual buffer is used in the quantizer selection process.

The regulation process for the physical buffer 1020 is governed by the following equations:

B.sub.i =B.sub.i-1 +b.sub.i -r                             (6)

Q.sub.i =f(B.sub.i-1)                                      (7)

where Bi is the buffer occupancy after coding macroblock i, bi is the number of bits used to code macroblock i, r is the number of bits per second per macroblock (equal to R over the number of macroblocks in a frame), and Qi is the quantizer step size selected for quantizing macroblock i. As discussed above, several forms are possible for the function f(.), one of which is given in Eq. (3).

In the embodiment shown in FIG. 10, Eq. (6) is only used to monitor the actual buffer 1020 occupancy so that macroblock skipping can be effected by sending flag 1021 to encoder 200. The quantizer selection is performed according to the virtual buffer as discussed below.

A system clock 1050 generates a system clock signal which causes buffer 1020 to output a predetermined amount of compressed video data. The clock signal is also applied to virtual buffer counter 1030 so that the counter 1030 can be incremented down by the predetermined amount. Likewise, the encoder 200 sends a macroblock bit count signal to counter 1030 whenever a macroblock of video data is encoded and passed to buffer 1020 so that the counter 1030 can be incremented up by the macroblock count. Counter 1030 also receives the target bit rate signal generated by the macroblock labeling circuit 1100 described above.

Buffer rate modulation is accomplished by manipulating the occupancy of the virtual buffer as follows. When a macroblock that has been assigned to object i is being encoded, the virtual buffer's occupancy, as determined by counter 1030, is modulated by the ratio Ri /R. This has the effect of increasing/decreasing the data output rate as from the perspective of the encoder, and results in appropriate quantizer modifications (finer/coarser respectively). Therefore, if a current macroblock (i) is assigned to an object k, Eq. (6) becomes ##EQU4## where B' indicates buffer occupancy for the virtual buffer, bi is equal to the number of bits used to encode the present macroblock (as indicated by the macroblock bit count signal), r is equal to the number of bits output by the physical buffer 1020, Rk is the target bit rate indicated by the target bit rate signal, and R is the average output bit rate to be maintained by the buffer 1020. Virtual buffer counter 1030 performs this calculation and provides a virtual buffer occupancy signal B' to rate controller 1040.

To assist the short-term convergence of the technique without affecting its long-term characteristics that are governed by Eq. (4), buffer size modulation is also used to smooth out quantization boundaries. The virtual buffer occupancy at object boundaries is modulated by rate controller 1040 by a factor greater/less than 1 depending on whether the encoder is entering an object region of finer/coarser quantization. The factor is stepped to 1 in a few macroblocks (i.e., its influence is faded out). This technique has the effect of immediately boosting the quantizer step size at lower/higher levels. With appropriate selection of modulation factors it is possible to achieve a smooth but fast change of quantization step sizes to the desired levels so that sharp quality differences between neighboring macroblocks are eliminated while the average bit rates Ri are maintained during encoding of macroblocks associated with the objects i.

Quantizer selection is performed by rate controller 1040 according to the formula ##EQU5## where si is a buffer size modulation factor for macroblock i. This factor is set to 1, except when crossing a region between macroblocks assigned to objects with different quantizer step sizes. When entering a region of lower step size, the factor is set to 1.4, and then decremented while the encoder 200 processes macroblocks of the same object with a step of 0.2 until it reaches the value 1. Similarly, when entering a region of higher step size, the factor is set to 0.6, and then incremented while the encoder 200 processes macroblocks of the same object with a step of 0.2 until it reaches the value 1.

FIG. 13 shows a buffer size modulation signal generation circuit which may be included in the rate controller 1040 to generate the factors si. Macroblock labels 1130 are input into register 1310. Whenever register 1310 receives a new macroblock labels the contents of register 1310 are shifted to register 1320, and the contents of registers 1310 and 1320 are compared in comparators 1330, 1340. If the contents of register 1310 is greater than the contents of register 1320, comparator 1330 outputs a "1"; otherwise comparator 1330 outputs a 0. If the contents of register 1310 is smaller than the contents of register 1320, comparator 1340 outputs a "1"; otherwise comparator 1330 outputs a 0. If registers 1310, 1320 have equal values, comparators 1330, 1340 output a 0.

The outputs of the comparators 1330 and 1340 are used as triggers for address generator 1350, which generates an address in modulation factor RAM 1360 that, in turn, contains the buffer size modulation factors si. As with Object Bit Rate RAM 1210, the contents of RAM 1360 are preferably integer values which are subsequently scaled by 128 in the quantizer selection portion of the rate controller 1040, in order to avoid fractional or floating point calculations. The buffer size modulation signal generation circuit 1300 is driven by the common system clock (not shown).

The address generator operates under the following state transition table:

              TABLE 1______________________________________T 1331    T 1341  Old State    New State                                 s.sub.i______________________________________0         0       0            0      1.00         0       1            2      1.40         0       2            3      1.20         0       3            0      1.00         0       4            5      0.60         0       5            6      0.80         0       6            0      1.00         1       X            4      0.61         0       X            1      1.4______________________________________

As shown in Table 1, when comparator 1330 outputs a 0 so that the value of trigger 1331 is a 0 and comparator 1340 outputs a 1 so that the value of trigger 1341 is a 1, the encoder 200 is compressing a macroblock of data that corresponds to a region in scene 110 that is closer to camera 100, and more bits will be allocated to that macroblock of data due to a higher target bit rate R value, as discussed above. In order to avoid an jump in quantization granularity, the address generator generates address value 4 which corresponds to si =0.6. In the next clock cycles, if the encoder 200 continues to compress macroblocks that belong to the same object, triggers 1331 and 1341 will both have the value 0 and the address generator will move from address 4 to address 5 to address 6 to address 0, as shown in Table 1, and the value of si will be faded out by increasing from 0.6 to 0.8 to 1.0.

Likewise, if when comparator 1330 outputs a 1 so that the value of trigger 1331 is a 1 and comparator 1340 outputs a 0 so that the value of trigger 1341 is a 0, the encoder 200 is compressing a macroblock of data that corresponds to a region in scene 110 that is farther to camera 100, and less bits will be allocated to that macroblock of data due to a lower target bit rate R value. In such circumstances, the address generator generates address value 1 which corresponds to si =1.4. In the next clock cycles, if the encoder 200 continues to compress macroblocks that belong to the same object, triggers 1331 and 1341 will both have the value 0 and the address generator will move from address 1 to address 2 to address 3 to address 0, as shown in Table 1, and the value of si will be faded out by decreasing from 1.4 to 1.2 to 1.0.

While the circuit described with reference to FIG. 13 assumes that macroblocks corresponding to objects closer to the camera 100 are assigned higher target bit rates R, other bit rate allocation schemes could be employed. A different state transition table could be employed under such circumstances. Alternatively, the architecture of circuit 1300 could be modified so that comparators 1330, 1330 compare the actual bit rate values assigned to the objects rather than the macroblock labels themselves. Such a modification could directly employ Object Bit Rate Ram 1210 by accessing the RAM 1210 on two successive clock cycles by the individual object labels, with the two outputs fed into comparators 1330, 1340 directly.

Referring again to FIG. 11, the factors si are provided via a line 1370 so that rate controller 1040 can calculate Eq. 9 and generate appropriate quantization selection signals Q. As will be appreciated by those skilled in the art, the calculation of Eq. 9 may be performed by a simple software routine run on a RAM (not shown) in rate controller 1040, or by appropriate micro-coded hardware. In this manner, the rate control circuit 1040 generates a quantization selection signal Q which is applied to the encoder 200 to select an appropriate quantization mode which is reflective of the perceptual importance of the object from which video data is presently being encoded in the encoder 200. The encoder 200 accordingly processes each macroblock in a frame in a left-to-right, top-to-bottom manner (except in the case of H.261, where the GOB structure must be followed), using the quantizer step size provided by the rate control circuit 1040.

While the foregoing has been directed to the use of depth information in field or frame based encoding techniques such as MPEG-2, the present invention has equal applicability to object-based video compression techniques such as MPEG-4. Although MPEG-4 is not unique in this capability, an embodiment of the present invention suitable for use in an MPEG-4 scheme is given to illustrate an application of the invention in an otherwise well-known object-based compression technique. As the MPEG-4 standard is subject to further modification as of the time of writing this specification, the discussion herein is based on the most recent version of the codec design, Verification Model 3.0.

The basic structure of MPEG-4 is similar to ITU H.263, except for the notion of a Video Object (VO) and a Video Object Plane (VOP). As illustrated in FIG. 14, VOs are independent objects 1410, 1420, 1430 that may exist in a scene, while VOPs are two dimensional images of arbitrary shape 1411, 1421, 1431. A frame is composed by overlaying VOPs; a sequence is composed of VOs that themselves consist of one VOP per frame.

VO compression is performed individually for each object. Since the VOPs of a VO have an arbitrary shape, both texture and shape compression must be utilized. Shape compression encodes the contour that defines the border of the VOP. Several well known techniques for shape coding, including chain coding, may be utilized.

With respect to texture coding, the techniques are almost identical to the ones used in the ITU H.263 specification, which in turn is similar to MPEG-2, with specific modifications for low bit rate applications. The codec operates on a macroblock mode, using the familiar motion-compensated DCT structure discussed above. In cases where the shape contour intersects a macroblock, the macroblock is "padded" by inserting filler pixel values at macroblock locations outside the shape contour to improve compression. Padded pixels are discarded at the receiver prior to display, according to the shape information that is simultaneously transmitted.

The architecture of an object-based encoder utilizing a depth-sensing camera in accordance with the present invention is shown in FIG. 15. The depth-sensing camera 100 provides a standard digital video or image signal 101 to MPEG-4 encoder 1600, and a second 102 consisting of depth information for the pixels contained in the normal video signal 101 to a video object segmentation circuit 1500. Video object segmentation circuit 1500 uses the depth information to robustly separate visual objects ("VOs"), and provides video object shape information to the MPEG-4 encoder 1600 which encodes them separately. Several VOs are multiplexed 1550 in the same bitstream to form the complete picture.

Referring to FIG. 16, an MPEG-4 encoder is shown. The video object shape information generated by video object segmentation circuit 1500 is provided to a shape coder 1610, while the standard digital video or image signal 101 is received and stored in a frame store 1601. The shape coder extends the current VOP bounding rectangle (i.e., the smallest rectangle that contains the entire VOP) at its bottom right side so as both its height and width are multiples of 16, to guarantee that the bounding rectangle contains an integer value of macroblocks. The individual macroblocks within the VOP are then processed with a modified MMR technique. of course, the coded information does not represent actual pixel values but rather whether or not a given pixel belongs to the current VOP or not.

The coded VOP shape information is provided to padding circuit 1620. Padding circuit 1620 retrieves the pixels from the stored standard digital image in 1601 that are identified by the coded VOP shape information, and performs a padding operation on those pixels by linearly interpolating pixels within a macroblock that are outside the VOP shape. The padding operation has the effect of improving coding performance by generating entire macroblocks for VOP's, which are more readily handled in a DCT compression technique.

Padded macroblocks for the VOPs are input to macroblock separator/adder 1630 and to motion estimation circuit 1640. A discrete cosign transform ("DCT") 1650 is applied to macroblock data from 1630. To achieve greater compression efficiency, DCT coefficients are then quantized 1651, Zig-Zag run-length encoded 1652 and variable length coded 1653. Quantized DCT coefficients are also inverse quantized 1655 and inverse discrete cosign transformed 1656 to provide past video data to the motion estimation circuit 1640 concurrently with present video data. The motion estimation circuit 1640 uses the past and present video data, which may be stored in frame store 1641, to generate motion vectors which are encoded 1641, 1642 and multiplexed 1660 with the compressed DCT coefficients.

In contrast with the segmentation process described with reference to FIG. 4, object-based compression requires not only the identification of areas of potential perceptual importance, but rather the identification of individual object areas. In the approach described with reference to FIG. 4, all objects that were within the same range from the camera were treated identically by object segmentation circuit 500, as that technique only affected rate control. Described below is an augmentation to that technique to include identification of individual objects.

Referring to FIG. 17, video object segmentation circuit 1500 is shown in greater detail. The circuit includes object map generation circuit 1710, region selector 1720, video object selector 1730, and demultiplexer 1740. The object map generation circuit 1710 generates an object map in the same manner as object segmentation circuit 500, and the hardware described with respect to that circuit is appropriate for use as the object map generation circuit 1710.

Region selector 1720 preprocesses the object map to identify which pixels are within each depth range (the area within range thresholds ti and ti+1). These pixels are marked with 1s, whereas the remaining ones are marked with 0s. This is called a depth region mask. Video object selector 1730, which may simply be a RAM preloaded with appropriate software, then proceeds to identify the one or more video objects that exist within that region. The VOP shape information for each video object located by video object selector 1730 is passed through DMUX 1740 and provided to the shape coder 1610. The process steps taken by video object selector 1730 are described below.

Referring again to FIG. 16, for VBR encoding, there is no output buffer and VOs can be independently assigned default quantizer levels in quantizer 1651. However, for CBR encoding, quantizer selection again becomes important as a means to regulate the generated bit rate.

As shown in FIG. 17, VOP shape information is also routed to a buffer 1750 and rate controller 1760 to effectuate rate control. The rate controller 1760 generates a quantization mode selection signal in the same manner as previously described with reference to FIG. 10, except that no macroblock labeling need occur: data from the same video object is always assigned the same target bit rate B. The video object identification is used directly instead of the macroblock label in FIGS. 12 and 13 to generate the target bit rate signal used in a virtual buffer counter (not shown in FIG. 17) and the buffer size modulation factor s. The quantization mode selection signal generated by rate controller 1760 may be applied to quantizer 1651 to ensure that the physical buffer (not shown in FIG. 16) does not overflow.

Referring to FIG. 18, a flow chart describing the rate control steps performed in accordance with the present invention are now described. Upon receiving video and depth information 1810, an object map is generated 1820 by computing a histogram 1821 of the received depth information to provide the number of pixels which have a predetermined depth value for a range of predetermined values, setting all values in the histogram which are below a predetermined threshold value t to zero in order to generate a clipped histogram 1822, scanning the clipped histogram 1823 to find boundaries of n regions with n different threshold depth values, and quantizing the depth information based on said n different threshold values.

Next, a current macroblock of video data is assigned 1830 to one of the regions of varying perceptual importance, and a signal indicative of a target bit rate associated with the assigned region is generated 1840. The target bit rate is used to generate a virtual buffer occupancy signal 1841 which, in turn, is used to generate a buffer size modulation signal 1851 whenever the assigned region is different from an immediately proceeding assigned region. A quantization mode signal is generated 1850 based on the object map and buffer size modulation signal to select a quantization mode reflective of the perceptual importance of the regions in the object map.

Finally, a compressed video signal is generated 1860, buffered 1870, and output 1880. If the buffer is filled, a signal indicative of a buffering overflow condition is generated 1871 and applied to step 1850. The virtual buffer count is increased 1861 whenever compressed video signal information is buffered; and decreased 1871 whenever buffered compressed video signal information is output.

Referring to FIG. 19, a flow chart describing the steps performed video object segmentation circuit 1500 are now described. Upon receiving video and depth information 1910, an object map is generated 1920 as described with reference to FIG. 18. The object map is preprocessed to generate an object mask 1920. One or more video objects that within the masked region are then identified as follows.

Starting from the upper left corner, the region mask is scanned 1940 left to right, top to bottom until a pixel with a value "1" is found. That pixel's location in the object mask is marked.

The neighborhood of that pixel is then searched 1950 (up-down, left-right, as well as diagonally) in the region mask to see if another "1" pixel exists 1960. If a neighboring "1" pixel exists, it's location in the object mask is also marked. If there are more neighboring pixels that have not been searched, the process is repeated 1961 (previously visited pixels are ignored). If there are no unsearched neighbors 1962, the algorithm terminates, and the object mask will contain a connected object region that contained the original starting pixel. This object mask is the shape information for this particular object that is provided to the encoder 1600.

Next, the pixels in the region mask that belong to the newly created object mask are themselves masked out 1970 (i.e., by setting their values to "0"), and the process is repeated 1980, 1981. If more that one objects exist in the same depth region, there will still be marked pixels left that will be identified as one or more additional objects. After all objects are processed this way, the region mask will be left with only zero valued pixels 1980, 1982.

Each object mask is labeled with a different number which, together with the depth range ID, uniquely identify an object. For example, two bytes can be used, with the first identifying the depth range, and the second identifying the object number within that range. Longer codes can be used to accommodate more ranges or objects per range. The benefit of using a hierarchical object ID is to easily identify which objects are in the foreground and which ones are in the background.

In order to take into account a temporal component, VOPs are tracked from one frame to the next, and a decision is made as to whether they belong to the same VO by considering both the shape difference from one frame to the next, and the depth range difference from one frame to the next. VOPs that have small shape and depth difference are considered to be part of the same VO. Although the encoder may operate at a low frame rate to minimize bandwidth requirements, the object detection subsystem can operate at the full 30 frames per second rate so that objects are adequately tracked.

Referring again to FIG. 19, one of the VOPs from the present frame is selected, as is one of the stored VOPs from the immediately proceeding frame 1990. The shape difference between the two VOPs is computed 2000 as the sum of the absolute values of the differences of the coordinates of the border pixels of the current VOP with their nearest pixels from the VOP from the previous frame. By summing up the differences in both horizontal and vertical directions, as well as the absolute value of the depth difference, an estimate of the dissimilarity between two VOPs is generated.

If there are more VOPs for the immediately proceeding frame 2010, 2011, a different past VOP is selected 2012, and the shape difference between the two VOPs is computed. When there are no more past frame VOPs 2010, 2013, the current VOP is assigned 2020 to the VO that provides the closest VOP in the previous frame. If there is no VOP in the previous frame, then a new VO is created.

If additional current VOPs exist 2030, 2031, the process is repeated by selecting a new current VOP 2032 and repeating steps 2000-2030. When all VOPs in the mask are assigned to a video object, a new mask may be generated 2040, 2041, and the process steps 1930-2030 are repeated. When all video objects have been identified 2040, 2042, the object identification signals are generated 2050 and sent to DMUX 1740 (FIG. 17).

The foregoing merely illustrates the principles of the invention. Various modifications and alterations to the described embodiments will be apparent to those skilled in the art in view of applicants' teachings, herein. For example, while the video object information generated in the embodiment shown in FIG. 15 is two-dimensional, it is also possible to encode the 3-D depth information generated by camera 100 directly and send it to the decoder. Such encoding would allow object manipulations such as rotations and custom lighting to be performed even after compression.

Further, more sophisticated rate control techniques can also be employed. For example, the system can try to identify the semantics of the different objects (e.g., human faces, sheets of paper, etc.) and automatically assign appropriate bit rates or quality levels to them. The semantics can either be obtained from the 2-D shape information, or from the 3-D depth information provided by the depth-sensing camera. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods which, although not explicitly shown or described herein, embody the principles of the invention and are thus within the spirit and scope of the invention.

Claims (49)

We claim:
1. An apparatus for encoding fields or frames of video information comprising a two dimensional array of pixels, and using a depth component of each of said pixels to enhance encoding, comprising:
(a) an encoder for receiving frames or fields of video information and generating a compressed video signal from said received frames or fields of video information, said encoder including a multi-mode quantizer for quantizing data which corresponds to a portion of said fields or frames of video information;
(b) an object segmentation circuit for receiving depth information which corresponds to said received video information and generating an object map to associate each pixel of said received field or frame with one of one or more regions of varying perceptual importance within said received frame or field; and
(c) a rate controller, coupled to said object segmentation circuit and to said multi-mode quantizer, for receiving said object map and for providing a signal, responsive to said object map, to said multi-mode quantizer to select a quantization mode therein, such that said selected quantization mode is reflective of said perceptual importance of said regions indicated by said object map.
2. The apparatus of claim 1, wherein said encoder is a variable bit rate encoder capable of generating an MPEG-2 compliant bit stream.
3. The apparatus of claim 1, wherein said object segmentation circuit comprises:
(1) a histogram generation circuit for receiving said depth information and for computing a histogram of said depth information to thereby provide the number of pixels which have a predetermined depth value for a range of predetermined values;
(2) a first logic circuit, coupled to said histogram generation circuit, for receiving said generated histogram and for setting all values in said histogram which are below a predetermined threshold value to zero to thereby generate a clipped histogram;
(3) a second logic circuit, coupled to said first logic circuit, for receiving said clipped histogram and for scanning said clipped histogram to find boundaries of n regions with n different threshold depth values; and
(4) a variable step quantization circuit, coupled to said second logic circuit, for receiving said n different threshold values and said depth information, and for quantizing said depth information based on said n different threshold values, to thereby generate said object map.
4. The apparatus of claim 3, wherein said histogram generation circuit comprises:
(i) a buffer for receiving and temporarily storing said depth information;
(ii) a memory, coupled to said buffer, for receiving said depth information from said buffer as memory addresses and for storing histogram values as said memory addresses; and
(iii) a logic circuit, coupled to said memory, for reading a histogram value from said memory at an address location, updating said histogram value, and providing said updated histogram value to said memory at said address location.
5. The apparatus of claim 1, wherein said encoder is a variable bit rate encoder, and further comprising:
(d) a video buffer having a preselected storage capacity, coupled to said encoder and to said rate controller, for receiving and temporarily storing said generated compressed video signal, and for providing a signal indicative of an overflow condition to said rate controller, wherein said signal provided by said rate controller to said multi-mode quantizer is also responsive to said overflow signal such that said selected quantization mode is reflective of said perceptual importance of said regions as constrained by said storage capacity of said video buffer.
6. The apparatus of claim 5, further comprising:
(e) a macroblock labeling circuit, coupled to said object segmentation circuit and to said rate controller, for assigning a current macroblock of video data to one of said regions of varying perceptual importance and for providing a signal indicative of said assigned region to said rate controller, wherein said signal provided by said rate controller to said multi-mode quantizer is reflective of said assigned region.
7. The apparatus of claim 6, wherein said current macroblock of video data is assigned to a region having the greatest perceptual importance of one or more regions identified by a location of said object map which corresponds to said macroblock of video data.
8. The apparatus of claim 6, wherein said encoder generates an output signal when compressed video data is output to said video buffer, and wherein said macroblock labeling circuit generates a signal indicative of a target bit rate associated with said assigned region, further comprising:
(f) a clock signal generating circuit, coupled to said video buffer, for providing a clock signal to said buffer, wherein said video buffer outputs a predetermined amount of said compressed video signal in response to said clock signal; and
(g) a counter, coupled to said encoder, to said clock signal generating circuit, to said macroblock labeling circuit and to said rate controller, for receiving said clock signal, said target bit rate signal and said encoder output signal, for counting the number of bits that are in a virtual buffer associated with said video buffer by adding to said count in response to said encoder output signal and subtracting from said count in response to said clock signal and to said target bit rate signal to thereby determine an occupancy of said virtual buffer, and for providing a virtual buffer occupancy signal indicative of said count to said rate controller;
wherein said signal provided by said rate controller to said multi-mode quantizer is also responsive to said virtual buffer occupancy signal such that said selected quantization mode is reflective of said perceptual importance of said regions indicated by said virtual buffer capacity as constrained by said storage capacity of said video buffer.
9. The apparatus of claim 8, wherein said virtual buffer occupancy Bi is determined by the equation Bi =Bi-1 +bi -r(Rk /R), where bi is equal to the number of bits used to encode the present macroblock indicated by said encoder output signal, r is equal to the number of bits output by said video buffer indicated by said clock signal, Rk is the target bit rate indicated by said target bit rate signal, and R is the average output bit rate to be maintained by said video buffer.
10. The apparatus of claim 8, wherein said rate controller further comprises a buffer size logic circuit, coupled to said macroblock labeling circuit, for receiving said assigned region and for generating a buffer size modulation signal whenever said assigned region is different from an immediately proceeding assigned region, wherein said signal provided by said rate controller to said multi-mode quantizer is also responsive to said buffer size modulation signal such that said selected quantization mode is reflective of said perceptual importance of said regions indicated by said virtual buffer capacity as modulated by said buffer size modulation signal and as constrained by said storage capacity of said video buffer.
11. An apparatus for encoding fields or frames of video information comprising a two dimensional array of pixels, and using a depth component of each of said pixels to enhance encoding, comprising:
(a) a depth sensing camera capable of generating in real-time both frames or fields of video information and depth information which corresponds to said video information;
(b) an encoder, coupled to said depth sensing camera and receiving said generated frames or fields of video information, for generating a compressed video signal from said frames or fields of video information, said encoder including a multi-mode quantizer for quantizing data which corresponds to a portion of said fields or frames of video information;
(c) an object segmentation circuit, coupled to said depth sensing camera and receiving said generated depth information, for generating an object map to associate each pixel of said received field or frame with one of one or more regions of varying perceptual importance within said received frame or field; and
(d) a rate controller, coupled to said object segmentation circuit and to said multi-mode quantizer, for receiving said object map and for providing a signal, responsive to said object map, to said multi-mode quantizer to select a quantization mode therein, such that said selected quantization mode is reflective of said perceptual importance of said regions indicated by said object map.
12. A method for encoding fields or frames of video information comprising a two dimensional array of pixels using a depth component of each of said pixels to enhance encoding, comprising the steps of:
(a) receiving frames or fields of video information and depth information which corresponds to said received video information;
(b) converting said received three information into an object map to thereby associate each pixel of said received field or frame with one of one or more regions of varying perceptual importance within said received frame or field;
(c) generating a quantization mode signal based on said object map to select a quantization mode reflective of said perceptual importance of said regions indicated by said object map; and
(d) generating a compressed video signal which corresponds to said received frames or fields of video information by quantizing data which corresponds to a portion of said received fields or frames of video information in accordance with said quantization mode selected by said quantization mode signal.
13. The method of claim 12, wherein said compressed video signal generated in step (d) is an MPEG-2 compliant bit stream.
14. The method of claim 12, wherein said converting step comprises:
(1) computing a histogram of said received depth information to thereby provide the number of pixels which have a predetermined depth value for a range of predetermined values;
(2) setting all values in said histogram which are below a predetermined threshold value to zero to thereby generate a clipped histogram;
(3) scanning said clipped histogram to find boundaries of n regions with n different threshold depth values; and
(4) quantizing said depth information based on said n different threshold values.
15. The method of claim 12, wherein said generated compressed video signal is a variable bit rate signal, and further comprising the steps of:
(e) buffering said generated compressed video signal; and
(f) providing a signal indicative of a buffering overflow condition, wherein said quantization mode signal is also responsive to said overflow signal such that said selected quantization mode is reflective of said perceptual importance of said regions as constrained by a buffering limitation.
16. The method of claim 15, further comprising the step of assigning, prior to generating said quantization mode signal, a current macroblock of video data within said received frames or fields of video information to one of said regions of varying perceptual importance, wherein said quantization mode signal is reflective of said assigned region.
17. The method of claim 16, wherein said assigning step further comprises assigning said current macroblock of video data to an region having the greatest perceptual importance of one or more regions identified by a location of said object map which corresponds to said macroblock of video data.
18. The method of claim 16, further comprising the step of generating a signal indicative of a target bit rate associated with said assigned region prior to generating said quantization mode signal.
19. The method of claim 18, further comprising the steps of:
(i) adding to a virtual buffer count indicative of a virtual buffer occupancy whenever compressed video signal information is buffered;
(ii) subtracting from said count whenever buffered compressed video signal information is output in an amount which dependant on said target bit rate signal; and
(iii) generating a virtual buffer occupancy signal indicative of said count, wherein said quantization mode signal is also responsive to said virtual buffer occupancy signal such that said selected quantization mode is reflective of said perceptual importance of said regions indicated by said virtual buffer occupancy as constrained by said buffering limitation.
20. The method of claim 19, wherein said virtual buffer occupancy Bi is determined by the equation Bi =Bi-1 +bi -r(Rk /R), where bi is equal to the number of bits used to encode the present macroblock indicated by said encoder output signal, r is equal to the number of bits output by said video buffer indicated by said clock signal, Rk is the target bit rate indicated by said target bit rate signal, and R is the average output bit rate required to prevent said buffering limitation from occurring.
21. The method of claim 16, further comprising the step of generating, prior to generating said quantization mode signal, a buffer size modulation signal whenever said assigned region is different from an immediately proceeding assigned region, wherein said quantization mode signal is also responsive to said buffer size modulation signal such that said selected quantization mode is reflective of said perceptual importance of said regions indicated by said virtual buffer capacity as modulated by said buffer size modulation signal and as constrained by said buffering limitation.
22. An apparatus for performing object-based encoding of video information using a depth component of said video information to enhance encoding, comprising:
(a) an object segmentation circuit for receiving depth information for a frame of video information and generating one or more object identification signals based on said received depth information indicative of a shape of one or more objects within said frame of video information; and
(b) an encoder, coupled to said object segmentation circuit, for receiving the frame of video information which corresponds to said received depth information and said one or more object identification signals, and for encoding a video signal representing only a portion of said video information, which portion substantially corresponds to said one or more objects identified by said one or more object identification signals.
23. The apparatus of claim 22, wherein said object segmentation circuit comprises:
(1) a object map generation circuit for receiving said depth information and for converting said depth information into an object map to thereby associate each pixel within said frame of video information with one of one or more regions of varying perceptual importance within said frame;
(2) a region masking circuit, coupled to said object map generation circuit and receiving said object map, for masking said object map to generate a depth region mask indicative of pixels within said frame which substantially correspond to a preselected region; and
(3) a video object selection circuit, coupled to said region masking circuit and receiving said generated region mask, for identifying one or more separate objects within said preselected region indicated by said region mask and generating said one or more object identification signals, such that each one of said object identification signals identifies one of said one or more identified separate objects.
24. The apparatus of claim 23, wherein said object map generation circuit comprises:
(i) a histogram generation circuit for receiving said depth information and for computing a histogram of said depth information to thereby provide the number of pixels which have a predetermined depth value for a range of predetermined values;
(ii) a first logic circuit, coupled to said histogram generation circuit, for receiving said generated histogram and for setting all values in said histogram which are below a predetermined threshold value to zero to thereby generate a clipped histogram;
(iii) a second logic circuit, coupled to said first logic circuit, for receiving said clipped histogram and for scanning said clipped histogram to find boundaries of n regions with n different threshold depth values; and
(iv) a variable step quantization circuit, coupled to said second logic circuit, for receiving said n different threshold values and said depth information, and for quantizing said depth information based on said n different threshold values, to thereby generate said object map.
25. The apparatus of claim 24, wherein said histogram generation circuit comprises:
(A) a buffer for receiving and temporarily storing said depth information;
(B) a memory, coupled to said buffer, for receiving said depth information from said buffer as memory addresses and for storing histogram values as said memory addresses; and
(C) a logic circuit, coupled to said memory, for reading a histogram value from said memory at an address location, updating said histogram value, and providing said updated histogram value to said memory at said address location.
26. The apparatus of claim 22, wherein said encoder includes a multi-mode quantizer for quantizing data which corresponds to a portion of said received fields of video information, further comprising:
(c) a rate controller, coupled to said an object segmentation circuit and to said multi-mode quantizer, for receiving said one or more object identification signals and for providing a signal, responsive to said object identification signals, to said multi-mode quantizer to select a quantization mode therein, such that for each portion of data to be quantized, said selected quantization mode is reflective of a perceptual importance of an object identified by one of said one or more object identification signals which is associated with said portion of data to be quantized.
27. The apparatus of claim 26, wherein said encoder is a variable bit rate encoder, and further comprising:
(d) a video buffer having a preselected storage capacity, coupled to said encoder and to said rate controller, for receiving and temporarily storing said generated compressed video signal, and for providing a signal indicative of an overflow condition to said rate controller, wherein said signal provided by said rate controller to said multi-mode quantizer is also responsive to said overflow signal such that said selected quantization mode is reflective of said perceptual importance of said object as constrained by said storage capacity of said video buffer.
28. The apparatus of claim 27, wherein said portion of data to be quantized is a macroblock, further comprising:
(e) a macroblock labeling circuit, coupled to said object segmentation circuit and to said rate controller, for assigning a current macroblock of video data to one of said objects and for providing a signal indicative of said assigned object to said rate controller, wherein said signal provided by said rate controller to said multi-mode quantizer is reflective of said assigned object.
29. The apparatus of claim 28, wherein said encoder generates an output signal when compressed video data is output to said video buffer, and wherein said macroblock labeling circuit generates a signal indicative of a target bit rate associated with said assigned object, further comprising:
(f) a clock signal generating circuit, coupled to said video buffer and providing a clock signal to said buffer, wherein said video buffer outputs a predetermined amount of said compressed video signal in response to said clock signal; and
(g) a counter, coupled to said encoder, to said clock signal generating circuit, to said macroblock labeling circuit and to said rate controller, for receiving said clock signal, said target bit rate signal and said encoder output signal, for counting the number of bits that are in a virtual buffer associated with said video buffer by adding to said count in response to said encoder output signal and subtracting from said count in response to said clock signal and to said target bit rate signal to thereby determine an occupancy of said virtual buffer, and for providing a virtual buffer occupancy signal indicative of said count to said rate controller;
wherein said signal provided by said rate controller to said multi-mode quantizer is also responsive to said virtual buffer occupancy signal such that said selected quantization mode is reflective of said perceptual importance of said object indicated by said virtual buffer capacity as constrained by said storage capacity of said video buffer.
30. The apparatus of claim 29, wherein said virtual buffer occupancy Bi is determined by the equation Bi =Bi-1 +bi -r(Rk /R), where bi is equal to the number of bits used to encode the present macroblock indicated by said encoder output signal, r is equal to the number of bits output by said video buffer indicated by said clock signal, Rk is the target bit rate indicated by said target bit rate signal, and R is the average output bit rate to be maintained by said video buffer.
31. The apparatus of claim 29, wherein said rate controller further comprises a buffer size logic circuit, coupled to said macroblock labeling circuit, for receiving said assigned object and for generating a buffer size modulation signal whenever said assigned object is different from an immediately proceeding assigned object, wherein said signal provided by said rate controller to said multi-mode quantizer is also responsive to said buffer size modulation signal such that said selected quantization mode is reflective of said perceptual importance of said object indicated by said virtual buffer capacity as modulated by said buffer size modulation signal and as constrained by said storage capacity of said video buffer.
32. An apparatus for performing object-based encoding of video information using a depth component of said video information to enhance encoding, comprising:
(a) a depth sensing camera capable of generating in real-time both frames of video information and depth information which corresponds to said video information;
(b) an object segmentation circuit, coupled to said depth sensing camera and receiving said generated depth information, for generating one or more object identification signals based on said received depth information indicative of a shape of one or more objects within said frame of video information; and
(c) an encoder, coupled to said object segmentation circuit and to said depth sensing camera and receiving said generated frame of video information which corresponds to said received depth information and said one or more object identification signals, for encoding a video signal representing only a portion of said video information, which portion substantially corresponds to said one or more objects identified by said one or more object identification signals.
33. A method for performing object-based encoding of video information using a depth component of said video information to enhance encoding, comprising the steps of:
(a) receiving frames of video information and depth information which corresponds to said received video information;
(b) generating one or more object identification signals based on said received depth information indicative of a shape of one or more objects within said frame of video information; and
(c) encoding a video signal representing only a portion of said received video information, which portion substantially corresponds to said one or more objects identified by said one or more object identification signals.
34. The method of claim 33, wherein step (b) comprises the steps of:
(1) converting said received depth information into an object map to thereby associate each pixel within said frame of video information with one of one or more regions of varying perceptual importance within said frame;
(2) masking said object map to generate a depth region mask indicative of pixels within said frame which substantially correspond to a preselected region;
(3) identifying one or more separate objects within said preselected region indicated by said depth region mask; and
(4) generating said one or more object identification signals such that each one of said object identification signals identifies one of said one or more identified separate objects.
35. The method of claim 34, wherein said converting step comprises the steps of:
(i) computing a histogram of said received depth information to thereby provide the number of pixels which have a predetermined depth value for a range of predetermined values;
(ii) setting all values in said histogram which are below a predetermined threshold value to zero to thereby generate a clipped histogram;
(iii) scanning said clipped histogram to find boundaries of n regions with n different threshold depth values; and
(iv) quantizing said depth information based on said n different threshold values.
36. The method of claim 34, wherein said identifying step comprises the steps of:
(i) scanning said depth region mask until a pixel with a nonmasked value is found;
(ii) searching neighboring pixels within said mask to find any other neighboring pixels with nonmasked values;
(iii) repeating said searching step until no neighboring pixels have a nonmasked value to identify all neighboring found pixels with nonmasked values as a video object plane which corresponds to an object within said depth region;
(iv) masking said object from said depth region mask; and
(v) repeating steps (i)-(iv) until all pixels within said depth region mask are masked to thereby identify one or more video object planes within said received frame of video information.
37. The method of claim 36, wherein said identifying step further comprises the steps of:
(vi) selecting one of said one or more video object planes, and one of one or more video object planes associated with an immediately preceding frame of video information;
(vii) comparing said selected video object plane and said selected previous frame video object plane to determine a depth difference therebetween;
(viii) repeating step (vii) after selecting a different one of said one or more previous frame video object planes unless all of said one or more previous frame video object planes have been selected;
(ix) assigning said selected video object plane to a video object which corresponds to one of said one or more previous frame video object planes for which a depth difference therebetween is minimized as compared to all of said determined depth differences; and
(x) repeating steps (vii)-(ix) after selecting a different one of said one or more video object planes unless all of said one or more video object planes have been selected, so that each of said one or more video object planes identifies an object.
38. The method of claim 33, wherein step (c) includes quantizing data which corresponds to a portion of said received fields of video information, and further comprising the step of
(d) generating a quantization mode signal based on an object identification signal which corresponds to said data to select a quantization mode reflective of a perceptual importance of an object indicated by said object identification signal, such that said data is quantized based on said selected quantization mode.
39. The method of claim 38, wherein said generated compressed video signal is a variable bit rate signal, and further comprising the steps of:
(e) buffering said generated compressed video signal; and
(f) providing a signal indicative of a buffering overflow condition, wherein said quantization mode signal is also responsive to said overflow signal such that said selected quantization mode is reflective of said perceptual importance of said object as constrained by a buffering limitation.
40. The method of claim 39, further comprising the step of assigning, prior to generating said quantization mode signal, a current macroblock of video data within said received frames or fields of video information to one of said objects, wherein said quantization mode signal is reflective of said assigned object.
41. The method of claim 40, further comprising the step of generating a signal indicative of a target bit rate associated with said assigned object prior to generating said quantization mode signal.
42. The method of claim 41, further comprising the steps of:
(i) adding to a virtual buffer count indicative of a virtual buffer occupancy whenever compressed video signal information is buffered;
(ii) subtracting from said count whenever buffered compressed video signal information is output in an amount which dependant on said target bit rate signal; and
(iii) generating a virtual buffer occupancy signal indicative of said count, wherein said quantization mode signal is also responsive to said virtual buffer occupancy signal such that said selected quantization mode is reflective of said objects indicated by said virtual buffer occupancy as constrained by said buffering limitation.
43. The method of claim 42, wherein said virtual buffer occupancy Bi is determined by the equation Bi =Bi-1 +bi -r(Rk /R), where bi is equal to the number of bits used to encode the present macroblock indicated by said encoder output signal, r is equal to the number of bits output by said video buffer indicated by said clock signal, Rk is the target bit rate indicated by said target bit rate signal, and R is the average output bit rate required to prevent said buffering limitation from occurring.
44. The method of claim 40, further comprising the step of generating, prior to generating said quantization mode signal, a buffer size modulation signal whenever said assigned object is different from an immediately proceeding assigned object, wherein said quantization mode signal is also responsive to said buffer size modulation signal such that said selected quantization mode is reflective of said object indicated by said virtual buffer capacity as modulated by said buffer size modulation signal and as constrained by said buffering limitation.
45. An object segmentation circuit for receiving depth information which corresponds to a frame of video information, and for identifying one or more separate objects within said frame of video information, comprising:
(a) a object map generation circuit for receiving said depth information, and for converting said depth information into an object map to thereby associate each pixel within said frame of video information with one of one or more regions of varying perceptual importance within said frame, wherein said object map generation circuit comprises:
(1) a histogram generation circuit for receiving said depth information and for computing a histogram of said depth information to thereby provide the number of pixels which have a predetermined depth value for a range of predetermined values;
(2) a first logic circuit, coupled to said histogram generation circuit, for receiving said generated histogram and for setting all values in said histogram which are below a predetermined threshold value to zero to thereby generate a clipped histogram;
(3) a second logic circuit, coupled to said first logic circuit, for receiving said clipped histogram and for scanning said clipped histogram to find boundaries of n regions with n different threshold depth values; and
(4) a variable step quantization circuit, coupled to said second logic circuit, for receiving said n different threshold values and said depth information, and for quantizing said depth information based on said n different threshold values, to thereby generate said object map;
(b) a region masking circuit, coupled to said object map generation circuit and receiving said object map, for masking said object map to generate one or more depth region masks indicative of pixels within said frame which substantially correspond to preselected regions of depth; and
(c) a video object selection circuit, coupled to said region masking circuit and receiving said generated one or more region masks, for identifying one or more separate objects within each of said one or more preselected regions indicated by each of said one or more region masks, such that each object associated with each depth region is identified as a separate object.
46. The apparatus of claim 45, wherein said histogram generation circuit comprises:
(i) a buffer for receiving and temporarily storing said depth information;
(ii) a memory, coupled to said buffer, for receiving said depth information from said buffer as memory addresses and for storing histogram values as said memory addresses; and
(iii) a logic circuit, coupled to said memory, for reading a histogram value from said memory at an address location, updating said histogram value, and providing said updated histogram value to said memory at said address location.
47. A method for identifying one or more separate objects within depth information which corresponds to a frame of video information, comprising the steps of:
(a) receiving said depth information;
(b) converting said received depth information into an object map to thereby associate each pixel within said frame of video information with one of one or more regions of varying perceptual importance within said frame, wherein said converting step comprises the steps of:
(1) computing a histogram of said received depth information to thereby provide the number of pixels which have a predetermined depth value for a range of predetermined values;
(2) setting all values in said histogram which are below a predetermined threshold value to zero to thereby generate a clipped histogram;
(3) scanning said clipped histogram to find boundaries of n regions with n different threshold depth values; and
(4) quantizing said depth information based on said n different threshold values;
(c) masking said object map to generate one or more depth region masks indicative of pixels within said frame which substantially correspond to preselected regions of depth; and
(d) identifying one or more separate objects within each of said one or more preselected regions indicated by said one or more region masks, such that each object associated with each depth region is identified as a separate object.
48. The method of claim 47, wherein said identifying step comprises the steps of:
(i) scanning a preselected depth region mask until a pixel with a nonmasked value is found;
(ii) searching neighboring pixels within said mask to find any other neighboring pixels with nonmasked values;
(iii) repeating said searching step until no neighboring pixels have a nonmasked value to identify all neighboring found pixels with nonmasked values as a video object plane which corresponds to an object within said depth region;
(iv) masking said object from said depth region mask; and
(v) repeating steps (i)-(iv) until all pixels within said depth region mask are masked to thereby identify one or more video object planes within said received frame of video information.
49. The method of claim 48, wherein said identifying step further comprises the steps of:
(vi) selecting one of said one or more video object planes, and one of one or more video object planes associated with an immediately preceding frame of video information;
(vii) comparing said selected video object plane and said selected previous frame video object plane to determine a depth difference therebetween;
(viii) repeating step (vii) after selecting a different one of said one or more previous frame video object planes unless all of said one or more previous frame video object planes have been selected;
(ix) assigning said selected video object plane to a video object which corresponds to one of said one or more previous frame video object planes for which a depth difference therebetween is minimized as compared to all of said determined depth differences; and
(x) repeating steps (vii)-(ix) after selecting a different one of said one or more video object planes unless all of said one or more video object planes have been selected, so that each of said one or more video object planes identifies an object.
US08723467 1996-10-09 1996-10-09 Methods and apparatus for performing digital image and video segmentation and compression using 3-D depth information Expired - Fee Related US6055330A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US08723467 US6055330A (en) 1996-10-09 1996-10-09 Methods and apparatus for performing digital image and video segmentation and compression using 3-D depth information

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
US08723467 US6055330A (en) 1996-10-09 1996-10-09 Methods and apparatus for performing digital image and video segmentation and compression using 3-D depth information
CA 2268191 CA2268191A1 (en) 1996-10-09 1997-10-09 Methods and apparatus for performing digital image and video segmentation and compression using 3-d depth information
PCT/US1997/018305 WO1998015915A1 (en) 1996-10-09 1997-10-09 Methods and apparatus for performing digital image and video segmentation and compression using 3-d depth information
JP51775998A JP2001509325A (en) 1996-10-09 1997-10-09 The method of three-dimensional depth information digital images and moving images using a split compresses and apparatus
KR19997003099A KR20000049031A (en) 1996-10-09 1997-10-09 Methods and apparatus for performing digital image and video segmentation and compression using 3-d depth information
EP19970910877 EP1012776A4 (en) 1996-10-09 1997-10-09 Methods and apparatus for performing digital image and video segmentation and compression using 3-d depth information

Publications (1)

Publication Number Publication Date
US6055330A true US6055330A (en) 2000-04-25

Family

ID=24906398

Family Applications (1)

Application Number Title Priority Date Filing Date
US08723467 Expired - Fee Related US6055330A (en) 1996-10-09 1996-10-09 Methods and apparatus for performing digital image and video segmentation and compression using 3-D depth information

Country Status (6)

Country Link
US (1) US6055330A (en)
EP (1) EP1012776A4 (en)
JP (1) JP2001509325A (en)
KR (1) KR20000049031A (en)
CA (1) CA2268191A1 (en)
WO (1) WO1998015915A1 (en)

Cited By (91)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6125142A (en) * 1997-12-02 2000-09-26 Daewoo Electronics Co., Ltd. Method and apparatus for encoding object information of a video object plane
US6259732B1 (en) * 1998-03-14 2001-07-10 Daewoo Electronics Co., Ltd. Method and apparatus for encoding interlaced macroblock texture information
US20010024471A1 (en) * 2000-03-15 2001-09-27 Philippe Bordes Process and device for coding video images
US20020012472A1 (en) * 2000-03-31 2002-01-31 Waterfall Andrew E. Method for visualization of time sequences of 3D optical fluorescence microscopy images
US20020048395A1 (en) * 2000-08-09 2002-04-25 Harman Philip Victor Image conversion and encoding techniques
WO2002052393A1 (en) * 2000-12-22 2002-07-04 The Government Of The United States Of America, As Represented By The Secretary, Department Of Health And Human Services Positioning an item in three dimensions via a graphical representation
WO2002060183A1 (en) * 2001-01-24 2002-08-01 Oy Gamecluster Ltd A method for compressing video information
US20020176625A1 (en) * 2001-04-04 2002-11-28 Mitsubishi Electric Research Laboratories, Inc. Method for segmenting multi-resolution video objects
US20030035484A1 (en) * 2000-04-17 2003-02-20 Pulsent Corporation Segmenting encoding system with image segmentation performed at a decoder and encoding scheme for generating encoded data relying on decoder segmentation
US6526161B1 (en) * 1999-08-30 2003-02-25 Koninklijke Philips Electronics N.V. System and method for biometrics-based facial feature extraction
US20030043912A1 (en) * 2001-08-23 2003-03-06 Sharp Laboratories Of America, Inc. Method and apparatus for motion vector coding with global motion parameters
US20030063676A1 (en) * 2000-04-17 2003-04-03 Pulsent Corporation Decoder for decoding segment-based encoding of video data using segmentation performed at a decoder
US20030123545A1 (en) * 1999-04-17 2003-07-03 Pulsent Corporation Segment-based encoding system using segment hierarchies
US20030123543A1 (en) * 1999-04-17 2003-07-03 Pulsent Corporation Segment-based encoding system using residue coding by basis function coefficients
US20030128759A1 (en) * 1999-04-17 2003-07-10 Pulsent Corporation Segment-based encoding system including segment-specific metadata
US6600786B1 (en) * 1999-04-17 2003-07-29 Pulsent Corporation Method and apparatus for efficient video processing
US20040028130A1 (en) * 1999-05-24 2004-02-12 May Anthony Richard Video encoder
US20040032488A1 (en) * 1997-12-05 2004-02-19 Dynamic Digital Depth Research Pty Ltd Image conversion and encoding techniques
US20040156433A1 (en) * 2001-06-11 2004-08-12 Comer Mary Lafuze Motion compensation for fine-grain scalable video
US6792154B1 (en) 1999-10-07 2004-09-14 World Multicast.com, Inc Video compression system and method using time
US20040184653A1 (en) * 2003-03-20 2004-09-23 Baer Richard L. Optical inspection system, illumination apparatus and method for use in imaging specular objects based on illumination gradients
US6879632B1 (en) * 1998-12-24 2005-04-12 Nec Corporation Apparatus for and method of variable bit rate video coding
US20050084007A1 (en) * 2003-10-16 2005-04-21 Lightstone Michael L. Apparatus, system, and method for video encoder rate control
US6898759B1 (en) * 1997-12-02 2005-05-24 Yamaha Corporation System of generating motion picture responsive to music
US20050232354A1 (en) * 2004-04-14 2005-10-20 Yi-Kai Chen Rate controlling method and apparatus for use in a transcoder
US20050244071A1 (en) * 2004-04-29 2005-11-03 Mitsubishi Denki Kabushiki Kaisha Adaptive quantization of depth signal in 3D visual coding
US6965379B2 (en) 2001-05-08 2005-11-15 Koninklijke Philips Electronics N.V. N-view synthesis from monocular video of certain broadcast and stored mass media content
US20050276323A1 (en) * 2002-09-27 2005-12-15 Vanguard Software Solutions, Inc. Real-time video coding/decoding
US20060002608A1 (en) * 2002-11-12 2006-01-05 Qinetiq Limited Image analysis
US20060007301A1 (en) * 2004-07-08 2006-01-12 Cho Gyoung I 3D television broadcasting system
US7020192B1 (en) * 1998-07-31 2006-03-28 Kabushiki Kaisha Toshiba Method of retrieving video picture and apparatus therefor
US20060071932A1 (en) * 2002-11-21 2006-04-06 Koninklijke Philips Electronics N.V. Method and apparatus for visualizing a sequece of volume images
US20060088206A1 (en) * 2004-10-21 2006-04-27 Kazunari Era Image processing apparatus, image pickup device and program therefor
US20060117268A1 (en) * 2004-11-30 2006-06-01 Micheal Talley System and method for graphical element selection for region of interest compression
US20060256081A1 (en) * 2002-07-27 2006-11-16 Sony Computer Entertainment America Inc. Scheme for detecting and tracking user manipulation of a game controller body
US20060256233A1 (en) * 2005-05-16 2006-11-16 Douglas Chin Systems, methods, and apparatus for video encoding
KR100656783B1 (en) 2004-12-14 2006-12-12 한국전자통신연구원 Apparatus and its method of transmitting an image associated with a depth and apparatus and its method of rendering stereoscopic images using it
US20060287087A1 (en) * 2002-07-27 2006-12-21 Sony Computer Entertainment America Inc. Method for mapping movements of a hand-held controller to game commands
US20070009046A1 (en) * 1997-04-24 2007-01-11 Shunichi Sekiguchi Method and apparatus for region-based moving image encoding and decoding
US7199836B1 (en) * 1998-02-13 2007-04-03 The Trustees Of Columbia University In The City Of New York Object-based audio-visual terminal and bitstream structure
US20070104271A1 (en) * 2000-12-26 2007-05-10 Nec Corporation Moving picture encoding system
US20070120844A1 (en) * 2005-11-28 2007-05-31 Baumer Optronic Gmbh Method and apparatus for the graphical operation of real-time image processing systems
US20070165139A1 (en) * 1997-02-14 2007-07-19 The Trustees Of Columbia University In The City Of New York Object-Based Audio-Visual Terminal And Bitstream Structure
US7321624B1 (en) * 2001-03-16 2008-01-22 Objectvideo, Inc. Bit-rate allocation system for object-based video encoding
US20080031327A1 (en) * 2006-08-01 2008-02-07 Haohong Wang Real-time capturing and generating stereo images and videos with a monoscopic low power mobile device
US20080056385A1 (en) * 1997-10-23 2008-03-06 Yoshimi Isu Image decoding apparatus, image coding apparatus, image communications system and coded bit stream converting apparatus
US20080062257A1 (en) * 2006-09-07 2008-03-13 Sony Computer Entertainment Inc. Touch screen-like user interface that does not require actual touching
US20080080789A1 (en) * 2006-09-28 2008-04-03 Sony Computer Entertainment Inc. Object detection using video input combined with tilt angle information
US20080095190A1 (en) * 2002-06-26 2008-04-24 Standard Microsystems Corp. Communication System and Method for Sending Isochronous Streaming Data Within a Frame Segment Using a Signaling Byte
US7372903B1 (en) * 1997-07-10 2008-05-13 Mediatek, Inc. Apparatus and method for object based rate control in a coding system
US20080174678A1 (en) * 2006-07-11 2008-07-24 Solomon Research Llc Digital imaging system
US20080273787A1 (en) * 2005-09-09 2008-11-06 Qinetiq Limited Automated Selection of Image Regions
US20080279285A1 (en) * 2004-07-02 2008-11-13 Koninklijke Philips Electronics, N.V. Video Processing
US20090009370A1 (en) * 2007-07-05 2009-01-08 Mega Chips Corporation Transcoder
US20090067504A1 (en) * 2007-09-07 2009-03-12 Alexander Zheludkov Real-time video coding/decoding
US20090066693A1 (en) * 2007-09-06 2009-03-12 Roc Carson Encoding A Depth Map Into An Image Using Analysis Of Two Consecutive Captured Frames
US20090122146A1 (en) * 2002-07-27 2009-05-14 Sony Computer Entertainment Inc. Method and apparatus for tracking three-dimensional movements of an object using a depth sensing camera
US20090148038A1 (en) * 2007-12-10 2009-06-11 Youichi Sawachi Distance image processing apparatus and method
US20100033617A1 (en) * 2008-08-05 2010-02-11 Qualcomm Incorporated System and method to generate depth data using edge detection
US20100061444A1 (en) * 2008-09-11 2010-03-11 On2 Technologies Inc. System and method for video encoding using adaptive segmentation
US20100124274A1 (en) * 2008-11-17 2010-05-20 Cheok Lai-Tee Analytics-modulated coding of surveillance video
US20100290712A1 (en) * 2009-05-13 2010-11-18 Seiko Epson Corporation Image processing method and image processing apparatus
US20120008672A1 (en) * 2010-07-07 2012-01-12 Gaddy William L System and method for transmission, processing, and rendering of stereoscopic and multi-view images
CN102326391A (en) * 2009-02-23 2012-01-18 国立大学法人名古屋大学 Multi-view image coding method, multi-view image decoding method, multi-view image coding device, multi-view image decoding device, multi-view image coding program, and multi-view image decoding program
US20120236024A1 (en) * 2009-12-04 2012-09-20 Panasonic Corporation Display control device, and method for forming display image
US20120281767A1 (en) * 2011-05-04 2012-11-08 Alberto Duenas Low latency rate control system and method
US8310656B2 (en) 2006-09-28 2012-11-13 Sony Computer Entertainment America Llc Mapping movements of a hand-held controller to the two-dimensional image plane of a display screen
US8313380B2 (en) 2002-07-27 2012-11-20 Sony Computer Entertainment America Llc Scheme for translating movements of a hand-held controller into inputs for a system
US20130003853A1 (en) * 2006-07-06 2013-01-03 Canon Kabushiki Kaisha Motion vector detection apparatus, motion vector detection method, image encoding apparatus, image encoding method, and computer program
EP2555527A1 (en) * 2010-03-26 2013-02-06 Sony Corporation Image processor, image processing method, and program
US8526488B2 (en) 2010-02-09 2013-09-03 Vanguard Software Solutions, Inc. Video sequence encoding system and algorithms
US20130279588A1 (en) * 2012-04-19 2013-10-24 Futurewei Technologies, Inc. Using Depth Information to Assist Motion Compensation-Based Video Coding
US8587655B2 (en) 2005-07-22 2013-11-19 Checkvideo Llc Directed attention digital video recordation
US8615140B2 (en) 2011-11-18 2013-12-24 Canon Kabushiki Kaisha Compression of image data in accordance with depth information of pixels
US8619198B1 (en) * 2009-04-28 2013-12-31 Lucasfilm Entertainment Company Ltd. Adjusting frame rates for video applications
US8638846B1 (en) * 2003-06-23 2014-01-28 At&T Intellectual Property Ii, L.P. Systems and methods for encoding and decoding video streams
US20140044347A1 (en) * 2011-04-25 2014-02-13 Sharp Kabushiki Kaisha Mage coding apparatus, image coding method, image coding program, image decoding apparatus, image decoding method, and image decoding program
US8693551B2 (en) 2011-11-16 2014-04-08 Vanguard Software Solutions, Inc. Optimal angular intra prediction for block-based video coding
WO2014090303A1 (en) * 2012-12-12 2014-06-19 Huawei Technologies Co., Ltd. Method and apparatus for segmentation of 3d image data
US20140219559A1 (en) * 2010-09-02 2014-08-07 Edge 3 Technologies, Inc. Apparatus and Method for Segmenting an Image
US20140267603A1 (en) * 2013-03-15 2014-09-18 Intuitive Surgical Operations, Inc. Depth based modification of captured images
WO2014150159A1 (en) * 2013-03-15 2014-09-25 Intel Corporation Variable resolution depth representation
US9106922B2 (en) 2012-12-19 2015-08-11 Vanguard Software Solutions, Inc. Motion estimation engine for video encoding
US9111127B2 (en) 2012-06-04 2015-08-18 Samsung Electronics Co., Ltd. Image processing apparatus and method for three-dimensional (3D) image
US9154799B2 (en) 2011-04-07 2015-10-06 Google Inc. Encoding and decoding motion via image segmentation
US20150350560A1 (en) * 2014-05-29 2015-12-03 Apple Inc. Video coding with composition and quality adaptation based on depth derivations
US9262670B2 (en) 2012-02-10 2016-02-16 Google Inc. Adaptive region of interest
US20160071281A1 (en) * 2012-12-12 2016-03-10 Giovanni Cordara Method and apparatus for segmentation of 3d image data
US9371099B2 (en) 2004-11-03 2016-06-21 The Wilfred J. and Louisette G. Lagassey Irrevocable Trust Modular intelligent transportation system
US9392272B1 (en) 2014-06-02 2016-07-12 Google Inc. Video coding using adaptive source variance based partitioning
US9578324B1 (en) 2014-06-27 2017-02-21 Google Inc. Video coding using statistical-based spatially differentiated partitioning

Families Citing this family (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6289129B1 (en) * 1998-06-19 2001-09-11 Motorola, Inc. Video rate buffer for use with push dataflow
US6804403B1 (en) 1998-07-15 2004-10-12 Digital Accelerator Corporation Region-based scalable image coding
US6917711B1 (en) 1998-08-10 2005-07-12 Digital Accelerator Corporation Embedded quadtree wavelets in image compression
RU2001111754A (en) * 1998-09-29 2003-05-27 Сименс Акциенгезелльшафт (DE) A method and apparatus for processing digitized image composed of picture elements
US20080232477A1 (en) * 2005-09-01 2008-09-25 Koninklijke Philips Electronics, N.V. Method and Device For Coding and Decoding of Video Error Resilience
JP5147650B2 (en) * 2007-12-10 2013-02-20 富士フイルム株式会社 Distance image processing apparatus and method, the distance image reproducing apparatus and method, and program
JP2009163717A (en) * 2007-12-10 2009-07-23 Fujifilm Corp Distance image processing apparatus and method, distance image reproducing apparatus and method, and program
KR101598855B1 (en) 2010-05-11 2016-03-14 삼성전자주식회사 Apparatus and Method for 3D video coding
KR20110135786A (en) * 2010-06-11 2011-12-19 삼성전자주식회사 Method and apparatus for encoding/decoding 3d video using depth transition data
JP5281623B2 (en) * 2010-09-29 2013-09-04 日本電信電話株式会社 Image coding method, image decoding method, image encoding apparatus, image decoding apparatus and the programs
JP5281624B2 (en) * 2010-09-29 2013-09-04 日本電信電話株式会社 Image coding method, image decoding method, image encoding apparatus, image decoding apparatus and the programs
JP5357199B2 (en) 2011-03-14 2013-12-04 日本電信電話株式会社 Image coding method, image decoding method, image encoding apparatus, image decoding apparatus, image encoding program and an image decoding program
KR101707463B1 (en) * 2011-12-29 2017-02-16 인텔 코포레이션 Variable Compression depth
KR20130079261A (en) * 2011-12-30 2013-07-10 (주)휴맥스 3d image encoding method and apparatus, and decoding method and apparatus
EP2742441A1 (en) 2012-05-24 2014-06-18 Qatar Foundation A method and system for creating depth signatures
KR101336139B1 (en) * 2012-06-11 2013-12-05 동의대학교 산학협력단 System and method for motion estimating using depth camera
KR101367673B1 (en) * 2012-07-03 2014-03-03 한밭대학교 산학협력단 Method and System for Selecting Motion Search Region for Image Sequences using Depth Camera Information
US20140321561A1 (en) * 2013-04-26 2014-10-30 DDD IP Ventures, Ltd. System and method for depth based adaptive streaming of video information
CN104427291B (en) * 2013-08-19 2018-09-28 华为技术有限公司 Kinds of image processing method and apparatus
US20150181168A1 (en) * 2013-12-20 2015-06-25 DDD IP Ventures, Ltd. Interactive quality improvement for video conferencing

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4893183A (en) * 1988-08-11 1990-01-09 Carnegie-Mellon University Robotic vision system
US5309232A (en) * 1992-02-07 1994-05-03 At&T Bell Laboratories Dynamic bit allocation for three-dimensional subband video coding
US5500673A (en) * 1994-04-06 1996-03-19 At&T Corp. Low bit rate audio-visual communication system having integrated perceptual speech and video coding
US5577130A (en) * 1991-08-05 1996-11-19 Philips Electronics North America Method and apparatus for determining the distance between an image and an object
US5617334A (en) * 1995-07-21 1997-04-01 The Trustees Of Columbia University In The City Of New York Multi-viewpoint digital video coder/decoder and method

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4893183A (en) * 1988-08-11 1990-01-09 Carnegie-Mellon University Robotic vision system
US5577130A (en) * 1991-08-05 1996-11-19 Philips Electronics North America Method and apparatus for determining the distance between an image and an object
US5309232A (en) * 1992-02-07 1994-05-03 At&T Bell Laboratories Dynamic bit allocation for three-dimensional subband video coding
US5500673A (en) * 1994-04-06 1996-03-19 At&T Corp. Low bit rate audio-visual communication system having integrated perceptual speech and video coding
US5617334A (en) * 1995-07-21 1997-04-01 The Trustees Of Columbia University In The City Of New York Multi-viewpoint digital video coder/decoder and method

Non-Patent Citations (6)

* Cited by examiner, † Cited by third party
Title
"Low Bitrate Video Coding with Depth Compensation" by Van Schalkwyk et al.; IEE Proc.-Vis. Image Signal Process, vol. 141, No. 3, Jun. 1994; pp. 148-153.
"Stereo Imaging in Low Bitrate Video Coding" by Venter et al.; Dept. of Electronic and Computer Engineering, University of Pretoria, South Africa; COMSIG 1989, Proceedings--Jun. 23, 1989; pp. 115-118.
"Visual Communications and Image Processing '94" by A.K. Katsaggelos; SPIE (The Society of Photo-Optical Instrumentation Engineers); vol. 2308; Sep. 25-29, Chicago, Illinois; pp. 1576-1591.
Low Bitrate Video Coding with Depth Compensation by Van Schalkwyk et al.; IEE Proc. Vis. Image Signal Process, vol. 141, No. 3, Jun. 1994; pp. 148 153. *
Stereo Imaging in Low Bitrate Video Coding by Venter et al.; Dept. of Electronic and Computer Engineering, University of Pretoria, South Africa; COMSIG 1989, Proceedings Jun. 23, 1989; pp. 115 118. *
Visual Communications and Image Processing 94 by A.K. Katsaggelos; SPIE (The Society of Photo Optical Instrumentation Engineers); vol. 2308; Sep. 25 29, Chicago, Illinois; pp. 1576 1591. *

Cited By (179)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8421923B2 (en) 1997-02-14 2013-04-16 The Trustees Of Columbia University In The City Of New York Object-based audio-visual terminal and bitstream structure
US8917357B2 (en) 1997-02-14 2014-12-23 The Trustees Of Columbia University In The City Of New York Object-based audio-visual terminal and bitstream structure
US20070165139A1 (en) * 1997-02-14 2007-07-19 The Trustees Of Columbia University In The City Of New York Object-Based Audio-Visual Terminal And Bitstream Structure
US8922717B2 (en) 1997-02-14 2014-12-30 The Trustees Of Columbia University In The City Of New York Object-based audio-visual terminal and bitstream structure
US8531608B2 (en) 1997-02-14 2013-09-10 The Trustees Of Columbia University In The City Of New York Object-based audio-visual terminal and bitstream structure
US8115873B2 (en) 1997-02-14 2012-02-14 The Trustees Of Columbia University In The City Of New York Object-based audio-visual terminal and bitstream structure
US20090245389A1 (en) * 1997-02-14 2009-10-01 Alexandros Eleftheriadis Object-Based Audio-Visual Terminal And Bitstream Structure
US20110007219A1 (en) * 1997-02-14 2011-01-13 Alexandros Eleftheriadis Object-Based Audio-Visual Terminal And Bitstream Structure
US8854552B2 (en) 1997-02-14 2014-10-07 The Trustees Of Columbia University In The City Of New York Object-based audio-visual terminal and bitstream structure
US20070263718A1 (en) * 1997-04-24 2007-11-15 Shunichi Sekiguchi Method and apparatus for region-based moving image encoding and decoding
US20070009046A1 (en) * 1997-04-24 2007-01-11 Shunichi Sekiguchi Method and apparatus for region-based moving image encoding and decoding
US20070263727A1 (en) * 1997-04-24 2007-11-15 Shunichi Sekiguchi Method and apparatus for region-based moving image encoding and decoding
US7372903B1 (en) * 1997-07-10 2008-05-13 Mediatek, Inc. Apparatus and method for object based rate control in a coding system
US20080056385A1 (en) * 1997-10-23 2008-03-06 Yoshimi Isu Image decoding apparatus, image coding apparatus, image communications system and coded bit stream converting apparatus
US8045624B2 (en) * 1997-10-23 2011-10-25 Mitsubishi Electric Corporation Video bit stream converting apparatus
US6125142A (en) * 1997-12-02 2000-09-26 Daewoo Electronics Co., Ltd. Method and apparatus for encoding object information of a video object plane
US6898759B1 (en) * 1997-12-02 2005-05-24 Yamaha Corporation System of generating motion picture responsive to music
US20040032488A1 (en) * 1997-12-05 2004-02-19 Dynamic Digital Depth Research Pty Ltd Image conversion and encoding techniques
US20040032980A1 (en) * 1997-12-05 2004-02-19 Dynamic Digital Depth Research Pty Ltd Image conversion and encoding techniques
US7054478B2 (en) 1997-12-05 2006-05-30 Dynamic Digital Depth Research Pty Ltd Image conversion and encoding techniques
US7894633B1 (en) * 1997-12-05 2011-02-22 Dynamic Digital Depth Research Pty Ltd Image conversion and encoding techniques
US7551770B2 (en) 1997-12-05 2009-06-23 Dynamic Digital Depth Research Pty Ltd Image conversion and encoding techniques for displaying stereoscopic 3D images
US7199836B1 (en) * 1998-02-13 2007-04-03 The Trustees Of Columbia University In The City Of New York Object-based audio-visual terminal and bitstream structure
USRE43130E1 (en) * 1998-03-14 2012-01-24 Daewoo Electronics Corporation Method and apparatus for encoding interlaced macroblock texture information
USRE41951E1 (en) * 1998-03-14 2010-11-23 Daewood Electronics Corp. Method and apparatus for encoding interlaced macroblock texture information
USRE41383E1 (en) * 1998-03-14 2010-06-22 Daewoo Electronics Corp. Method and apparatus for encoding interplaced macroblock texture information
USRE43061E1 (en) * 1998-03-14 2012-01-03 Daewoo Electronics Corporation Method and apparatus for encoding interlaced macroblock texture information
USRE43060E1 (en) * 1998-03-14 2012-01-03 Daewoo Electronics Corporation Method and apparatus for encoding interlaced macroblock texture information
US6259732B1 (en) * 1998-03-14 2001-07-10 Daewoo Electronics Co., Ltd. Method and apparatus for encoding interlaced macroblock texture information
USRE43129E1 (en) * 1998-03-14 2012-01-24 Daewoo Electronics Corporation Method and apparatus for encoding interlaced macroblock texture information
US7020192B1 (en) * 1998-07-31 2006-03-28 Kabushiki Kaisha Toshiba Method of retrieving video picture and apparatus therefor
US20060109908A1 (en) * 1998-07-31 2006-05-25 Noboru Yamaguchi Method of retrieving video picture and apparatus therefor
US6879632B1 (en) * 1998-12-24 2005-04-12 Nec Corporation Apparatus for and method of variable bit rate video coding
US20060153304A1 (en) * 1999-04-17 2006-07-13 Pts Corporation Segment-based encoding system using residue coding by basis function coefficients
US20030123545A1 (en) * 1999-04-17 2003-07-03 Pulsent Corporation Segment-based encoding system using segment hierarchies
US20060262845A1 (en) * 1999-04-17 2006-11-23 Adityo Prakash Segment-based encoding system using segment hierarchies
US9064298B2 (en) * 1999-04-17 2015-06-23 Altera Corporation Segment-based encoding system using residue coding by basis function coefficients
US20030123543A1 (en) * 1999-04-17 2003-07-03 Pulsent Corporation Segment-based encoding system using residue coding by basis function coefficients
US6879633B2 (en) * 1999-04-17 2005-04-12 Pts Corporation Method and apparatus for efficient video processing
US20030128759A1 (en) * 1999-04-17 2003-07-10 Pulsent Corporation Segment-based encoding system including segment-specific metadata
US9547916B2 (en) * 1999-04-17 2017-01-17 Altera Corporation Segment-based encoding system including segment-specific metadata
US7050503B2 (en) * 1999-04-17 2006-05-23 Pts Corporation Segment-based encoding system using residue coding by basis function coefficients
US20060256852A1 (en) * 1999-04-17 2006-11-16 Adityo Prakash Segment-based encoding system including segment-specific metadata
US20040076231A1 (en) * 1999-04-17 2004-04-22 Pulsent Corporation Method and apparatus for efficient video processing
US7085319B2 (en) * 1999-04-17 2006-08-01 Pts Corporation Segment-based encoding system using segment hierarchies
US7082162B2 (en) * 1999-04-17 2006-07-25 Pts Corporation Segment-based encoding system including segment-specific metadata
US6600786B1 (en) * 1999-04-17 2003-07-29 Pulsent Corporation Method and apparatus for efficient video processing
US20040028130A1 (en) * 1999-05-24 2004-02-12 May Anthony Richard Video encoder
US6526161B1 (en) * 1999-08-30 2003-02-25 Koninklijke Philips Electronics N.V. System and method for biometrics-based facial feature extraction
US6792154B1 (en) 1999-10-07 2004-09-14 World Multicast.com, Inc Video compression system and method using time
US6909747B2 (en) * 2000-03-15 2005-06-21 Thomson Licensing S.A. Process and device for coding video images
US20010024471A1 (en) * 2000-03-15 2001-09-27 Philippe Bordes Process and device for coding video images
US20020012472A1 (en) * 2000-03-31 2002-01-31 Waterfall Andrew E. Method for visualization of time sequences of 3D optical fluorescence microscopy images
US6888894B2 (en) * 2000-04-17 2005-05-03 Pts Corporation Segmenting encoding system with image segmentation performed at a decoder and encoding scheme for generating encoded data relying on decoder segmentation
US20030035484A1 (en) * 2000-04-17 2003-02-20 Pulsent Corporation Segmenting encoding system with image segmentation performed at a decoder and encoding scheme for generating encoded data relying on decoder segmentation
US7082166B2 (en) * 2000-04-17 2006-07-25 Pts Corporation Decoder for decoding segment-based encoding of video data using segmentation performed at a decoder
US20030063676A1 (en) * 2000-04-17 2003-04-03 Pulsent Corporation Decoder for decoding segment-based encoding of video data using segmentation performed at a decoder
US20020048395A1 (en) * 2000-08-09 2002-04-25 Harman Philip Victor Image conversion and encoding techniques
US7035451B2 (en) * 2000-08-09 2006-04-25 Dynamic Digital Depth Research Pty Ltd. Image conversion and encoding techniques
WO2002052393A1 (en) * 2000-12-22 2002-07-04 The Government Of The United States Of America, As Represented By The Secretary, Department Of Health And Human Services Positioning an item in three dimensions via a graphical representation
US20070104271A1 (en) * 2000-12-26 2007-05-10 Nec Corporation Moving picture encoding system
WO2002060183A1 (en) * 2001-01-24 2002-08-01 Oy Gamecluster Ltd A method for compressing video information
US7894525B2 (en) 2001-01-24 2011-02-22 Oy Gamecluster Ltd. Method for compressing video information
US20040095999A1 (en) * 2001-01-24 2004-05-20 Erick Piehl Method for compressing video information
US7321624B1 (en) * 2001-03-16 2008-01-22 Objectvideo, Inc. Bit-rate allocation system for object-based video encoding
US20020176625A1 (en) * 2001-04-04 2002-11-28 Mitsubishi Electric Research Laboratories, Inc. Method for segmenting multi-resolution video objects
US6859554B2 (en) 2001-04-04 2005-02-22 Mitsubishi Electric Research Laboratories, Inc. Method for segmenting multi-resolution video objects
US6965379B2 (en) 2001-05-08 2005-11-15 Koninklijke Philips Electronics N.V. N-view synthesis from monocular video of certain broadcast and stored mass media content
US20040156433A1 (en) * 2001-06-11 2004-08-12 Comer Mary Lafuze Motion compensation for fine-grain scalable video
US7912124B2 (en) 2001-06-11 2011-03-22 Thomson Licensing Motion compensation for fine-grain scalable video
US20030043912A1 (en) * 2001-08-23 2003-03-06 Sharp Laboratories Of America, Inc. Method and apparatus for motion vector coding with global motion parameters
US7050500B2 (en) * 2001-08-23 2006-05-23 Sharp Laboratories Of America, Inc. Method and apparatus for motion vector coding with global motion parameters
US7809023B2 (en) * 2002-06-26 2010-10-05 Standard Microsystems Corporation Communication system and method for sending isochronous streaming data within a frame segment using a signaling byte
US20080095190A1 (en) * 2002-06-26 2008-04-24 Standard Microsystems Corp. Communication System and Method for Sending Isochronous Streaming Data Within a Frame Segment Using a Signaling Byte
US8570378B2 (en) 2002-07-27 2013-10-29 Sony Computer Entertainment Inc. Method and apparatus for tracking three-dimensional movements of an object using a depth sensing camera
US20060256081A1 (en) * 2002-07-27 2006-11-16 Sony Computer Entertainment America Inc. Scheme for detecting and tracking user manipulation of a game controller body
US9381424B2 (en) 2002-07-27 2016-07-05 Sony Interactive Entertainment America Llc Scheme for translating movements of a hand-held controller into inputs for a system
US9393487B2 (en) 2002-07-27 2016-07-19 Sony Interactive Entertainment Inc. Method for mapping movements of a hand-held controller to game commands
US20090122146A1 (en) * 2002-07-27 2009-05-14 Sony Computer Entertainment Inc. Method and apparatus for tracking three-dimensional movements of an object using a depth sensing camera
US20060287087A1 (en) * 2002-07-27 2006-12-21 Sony Computer Entertainment America Inc. Method for mapping movements of a hand-held controller to game commands
US8313380B2 (en) 2002-07-27 2012-11-20 Sony Computer Entertainment America Llc Scheme for translating movements of a hand-held controller into inputs for a system
US20050276323A1 (en) * 2002-09-27 2005-12-15 Vanguard Software Solutions, Inc. Real-time video coding/decoding
US7336720B2 (en) * 2002-09-27 2008-02-26 Vanguard Software Solutions, Inc. Real-time video coding/decoding
US7555161B2 (en) * 2002-11-12 2009-06-30 Qinetiq Limited Image analysis
US20060002608A1 (en) * 2002-11-12 2006-01-05 Qinetiq Limited Image analysis
US20060071932A1 (en) * 2002-11-21 2006-04-06 Koninklijke Philips Electronics N.V. Method and apparatus for visualizing a sequece of volume images
US20040184653A1 (en) * 2003-03-20 2004-09-23 Baer Richard L. Optical inspection system, illumination apparatus and method for use in imaging specular objects based on illumination gradients
US8638846B1 (en) * 2003-06-23 2014-01-28 At&T Intellectual Property Ii, L.P. Systems and methods for encoding and decoding video streams
US7535959B2 (en) * 2003-10-16 2009-05-19 Nvidia Corporation Apparatus, system, and method for video encoder rate control
US20050084007A1 (en) * 2003-10-16 2005-04-21 Lightstone Michael L. Apparatus, system, and method for video encoder rate control
US20050232354A1 (en) * 2004-04-14 2005-10-20 Yi-Kai Chen Rate controlling method and apparatus for use in a transcoder
US7397855B2 (en) * 2004-04-14 2008-07-08 Corel Tw Corp. Rate controlling method and apparatus for use in a transcoder
US20050244071A1 (en) * 2004-04-29 2005-11-03 Mitsubishi Denki Kabushiki Kaisha Adaptive quantization of depth signal in 3D visual coding
US7558432B2 (en) 2004-04-29 2009-07-07 Mitsubishi Electric Corporation Adaptive quantization of depth signal in 3D visual coding
US20080279285A1 (en) * 2004-07-02 2008-11-13 Koninklijke Philips Electronics, N.V. Video Processing
US20060007301A1 (en) * 2004-07-08 2006-01-12 Cho Gyoung I 3D television broadcasting system
US8537204B2 (en) * 2004-07-08 2013-09-17 Gyoung Il Cho 3D television broadcasting system
US20060088206A1 (en) * 2004-10-21 2006-04-27 Kazunari Era Image processing apparatus, image pickup device and program therefor
US7643672B2 (en) * 2004-10-21 2010-01-05 Kazunari Era Image processing apparatus, image pickup device and program therefor
US9371099B2 (en) 2004-11-03 2016-06-21 The Wilfred J. and Louisette G. Lagassey Irrevocable Trust Modular intelligent transportation system
US20060117268A1 (en) * 2004-11-30 2006-06-01 Micheal Talley System and method for graphical element selection for region of interest compression
KR100656783B1 (en) 2004-12-14 2006-12-12 한국전자통신연구원 Apparatus and its method of transmitting an image associated with a depth and apparatus and its method of rendering stereoscopic images using it
US20060256233A1 (en) * 2005-05-16 2006-11-16 Douglas Chin Systems, methods, and apparatus for video encoding
US8587655B2 (en) 2005-07-22 2013-11-19 Checkvideo Llc Directed attention digital video recordation
US20080273787A1 (en) * 2005-09-09 2008-11-06 Qinetiq Limited Automated Selection of Image Regions
US8265370B2 (en) * 2005-09-09 2012-09-11 Qinetiq Limited Automated selection of image regions
US20070120844A1 (en) * 2005-11-28 2007-05-31 Baumer Optronic Gmbh Method and apparatus for the graphical operation of real-time image processing systems
US9264735B2 (en) * 2006-07-06 2016-02-16 Canon Kabushiki Kaisha Image encoding apparatus and method for allowing motion vector detection
US20130003853A1 (en) * 2006-07-06 2013-01-03 Canon Kabushiki Kaisha Motion vector detection apparatus, motion vector detection method, image encoding apparatus, image encoding method, and computer program
US20080174678A1 (en) * 2006-07-11 2008-07-24 Solomon Research Llc Digital imaging system
US7612805B2 (en) * 2006-07-11 2009-11-03 Neal Solomon Digital imaging system and methods for selective image filtration
US8970680B2 (en) * 2006-08-01 2015-03-03 Qualcomm Incorporated Real-time capturing and generating stereo images and videos with a monoscopic low power mobile device
US9509980B2 (en) 2006-08-01 2016-11-29 Qualcomm Incorporated Real-time capturing and generating viewpoint images and videos with a monoscopic low power mobile device
US20080031327A1 (en) * 2006-08-01 2008-02-07 Haohong Wang Real-time capturing and generating stereo images and videos with a monoscopic low power mobile device
US8395658B2 (en) 2006-09-07 2013-03-12 Sony Computer Entertainment Inc. Touch screen-like user interface that does not require actual touching
US20080062257A1 (en) * 2006-09-07 2008-03-13 Sony Computer Entertainment Inc. Touch screen-like user interface that does not require actual touching
US20080080789A1 (en) * 2006-09-28 2008-04-03 Sony Computer Entertainment Inc. Object detection using video input combined with tilt angle information
US8310656B2 (en) 2006-09-28 2012-11-13 Sony Computer Entertainment America Llc Mapping movements of a hand-held controller to the two-dimensional image plane of a display screen
US8781151B2 (en) 2006-09-28 2014-07-15 Sony Computer Entertainment Inc. Object detection using video input combined with tilt angle information
US20090009370A1 (en) * 2007-07-05 2009-01-08 Mega Chips Corporation Transcoder
US7714751B2 (en) * 2007-07-05 2010-05-11 Megachips Corporation Transcoder controlling generated codes of an output stream to a target bit rate
US20090066693A1 (en) * 2007-09-06 2009-03-12 Roc Carson Encoding A Depth Map Into An Image Using Analysis Of Two Consecutive Captured Frames
US8023562B2 (en) 2007-09-07 2011-09-20 Vanguard Software Solutions, Inc. Real-time video coding/decoding
US20090067504A1 (en) * 2007-09-07 2009-03-12 Alexander Zheludkov Real-time video coding/decoding
US8665960B2 (en) 2007-09-07 2014-03-04 Vanguard Software Solutions, Inc. Real-time video coding/decoding
US20090148038A1 (en) * 2007-12-10 2009-06-11 Youichi Sawachi Distance image processing apparatus and method
WO2010017239A1 (en) * 2008-08-05 2010-02-11 Qualcomm Incorporated System and method to generate depth data using edge detection
US8184196B2 (en) * 2008-08-05 2012-05-22 Qualcomm Incorporated System and method to generate depth data using edge detection
US8477232B2 (en) 2008-08-05 2013-07-02 Qualcomm Incorporated System and method to capture depth data of an image
US20100033617A1 (en) * 2008-08-05 2010-02-11 Qualcomm Incorporated System and method to generate depth data using edge detection
US9924161B2 (en) 2008-09-11 2018-03-20 Google Llc System and method for video coding using adaptive segmentation
US8325796B2 (en) 2008-09-11 2012-12-04 Google Inc. System and method for video coding using adaptive segmentation
US20100061444A1 (en) * 2008-09-11 2010-03-11 On2 Technologies Inc. System and method for video encoding using adaptive segmentation
US20100124274A1 (en) * 2008-11-17 2010-05-20 Cheok Lai-Tee Analytics-modulated coding of surveillance video
US9215467B2 (en) * 2008-11-17 2015-12-15 Checkvideo Llc Analytics-modulated coding of surveillance video
CN102326391B (en) 2009-02-23 2014-07-09 日本电信电话株式会社 Multi-view image coding device, multi-view image decoding method, multi-view image decoding device, multi-view image decoding method
CN102326391A (en) * 2009-02-23 2012-01-18 国立大学法人名古屋大学 Multi-view image coding method, multi-view image decoding method, multi-view image coding device, multi-view image decoding device, multi-view image coding program, and multi-view image decoding program
US8548228B2 (en) 2009-02-23 2013-10-01 Nippon Telegraph And Telephone Corporation Multi-view image coding method, multi-view image decoding method, multi-view image coding device, multi-view image decoding device, multi-view image coding program, and multi-view image decoding program
US8619198B1 (en) * 2009-04-28 2013-12-31 Lucasfilm Entertainment Company Ltd. Adjusting frame rates for video applications
US20100290712A1 (en) * 2009-05-13 2010-11-18 Seiko Epson Corporation Image processing method and image processing apparatus
US8542932B2 (en) * 2009-05-13 2013-09-24 Seiko Epson Corporation Image processing method and image processing apparatus using different compression methods
US20120236024A1 (en) * 2009-12-04 2012-09-20 Panasonic Corporation Display control device, and method for forming display image
US8526488B2 (en) 2010-02-09 2013-09-03 Vanguard Software Solutions, Inc. Video sequence encoding system and algorithms
EP2555527A1 (en) * 2010-03-26 2013-02-06 Sony Corporation Image processor, image processing method, and program
US8879840B2 (en) 2010-03-26 2014-11-04 Sony Corporation Image processor, image processing method, and program for shift-changing depth data of an image
EP2555527A4 (en) * 2010-03-26 2014-01-01 Sony Corp Image processor, image processing method, and program
US20120008672A1 (en) * 2010-07-07 2012-01-12 Gaddy William L System and method for transmission, processing, and rendering of stereoscopic and multi-view images
US8774267B2 (en) * 2010-07-07 2014-07-08 Spinella Ip Holdings, Inc. System and method for transmission, processing, and rendering of stereoscopic and multi-view images
US20140219559A1 (en) * 2010-09-02 2014-08-07 Edge 3 Technologies, Inc. Apparatus and Method for Segmenting an Image
US9154799B2 (en) 2011-04-07 2015-10-06 Google Inc. Encoding and decoding motion via image segmentation
US20140044347A1 (en) * 2011-04-25 2014-02-13 Sharp Kabushiki Kaisha Mage coding apparatus, image coding method, image coding program, image decoding apparatus, image decoding method, and image decoding program
WO2012151548A1 (en) * 2011-05-04 2012-11-08 Cavium, Inc. Low latency rate control system and method
US9445107B2 (en) 2011-05-04 2016-09-13 Cavium, Inc. Low latency rate control system and method
US8831108B2 (en) * 2011-05-04 2014-09-09 Cavium, Inc. Low latency rate control system and method
US20120281767A1 (en) * 2011-05-04 2012-11-08 Alberto Duenas Low latency rate control system and method
US8693551B2 (en) 2011-11-16 2014-04-08 Vanguard Software Solutions, Inc. Optimal angular intra prediction for block-based video coding
US9451266B2 (en) 2011-11-16 2016-09-20 Vanguard Video Llc Optimal intra prediction in block-based video coding to calculate minimal activity direction based on texture gradient distribution
US9131235B2 (en) 2011-11-16 2015-09-08 Vanguard Software Solutions, Inc. Optimal intra prediction in block-based video coding
US9307250B2 (en) 2011-11-16 2016-04-05 Vanguard Video Llc Optimization of intra block size in video coding based on minimal activity directions and strengths
US8891633B2 (en) 2011-11-16 2014-11-18 Vanguard Video Llc Video compression for high efficiency video coding using a reduced resolution image
US8615140B2 (en) 2011-11-18 2013-12-24 Canon Kabushiki Kaisha Compression of image data in accordance with depth information of pixels
US9262670B2 (en) 2012-02-10 2016-02-16 Google Inc. Adaptive region of interest
CN104396236A (en) * 2012-04-19 2015-03-04 华为技术有限公司 Using depth information to assist motion compensation-based video coding
US9584806B2 (en) * 2012-04-19 2017-02-28 Futurewei Technologies, Inc. Using depth information to assist motion compensation-based video coding
US20130279588A1 (en) * 2012-04-19 2013-10-24 Futurewei Technologies, Inc. Using Depth Information to Assist Motion Compensation-Based Video Coding
US9111127B2 (en) 2012-06-04 2015-08-18 Samsung Electronics Co., Ltd. Image processing apparatus and method for three-dimensional (3D) image
WO2014090303A1 (en) * 2012-12-12 2014-06-19 Huawei Technologies Co., Ltd. Method and apparatus for segmentation of 3d image data
US10096116B2 (en) * 2012-12-12 2018-10-09 Huawei Technologies Co., Ltd. Method and apparatus for segmentation of 3D image data
US20160071281A1 (en) * 2012-12-12 2016-03-10 Giovanni Cordara Method and apparatus for segmentation of 3d image data
CN104982032A (en) * 2012-12-12 2015-10-14 华为技术有限公司 Method and apparatus for segmentation of 3D image data
US20160005181A1 (en) * 2012-12-12 2016-01-07 Huawei Technologies Co., Ltd. Method and apparatus for segmentation of 3d image data
US9106922B2 (en) 2012-12-19 2015-08-11 Vanguard Software Solutions, Inc. Motion estimation engine for video encoding
US20140267603A1 (en) * 2013-03-15 2014-09-18 Intuitive Surgical Operations, Inc. Depth based modification of captured images
US9860510B2 (en) * 2013-03-15 2018-01-02 Intuitive Surgical Operations, Inc. Depth based modification of captured images
WO2014150159A1 (en) * 2013-03-15 2014-09-25 Intel Corporation Variable resolution depth representation
US20150350560A1 (en) * 2014-05-29 2015-12-03 Apple Inc. Video coding with composition and quality adaptation based on depth derivations
US9876964B2 (en) * 2014-05-29 2018-01-23 Apple Inc. Video coding with composition and quality adaptation based on depth derivations
US9392272B1 (en) 2014-06-02 2016-07-12 Google Inc. Video coding using adaptive source variance based partitioning
US9578324B1 (en) 2014-06-27 2017-02-21 Google Inc. Video coding using statistical-based spatially differentiated partitioning

Also Published As

Publication number Publication date Type
KR20000049031A (en) 2000-07-25 application
EP1012776A1 (en) 2000-06-28 application
EP1012776A4 (en) 2002-04-24 application
CA2268191A1 (en) 1998-04-16 application
WO1998015915A1 (en) 1998-04-16 application
JP2001509325A (en) 2001-07-10 application

Similar Documents

Publication Publication Date Title
US5912706A (en) Video coding/decoding apparatus which codes information indicating whether an intraframe or interframe predictive coding mode is used
US5949490A (en) Distributing video buffer rate control over a parallel compression architecture
US5764814A (en) Representation and encoding of general arbitrary shapes
US6148026A (en) Mesh node coding to enable object based functionalities within a motion compensated transform video coder
US5687097A (en) Method and apparatus for efficiently determining a frame motion vector in a video encoder
US6088061A (en) Video coding device and video decoding device
US6052414A (en) Moving picture coding method and apparatus for low bit rate systems using dynamic motion estimation
US5764803A (en) Motion-adaptive modelling of scene content for very low bit rate model-assisted coding of video sequences
US5661524A (en) Method and apparatus for motion estimation using trajectory in a digital video encoder
US5295201A (en) Arrangement of encoding motion image signals using motion compensation and orthogonal transformation
US5832115A (en) Ternary image templates for improved semantic compression
US5438374A (en) System and method for filtering video signals
US6983018B1 (en) Efficient motion vector coding for video compression
US6128041A (en) Method and apparatus for binary shape encoding
US5751378A (en) Scene change detector for digital video
US6625316B1 (en) Image processing apparatus and method, and image processing system
US5731850A (en) Hybrid hierarchial/full-search MPEG encoder motion estimation
US5729295A (en) Image sequence encoding device and area extracting device
US6678322B1 (en) Video data coding device, video data coding method, video data transmitting device, and video data recording medium
US5559557A (en) Motion video coding with adaptive precision for DC component coefficient quantization and variable length coding
US6754269B1 (en) Video encoding apparatus and video decoding apparatus
Fukuhara et al. Very low bit-rate video coding with block partitioning and adaptive selection of two time-differential frame memories
US5617334A (en) Multi-viewpoint digital video coder/decoder and method
US5883678A (en) Video coding and video decoding apparatus for reducing an alpha-map signal at a controlled reduction ratio
EP0469835A2 (en) Image processing apparatus and method

Legal Events

Date Code Title Description
AS Assignment

Owner name: COLUMBIA UNIVERSITY, TRUSTEES OF, THE, IN THE CIT

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ELEFTHERIADIS, ALEXANDROS;ANASTASSIOU, DIMITRIS;CHANG, SHIF-FU;AND OTHERS;REEL/FRAME:008271/0031;SIGNING DATES FROM 19961113 TO 19961127

CC Certificate of correction
REMI Maintenance fee reminder mailed
LAPS Lapse for failure to pay maintenance fees
FP Expired due to failure to pay maintenance fee

Effective date: 20040425