KR20050032118A - System and method of streaming 3-d wireframe animations - Google Patents

Abstract

Optimal resilience to errors in packetized streaming 3-D wireframe animation is achieved by partitioning the stream into layers and applying unequal error correction coding to each layer independently to maintain the same overall bitrate. The unequal error protection scheme for each of the layers combined with error concealment at the receiver achieves graceful degradation of streamed animation at higher packet loss rates than approaches that do not account for subjective parameters such as visual smoothness.

Description

System and method of streaming 3-D wireframe animations

This application claims the priority of US Provisional Patent Application No. 60 / 404,410, filed August 20, 2002, which is incorporated herein by reference.

Non-Application No. 10 / 198,129, filed on July 19, 2002, entitled "Coding of Animated 3-D Wireframe Models For Internet Streaming Applications: Methods, Systems and Program Products," was assigned to the applicant of the present application, Reference is made herein.

The present invention relates to a method and system for streaming data, more specifically 3-D wireframe animations.

The Internet has been rapidly developing in recent years from a low bandwidth, text-only collaboration medium to a rich, interactive, real-time audiovisual virtual world. It includes a number of applications, environments and users in which 3-D animations constitute a driving force. Animated 3-D models enable intuitive and realistic interaction with displayed objects and allow effects that cannot be achieved with conventional audiovisual animations. Thus, the current challenge is to consolidate the animated 3-D geometry as a new data stream in the current situation involving the infrastructure of the Internet, strengthening existing network environments and taking into account limited resources. While static 3-D mesh geometry compression has been actively studied for the past decade, little research has been done to compress dynamic 3-D geometry that extends static 3-D meshes into the time domain.

The most common representations for 3-D static models are polygonal or triangular meshes. The representations allow for proximity models of any shape and topology within the desired precision or quality. Effective algorithms and data structures exist to create, modify, compress, transmit and store the static meshes. In the future, non-static, stream types that introduce a time dimension will require scalable solutions to withstand the limited resources (bandwidth) and characteristics (channel errors) of the network.

The problem of 3-D wireframe animation streaming described herein can be described as follows: (i) A time-dependent 3-D mesh is compressively compressed into a sequence of wireframe animation frames, (ii) available The rate R is known (or determined with respect to the corresponding TCP-friendly rate), (i) the channel error characteristics are known, and (i) the fractional part of the available rate C (C <R) It is assumed that it can be left for channel coding. The issue is then to identify the optimal number of bits to be assigned to each critical level (layer) in the animation scene to maximize the perceived quality of the time-dependent mesh on the receiver side.

Most animation coding approaches use objective metrics to achieve hierarchical coding of static 3-D meshes. There is a need for animation approaches that use subjective quantities such as visual smoothness to provide an improved appearance of animation. Described here are forward error correction (FEC) codes associated with a 3-D wire frame animation codec and its bitstream content. In addition to the visual distortion metrics, unequal error protection (UEP) methods and receiver-based concealment methods are also described.

1A is a block diagram of a 3-D animation codec.

1B is a block diagram of a decoder.

2 is a comparison of distortion metrics including PSNR, Hausdorff Distance, and visual smoothness.

3A is a flow diagram of a method of error tolerant wireframe streaming.

3B illustrates a flowchart in accordance with an aspect of the present invention.

4 is a comparison of three error concealment methods for a sequence of wireframe animation TELLY.

5 is a comparison of visual smoothness (VS) in which frames of three layers of wireframe animation TELLY are transmitted and decoded.

FIG. 6 is a comparison of visual smoothness in which frames of two layers of wireframe animation BOUNCEBALL are transmitted and encoded. FIG.

The present invention focuses on source and channel coding techniques for error tolerant time-dependent 3-D mesh streaming over the Internet taking into account network bandwidth and taking into account the bursty loss nature of the channel.

An exemplary embodiment of the present invention includes calculating visual smoothness for each node in a wireframe mesh, and associated with the wireframe mesh such that the average visual smoothness value associated with each layer reflects the importance of the individual layers in the animation sequence. A method of streaming data comprising stratifying data into a plurality of layers. Still other embodiments of the present invention may include a bitstream generated according to processing similar to the above methods and apparatus for generating and transmitting bitstreams or receiving bitstreams.

Additional features and advantages of the invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the invention. The features and advantages of the invention may be realized and obtained by means and combinations particularly pointed out in the appended claims. These and other features of the present invention will become more fully apparent from the following specification and appended claims, and will also be learned by practicing the invention as described herein.

BRIEF DESCRIPTION OF DRAWINGS To describe the above and other advantages and features of the present invention in a manner that can be obtained, the more specific description of the present invention described above will be briefly described with reference to specific embodiments shown in the accompanying drawings. It is to be understood that these drawings describe only exemplary embodiments of the invention and are not to be considered limiting of its scope, for the invention will be described and described in more detail through the use of the accompanying drawings.

In general, many studies are in progress for streaming video over computer networks, especially the Internet. There is very little research in the field of streaming 3-D wireframe animation. Both treatments have some similarities, but the two treatments are mostly different. Different data passes through the network, and thus losses affect signal reconstruction differently. The perceptual effects of such losses are rarely addressed in the art. Many studies in this range have relied on objective measures such as PSNR instead of considering subjective effects.

The present invention combines the concepts from a number of disciplines to address the problems of how to achieve optimal resilience to errors by perceptual effects at the receiver side. In this respect, the present invention relates to subjective quality of animation, such as, for example, mesh surface smoothing. In order to achieve an improved coding scheme that takes into account subjective factors, one aspect of the present invention maintains the same overall bit rate while minimizing the perceptual effects of error, as measured by the distortion metric for static 3-D mesh compression. And independently applying Reed-Solomom (RS) forward error correction (FEC) codes for each layer and dividing the animation stream into multiple layers. Graceful degradation of animations streamed at higher packet loss rates than other approaches is achieved by unequal error protection (UEP), which combines error concealment (EC) and an efficient packetization scheme. Can be.

The present disclosure first provides an overview of a 3-D animation codec that introduces an associated notation, an overview of error correction RS codes, derivation of a channel model, and an exemplary description of UEP packetization for an encoded bitstream.

Vertices m _j of a time-dependent 3-D mesh form an indexed set M _t = {m _jt ; j = 1,2, ..., n} at time t, where n is the mesh Is the number of my vertices. Assuming a vertex has three spatial components (x _j , y _j , z _j ) and no connection changes occur in time, at time t, the indexed set of data is represented by the position matrix M _t as Can indicate:

The indexed set of vertices is divided into natural divisions by intuition called nodes. The term "node" is used to emphasize the correspondence of nodes with those as defined in VRML. The position matrix corresponding to the i th node is denoted by N _{i, t} . Note that without losing generality, the vertex matrix can now be expressed as follows for k nodes:

For ease of presentation, the terms N _{i, t} (i = 1,2, ..., k) denote a matrix and may be used to refer to the i th node as well. The purpose of the 3D-animation compression algorithm is to compress the sequence M _t of the metrics that form the composite animation for transmission over the communication channel. Clearly, for free-form animations of a 3-D mesh, the mesh's coordinates may indicate a high variance that makes it unsuitable for compressing the M _t matrices. Thus, the signal can be defined as a set of nonzero displacements of all vertices at all nodes at time t:

According to this notation, it can be represented by the displacement matrix D _t as follows:

Or equally, using node matrices:

Where F _{i, t} is the displacement matrix of node i, for l nodes (i = 1,2, ..., l). Note that since D _t does not include vertices with zero displacement on all axes, the dimension of D _t is reduced to p ≦ n compared to M _t . Note also that if the vertices at the node do not change (F _{i; t} = 0), 1 ≦ k is maintained. In 3D-animation techniques, sparse animations are referred to as sequences with p <n and 1 <k, and when p = n and 1 = k, the animation is called dense. If the encoder can control the parameters p and l, it can generate a layered bitstream (by adjusting parameter l), where all layers L are scalable (by adjusting parameter p) The point is clear. The sparsity (or density) of the animation is limited to the density factor in the range [0..1] as defined below:

Where F is the number of animation frames and k is the number of nodes in the reference model. If p → n and l → k, then df → 1, thus completing the animation.

The concepts described and summarized in equation (1) above are suitable for DPCM coders, as described below. The coding process assumes that an initial wireframe model M ₀ , referred to herein as a reference model, already exists at the receiver. If the transmission is assumed to be on the same lossy channel as the time-dependent mesh, then the reference model can be compressed and streamed in a conventional manner for static 3-D mesh transmission, with error protection. The existing methods can interoperate with and accommodate static mesh transmissions and will not be discussed in more detail here.

In the context of the 3D-animation codec, the I-frame describes the changes from the reference model M ₀ to the model at the current time instant t. The P-frame describes changes in the model from the previous time instant t-1 to the current time instant t. The corresponding position and displacement matrices for the I and P frames are respectively Is displayed.

1A shows an exemplary block diagram 100 of coding processing in accordance with an aspect of the present invention. The diagram shows a DPCM encoder using the temporal correlation of the displacement of each vertex along all axes in 3-D space. To encode a P-frame, the decoded set (animation frame or displacement matrix) of the previous instance is estimated value (108, 106). Used as (Samely for I-frame encoding, the predicted matrix is And this is the displacement matrix for the reference model at t = 0.) Then, the prediction error E _t , i.e., the difference 106 between the current displacement matrix and the predicted matrix is calculated 102 and quantized 104 ( )do. Finally, the quantized samples are entropy coded (C _t ) using arithmetic coding algorithm 110 adapted to adjust for unknown data statistics. The prediction scheme prevents quantization error accumulation.

DPCM decoder 120 is shown in FIG. 1B. Decoder 120 first arithmetically decodes received samples 122 (C ′ _t ) and calculates decoded samples 124, 126 ( ). The quantization range of each node is determined by its rectangular bounding box. The quantization step size may be assumed to be the same for all nodes, or may vary to shape the encoded bitstream rate. Allowing different quantization step sizes for different nodes can cause artifacts such as mesh cracks, especially at the boundaries between nodes.

Since D _t does not include vertices with zero displacement on all axes, the present disclosure described that the dimension of D _t is reduced to p ≦ n compared to M _t . This feature provides the advantage of not reducing animation frames compared to the BIFS-animation of MPEG-4. For sparse D _t matrices, it may be the case that the entire node is not animated to allow good animation immunity and produce a scalable bitstream. Further, in the case of F _{i, t} = 0,? I∈ [1..l], it causes an "empty" frame, so that the displacement matrix D _t is zero. This property is similar to the inherent silence period in voice audio streams and can be utilized at the application layer of RTP-based receivers to absorb network jitter. Major inter-stream synchronization can also be achieved in many applications (eg, lip synchronization of a 3-D animated virtual salesman with packet voice).

Next, the channel model and error correction codes are described. The idea of forward error correction (FEC) is to send additional redundant packets that can be used to reconstruct lost packets at the receiver side. In FEC processing according to a preferred embodiment of the present invention, Reed-Solomon (RS) codes are used for packets. RS codes are the only known non-trivial maximum distance separable codes and are therefore suitable for protection against packet losses over massive loss channels. RS length of n (n, k) code, and D k are Galois field (Galois field) is defined on the GF (2 ^q), k q- bit information symbols of the n number of code words of the symbols, that is, ^q n≤2 Encode to -1. The sender needs to store copies of k information packets in order to calculate nk extra packets. The resulting n packets are stacked in a block of packet (BOP) structure. Such BOP structures are known in the art and are therefore not described in more detail here. To maintain a constant overall channel data rate, the source rate is reduced by a fractional k / n called code rate, resulting in an initial reduced animation quality. The receiver starts decoding as soon as it receives any k correction symbols, or packets of a BOP.

Indeed, the underlying bursty loss process of the Internet is very complex, but can be very approximated with a two-state Markov model. The two states are state G (good), in which packets are received correctly and timely, and B (bad), delayed by the point in time at which packets can be considered lost or lost. Since state transition probabilities p _GB and p _BG describe the model sufficiently, but it is not intuitive enough, the model can be expressed as follows using the average loss probability P _B and the average burst length L _B :

For the selection of RS code parameters, the probability that the BOP cannot be reconstructed by the erasure decoder as a function of the channel and RS code parameters needs to be known. For RS (n, k) codes, this is the probability that more than nk packets are lost in the BOP, which is called the block error rate P _BER . P (m, n) is the probability of m missing packets within a block of n packets, which is also called a block error density function. The calculation is then as follows:

The average loss probability P _B and the average loss burst L _B corresponding to the two-state Markov model described above relate to the block error density function P (m, n). The exact nature of the relationship is widely studied and derived from the paper. Here we adapt the derivation for the bit error channels to the packet loss channel.

The Markov model as described above is a renewal model, i.e. a loss event resets the loss processing. The model is determined by the distribution of error-free intervals (gaps). If an event of gap length v occurs such that v-1 packets are received between two missing packets, the gap density function g (v) gives the probability of gap length v, i.e. g (v) = Pr (0 ^v-1 | 1). The gap distribution function G (v) gives a probability of gap length greater than v-1, that is, G (v) = Pr (0 ^v-1 | 1). If all packets are lost in state B of our model and all packets are received in state G, we get:

Let R (m, n) be the probability of m-1 packet losses in the next n-1 packets after the lost packet. The probability can be calculated from the iterations:

Then, the block error density function P (m, n) or the probability of m missing packets in a block of n packets is given by:

Where P _B is the mean error probability.

From Equation 5, note that P (m, n) determines the performance of the FEC scheme and can be expressed as a function of P _B , L _B using Equations 3 and 4. As described below, the representation of P (m, n) can be used in RS (m, n) FEC schemes for optimized source / channel rate allocation that minimizes visual distortion.

Next, the bitstream format and packetization process are described. The output bitstream of the 3-D animation codec needs to be properly packetized for streaming to an application-level transfer protocol such as RTP. The above process for a single layer bit-stream is known and its main features are summarized in the following three concepts:

(1) To describe which nodes of the model are animated, animation masks, nodemasks and vertexmasks are defined in a manner similar to BIFS-animation. The nodemask is essentially a bit-mask where each bit indicates if the corresponding node in the node table will be animated, if set. Since the reference wireframe model already exists, the node table (an ordered list of all nodes in the scene) is known a priori to the receiver or downloaded by other means. In a similar manner, vertex masks are defined one per axis for the vertices to be animated.

(2) In its simplest form, one frame (representing one application data unit (ADU)) is contained in one RTP packet. In this regard, the output bitstream of the 3D-animation codec is 'naturally packetizable' according to the known Application Level Framing (ALF) principle. The RTP packet payload format is considered to begin with node and vertex masks, followed by samples encoded along each axis.

(3) The M bits in the RTP header should be set first in a series of "empty" frames, which (if present) can be grouped together.

The simple form is sufficient for simple animations with an appropriate number of vertices. However, sequences or high-resolution meshes with high scene complexity may generate multiple coded data after compression, potentially causing frames that exceed the path MTU. In such cases, raw packetization at a single layer requires the specification of fragmentation rules for the RTP payload, which may not always be constant for ALF. In addition, frames packetized directly in the RTP as described above create a variable bit rate stream due to their varying lengths.

In order to (a) accommodate layered bitstreams and (b) generate a constant bitrate stream, a more efficient packetization scheme is obtained that satisfies the requirements described above. The efficiency can be achieved by suitably adapting a block structure known as a block of packets (BOP). In the method, the encoded frames of the single layers S _P - open grid structure is subsequently placed in a line in the order of the n- line by (S _P -column grid structure), then the RS codes as vertical grid Is generated. For data frames protected by the RS (n, k) erasure code, error tolerance information is appended so that the length of the grid is n for k source data frames. The method is most suitable for packet networks with burst packet errors and can be fully described by the sequence frame rate FR, packet size S _P , data packet rate F _{BOP at BOP} , and RS code (n, k).

Intuitively, for a _BOP that constitutes F _BOP data frames with long packets of S _P bytes, at the FR frame rate, the total source and channel bit rate R are given by:

The equation serves as a guide to the design of efficient packetization schemes by properly balancing the parameters F _BOP , n and S _P. It also includes a trade-off between delay and immunity. For layered bitstreams, a design is needed for one BOP structure per layer. By varying the parameters in Equation 6, different RS code rates can be assigned to each layer, thus providing each layer with an uneven level of error protection. In practice, the following describes how the parameters are adjusted for an application of 3-D animation streaming that takes into account the measure of visual error.

The original mesh M _t and its decoded mesh at time t to measure visual loss resulting from incomplete reconstruction of the animated mesh at the receiver A metric that can capture the visual difference between the two is required. The simplest measure is the RMS geometric distance between the corresponding vertices. Alternatively, Hausdorff distance is generally used as the error metric. In this case, Hausdorf Street, Distance of all points in Two sets M _t and in such a way that all points M _t lie within It is defined as the maximum minimum distance between the vertices of. The above can be expressed as follows:

here

And Is the two vertices m _t and Euclidean distance between. Many other distortion metrics can be derived equally to natural video coding such as SNR and PSNR, but fail to give a uniform measure of the user whose distortion is detected on multiple signals, and encode the methods through different media. The specific characteristics of the signal they encode are then matched. In addition, especially for 3-D meshes, all the above metrics give only objective indications or noise-to-signal ratios of geometric approximation, and fail to capture the more subtle visual characteristics perceived by the human eye such as surface smoothing.

2 shows a comparison diagram 200 of distortion metrics such as PSNR, Hausdorff distance, and visual smoothness for 150 frames of an animated sequence BOUNCEBALL having an I-frame at 8 Hz frequency. The two upper points (PSNR-Huisdorf) show the predicted correlation between the corresponding metrics of the Hausdorf distance and the geometric distance they represent. Conversely, the two lower points indicate that the visual distortion (Equation 8) can be low in cases where the geometric distance is high.

One attempt was made by Karni and Gotsman to use surface smoothing as being undertaken while evaluating the spectral compression algorithm for 3-D mesh geometries. See "Spectral compression for mesh geometry" by Zachi Karni and Craig Gotsman in Siggraph 2000, Computer Graphics Proceedings , Kurt Akeley, Ed. 2000, pp. 279-286, ACM Press / ACM SIGGRAPH / Addison Wesley Longman. Here, the proposed 3-D mesh distortion metric normalizes the objective error calculated as the Euclidean distance between two vertices, with each vertex distance to adjacent vertices. The type of error metric captures the surface smoothness of the 3-D mesh. This can be accomplished by the Laplacian operator taking into account the topology and geometry. At vertex v _i , the value of this geometric Laplacian is

Where n (i) is the index set of adjacent ones of vertex i and l _ij is the geometric distance between vertices i and j. Thus, the new metric is defined as the mean of the geometric distance between the meshes and the basis of the Laplacian difference (m _t , Are each vertex sets of meshes And n is Is the size of the set):

The above metric in Equation 8 is well used in the present invention and will be referred to as visual smoothness metric (VS) hereinafter. Other equations regarding the visual smoothness of the mesh can also be used.

The VS metric requires connection information such as adjacent vertices of all vertices m _t . In the case of the 3D-animation codec, if it is assumed that the connection is not changed during the animation, vertex neighbors may be precomputed.

The BOP structure described above is suitable for the design of an efficient packetization scheme that employs redundancy information based on RS erasure codes. The relationship of the design parameters is also shown in equation 6. However, the above expression does not reflect any information about stratification. A formal tiered design approach is described next, following the proposed error tolerance method for 3-D wireframe streaming.

The stratification is performed in such a way that the average VS value of each layer reflects its importance in the animation sequence. To achieve this, VS from equation 8 is independently calculated for all nodes in the mesh, which nodes are ordered according to their average VS in the sequence. The node, or group of nodes, having the highest mean VS forms the first and visually most important layer L ₀ . This is a layer that is more resistant to packet errors than other layers. Subsequent critical layers L ₁ , ..., L _M are created in VS order, corresponding to subsequent nodes, or groups of nodes.

If the 3-D mesh has more nodes than the desired number of layers, the number of nodes grouped in the same layer is a design choice and refers to the output bit rate of that layer. For meshes with only a few nodes but with multiple vertices per node, node segmentation may be desirable. The segmentation will reconstruct the vertices of the 3-D mesh in a new mesh with more nodes than the original. This process will affect the connection, but is not a model in its entirety. If possible, the mesh dividing into nodes is arbitrary, but will reflect more natural objects, and these new nodes will represent the 3-D scene and its corresponding motion. If segmentation is not possible in the scene, then the mesh may be split in sub-meshes (nodes) of any size to be assigned to the same layer. Mesh segmentation will require complex pre-processing steps that will be understood by those skilled in the art. Recall, however, that the 3D-animation codec assumes a static connection.

It is common to build layers with cumulative effects in "natural video". That is, the layer L _j data adds data of the layer L _j-1 and improves the overall video quality. However, only up to layer L _j-1 can be decoded and the refinement layers are lost. This approach can be taken in an adaptive streaming scenario, in which the sender can choose to transmit only j-1 layers during congested network conditions, where j or more layers have improved bandwidth, i.e. more bandwidth It can be selected to be sent when it becomes available.

The nature of the 3-D animation layers disclosed herein is not always cumulative in the same scene. The decoding layer L _j (built with proper node grouping or node segmentation) is the previous layers It is not necessary to refine the quality of the data contained in L ₀ , ... L _j-1 , but it is possible to add animation detail to an animated model, for example, by adding animation to more vertices in the model. Add.

As an example, consider the sequence TELLY (discussed fully below), which is a head-and-shoulders talking avatat. TELLY always faces the camera (fixed camera). Since the camera does not move, a lot of bandwidth is wasted to animate the back of the hair. However, with the proper segmentation of the node "hair" assigning visible parts to layer L _j-1 and assigning invisible parts to layer L _j , the visible and invisible parts of the hair (set of vertices) are easily detected. can do. For fixed cameras (and no interaction allowed), layer L _j is not transmitted. Thus, when a user views a wireframe mesh or animation in a fixed model, only visible portions of the animation can be seen since the animation does not rotate or move. If the user can inspect the animation or see the back side of it by rotating or zooming on the avatar (or other model), layer L _j is shown. In this case, the user sees the animation in an interactive mode that allows the user to also see parts of the animation that are not visible in the fixed mode due to lack of motion in the animation. However, layer L _j does not refine the animation of visible nodes of hair in layer L _j-1 . It contains additional animation data for invisible vertices. This provides an exemplary result of the segmentation method.

Also, the "interaction mode" does not necessarily require animation and user interaction. Interaction mode is referred to as any viewing mode in which the animation can be moved or rotated to expose a portion of the animation that is previously hidden. Thus, if the viewer simply views the animation, the animation can move and rotate in a more human or natural way while speaking. In this case, the L _j layer or other invisible layers may be transmitted to provide additional animation data to complete the viewing experience. In this regard, the fixed or interactive mode may depend on the available bandwidth. That is, if there is enough bandwidth to transmit both visible and invisible layers of animation, the animation can be viewed in interactive mode instead of fixed mode. In another aspect of the invention, the user can select a fixed or interactive mode, thus controlling which layers are transmitted.

3A shows a set of exemplary steps in accordance with an aspect of the present invention. The method involves partitioning a 3-D wireframe mesh (302), calculating VS values for each node in the mesh (304), and the average VS value associated with each layer reflecting the importance of each layer in the animation sequence. Layering 306 the data associated with the wireframe mesh into a plurality of layers. The same overall bit rate is maintained when transmitting multiple layers by applying an error correction code that is not identical in the layer to each layer according to the importance of the layer.

The term "split" as used herein may mean a preprocessing step, such as splitting a mesh into any or non-random sub-meshes to be assigned to the same layer. In addition, the term may have another application, such as a process for generating various layers including one or more nodes.

3B shows a flowchart of another aspect of the present invention. The method includes allocating more redundancy 320 to one of the plurality of layers representing the largest visual distortion. For example, this may be a layer containing visually coarse information. The redundancy is then gradually reduced (322) in the layers that contribute less to visual smoothness. If the loss of unrecoverable packets only occurs within an individual layer, interpolation based concealment is applied to each layer at the receiver (324). As packets belonging to a particular layer travel through the communication network, they can take different paths from the transmitter to the receiver and thus suffer from variable delays and losses. When the receiver knows the overall packet loss rate, the receiver will want to reduce the loss rate by using the extra information (FEC) provided separately for each layer. The amount of FEC will not be enough to recover all lost packets (remaining packets). Interpolation based concealment can be applied independently to each layer to reduce distortion introduced by residual packet loss. In general, steps 320 and 322 are performed at the coding / transmitter end and step 324 is performed at the receiver via a communication network such as peer-to-peer networks.

The time t, and the estimated distortion of the animation at the receiver is the sum of gopin P _jt · D _jt of both, where j is the layer index, D _jt is the visual distortion resulting in lost your information layer j at time t, P _jt is the probability of having an irrecoverable packet loss in layer j. By reconstructing the layers, the probability P _jt is independent and the burst packet loss in the layer provides its own visual distortion D _jt in the decoded sequence. Formally, at time t, the visual smoothness BS _(t) predicted at the decoder can be expressed as:

Where L is the number of layers. Where P _jt is the block error rate P _BER as given by equation 5 or the probability of losing nk _j or more packets in layer j. Using the block error density function P (m, n), the following is derived:

From equations 9 and 10, VS _(t) can be described as follows:

Equation 11 estimates the predicted visual smoothness experienced per frame at the decoder in a statistical manner. The aim is to minimize the distortion on the value of k _jt in equation (11). From the way the bitstream is divided into layers, the optimization that allocates more redundancy to the layer that represents the largest visual distortion (coarse layer) and gradually reduces the redundancy rate on the layers that make the most sophisticated contribution to overall smoothing Processing is foreseen. Conditions 0≤k _jt ≤n, There are L values of k _jt that need to be computed at every time t that follows, where R _C is the redundancy bit and q is the symbol size. This formula leads to a nonlinear constraint optimization problem that can be solved numerically.

The expected behavior of the model for P _B = 0 is to produce the same values for k _jt , and _unequally varying k _jt at high P _B will be obtained. Note that for the calculation of the smoothing distortions in Equation 11, no error concealment occurs at the receiver.

It can be seen that techniques based on vertex linear interpolation are a sufficient and efficient error concealment method for 3D-animated frames. This relies on 'locality of the reference principle', as high-frame animations are not similar to representing vertex trajectories other than linear or piece-wise linear. If higher complexity can be accommodated, higher order interpolation may be employed using information from neighboring frames. Some known interpolation and other concealment methods that can be used by any other decoder are common.

4 shows the experimental parameters of this task, namely A graph 400 showing the relative performances of three error concealment methods adapted to P _B = [0..30], L _B = 4. It is clear that linear interpolation outperforms frame repetition or motion vector-based methods. The figure shows the mean value for eight iterations with different loss patterns. The interpolation concealment method verifies the locality of the reference principle and shows very low variance (average loss burst length L _B = 4 is much less than the sequence frame rate of 30 Hz). Therefore, the present invention preferably uses error concealment based on interpolation at the receiver when the channel decoder receives nk _jt or less BOP packets. In fact, k _jt , which provides a solution to the optimization problem, will also provide minimal distortion if combined with concealment techniques. In such cases, if there is no error concealment, the predicted distortion will be lower than the distortion.

The following describes a method and experimental procedure for tuning the optimization process and values for 3-D wireframe animation in the real world case, with discussion of experimental results using the present invention.

The following experiments are demonstrated through simulating the effectiveness of the proposed non-identical error protection (UEP) combined with error concealment (EC) to stream 3-D wireframe animations. In particular, using UEP with EC is compared to simple UEP, Equal (Equal Error Protection) and No Protection (NP). The comparison is based on a visual smoothness metric known to result in a distortion measure that captures the surface smoothness of the time-dependent mesh during animation. For the calculation of the parameters k _jt , the channel rate R _C is given so that the constrained minimization problem of equation 11 is solved numerically. In addition, n is calculated from Equation 6 so that the ratio characteristics of the original source signal match the specific design of the BOP. The other parameters used in Equation 6 are given below for the two sequences in the experiments and are also summarized in Table I.

Table Ι

Animation Sequence Parameters Used in Redundancy Experiments

TELLY & BOUNCEBALL

In the case of EEP, it is assumed that the constant k can be derived directly from the selection of the channel rate which is set to 15%. For NP, all available channel rates for the source are allocated. Finally, the EC scheme is used based on interpolation for the case of UEP with residual losses. In all experiments L _B = 4 is used.

The sequences TELLY and BOUNCEBALL were used as density factors of df _TELLY = 0.75 and df _BBALL = 1.0 given in Equation 2. TELLY includes nine nodes (three state sparse, six completed) and a total of 780 frames at 30 Hz, as shown in Table I. The average source bit rate is R _{S, TELLY} = 220 Kbps. BOUNCEBALL has one node and 528 frames at 24 Hz, forming one layer of average source rate R _{S, BALL} = 61 Kbps. Both sequences are coded with I-frames every 15 frames. Approximately, it was allowed channel coding redundancy of 15%, to induce the full source and channel coding redundancy, and R and R _TELLY = 253Kbps _BBALL = induce the full source and a coding rate of 70.15Kbps. By selecting n = 32 parameters from equation 6, the calculations for packetization of each layer are made in Table I. Since higher n makes RS codes more resistant at the expense of delay and buffer space, the value of n is chosen as a compromise between delay and efficiency.

According to the proposed layering method provided in section V, it is divided into three layers, each of which constitutes the nodes shown in Table I. Of the total number of animated vertices in the 3-D mesh, the fractional part of each layer averages (L ₀ , L ₁ , L ₂ ) = (0.48,0.42,0.10). This division is expected to reflect the source bit rate of each layer proportional term. It is known that the proposed layering scheme assigns two sparse nodes of three to the same layer L ₁ . The total number of vertices of these two sparse nodes represents 65% of the vertices in the reference mesh. The third sparse node, nostril, is assigned to layer L ₂ , but the individual motion is about a very small fraction of the total number of vertices of the model (about 1.3%). If we want to relate the node-to-layer allocation (using the VS metric) to the calculated density factor df _L per layer ⁴ (Equation 2) and the output bit rate, this fact can bear some weight. If such a relationship exists, a dynamic tiering scheme can be developed for the applications as needed.

The sequence BOUNCEBALL initially contains only one node. The sequence represents a soft ball with an inherent symmetry around the center point as an enclosing shape. Given the shape symmetrically, it was decided to split the mesh into two nodes of the same number of vertices without considering the VS metric for each node. The logic behind the partitioning is an attempt to verify the effect of the VS metric having the proposed UEP immunity scheme. All other source coding parameters, most importantly the quantization step size, are constant between the two layers. It is anticipated that the two layers will receive approximately the same average guard bit so that UEP performance approaches EEP.

5 shows a first diagram 502 illustrating VS as a function of average packet loss rate P _B for TELLY. Four curves in the figure represent each proposed resilience method for codes 31 and 22. The average of the codes computed for the UEP (approximately the nearest integer) is as follows: . It is clear that UEP and UEP + EC outperform NP and EEP for medium for high loss rates of P _B > 9%. Recall that stratification is performed in such a way that the lowest layer is represented by a high average visual distortion. Since the UEP method assigns higher codes to the lower layer (L ₀ ), better immunity is predicted for the high loss rate L ₀ . This factor is superior in mean distortion, resulting in better performance. Note that since the RS codes are sufficient to recover all or most of the errors, the EEP and UEP operate in almost the same way at low loss rates. It is also noted that the NP method under conditions without loss is better than the other methods. This is an understandable result, since resource information takes all available channel rates, thus allowing better encoding of the signal. It is also worth noting the effects of EC: The distortion of the UEP + EC scheme is slightly improved through a simple UEP case. This is also predicted.

The results for the (31, 27) RS code on the sequence TELLY are very similar to that shown in the diagram 504 of FIG. In this specification, the threshold at which UEP methods replace EEP or NP (with or without EC) is approximately P _B = 7%. Initial NP performance (low P _B ), as compared with (31, 22) where channel coding bits repeatedly emphasize the factor that is actually 'consumed' because it does not distribute large immunity in the low loss region at the expense of resource rate. Note how steep this is. The corresponding average codes per floor are: to be. It is also improved once in the performance of the UEP methods from the interpolation algorithm of error concealment. As this quantity is not considered in the optimization problem, it is expected to contribute a small reduction to the visual error.

FIG. 6 shows the results 602 obtained for the same observation repeated through a 'symmetrically' layered BOUNCEBALL sequence as described above in this section. The same (31, 22) EEP code was used before the comparison. Graph 602 shows the same trend and relative performance in TELLY with UEP + EP providing the best overall performance. However, note that the distance of the UEP curves from the EEP is significantly reduced compared to the TELLY sequence at high P _B 's. The average integer calculated in RS codes for the UEP case is: That is, equivalent to the EEP case. This may seem surprising, but careful reasoning involves animations that correspond to visually balanced distortions (the same number of vertices, nodes, very similar motion in the scene, and identical encoding parameters). We propose layers that are equally balanced by the amount of. This is exactly the result predicted in the stratification for the BOUNCEBALL sequence described above. Indeed, the actual values of k _0t , k _1t calculated as a solution to the optimization problem change around an average integer value of 22. Moreover, recall that the original symmetric BOUNCEBALL mesh is split into two arbitrary nodes without considering those individual visual distortions that are similarly assumed. Indeed, deformation of the softball at the bouncing points reduces the symmetry of the circular shape. The above facts reasonably explain why the UEP and EEP curves do not exactly match at higher P _B 's as typically predicted. Finally, the UEP + EC method is difficult to perceive for visual distortions as in the above observations but provides some improvement.

The present invention addresses the basic problem of a method for best utilizing available channel capacity for streaming 3-D wireframe animation in such a way as to obtain optimal subjective immunity to error. Briefly, the present invention links channel coding, packetization, and layering with subjective parameters that measure visual smoothness in reconstructed images. Based on this, we trust that the results can help disclose how 3-D animation can be an important networked media type. The disclosed methods exploit optimization of the distribution of the bit budget allocation left for channel coding between different layers using a metric reflecting the visual characteristics of the human eye that detects surface smoothness on time-dependent meshes. Try. In the use of the metric, the encoded bitstream is initially divided into visual importance layers, and the UEP combined with the EC shows good protection against sudden packet errors occurring on the Internet.

Embodiments within the scope of the present invention may also include computer readable media for carrying or possessing computer executable instructions or data structures stored in the instructions. The computer readable media is any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, the computer readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage that may be used to carry or store desired program code means in the form of computer executable instructions or data structures. Device, magnetic disk storage or other magnetic storage devices, or any other medium. When information is transmitted or provided to a computer (either hardwired, wireless, or a combination thereof) over a network or other communication connection, the computer properly observes the connection of either wireless or wired as a computer readable medium. . As such, any such connection is appropriately referred to as a computer readable medium. Combinations of the above may also be included within the scope of computer-readable media.

For example, computer executable instructions include instructions and data that cause a general purpose computer, special purpose computer, or special purpose processing device to execute a particular function or group of functions. Computer executable instructions also include program modules executed by a standalone computer or a network environment computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or execute particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of program code means for executing the steps of the methods disclosed herein. The particular sequence of executable instructions or related data constructs represents examples of corresponding actions for executing the functions described in the above steps.

Those skilled in the art will appreciate that other embodiments of the present invention include personal computers, handheld devices, multiprocessor systems, microprocessor-based or programmable electronics, network PCs, small computers, mainframe computers, and the like. It is understood that the present invention can be practiced in many types of networked computer environments. Embodiments may also be practiced in distributed computer environments where tasks are performed by remote processing devices that are linked locally or by a communication network (either by hardwire links, wireless links, or a combination thereof). Can be. For example, environments distributed peer to peer provide an ideal communication network to which the principles of the present invention are applied and beneficial. In a distributed computer environment, program modules may be located on both sides of local and remote memory storage devices.

Although the description contains specific details, they are not to be construed in any way to limit the claims. Other configurations of the above embodiments of the invention are part of the scope of the invention. Accordingly, the appended claims and their legal equivalents define only the invention rather than provide any specific examples.

Claims (21)

As a way of streaming data, a) calculating visual smoothness for each node in a wireframe mesh; And b) stratifying the data associated with the wireframe mesh into a plurality of layers such that the average visual smoothness value associated with each layer reflects the importance of the individual layers in the animation sequence. 2. The method of claim 1, further comprising maintaining the same overall bit rate when transmitting the plurality of layers. 2. The method of claim 1, wherein each layer of the plurality of layers comprises a node or group of nodes in the wireframe mesh. The method of claim 1, wherein the most significant layer contains the highest average visual smoothness value and is most resistant to packet errors. The method of claim 1, wherein the number of nodes included in a particular layer is associated with an output bit rate of the respective layer. 2. The method of claim 1, further comprising generating a 3-D packetized streaming signal representing a scene comprising an animation. 2. The method of claim 1, wherein maintaining the same overall bit rate occurs by applying to each layer an error correction code that is not identical in the layer according to the importance of the individual layers in the animation sequence. 2. The method of claim 1, further comprising dividing the wireframe mesh to create a segmented mesh having more nodes than the wireframe mesh before stratifying the data. 10. The method of claim 8, wherein the segmenting further comprises segmenting the wireframe mesh according to the objects in the scene represented by the nodes and the corresponding motion of the objects. 9. The method of claim 8, wherein partitioning further comprises dividing the wireframe mesh into sub-meshes of any size to be assigned to the same layer. A method of receiving streamed data, wherein the unequal error protection scheme for the wireframe mesh is applied to the streamed data to divide the wireframe mesh into a plurality of layers. In the way: Applying an error concealment scheme to the plurality of layers, wherein graceful degradation of a streamed animation with a high packet loss rate is achieved at a receiver. 12. The method of claim 11, wherein the non-identical error protection scheme comprises optimizing the distribution of bit budget allocation among the plurality of layers. 13. The method of claim 12, wherein the non-identical error protection scheme uses a visual smoothness metric. 12. The method of claim 11, wherein the plurality of layers are divided according to visual importance. 12. The method of claim 11, wherein interpolation-based concealment is applied to residual packets that do not recover from error-concealment. A method for decoding streaming data associated with a wireframe mesh at a receiver, the streaming data comprising a plurality of layers organized according to the visual importance of the wireframe mesh, and a sender having the plurality of layers exhibiting greater visual distortion. In the streaming data decoding method, assigning more redundancy to one of the layers: Using redundant information in individual layers in the streaming data to recover lost packets; And Applying error concealment based on interpolation to reduce distortion due to residual packet loss. 17. The method of claim 16, wherein the layer representing the largest visual distortion is a coarse layer. 17. The method of claim 16, wherein the concealment based on interpolation is applied to each layer independently. A method of streaming wireframe mesh data, Dividing the wireframe mesh into layers including a layer having visible portions and a layer having non-visible portions; When the user views the wireframe mesh in a fixed mode, transmitting only the layer with visible portions; And When the user views the wireframe mesh in an interactive mode, transmitting a layer having visible portions and a layer having invisible portions. 20. The method of claim 19, wherein transmitting only layers with visible portions or transmitting a layer with visible portions and a layer with invisible portions is dependent on the available bandwidth. 20. The method of claim 19, wherein transmitting only layers with visible portions or transmitting a layer with visible portions and a layer with invisible portions is selected by a user.