WO2003007486A2 - Modelisation d'images par la geometrie et la brillance - Google Patents

Modelisation d'images par la geometrie et la brillance Download PDF

Info

Publication number
WO2003007486A2
WO2003007486A2 PCT/IL2002/000563 IL0200563W WO03007486A2 WO 2003007486 A2 WO2003007486 A2 WO 2003007486A2 IL 0200563 W IL0200563 W IL 0200563W WO 03007486 A2 WO03007486 A2 WO 03007486A2
Authority
WO
WIPO (PCT)
Prior art keywords
image
brightness
cross
line
skeleton
Prior art date
Application number
PCT/IL2002/000563
Other languages
English (en)
Other versions
WO2003007486A3 (fr
Inventor
Yosef Yomdin
Yoram Elichai
Original Assignee
Vimatix Technologies Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US09/902,643 external-priority patent/US6801210B2/en
Application filed by Vimatix Technologies Ltd. filed Critical Vimatix Technologies Ltd.
Priority to EP02747643A priority Critical patent/EP1417647A2/fr
Priority to AU2002318027A priority patent/AU2002318027A1/en
Priority to PCT/IL2002/000935 priority patent/WO2003045045A2/fr
Priority to AU2002353468A priority patent/AU2002353468A1/en
Publication of WO2003007486A2 publication Critical patent/WO2003007486A2/fr
Priority to US10/483,786 priority patent/US20040174361A1/en
Publication of WO2003007486A3 publication Critical patent/WO2003007486A3/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/001—Model-based coding, e.g. wire frame

Definitions

  • the present invention relates to representation of images, for example for transmission . and/or storage.
  • Vectorization is the representation of a visual image by geometric entities, like vectors, curves and the like. Vectorized images are usually significantly more compact and easier to process than those images represented by conventional techniques, including use of pixels.
  • skeletons are used in computer graphics, and especially in virtual reality applications.
  • a skeleton of a graphic is formed of a plurality of "bones", each of which is associated with a structure of the graphic.
  • a controller states the movements to be applied to each bone and the associated structures are moved accordingly, by a computer handling the graphic.
  • a broad aspect of some embodiments of the invention relates to a high quality compressed format for representing images using geometric and brightness modeling.
  • the format includes representing characteristic lines of a compressed image by one or more color cross sections of the line and representing the remaining portions of the image in another manner, for example by a selected number of background pixels and/or patches, from which the remaining pixels of the image may be derived using interpolation methods and/or other methods.
  • patches refers herein to groups of pixels having same or similar color, surrounded by points having very different colors. In an exemplary embodiment of the invention, such patches are generally used for very small points which are not represented properly by background points.
  • An aspect of some embodiments of the invention relates to a compressed format for representing images using characteristic lines, in which the characteristic lines are represented by cross-section profiles including at least one background parameter, for example a brightness of a background point along the cross section of the characteristic line.
  • the background parameter optionally has a value which differs substantially from an equivalent parameter value of the line itself.
  • the cross section profiles include three color parameters which describe the color change of the line and two or more color parameters which describe the background adjacent the line.
  • the profile includes more than 25% background parameters.
  • the brightness of the background at edges of the cross-section is determined and stored therewith.
  • background points are generated along the line, adjacent thereto, with the stored parameter values.
  • the parameters of the images are aggregated along the Characteristic Lines, at the Crossings of characteristic lines in the represented image and/or between the Characteristic Lines and the Background. This may include, for example, storing the Cross-Section parameters of Characteristic Lines at their Crossings, then interpolating them along the Line and using these interpolated data either as the final one or as a prediction. In the last case the actual Cross-Section parameters at the Line Points are optionally given by the corrections to the predicted values. The same procedure can be performed for the Background values at the Crossings and along the lines. Thus, the stability of the image representation is increased and the amount of data required for representation is reduced.
  • An aspect of some embodiments of the invention relates to a compressed format for representing images using characteristic lines of a plurality of different types with respect to the relation of the lines to the background points.
  • the characteristic lines are classified as separating or non-separating lines. Separating lines prevent or reduce the influence of background points included in the compressed format on reconstructed points on the other side of the separating lines.
  • Non-separating lines are optionally considered for the purpose of background construction, as if the lines are non-existent.
  • one or more partially separating lines are defined, which reduce the influence of points on opposite sides of the line according to their level of separation.
  • each characteristic line is associated with a separation level parameter.
  • edges i.e., lines which are boundaries between two image regions with different colors
  • Ridges are optionally classified as either separating or non-separating lines, for example according to the difference between the background color, between the sides of the line. If the background color has a substantial visual difference between the sides of the line, the line is optionally classified as separating, while if the difference is non-substantial or non-existent, the line is optionally classified as non-separating.
  • the point in reconstructing a value of a point, the point receives a weighted average of the color values of points specified in the compressed format.
  • the weights optionally depend on the distance between the lines and the separation levels of the lines between the points if such lines exist.
  • the distance between two points is defined as the shortest path between the points, which does not cross lines having a separation value above a predetermined threshold.
  • An aspect of some embodiments of the invention relates to a compressed format for representing images using characteristic lines, background points and/or patches, in which at least some of the characteristic lines, background points and/or patches have depth values for three dimensional representations.
  • a third coordinate states the depth of the line.
  • depth is associated with other points along the line.
  • some or all of the parameters of the line cross-sections are associated with a depth parameter, which represents the depth of the line in the corresponding cross-section point.
  • patch center points and/or background points are associated with a depth parameter, in addition to their 2D coordinates.
  • points for which a depth is not stated are assumed to have a default (e.g., unit) depth.
  • points for which a depth is not stated are assumed to have a depth interpolated from depth values stated from neighboring elements, to which a depth was assigned, hi some embodiments of the invention, the depth is added to the image by a human operator, optionally using a suitable user interface.
  • a different parameter is associated, for example, a rendering parameter such as opacity, reflectivity or material type.
  • the different parameter is used by a function, for example one provided with the image or with the rendering system, for example, such a function can decide a pixel value or a pixel value modification based on the value of the parameter or base don other information such as user input, hi one example, this parameter is used to indicate self illuminating points on an image, so when a user changes a general lighting level during display, the intensity of these points is adjusted to compensate for a general reduction in intensity of the image.
  • the parameter is used as part of a user interface, for example, to indicate a value to be returned if a user selected that point or an area near it.
  • An aspect of some embodiments of the invention relates to a format of representing the geometry of a characteristic line of an image in a compression format.
  • the format allows using one of a plurality of different segment shapes for representing each of the segments of a characteristic line.
  • the different segment shapes include one or more of parabolic, elliptic and circular.
  • straight lines and/or other fine shapes, such as splines or non-functional shapes, such as freedrawn shapes, are used.
  • each segment is represented by the coordinates of its ends, the section shape and the height of its curve above a straight line connecting the ends.
  • the use of different segment shapes allows for better fitting of image data by the characteristic lines.
  • An aspect of some embodiments of the invention relates to a format of representing the geometry of a characteristic line of an image, using elliptic and/or circular shapes.
  • image characteristic lines such as synthetic global circular and elliptic curves which commonly appear in images of animals and humans
  • visual contours of near-circular, near- spherical or near-ellipsoidal 3D objects which commonly appear in some handwritten characters
  • the use of elliptic and/or circular shapes better represents at least some portions of the lines.
  • a human operator may indicate to a compression unit or software, for a specific image and/or for specific lines of an image, which segment shape is to be used.
  • an image creation software utilized to generate the image assigns to each created line (or to each segment thereof) a line type which is to be used in representing the line.
  • the compression software attempts to fit the line into a plurality of different shapes, and selects the shape which provides the smallest error and/or requires the least representation data.
  • Various error criteria as known in the art, for example, may be used.
  • the compression software in representing a line, the compression software begins from a first direction attempting to progress each time for a longest segment of the line, without exceeding a predetermined error level. The segment shape which allows for a longest segment is used. This procedure optionally proceeds over the entire line.
  • An aspect of some embodiments of the invention relates to a method of displaying (or otherwise reconstructing) images represented in a compressed format using geometric and/or brightness modeling, including a plurality of links (e.g., segments of characteristic lines).
  • the method includes opening each link separately, and using for each pixel the value calculated for a single link, optionally the nearest link.
  • the nearest link is optionally determined using a suitable distance function. This method achieves efficient display of the image with a relatively low reduction in quality due to the selection of only a single adjacent link.
  • the values from a plurality of link are merged, for example, by averaging, possible distance weighted.
  • the number of links used may be, for example, all the links, some of the links, the links within a certain distance and/or a fixed number of links.
  • each link in opening each link, brightness values are determined for a plurality of pixels in the vicinity of the link. Thereafter, for each pixel the value of the closest link is selected. Alternatively, each pixel is first associated with a link, and in opening the links, values are assigned only to pixels associated with the opened link.
  • the distance function uses elliptic equidistant curves at link ends, to provide a correct choice of the link side. This simplifies computations and strongly increases their stability.
  • An aspect of some embodiments of the invention relates to a method of displaying (or otherwise decompressing) images represented in a compressed format using separating lines and background points, hi an exemplary embodiment of the invention, the method includes calculating the distance from a pixel whose value is determined for display, by propagating waves in concentric circles in all directions, until the background points are reached. When a wave hits a separating line it optionally does not continue. Alternatively or additionally, when a wave reaches a partially separating line, the wave is delayed for a number of steps according to the separation level of the line. Alternatively or additionally, other distance calculation or estimation methods are used, for example, non-propagating methods or propagating in a plurality of vector directions (e.g., 4-8) and interpolating between the end points of the vector directions.
  • a plurality of vector directions e.g., 4-8
  • An aspect of some embodiments of the invention relates to representing an image by a plurality of layers, at least some of which are represented using contour characteristic lines having cross-section profiles. Dividing an image into a plurality of layers which are represented separately, may allow easier manipulation of the layers of the image, for example for animation.
  • one or more objects including a plurality of layers with predetermined relations relative to each other, are defined.
  • the layers are selected by finding image portions which are surrounded entirely by a separating line. Such portions are defined as separate layers.
  • layers may be defined by a human operator, hi some embodiments of the invention, a human operator defines a layer in a first image of a sequence of images and an image compression system finds the layer in other images of the sequence.
  • separate layers are defined for objects which are expected to move in an image sequence, such as humans, animals, balls and vehicles.
  • an animation or video set of images includes one or more base images and additional images defined relative to one or more base images.
  • the base images are optionally defined in terms of layers (i.e., sub-textures) formed of characteristic lines with their cross-section profiles, patches and background points.
  • the additional images are optionally defined in terms of motion of objects and/or layers in a three dimensional space and/or color transformations of layers and/or objects.
  • the additional images are defined in terms of changes in one or more parameters of any the characteristic lines, patches and/or background points relative to the base image. It is noted that since the CORE representation generally follows a visual meaning of the image, the changes between images in a video sequence are generally limited to the geometry of the characteristic lines, patches and/or background points and/or the brightness (color) parameters thereof, without changing the entire representation.
  • the additional images are stated in terms of changes (e.g., movements and/or deformations) in a skeleton associated with image sequence, relative to the form of the skeleton associated with the base image.
  • An aspect of some embodiments of the invention relates to a method of generating animation based on a two-dimensional image.
  • the method include associating a skeleton with the two-dimensional image, by stating for each of one or more bones (i.e., parts) of the skeleton, the portion of the image to move with the bone. Thereafter, movements are optionally defined for the skeleton, and the portions of the image move relative to each other according to the skeleton movement.
  • the image is cut into a plurality of layers, and each bone of the skeleton is associated with one or more of the layers.
  • the points of the image are represented by Cartesian coordinates.
  • the image and/or its layers are represented as bit maps.
  • one or more of the layers or the entire image is represented using characteristic lines and background points.
  • the image representation comprises characteristic lines, at least some of which are associated with bones of the skeleton.
  • the bones of the skeleton comprise characteristic lines of the image.
  • Special markers are optionally used to specify each of the roles of a specific line. This leads to a dramatic data reduction and simplification of processing.
  • one or more separate lines are defined for the bones.
  • the motion of any point, associated to the skeleton, produced by the skeleton motion is defined as follows: the coordinates of the new point with respect to the coordinate system for the moved skeleton are the same as the coordinates of the initial point with respect to the coordinate system of initial skeleton. Alternatively, only some of the motion of the points is defined using this method.
  • some image pixels are associated to the skeleton (usually, pixel layers). Their motion is inferred by the skeleton motion in the same way.
  • an animation sequence may include indication of stretching and/or bending of a bone, resulting in a corresponding change in the associated image portion.
  • the image in preparing animation based on an image, is cut into layers.
  • the hands, legs and body of the person may each be cut into a separate layer, for example manually or automatically (e.g., based on color or differences between images of a series of images).
  • Each of the layers is then optionally converted into a CORE or VIM representation.
  • the image is first compressed into the VIM representation and then is cut into layers.
  • a skeleton is optionally associated with the image.
  • a library skeleton is overlaid by an operator on the image, using a suitable graphic human interface.
  • the operator draws bones of the skeleton on the image, using a suitable computer interface, for example freehand drawing or selecting a template from a library.
  • the operator indicates for each bone of the skeleton with which layer it is to be associated.
  • the computer associates each bone with the closest layer and/or with the layer with a similar shape, hi some embodiments of the invention, the computer replaces one or more defined bones , with substantially parallel characteristic lines of the associated layer, so as to reduce the amount of data stored for the image.
  • an animator defines the movement of the layers of the image, by defining movements of the skeleton.
  • a method of representing an image comprising: determining one or more characteristic lines over which the profile of the image changes considerably; and storing, for each of the one or more characteristic lines, one or more cross section profiles including data on the brightness of the line and of one or more background points adjacent the line having a brightness substantially different from the brightness of the line.
  • determining the one or more characteristic lines comprises determining a line which is different considerably from both its banks.
  • determining the one or more characteristic lines comprises determining a line which is different considerably from one of its banks.
  • storing one or more cross section profiles comprises storing, for each cross section profile, brightness values for at least three points along the cross section.
  • storing one or more cross section profiles comprises storing, for each cross section profile, brightness values for at least five points along the cross section.
  • storing one or more cross section profiles comprises storing, for each cross section profile, brightness values for at least two background points along the cross section.
  • storing one or more cross section profiles comprises storing, for each cross section profile, data on the background points, which includes at least 25% of the stored data of the profile.
  • storing data on the brightness comprises storing data on the color of the line.
  • a method of representing an image comprising: providing one or more characteristic lines of the image; and classifying each of the one or more lines as belonging to one of a plurality of classes with respect to its effect on constructing background points of the image.
  • classifying the lines comprises stating a separation level.
  • classifying the lines comprises classifying into one of two classes.
  • the two classes comprise separating lines and non-separating lines.
  • a method of representing an image comprising: determining one or more characteristic lines over which the profile of the image changes considerably; and storing, for at least one of the characteristic lines, a three-dimensional geometry of the line.
  • the method comprises storing for the one or more characteristic lines a cross section profile.
  • storing the three-dimensional geometry comprises storing a two-dimensional geometry of the line along with at least one depth coordinate of the line.
  • a method of representing an image comprising: determining a characteristic line over which the brightness profile of the image changes considerably; dividing the characteristic line into one or more segments according to the shape of the line; selecting for each segment one of a plurality of model shapes to be used to represent the segment; and determining, for each selected model, one or more parameter values, so that the model approximates the segment.
  • selecting the model shape comprises selecting from a group comprising an elliptic model.
  • selecting the model shape comprises selecting from a group comprising a circular model.
  • selecting the model shape comprises selecting from a group comprising a parabolic model.
  • determining the one or more parameters comprises determining end points of the segment and a maximal distance of the segment from a straight line connecting the end points.
  • a method of displaying a compressed image including a plurality of links comprising: determining for at least some of the pixels of the image, an associated link; decompressing each of the links separately, without relation to the other links, so as to generate brightness values for a group of pixels in the vicinity of the link; and selecting for the at least some of the pixels of the image a brightness value from the decompression of the associated link of the pixel.
  • determining the associated link is performed before the decompressing of the links.
  • determining the associated link is performed after the decompressing of the links.
  • determining the associated link comprises determining a nearest link.
  • a method of determining a display value of a pixel in an image represented by lines and a plurality of background points comprising: propagating lines from the pixel in a plurality of directions within the image; delaying or terminating the propagating upon meeting lines of the image, according to a separation level of the line; determining the distance to background points in the vicinity of the pixel, according to the propagating; and calculating a brightness value for the pixel, according to the brightness values of the background points and determined distances to the background points.
  • propagating comprises propagating in a circular fashion.
  • a method of defining animation for an image comprising: providing a two-dimensional image; providing a skeleton including at least one bone; associating at least one of the bones of the skeleton with one or more portions of the image; defining movements of the skeleton; and moving the pixels of the portions of the image associated with moving bones of the skeleton, responsive to the movements of the bones.
  • defining movements of the skeleton comprises defining stretching of one or more bones of the skeleton.
  • Fig. 1 is general representation of an image, in accordance with an embodiment of the present invention
  • Fig. 2. shows a segment of a characteristic line with a color cross-section interpolated along it, in accordance with an embodiment of the present invention
  • Fig. 3 shows an edge cross-section of a characteristic line, in accordance with an embodiment of the present invention
  • Fig. 4 shows a separating ridge cross-section of a characteristic line, in accordance with an embodiment of the present invention
  • Fig. 5 shows a non-separating ridge cross-section of a characteristic line, in accordance with an embodiment of the present invention
  • Fig. 6 shows a parabolic mathematical model, representing patch, in accordance with an embodiment of the present invention
  • Fig. 7 shows a background partition by characteristic lines, in accordance with an embodiment of the present invention.
  • Fig. 8 shows background representing points in different parts of the background partition, in accordance with an embodiment of the present invention
  • Fig. 9 shows crossing of characteristic lines, in accordance with an embodiment of the present invention.
  • Fig. 10 shows splitting of characteristic lines, in accordance with an embodiment of the present invention
  • Fig. 11 shows correlation of cross-section parameters of characteristic lines at their crossing, in accordance with an embodiment of the present invention
  • Fig. 12 is a block-diagram of an image reconstruction method, in accordance with an embodiment of the present invention
  • Fig. 13 shows the representation of the edge cross-section by parabolic pieces, in accordance with an embodiment of the present invention
  • Fig. 14 shows the representation of the ridge cross-section by two edge cross-sections, in accordance with an embodiment of the present invention
  • Fig. 15 illustrates gluing edge cross-section to the background, in accordance with an embodiment of the present invention
  • Fig. 16 shows the weight functions used in the reconstruction algorithm, in accordance with an embodiment of the present invention.
  • Figs. 17 and 18 show construction of the distance to the line segment, in accordance with an embodiment of the present invention.
  • Fig. 19 shows construction of the distance function near the ends of the line segments, in accordance with an embodiment of the present invention.
  • Figs. 20A and 20B show patch and patch weight functions respectively, in accordance with an embodiment of the present invention
  • Fig. 21 shows some possible shapes of the background weight function, in accordance with an embodiment of the present invention
  • Fig. 22 illustrates the construction of the distance between the background points, taking into account separating lines, in accordance with an embodiment of the present invention
  • Fig. 23 illustrates signal expansion process for constructing distance between background points, in accordance with an embodiment of the present invention
  • Fig. 24 shows construction of margin background points near characteristic lines, in accordance with an embodiment of the present invention.
  • Fig. 25 is a flowchart of animation creation, in accordance with an embodiment of the present invention.
  • Fig. 26 shows a block-diagram, representing steps of a skeleton construction, in accordance with an embodiment of the present invention.
  • Fig. 27A-C show bending and stretching of bones, in accordance with an embodiment of the present invention.
  • DETAILED DESCRIPTION OF EMBODIMENTS hi the present application, as in the documents listed in the related applications section above, the following terms are used interchangeably:
  • VIM Vector Imaging
  • the lines used to define the images are referred to as Characteristic Lines and Lines (LN).
  • the lines are formed of segments referred to as Line Segments (LS), Links and Arcs.
  • a single point along a characteristic line is referred to as a Line Point (LP) or a vertex.
  • LP Line Point
  • LC Line Color Profiles
  • Cross-Sections which define the change in the image across the line.
  • the image is further defined by points each of which is referred to as an Area Color Point (AC), a Background representing point or a Background point.
  • AC Area Color Point
  • Background representing point
  • Background point a Background point
  • Some images may be defined by a plurality of separate portions referred to as Sub- Textures (ST) and Layers.
  • ST Sub- Textures
  • Layers Layers
  • FIG. 1 is general representation of an image, in accordance with an embodiment of the present invention.
  • Fig. 2. shows a segment of a characteristic line with a color cross-section interpolated along it, in accordance with an embodiment of the present invention.
  • Characteristic Lines and their Color Cross-Sections have been defined in U.S. Patent applications 09/716,279 and 09/902,643, the disclosures of which documents is incorporated herein by reference.
  • characteristic lines are lines on the image along which the image visual pattern consistently repeats itself.
  • Characteristic lines are optionally represented by their "central lines” and "brightness cross-sections". The central line captures in the most accurate way the geometric shape of the characteristic line, while the brightness cross-section describes the brightness (color) behavior in the orthogonal sections of the line.
  • Cross-sections optionally represent the visual pattern, that repeats itself along the line.
  • the central line is represented by a second or third order spline curve (preferably a second order).
  • each cross-section is represented by one of a number of model shape types, each characterized by a small number of parameters.
  • the model shape types include parabolic, elliptic and circular curves. Alternatively or additionally, other curves may be used.
  • Cross-sections are stored at predefined points (referred to as "cross-section control points") on the central line, and interpolated between these control points. Parameters of Characteristic Lines' Geometry
  • the initial point ZQ and the end point z n may coincide.
  • each P-curve Being represented by an ordered list of consecutive points, each P-curve possesses a natural orientation.
  • either parabolic, elliptic or circular arcs S ⁇ are used, in such a way that for each [z[. ⁇ , z ⁇ the arc Si is completely characterized by the height hi of its center over the segment [z[. ⁇ , ZJ].
  • this height is optionally taken with the sign plus, if the arc is curved to the left side of the P-curve, and with the sign minus, if the arc is curved to the right.
  • the parabolic arcs Si are chosen, they are optionally taken to be symmetric with respect to the line passing through the center of the straight segment [z ⁇ . ⁇ , z ⁇ and orthogonal to it.
  • the symmetric parabolic arc Si is completely characterized by the height hi of its center over the segment [zi_ ⁇ , z .
  • a P-spline curve is given by the following parameters:
  • the arcs Si of the P-curves, representing poly-links, are referred to below as "links".
  • each poly-link is optionally given by an ordered list of its vertices zi, starting at one of the endpoints of the poly-link and continuing following the natural order of the vertices, together with the heights of the corresponding links.
  • the vectors [zi_ ⁇ , zi] and the heights can be specified.
  • Cross-sections are optionally specified at the cross-section control points along each poly-link, including their end points.
  • the cross-section control points optionally coincide with the vertices of the P-curves, representing the poly-link
  • each cross-section has a type, optionally either edge or ridge.
  • an edge is a line, separating two areas of the image with different brightness (color) levels.
  • a ridge is a line, representing a thin strip, which differs in its brightness (color) from the brightness (color) on each of its sides.
  • the type of the cross-section does not change along the poly-links between the joints.
  • each ridge i.e. a poly-line with the ridge type of the cross- section
  • Edges are optionally always separating.
  • Fig. 3 shows an edge cross-section, in accordance with an embodiment of the present invention.
  • the edge cross section is described by the following parameters: Left width WL, Right width WR, Left brightness LB1, Left brightness LB2, Right brightness RBI and Right brightness RB2.
  • the Left width WL and the Right width WR are always assumed to be equal.
  • the Left width WL and Right width WR are equal by default unless otherwise specified.
  • the edge cross-section is assumed to have
  • umps defined by the differences between RBI and RB2 and LB1 and LB2, respectively.
  • bumps are used to better represent natural images, either scanned or taken by a video or a still digital camera.
  • the bumps are generally caused by the physics of the light propagation, by specifics of scanning procedures and/or by specifics of human visual perception.
  • the "margin” parameters LB1 and RBI being important cross-section features, are stored together with other cross-section parameters. However, they have another natural visual interpretation as the background brightness along the edge on its different sides. It is this interpretation, that is optionally used in the recommended reconstruction algorithm: the margin cross-section parameters enter the Background reconstruction Procedure, while the rest of the cross-section parameters are treated in the Cross-section reconstruction Procedure.
  • the "height" of the cross-section bump can be taken to be a certain fixed fraction of the total height of the cross section. This fact is optionally utilized in the "aggregation" level of the VIM representation, by encoding of the actual heights as the co ⁇ ections to the default ones.
  • the parameters LB2 and RB2 are eliminated and replaced by their default values, computed through the rest of the parameters.
  • Separating Ridge Cross-Section Fig. 4 shows a separating ridge cross-section, in accordance with an embodiment of the present invention.
  • a ridge cross-section is described by the following parameters: Left width WL, Right width WR, Left brightness LB1, Left brightness LB2, Central brightness CB, Right brightness RBI and Right brightness RB2.
  • Fig. 5 is an exemplary cross-section graph of a non-separating ridge, in accordance with an embodiment of the present invention.
  • This type of cross-section has the same parameters as the separating one, besides the margin parameters LBl and RBI, which are not defined.
  • the right and the left width of the ridge are usually roughly equal, so a single width parameter can optionally be used.
  • the right and left brightness values of a non-separating ridge can be identified with the background values at the co ⁇ esponding points.
  • the type and width of the cross-sections are optionally the same for each color component, while the brightness parameters are specified independently for each color.
  • the brightness of the bumps can be stored as a correction to a default value, equal to a certain percentage (typically around 10%) of the total brightness jump.
  • a closed chain of poly-links without self-intersections is optionally marked as a boundary contour of the VIM image. Patches
  • Fig. 6 is a schematic illustration of a patch, in accordance with an embodiment of the present invention.
  • Patches optionally capture fine scale brightness maxima and minima. They are represented by Gaussian-shaped or parabolic-shaped mathematical models, blended with the background along their elliptic-form margins, hi some embodiments of the invention, each patch is specified by the following parameters: the coordinates (Cx, Cy) of its center, the sizes RI and R2 of the bigger and the smaller semi-axes of a base ellipse, the direction a (angle with the x-axis) of the main semi-axis of the base ellipse, the brightness value CB at the center and the "margin" brightness value MB.
  • the brightness values at the center of each patch are specified independently for each color separation.
  • This background refers herein to the part of the VIM image, "complementary" to the characteristic lines.
  • the background includes slow scale image components.
  • the background is optionally determined by one or more of the following elements:
  • VIM Sub-Textures Some of the image subparts, completely bounded by separating characteristic lines or/and the image borders, are provided with their single global background brightness value GB. In fact, these subparts, called VIM Sub-Textures, play an important role on higher representation and animation levels. They are specified by Sub-Texture numbers, and the values GB are attached to the Sub-Textures. Figure 7 shows background parts with different GB values.
  • Crossings of characteristic lines are represented by crossings of their central lines and by blending of their brightness values near the crossings. See Fig. 9.
  • Splitting is a special type of crossing, where the characteristic line splits into two or more new characteristic lines, according to a splitting of the cross-section into the corresponding sub-pieces. See Fig. 10, for example.
  • crossings and splittings are optionally represented by joints. At each joint cross-sections are given for each of the characteristic lines starting (or ending) at this joint. No compatibility conditions between these cross-sections are assumed in the basic VIM profile.
  • the side brightness (color) values on appropriate sides of the adjacent characteristic lines at their crossing are roughly equal. This fact is optionally used on the higher aggregation level to reduce the data size. (See Fig. 11, for example).
  • Splitting of a ridge is a special type of crossing, where a ridge, becoming wider, splits into two edges. This is a frequently appearing type of a crossing, and its specific structure
  • the parameters of the images are optionally aggregated along the Characteristic Lines, at the Crossings of characteristic lines in the represented image and/or between the Characteristic Lines and the Background. This includes storing the Cross-Section parameters of Characteristic Lines at their Crossings, then interpolating them along the Line and using these interpolated data either as the final one or as a prediction. In the last case the actual Cross-Section parameters at the Line Points are given by the co ⁇ ections to the predicted values. The same procedure can be performed for the Background values at the Crossings and along the lines. Thus, the stability of the image representation is increased and the amount of data required for representation is reduced. Associating depth to CORE/VIM images
  • Depth value can be associated to each of the CORE/VIM elements, thus making the image three-dimensional.
  • depth value is added to the coordinates of each of the control points of the splines, representing characteristic lines, to the center coordinates of patches and to the coordinates of the background points.
  • depth values are associated with the Color Cross-section parameters of the characteristic lines.
  • Layers of VIM images i some embodiments of the invention, some parts of the VIM image, optionally bounded by characteristic lines, are defined as Layers.
  • VIM Layers can move independently of one another, and, being three-dimensional objects, they may occlude one another.
  • VIM Layers are also called VUVI Sub-Textures. Objects of VIM images
  • VIM Layers are defined as VIM Objects.
  • a rigid motion of VIM Objects in the ambient 3D space is allowed.
  • VIM Skeletons are normally associated to VIM Objects, as described below. Reconstruction algorithm
  • the reconstruction algorithm starts with the input parameters, described above, and comprises the following main procedures, exemplary embodiments of which are described below:
  • u is the distance from the central line (with the sign)
  • t is the coordinate (along the line) of the projection on the central line.
  • DL the coordinates (u,t)
  • the brightness of the cross-sections (computed in the Procedure CS) is interpolated between the control points to a complete neighborhood of the characteristic line.
  • the brightness of a patch is optionally computed using a certain Gaussian - type brightness function, with the basis (in the image plane) - for example an ellipse, defined by the input patch parameters, and the height of the vertex equal to the input brightness parameter.
  • a specific choice of the model Gaussian - type brightness function is influenced by the same considerations as the choice of the cross-sections model shapes, hi particular, it can be taken to be a paraboloid with the elliptic basis and the vertex as specified by the input parameters (procedure BP).
  • Background brightness values are computed by an interpolation between the brightness values of closed background components, the margin brightness values of the separating characteristic lines and the brightness values of the background representing points.
  • the interpolation process is performed in such a way that the interpolation does not "cross" the separating characteristic lines (Procedure BB).
  • the interpolation process includes a "signal expansion algorithm", in which the background representing points (and the margins of separating lines) transmit their brightness value to the neighboring pixels, which in turn transmit it to their neighbors, etc.
  • the signal is transmitted only to the neighboring pixels, lying on the same side of each of the separating characteristic lines.
  • the background brightness value is computed at each pixel as a weighted average of the brightness values, received by this pixel in the process of signal expansion.
  • the weights reflect the distances to co ⁇ esponding background representing points.
  • the range of the signal expansion from each background representing point is limited to a certain constant, reflecting the density of the background representing points.
  • the final brightness values of a VIM image are computed as the values of the characteristic lines, patches or the background at the interior pixels of the co ⁇ esponding parts of the image. At the margin areas of the characteristic lines and of the patches the final brightness is computed by averaging their brightness values with the background brightness, with the weights compute in the Procedures WL and WP, respectively. To simplify the presentation we assume below that the right width WR and the left width WL of the ridge cross-sections are always the same and use a single width parameter W.
  • Fig. 12 is a flowchart of acts performed in reconstructing an image from a CORE representation, in accordance with an embodiment of the present invention.
  • the reconstruction algorithm starts with the input parameters, as above, and computes the brightness (color) value of the VIM image at each given point z in the image plane. In the case of a color image these computations are performed for each color component.
  • Procedure MAIN Computing the Final Brightness Values
  • BB(z), BL(z) and BP s (z) are the brightness functions of the background, of the characteristic lines and of the patches, computed in the Procedures BB, BL and BP, respectively, and the sum ⁇ runs over all the patches P s .
  • division by the sum of all weight functions in formula (1) guarantees that their sum is identically 1 and that formula (1) is a true averaging.
  • Procedure BL Brightness of Characteristic Lines
  • This procedure computes for any point z on the image the brightness BL(z), as defined by the cross-sections of the characteristic lines.
  • BL(z) needs to be computed only for those z which are "close enough" to at least one of the characteristic lines in the texture, as expressed by the weight function WL.
  • the most intuitive and natural way to define the brightness of a characteristic line is to associate to it a coordinate system (uu, tt), with uu(z) the (signed) distance of the point z to the line along the normal direction, and tt(z) the length parameter along the curve of the orthogonal projection pp(z) of z onto the line. Then the brightness cross-sections are computed according to the coordinate uu and interpolated along the line with respect to the coordinate tt.
  • the co ⁇ esponding algorithm can be constructed. However, it provides some serious drawbacks:
  • S(z), u(z) and t(z) are computed by the Procedure DL, described below.
  • the Procedure BL splits according to whether the link S(z) has a "free end” (i.e. an endpoint, not belonging to any other link) or not. The case where S(z) does not have "free ends"
  • Ci and C2 denote the equations of the two cross-sections (normalized to the unit width, as described in the Procedure CS below) at the two endpoints of the link S(z).
  • Wi and W2 denote the respective right widths RWi and RW 2 of the cross-sections at these points.
  • W t and W 2 denote the respective left widths LWi and LW of the cross-sections at these points.
  • W(z) t(z) - W 1 + (l -t(z))- W 2 u(z) f u(z) and the values C l and C are computed by the procedure CS.
  • Fig. 2 illustrates this construction. The case where S(z) has a "free end"
  • BL(z) BM.
  • DE is a positive tuning parameter, defining the shape of the end of a characteristic line.
  • BM is half of the sum of the brightness parameters LB2 and RB2 of the cross-section at the
  • t(z) is replaced by l-t(z) in the above formula.
  • the formula above provides one of possible choices of the shape of characteristic lines near their ends. It assumes that the cross-section brightness gradually descends to the "middle basis" value BM inside the prescribed distance DE. Other shapes can be defined, by properly computing the width and the brightness in the neighborhood of the end point. This is done by specifying a shape of the characteristic line near the end (for example, circular, elliptic or parabolic), computing the cross-section width according to the chosen shape, and rescaling the brightness cross-section accordingly.
  • Procedure CS Cross-Sections of Characteristic Lines
  • the CS procedure computes a brightness value of an edge or a ridge (unit width) cross- section CS(u) for any given cross-section "interior" brightness parameters, as described above, and for any value of u.
  • u is the distance u(z) to the line, normalized by the width W(z) of the line, so the width parameter W is taken into account inside the BL, and it does not appear below.
  • the margin brightness parameters LBl and RBI enter the computations in the Background brightness Procedure BB.
  • the recommended edge cross-section is composed of two symmetric parabolic segments.
  • CS(u) LB2 + (RB2 - LB2)*NEC(u) Ridge Cross-Section
  • the width of the ridges is taken into account in the Procedure BL.
  • This block computes the weight function WL(z), which is used in a final blending of the characteristic lines with the background.
  • the function WL(z) is equal to one in a certain neighborhood of the characteristic lines, and is zero outside of a certain larger neighborhood. More accurately:
  • the distance u(z) is computed in the Procedure DL.
  • XJ ⁇ and UL are tuning parameters, see the last section "Tuning Parameters".
  • Fig. 16 shows a typical cross- section and a general shape of the weight function WL(z), in accordance with an embodiment of the present invention.
  • Procedure DL Distance to Characteristic Lines
  • This Procedure optionally computes for any point z in the texture the point p(z) on the characteristic lines which is nearest to z, i.e. its the "projection" of z onto the set of characteristic lines, the distance u(z) between z and p(z), the link S(z) on which p(z) resides and the proportion t(z) in which p(z) divides the link S(z).
  • u(z), p(z) and t(z) are not exactly the Euclidean distance, the co ⁇ esponding mathematical projection and proportion respectively; however, in most cases they give a reasonable approximation for these mathematical entities.
  • the Euclidean distances and/or other distances are used.
  • S(z) is defined as the link Sj, for which the minimum of the absolute values
  • the straight oriented segment [a, d], joining the end points of the link S is constructed, with the orientation, induced from the orientation of the poly-link, containing S.
  • Ii is the straight line, containing the segment [a, d].
  • 1 2 and 1 3 are the straight lines, orthogonal to li and passing through a and d, respectively.
  • the function u(z) is constructed as follows:
  • is the length of the segment, joining z and S and orthogonal to li.
  • 11 be an oriented line, formed by the interval of the line 1] from infinity to a, then by the link S from a to d, and then by the interval of the line li from d to infinity. For z right to 11
  • the sign of u(z) is "+".
  • the sign of u(z) is "-”.
  • the projection p(z) is the intersection point of S and of the segment, joining z and S and orthogonal to l ⁇ .
  • p(z) is a, and for z right to 1 3 , p(z) is d.
  • M be the linear transformation of the plane, fransforming the basis ellipse of the patch to the unit circle.
  • M is a product of the translation by (-xO, -yO), the rotation matrix to the angle -a, and the rescaling 1/Rl and 1/R2 times along the x and y axes, respectively.
  • BP s (z) BM + (BP - BM)(1 - x'(z) 2 + y'(z) 2 ) for x'(z) 2 + y'(z) 2 ⁇ 1.
  • WP s (z) 0 for uu(z) > UP 1
  • WP s (z) 1 for uu(z) ⁇ UP2
  • WP s (z) (UPl-uu(z))/(UPl-UP2) for uu(z) between UP2 and UPl
  • Procedure BB Brightness of Background This Procedure computes the brightness value of the background at any point z of the image. This value is obtained as a result of interpolation between the "global" background brightness values, the margin brightness values of the characteristic lines and the brightness values at the background representing points. The main difficulty is that the interpolation is not allowed to cross the separating lines. To overcome this difficulty a special "distance" d between the points on the image is introduced and computed in the Procedure SE below. Then averaging weights are computed through the distance d.
  • This block uses as an input a certain collection of the background representing points Z , (containing the input background representing points, as described in the Addendum A above, and the margin representing points, produced by the block "MRP", described below). At each point Zi the brightness value Bbi is given.
  • the background brightness value BB(z) is finally produces by the block BB as follows: BB(z) is the weighted sum of the global brightness GB and of the Local brightness functions Bbi(z) over all the background representing points Zi :
  • GW(z) is zero at any z, where at least one of the weights of the representing points is 1, and GW(z) is one at any z where all the weights of the background representing points vanish.
  • This Procedure computes the global background value GB, which appears in the expression (2) in the Procedure BB.
  • GB is equal to this global value GB r .
  • GB is equal to the default global value DGB. If DGB is not defined, GB is equal to zero.
  • the cu ⁇ ent procedure consists in a signal expansion, that transmits to each pixel its Sub-Texture number. We describe it shortly, since it essentially belongs to a higher data representation layer.
  • the procedure MRP is applied, which creates margin representing points, carrying the co ⁇ esponding Sub-Texture numbers. These numbers are taken from the co ⁇ esponding poly-links.
  • the Signal Expansion Procedure is applied to the margin representing points, essentially as in the block SE, with the following difference: only the marking and the number of the Sub-Texture is transmitted between the pixels on each step of signal expansion.
  • each pixel in the image memorizes the number of the Sub-Texture, to which it belongs.
  • Bbi(z) Two types of the local brightness functions Bbi(z) are used.
  • Bbi(z) is identically equal to the input brightness value Bbi a * me point Zi .
  • LBF is zero for the zero order and LBF is one for the first order of the functions Bbi(z).
  • N is an integer valued tuning parameter.
  • Typical value of N is 4 or 9: usually the background representing points form a regular or an almost regular grid, and the nearest neighbors are taken at each point 2 to construct the linear function Li(z).
  • the weights WR(d(z, Zi)) depend only on the distance d(z, Zi) from the point z to the background representing point Zi.
  • > UBl, RGWF(t) 1 for
  • ⁇ UB2, and WR(t) BVS(3v 2 - 2v 3 ) + (1 - BVS)v , for UB2 ⁇
  • ⁇ UBl, where v (
  • D denote the domain of the VIM image, with "cuts" along all the separating poly- links PI .
  • the distance dd(z ls z 2 ) is defined as the (Euclidean) length of the shortest path, joining ⁇ and z in D and avoiding all the cuts PL- See Fig. 22, for example. It is natural to assume that the influence of the color at zi to the color at z decreases as the distance dd(z 1? z ) increases. However, a precise computation of the distance dd is a rather complicated geometric problem. Consequently, we use instead of the distance dd(z l5 z 2 ) its approximation d(z 1? z 2 ), which is computed through a "signal expansion algorithm", as described below.
  • the block SE computes the distance d(z ls z 2 ) for any two points z ⁇ and z 2 in the image plane.
  • the algorithm is not symmetric with respect to z ⁇ and z 2 : in fact, for a fixed point zi, the distance d(z l5 z ) is first computed for any pixel z 2 of the image. Then an additional routine computes d(z l5 z ) for any given z2 (and not necessarily a pixel).
  • a "neighboring pixel" is used. It is defined as follows: for z not a pixel, the four pixels at the corners of the pixel grid cell, containing z, are the neighbors of z.
  • the distance d(z ls z ) is computed in the following steps: For any pixel p the distance u(p) to the separating poly-links PL; is computed and stored at this pixel. The computation of u(p) is performed by the procedure DL, described above, applied only to separating poly-links PL . Those pixels p, for which u(p) ⁇ FU, are marked as
  • forbidden pixels are excluded from all the rest of computations, and those pixels that are not forbidden, are called below “free” ones.
  • FU is a tuning parameter.
  • each of the free neighbor pixels of zi is marked, and this pixel memorizes its Euclidean distance from zi as the auxiliary distance dd, to be computed.
  • any free unmarked pixel p at least one of whose free neighboring pixels was marked in the previous steps, is marked.
  • This pixel memorizes as its auxiliary distance dd(p) from z ⁇ , the minimum of dd at the neighboring free pixels plus one.
  • This process is continued the number of steps, equal to the maximal dimension of the image (in pixels).
  • each free pixel p on the image plane memorizes its auxiliary distance dd(p) from zi.
  • d(z ls z ) from is computed as maximum of DI and D2, where DI is the Euclidean distance of z 2 to z 1 , and D2 is the minimum over the free neighboring to z pixels p, of dd(p) + the Euclidean distance of z 2 to p. This optionally completes the computation of the distance d(z ⁇ , z ). See Fig. 23.
  • the tuning parameter FU optionally determines the size of a neighborhood of the separating poly-links, where all the pixels are marked as forbidden. Taking any value of FU, larger than 0.8 excludes a possibility of signal expansion crossing separating lines. Indeed, for any two neighboring pixels, which are on different sides of a separating line, at least one is closer to the line than 0.8 and hence is marked as forbidden. To provide stability of finite accuracy computations a bigger value of U may be taken. However, in this case signal expansion will not pass a "bottle-neck" between two separating lines, which are closer to one another than 2FU. Normally such regions will be covered by the cross-sections of these lines. However, a sub-pixel grid can be used to guarantee that signal expansion passes thin "bottle- necks".
  • the image is subdivided into blocks in 2 - 3 scales (say, blocks of 16x16, 8x8 and 4x4 pixels), hi a first stage, signal expansion is performed between the highest scale blocks (say, 16x16), as described above. Forbidden are the blocks, crossed by separating characteristic lines. In the second stage the forbidden blocks are subdivided into 8x8 sub- blocks, and the expansion is performed for them. The new forbidden sub-blocks are subdivided into the 4x4 ones, and the expansion is repeated, hi the last stage the expansion is completed on the pixels level.
  • the distance d(z l5 z 2 ) is optionally computed only until the threshold UBl is reached, since for larger distances the weight functions vanish. This restricts the number of steps in signal expansion to UBl + 1.
  • signal expansion and memorization of the distances at the free pixels can be implemented for all the background representing points at once (especially since the above distance restriction usually makes for any pixel only the information relevant, concerning a few neighboring background grid points).
  • all the mathematical data required in the interpolation block is computed incrementally, by using well known formulae for incremental computation of polynomials on grids.
  • Procedure MRP Margin Representing Points
  • This Procedure constructs a grid of representing points on the margins of all the characteristic lines together with the background brightness values at these points. Later the constructed margin points are used (together with the original background representing points) in the interpolation process in the block BB.
  • the margin representing points MZJ are produced in the following steps: a. On each poly-link, the points W are built with the distance UM1 from one another, starting with one of the ends (the distance is measured along the poly-link). The last constructed point on each poly-link may be closer to the end joint of this poly-link, than UMl. b. At each W k the line l orthogonal to the poly-link and intersecting it at W k is drown.
  • S(MZJ) and t(Mz j ) are optionally computed by the Procedure DL.
  • UM1 and UM2 are optionally tuning parameters (the first one absolute and the second relative to the width), satisfying UM1 ⁇ UBl, 1 ⁇ UM2 ⁇ UL1. See Fig. 24.
  • depth values are optionally stored in VUVI images in the same way as each of the color components.
  • the depth value is optionally reconstructed for each image pixel using the same procedure, as used for reconstructing the color values of the image.
  • the depth may be used in displaying images with layers, in order to determine how overlapping layers are to be displayed, for example using the "z-buffering" procedure, known in the art.
  • the following operations are performed in reconstruction of a multi-layer VIM image:
  • Color values and depth values are reconstructed for each layer separately (as described above) at each pixel of the bounding rectangle of the layer. Pixels outside the layer are marked accordingly.
  • a buffer is created for the entire image. In this buffer for each pixel its color and depth values, obtained from each of the layers, are stored.
  • the smallest depth of the layers is chosen (i.e. the layer, nearest to the viewer).
  • depth sorting can be performed successively for each layer, in the process of computing the color and the depth values of this layer.
  • VIM Animation hi an exemplary embodiment of the invention, animation of VIM images is provided by any evolution in time of the parameters of VIM/CORE elements. Since CORE representation faithfully captures any image, and behaves consistently in video-sequences, usually a proper evolution in time of the parameters of VIM/CORE elements, with only eventual replacing of these elements, allows for faithfully representing video-sequences.
  • any of the geometric and brightness parameters of the CORE images may evolve with time independently of the others.
  • a sub- group of parameters may change.
  • change values for a first group of parameters is specified for each image, while the change values of a second group of parameters are specified only when necessary.
  • the change parameters are specified relative to the previous base image.
  • the change values are specified relative to the previous image.
  • change values are specified for the following parameters: Rigid motion of the VIM Objects of the image in 3D space, Color transformations for each Layer of the image, Mutual motion of Layers in a VJJVI Object and Deformation of Layers in a VIM Object.
  • the mutual motion of layers and/or the deformation of layers are defined via the Skeleton Motion, as described below.
  • the animation process is performed as follows: A given image or a sequence of images are transformed into CORE format by a CORE transformation, as described in U.S. Patent applications 09/716,279 and/or 09/902,643.
  • initial image analysis and/or detection of edges and ridges are performed as described in U.S. provisional patent application 60/379,415.
  • a CORE image is created synthetically, for example using a graphical editing tool.
  • the layers of the VUVI image are defined by a user who manually marks the contours of each layer.
  • an automatic layer identification procedure identifies layers, for example by identifying textures su ⁇ ounded by closed characteristic lines.
  • depth is added to the VIM Layers.
  • VIM objects are optionally defined by a human operator as combinations of some of the created VUVI Layers.
  • VIM (or CORE) Skeletons are optionally inserted into VIM Objects.
  • new skeletons can be created using the Skeleton IPI, as described below (or using conventional graphic editors).
  • Library skeletons can also be inserted into new VUVI Objects.
  • An animation scenario is optionally created, using Rigid motion of VIM Objects in 3D space, Color transformations for each Layer and/or Objects animation via their skeletons.
  • the explicit animation is usually created for a sequence of Key-frames and further interpolated to the intermediate frames.
  • Skeleton animations can be created using Skeleton IPI (or other graphic editing tools), or they can be taken from the skeleton animation libraries. If necessary, additional time evolution of any required VIM parameter is created.
  • Fig. 25 represents main steps of animation creation, in accordance with an embodiment of the present invention.
  • a skeleton is a symbolic simplified representation of an object, allowing for an easy and intuitive control of its motion, h some embodiments of the invention, the skeleton is formed partially or entirely from characteristic lines of the object. Consequently, the CORE skeleton controls directly the geometry and the motion of CORE models, without creating any auxiliary geometric and/or kinematical structure.
  • the CORE skeleton may be just a part of the CORE image representation: in this case some of the CORE Lines and points, while expressing color information, serve additionally as the skeleton elements.
  • the same CORE models serve for capturing the color of the object, its geometric 2D and/or 3D shape and its kinematics. Special markers (flags) are used to specify each of these roles. This leads to a dramatic data reduction and simplification of processing.
  • CORE skeletons which are basically applied to (flat) CORE objects, as they appear on the original image.
  • VIM skeletons applied to three-dimensional CORE objects and to VIM objects.
  • R-skeletons applied to any raster image or object.
  • CORE skeletons are essentially a special case of VIM skeletons.
  • a CORE skeleton is obtained by reducing a VIM skeleton to the image plane, h the later stages of construction of VIM objects and scenes the CORE skeleton may be embedded (together with the co ⁇ esponding CORE object) into the 3D space, hi this case it becomes a full-scale VIM skeleton.
  • the R-skeleton uses the same mathematical solutions as the CORE and the VIM skeletons, but can act directly on the pixels of a raster image, as described below.
  • the VIM skeleton optionally comprises a wire-frame, a kinematical model and an influence model, as is now described.
  • each skeleton is associated with a Skeleton Image Processing Interface (IPI), which allows for a creation and an interactive control of a skeleton.
  • IPI Skeleton Image Processing Interface
  • the wire-frame optionally comprises one or several connected 3D
  • - curves represented by splines (preferably of order 1, 2 or 3).
  • these curves are formed by spline segments, joining one another at the join points.
  • Some join points on some of the skeleton components may be marked, and the relative position of certain groups of these marked points with respect to one another, may be fixed.
  • skeleton curves are represented by P- splines.
  • the skeleton Lines are specified among the rest of the VIM's Lines by the special "Control Flag".
  • some or all of the skeletons are specified by a plurality of joints and the frame is determined by connecting the joints with default segments.
  • the default segments comprise straight lines.
  • a full kinematical model allows for a free choice of the 3D position of the entire skeleton, of the 3D position of any of the (non-fixed) join (end) points of the spline segments, and the control of the parameters of the spline segments between the join (end) points, hi some embodiments of the invention, to make animation easier, some of the degrees of freedom of the skeleton may be frozen in a default mode of operation (according to the kinematics of the animated object or to the anatomy of the animated character), and become available to the operator only in a special operation mode.
  • some degrees of freedom may be restricted partially or completely, or related to other degrees of freedom, according to the kinematics, the anatomy and the geometry of the animated character or object.
  • the (restricted) kinematical model of the skeleton is fixed.
  • Properly constructed kinematical model of the skeleton makes animation especially easy for a nonprofessional operator, since any motion produced will look natural.
  • Skeleton IPI Image Processing Interface
  • the interactive control of the skeleton includes the control of the 3D position of the entire skeleton, of the 3D position of any of the join (end) points of the spline segments, and the control of the parameters of the spline segments between the join (end) points.
  • the IPI for an interactive control of the skeleton can be constructed on the base of the Geometric IPI: The Geometric IPI
  • the structure of the CORE image representation allows for a free access and variation of any of the geometric parameters of a characteristic line. Consequently, any operation on these parameters can be performed automatically or interactively.
  • the Geometric IPI can be constructed as follows. The operator indicates a certain end- point of the spline segments on the central line of a characteristic line, displayed on the screen. Then the indicated point is interactively moved into a desired position. The spline segments follow the motion of the end-point. By moving the central point of a spline segment, its curvature is controlled.
  • the central line can be represented by a Bezier spline, and the operator then controls its shape changing interactively positions of the spline control points.
  • Well known conventional interfaces can be used in this step.
  • the operator controls the projection of the skeleton on the screen plane.
  • the depth of each of the join points and of the spline segments can be controlled by pointing with the mouse at the join point (an interior point of the spline segments, respectively), simultaneously pressing one of the buttons, assigned as the depth control button.
  • Any conventional 3D interface can also be used to control the VIM skeleton.
  • the same Skeleton IPI allows for an interactive creation of any desired VUVI skeleton, hi an embodiment, the operator first draws the plane wire-frame, as seen in the screen plane. Then the fixed groups of the join points are marked and the depth is inserted. Next the operator defines the kinematical restrictions of the skeleton.
  • Fig. 26 shows a block-diagram, representing steps of a skeleton construction, in accordance with an embodiment of the present invention.
  • the influence model of the VIM skeleton defines in what way the motion of the skeleton and of its parts influences the motion of the entire VIM object and the motion and geometric transformations of each of the 3D CORE objects, from which the VIM object is constructed.
  • the influence model comprises the following two parts: 1. Influence scheme, prescribing the parts of the object, influences by different parts of the skeleton. 2. Motion transfer model, prescribing the way in which the motion of the skeleton is translated into the motion of any nearby point in 3D space. As the influence model of the VJJVI skeleton is fixed, the motion of the skeleton is optionally translated into the motion of the VIM object in a straightforward way: - For each 3D CORE object inside the VIM object only the motion of the part of the skeleton, prescribed by the influence scheme, is taken into account.
  • Each control point of the splines forming the central lines of the characteristic lines in the CORE object (in particular, the join points and the middle points in the case of parabolic splines), is transformed according to the motion transfer model. This transformation includes the depth, associated to this control point.
  • Each control point of the splines, forming the cross-sections of the characteristic lines in the CORE object, is transformed according to the motion transfer model. This transformation includes the depth, associated to this control point.
  • the influence scheme associates to each 3D object inside the animated VIM object a certain part of the skeleton (usually consisting of some of the skeleton components). Only this part influences the motion of this CORE object, hi a specific implementation, described above, the influence scheme of the skeleton is based on the specification of the Layers (Sub-Textures), affected by each part of the skeleton.
  • Motion transfer model
  • the functioning of the skeleton is optionally based on the fact that its motion is translated into the motion of the nearby points, in particular, the motion of the nearby CORE models.
  • the present invention utilizes the following general scheme to transfer the skeleton motion to the nearby points: -
  • the coordinate frame is constructed, comprising coordinate systems around each skeleton component.
  • a new point p' is found, whose coordinates with respect to the transformed skeleton component are the same, as the coordinates of p with respect to the original component.
  • p' is the result of the desired motion of p.
  • the result of its motion is obtained by averaging the points p', obtained with each of the influence regions involved.
  • the averaging weights can be taken to be inverse proportional to the distances of p to the co ⁇ esponding components.
  • each "bone” has its geometric influence zone, and to each point a weight is associated, reflecting the influence of each bone.
  • the actual motion of the point is a weighted average of the bone's motions.
  • the coordinate frame of the VDVI skeleton optionally comprises special coordinate systems, associated to each of the skeleton components, and of the "influence regions" of these components.
  • the following coordinate system (u,t,w) is associated to each component of the skeleton: for any point p in the 3D space, u is the distance of this point from the considered skeleton component.
  • the t coordinate of p is defined as the coordinate of the projection of the point p onto this component (i.e. the distance, along the component, of the projection of p from one of the end points), h turn, the w coordinate of p is defined as the rotation angle of the vector, joining the point p with its projection on the component, and the reference direction at the projection point.
  • the reference direction is mostly chosen to be the same at any point of the component (for example, the orthogonal direction to the component in the image plane). However, interactive choice of the reference directions at the join points and their interpolation along the component are also allowed.
  • the new coordinates (u',t',w') for a point s in 3D space are defined by the same expressions as above: u' is the distance of the point s from the transformed skeleton component.
  • the t' coordinate is the coordinate of the projection of s onto the transformed component (i.e.
  • the w' coordinate of s is defined as the rotation angle of the vector, joining the point s with its projection on the transformed component, and the reference direction at the projection point.
  • the evolution of the reference direction following the prescribed evolution of the skeleton, can be defined as follows, h some embodiments of the invention, for rigid transformations of the entire skeleton, the reference direction is transformed exactly in the same way, while for restricted motions of the parts of the skeleton, the reference direction is kept unchanged. However, if the parts motions involve relatively strong 3D rotations, the reference directions at the j oin points follow these rotations.
  • the construction of the coordinate frame of a VIM skeleton involves an algorithmic problem that should be addressed in an implementation.
  • the problem is that if a skeleton component has a complicated geometric shape, and if the point p is relatively far away from this component, the projection of p on the component (and hence the distance u, the coordinate t and the rotation angle w) are not uniquely defined. This non-uniqueness, in turn, leads to numerical instability of the relevant computations.
  • This problem is optionally settled as follows:
  • the influence regions of the skeleton components consist of all the points in 3D space, that are closer to this component than a certain threshold S.
  • the threshold S is optionally chosen according to the actual shape of the object to be animated.
  • the operator can define more complicated influence regions interactively, using Geometric 's or any conventional 3D editor. More complicated shapes of the influence region can be chosen, for example, those used in MPEG-4 Human animation.
  • the CORE skeleton is a special case of the VIM skeleton. Its only distinction is that it is restricted to the image plane. Hence it can be applied to any CORE object, without transforming it to a 3D CORE object. All the description above remains valid, with appropriate simplifications. In other embodiments of the invention, the skeleton is allowed a
  • the nearest component of the skeleton is pre- computed, as well as the coordinates (u,v,t) with respect to this nearest component.
  • the new position of the point is determined by the requirement that its coordinates (u',v',f) with respect to the transformed skeleton component be the same as the initial coordinates (u,v,t).
  • CORE and VUVI skeletons provide a flexible and convenient tool to create a desired motion of the CORE (VIM) object, in accordance with an exemplary embodiment of the invention.
  • the user optionally creates a desired evolution in time of the skeleton, and the object follows this evolution.
  • some embodiments of the invention usually, only a sequence of key-frame positions of the skeleton must be created, while the intermediate skeleton positions are obtained by interpolation.
  • the motion scenario is a sequence of the skeleton parameters, co ⁇ esponding the chosen key-frame positions. An important issue is a choice of the skeleton parameters to be memorized in the scenario.
  • skeleton parameters are parameterized by elements of the group 0(3) of 3x3 orthogonal matrices, defined in the MPEG-4 standard. Alternatively or additionally, Quaternion parameters, as discussed in MPEG-4 standard, are used instead of elements of the 0(3) group.
  • the motion scenarios represented in this invariant form (i.e. with rotation angles and/or matrices, can be used without any modification for any rescaled and spatially displaced object.
  • the interpolation of the VIM (CORE) skeleton parameters can be implemented in various forms. One convenient way includes a linear interpolation of the positions of the join and end points of spline segments and of the interior spline segments parameters. Higher order interpolation, taking into account more than two nearest neighbor key-frames, can be performed. However, the choice of the kinematical model and of the scenario parameters can prescribe another type of interpolation.
  • interpolation of the rotation parameters described above must be performed inside the group 0(3) of 3x3 orthogonal matrices, to preserve the visual integrity of the rigid parts of the animated object.
  • Such an interpolation can be performed along geodesies in the group 0(3).
  • higher order interpolation can be applied to preserve a natural and smooth character of the motion.
  • R-skeleton R-skeleton is mathematically identical to the VIM skeleton. However, it can be applied to raster images and their Layers. This is optionally done by applying the above given formulae to each pixel of the Layer, associated to the co ⁇ esponding part of the skeleton. Thus, a nonlinear "warping" is performed over some or all the pixels of the Layer. This potentially allows one to reproduce in a rather accurate way the most complicated skeleton motions.
  • An alternative way to impose the skeleton motion onto the Layer pixels is to translate it into a certain kind of mathematical plane transformations: projective, bilinear, affine, or rigid movements and rescaling.
  • This implementation is computationally simpler, but it introduces a mathematical rigidity into the possible layer motions.
  • a Character is subdivided into smaller Layers, whose, for example, affine motions reconstruct with a sufficient accuracy the original skeleton motion.
  • each of some or all of the bones of the skeleton has the following three additional parameters:
  • the usage of the additional degrees of freedom, provided by explicit stretching and bending of Bones does not generally create any additional load on the animator. Indeed, the stretching and bending of any bone may be naturally related with the Bone's position and orientation, but “uncoupled” with other Bones parameters. As a result, it can be incorporated into a simple and intuitive Animation Authoring Tool almost seamlessly. This "localization" of the stretching and bending parameters and their independence of the rest of the motion controls is especially important in a full 3D environment where coordination of the character's Skeleton 3D motion is usually a difficult animation task.
  • a times linear stretching is performed of all the space in the direction of the Bone Vector VI .
  • Bending is defined by the bending amplitude parameter b and the bending direction parameter f.
  • the bending direction is given by a unit vector W at the central point of the Bone, which is orthogonal to the Bone vector VI, and it is completely determined by the angle f, that W forms with the unit vector V2.
  • V pVl + qV2 + rV3
  • V pVl + qV2 + rV3 + b*p(l-p)W, for p between 0 and 1
  • V V, for p outside of the interval [0,1], i.e. for the projection of the point V on the Bone's line outside the Bone.
  • the suggested formula symmetric parabolic bending of the Bone is used.
  • Other formulae, specified by the same two parameters, can be used, for example, bending via circular segments.
  • the total shift is a weighted sum of the shifts, imposed by each of the Bones, with the weights defined as described in "Skin and Bones" MPEG-4 documents.
  • the term brightness used above refers to both gray scale levels in black and white images and to any color components of color images, in accordance with substantially any color scheme, such as RGB. It is noted that the present invention is not limited to any specific images and may be used with substantially any images, including, for example, real life images, animated images, infra-red images, computer tomography (CT) images, radar images and synthetic images (such as appearing in Scientific visualization).
  • CT computer tomography

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Generation (AREA)
  • Image Processing (AREA)

Abstract

L'invention concerne un procédé de représentation d'images. Ce procédé consiste: à déterminer au moins une ligne caractéristique sur laquelle le profil de l'image change considérablement; et à stocker pour chaque ligne caractéristique au moins un profil transversal comprenant des données sur la brillance de la ligne et d'au moins un point d'arrière-plan adjacent à la ligne, ce point présentant une brillance sensiblement différente de la brillance de la ligne.
PCT/IL2002/000563 2000-10-13 2002-07-11 Modelisation d'images par la geometrie et la brillance WO2003007486A2 (fr)

Priority Applications (5)

Application Number Priority Date Filing Date Title
EP02747643A EP1417647A2 (fr) 2001-07-12 2002-07-11 Modelisation d'images par la geometrie et la brillance
AU2002318027A AU2002318027A1 (en) 2001-07-12 2002-07-11 Geometric and brightness modeling of images
PCT/IL2002/000935 WO2003045045A2 (fr) 2001-11-23 2002-11-21 Codage d'images geometriques modelisees
AU2002353468A AU2002353468A1 (en) 2001-11-23 2002-11-21 Encoding of geometric modeled images
US10/483,786 US20040174361A1 (en) 2000-10-13 2004-01-12 Geometric and brightness modeling of images

Applications Claiming Priority (12)

Application Number Priority Date Filing Date Title
US30441501P 2001-07-12 2001-07-12
US60/304,415 2001-07-12
US09/902,643 2001-07-12
US09/902,643 US6801210B2 (en) 2001-07-12 2001-07-12 Method and apparatus for image representation by geometric and brightness modeling
US31048601P 2001-08-08 2001-08-08
US60/310,486 2001-08-08
US33205101P 2001-11-23 2001-11-23
US60/332,051 2001-11-23
US33407201P 2001-11-30 2001-11-30
US60/334,072 2001-11-30
US37941502P 2002-05-13 2002-05-13
US60/379,415 2002-05-13

Publications (2)

Publication Number Publication Date
WO2003007486A2 true WO2003007486A2 (fr) 2003-01-23
WO2003007486A3 WO2003007486A3 (fr) 2004-03-04

Family

ID=31982755

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IL2002/000563 WO2003007486A2 (fr) 2000-10-13 2002-07-11 Modelisation d'images par la geometrie et la brillance

Country Status (3)

Country Link
EP (1) EP1417647A2 (fr)
AU (1) AU2002318027A1 (fr)
WO (1) WO2003007486A2 (fr)

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6300960B1 (en) * 1997-08-04 2001-10-09 Pixar Animation Studios Realistic surface simulation in computer animation

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6300960B1 (en) * 1997-08-04 2001-10-09 Pixar Animation Studios Realistic surface simulation in computer animation

Also Published As

Publication number Publication date
AU2002318027A1 (en) 2003-01-29
EP1417647A2 (fr) 2004-05-12
WO2003007486A3 (fr) 2004-03-04

Similar Documents

Publication Publication Date Title
US9972129B2 (en) Compression of a three-dimensional modeled object
US5742294A (en) Method and apparatus for synthesizing images
US5544291A (en) Resolution-independent method for displaying a three dimensional model in two-dimensional display space
US7095879B2 (en) System and method for face recognition using synthesized images
JP3954211B2 (ja) 三次元シーンにおける形状及び模様の復元方法及び装置
US6907140B2 (en) Image recognition/reproduction method and apparatus
KR20220084407A (ko) 포인트 클라우드 표현을 통한 메시 압축
US11398059B2 (en) Processing 3D video content
US20050063596A1 (en) Encoding of geometric modeled images
KR20220137937A (ko) 투영 기반 메시 압축
JPH0877356A (ja) 三次元多眼画像の処理方法及び処理装置
JPH06503663A (ja) 動画作成装置
US20200027261A1 (en) Rendering 360 depth content
US20070103466A1 (en) System and Computer-Implemented Method for Modeling the Three-Dimensional Shape of An Object by Shading of a Two-Dimensional Image of the Object
US5960118A (en) Method for 2D and 3D images capturing, representation, processing and compression
US20040174361A1 (en) Geometric and brightness modeling of images
WO2000004508A1 (fr) Balayage automatique de scenes en 3d provenant d'images mobiles
WO2003007486A2 (fr) Modelisation d'images par la geometrie et la brillance
Borshukov New algorithms for modeling and rendering architecture from photographs
CN109792490B (zh) 用于流图像压缩的球面视频的改进的伪圆柱映射
WO2003045045A2 (fr) Codage d'images geometriques modelisees
Karpouzis et al. Compact 3D model generation based on 2D views of human faces: application to face recognition
Chang Creating interactive 3D media with projector-camera systems
JPH0935082A (ja) 画像処理方法および画像処理装置
EP3598395A1 (fr) Rendu de contenu de profondeur à 360 degrés

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A2

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ OM PH PL PT RO RU SD SE SG SI SK SL TJ TM TN TR TT TZ UA UG US UZ VN YU ZA ZM ZW

AL Designated countries for regional patents

Kind code of ref document: A2

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE BG CH CY CZ DE DK EE ES FI FR GB GR IE IT LU MC NL PT SE SK TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
WWE Wipo information: entry into national phase

Ref document number: 10483786

Country of ref document: US

WWE Wipo information: entry into national phase

Ref document number: 2003513135

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 2002747643

Country of ref document: EP

WWP Wipo information: published in national office

Ref document number: 2002747643

Country of ref document: EP

REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

NENP Non-entry into the national phase in:

Ref country code: JP

WWW Wipo information: withdrawn in national office

Ref document number: 2002747643

Country of ref document: EP