JP2012256280A - Dynamic image processing method, dynamic image processor and dynamic image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To present a dynamic image with vector data.SOLUTION: A dynamic image processing method includes steps of: dividing a closed region with similar color values within a start frame image of dynamic image data into quadrangular curved-surface patches; tracing peaks and control points of the curved-surface patches in chronological order with reference to a frame of the dynamic image data; approximating a locus of the peaks and the control points in a time axis direction by a parametric curve; acquiring each color value corresponding to the peak and the control point of the parametric curve from the frame image of the dynamic image data; and outputting the color value corresponding to each spatiotemporal coordinate value of the peak and the control point of the parametric curve. Accordingly, the dynamic image data can be presented with vector data.

Description

  The present invention relates to a technique for expressing a moving image by vector data.

  In general, an image taken with a digital camera, a video camera, or the like is often stored and used in a bitmap format. In the bitmap format, the image is divided into pixels, and the image is handled as a set of pixels of various colors, so that the realism is excellent. However, it is not easy to edit in units of objects shown in the image, and there is a problem that the image quality deteriorates when the image is enlarged. Vector-based data based on figures is easy to edit on an object-by-object basis because vector data is represented by functional numerical values and the object itself is drawn based on mathematical formulas. Image quality is preserved. Note that an object means a meaningful collection taken in an image, and is represented by, for example, an image of a person, a head, a desk, a vehicle, or the like.

  Against this background, a technique for converting a bitmap format image into vector format data has been proposed (see Non-Patent Documents 1-3). These methods are characterized in that they are converted into vector format data having as few graphic elements as possible without impairing the image quality (realism) of the original bitmap format image as much as possible. Specifically, an object in a bitmap format image is regarded as a closed region, and the outline of the closed region and the region having a similar color value for each region have a “triangle composed of parametric curves” or “side It is divided into parametric curved surface patches such as “a quadrangle composed of parametric curves”, the contour of the closed region is approximated by a parametric curve, and the color inside the parametric curved surface patch is generated by interpolation. Hereinafter, the parametric curved patch may be referred to as a “triangular patch”, a “rectangular patch”, a “curved patch”, or simply a “patch”.

  FIG. 13 shows how a bitmap format image is converted to vector format data. FIG. 13A shows a bitmap-format image in which each pixel has a color value. FIG. 13B shows an example approximated by a quadrangular patch, and FIG. 13C shows an example approximated by a triangular patch. At this time, the rectangular patch and the triangular patch can geometrically have a height in the direction perpendicular to the image plane, but by expressing the value corresponding to this “height” as a “color value”, The color inside the patch can be interpolated. In Non-Patent Document 3, a parametric curve is used only for the outline of the region, and thin plate spline interpolation is used for internal color interpolation.

  Here, examples of a rectangular patch and a triangular patch will be described.

The bicubic Bezier square patch is a coordinate value P _ij (i = 0,1,2,3; j = 0,1,2,3) of 4 × 4 control points arranged in a grid in a three-dimensional space. The point P inside the region surrounded by the four vertices P ₀₀ , P ₀₃ , P ₃₀ and P ₃₃ is expressed by the following equation (1).

B _i ³ (u) and B _j ³ (v) are third-order Bernstein basis functions, respectively. u and v (0 ≦ u ≦ 1, 0 ≦ v ≦ 1) are parameters of the parametric space.

The one-variable third-order Bernstein basis function for the parameter s is expressed by the following equation (2).

  That is, a curved surface passing through the four vertices constituting the quadrangle can be formed using the three-dimensional coordinate values of the 16 control points including the four vertices constituting the quadrangle. Although the parameters u and v are cubic, the number of control points increases by one when the order increases by one.

In addition, the bicubic Bezier triangular patch has three vertexes, two control points of the Bezier curve constituting each side, and one control point P _{ijk in} total within the triangular patch. (I = 0,1,2,3; j = 0,1,2,3; k = 0,1,2,3; i + j + k = 3) Can do.

B _ijk ³ (u, v) is a two-variable cubic Bernstein basis function. u and v (0 ≦ u ≦ 1, 0 ≦ v ≦ 1) are parameters of the parametric space.

A two-variable cubic Bernstein basis function for parameters u and v is expressed by the following equation (4).

  On the other hand, parametric solids have been proposed for the purpose of deforming the shape of a three-dimensional object represented by computer graphics. The 3D object is covered with a parametric solid, the coordinate values of each point constituting the 3D object are associated with the parameters for generating the parametric solid, and the control points of the parametric solid are deformed to change the shape of the 3D object accordingly. It is a technology to deform. For example, Non-Patent Document 4 uses a Bezier solid in which each side or surface of the solid is represented by a Bezier curve or a Bezier patch as a parametric solid.

The cubic Bézier solid is a coordinate value P _ijk (i = 0,1,2,3; j = 0,1,2,3) of 4 × 4 × 4 control points arranged in a three-dimensional space in a three-dimensional space. k = 0,1,2,3) and the parameters u, v, and w can be used to express the point P inside the solid as the following equation (5).

B _i ³ (u), B _j ³ (v), and B _k ³ (w) are the third-order Bernstein basis functions represented by Expression (2), respectively. u, v, w (0 ≦ u ≦ 1, 0 ≦ v ≦ 1, 0 ≦ w ≦ 1) are parameters of the parametric space.

Similarly, the Bezier triangular patch represented by the expression (3) is regarded as the bottom surface, and the triangular prism whose surface is a curved surface is defined as a coordinate value P _{ijk, m} (i = 0,1,2, 2) of 10 × 4 control points. 3; j = 0,1,2,3; k = 0,1,2,3; m = 0,1,2,3; i + j + k = 3) and parameters u, v, w The point P inside the solid body can be expressed by the following formula (6).

B _m ³ (w) is a one-variable cubic Bernstein basis function represented by Expression (2), and B _ijk ³ (u, v) is a two-variable cubic Bernstein basis function represented by Expression (4).

Japanese Patent No. 4189276 Japanese Patent No. 2834790

Brian Price and William Barrett, “Object-Based Vectorization for Interactive Image Editing”, Visual Computer, 22, August 2006, p.661-670 Yu-Kun Lai, Shi-Min Hu and Ralph R. Martin, “Automatic and Topology-Preserving Gradient Mesh Generation for Image Vectorization”, ACM Transactions on Graphics, Vol. 28, No. 3 (SIGGRAPH 2009), August 2009 , Article 85: 1-8 Tian Xia, Binbin Liao, Yizhou Yu, “Patch-based Image Vectorization with Automatic Curvilinear Feature Alignment”, ACM Transactions on Graphics, Vol. 28, No. 5 (SIGGRAPH Asia 2009), December 2009, Article 115: 1 -Ten Thomas W. Sederberg and Scott R. Parry, “Free-Form Deformation of Solid Geometric Models”, Computer Graphics, Vol. 20, No. 4 (Proceedings of SIGGRAPH 1986), 1986, p. 151-160 Jue Wang, Yingqing Xu, Heugn-Yeung Shum and Michael F. Cohen, “Video tooning”, ACM Transactions on Graphics, Vol. 23, No. 3 (SIGGRAPH 2004), August 2004, p.574-583

  With regard to the vector data of a moving image, for example, it is conceivable that the motion of a graphic element such as a circle or a rectangle is represented by a trajectory or motion equation along the time axis direction, that is, an animation. However, it is not easy to express a real moving image shot with a video camera or the like as such an animation. This is because an object that is a subject is accompanied by a change in shape or concealed, and it is difficult to restore the shapes and motions of all subjects captured in the moving image as mathematical expressions.

  On the other hand, a method for linearly approximating the color information of all the pixels in the initial frame and the temporal change of the pixels by focusing on the pixels rather than the objects in the moving image is disclosed. For example, Patent Document 1 discloses a method of generating an image at an arbitrary time on the way by connecting pixels corresponding to an initial frame and an end frame with a straight line. Patent Document 2 discloses a method of tracking a pixel of the first frame over the subsequent frames and approximating it with a straight line for a moving image taken by a camera that moves a stationary rigid body at a constant linear velocity.

  However, these methods have a problem that it is difficult to apply when the subject moves or deforms. In Non-Patent Document 5, for a moving image in which a subject moves or deforms, a method of generating an illustration-like image by approximating the boundaries of pixels and the subject with a smooth curve and connecting them smoothly in the time axis direction. However, it is difficult to realistically reconstruct the original image.

  The present invention has been made in view of the above, and an object thereof is to express a moving image as vector data.

  A moving image processing apparatus according to a first aspect of the present invention includes a reading unit that reads moving image data and stores the moving image data in a storage unit, reads a frame image designated by the moving image data from the storage unit, and performs similar processing in the frame image. Vectorization means that divides a closed region having the specified color value into curved surface patches, registers the edge of the edge of the curved surface patch as a vertex in the vertex list, and registers the points for configuring that edge as control points in the control point list A frame image is read from the moving image data in chronological order, and for each curved surface patch, a corresponding point corresponding to a vertex registered in the vertex list is detected from the read frame image, and a vertex is used as a vertex in the frame image. Vertex tracking means to be registered in the list, and each curved surface patch is registered in the vertex list for each frame image read in chronological order. When the determined vertex is the end of the side of the curved patch, the position of the control point for constructing the side of the curved patch is estimated based on the positional relationship of the vertex and controlled as the control point in the frame image Control point tracking means for registering in the point list, and curve approximation that refers to the vertex list and the control point list, and approximates the trajectory in the time axis direction of each vertex and control point registered by the vectorization means with a parametric curve Means, color value acquisition means for acquiring color values corresponding to vertices and control points of the parametric curve approximated by the curve approximation means, from a frame image of the moving image data, and a time space of the vertices and control points of the parametric curve Output means for outputting a color value corresponding to the coordinate value.

  According to a second aspect of the present invention, there is provided a moving image processing method, comprising: a step of reading moving image data by a reading unit; and storing the moving image data in the accumulating unit; and a specified frame image of the moving image data from the accumulating unit by a vectorizing unit. Read out, divide a closed region with similar color values in the frame image into curved patches, register the edge of the curved patch edge as a vertex in the vertex list, and control the points that constitute that edge as control points A step of registering in a point list, and a vertex tracking unit that reads frame images from the moving image data in chronological order, and corresponding points corresponding to vertices registered in the vertex list from the read frame image for each curved surface patch Are detected and registered in the vertex list as vertices in the frame image, and read out in chronological order by the control point tracking means For each frame image, for each surface patch, the position of the control point for configuring the side of the surface patch when the vertex registered in the vertex list is the end of the surface of the surface patch, A step of estimating based on the positional relationship and registering it in the control point list as a control point in the frame image; and referring to the vertex list and the control point list by a curve approximating unit, and a vertex registered by the vectorizing unit And a step of approximating the trajectory in the time axis direction of each control point with a parametric curve, and the color value corresponding to the vertex of the parametric curve approximated by the curve approximation means and the control point by the color value acquisition means of the moving image data Obtaining from the frame image, and the vertices and control points of the parametric curve by the output means And having the steps of: outputting a color value corresponding to the spatial coordinates.

  A moving image processing program according to a third aspect of the present invention causes a computer to function as the moving image processing apparatus.

  According to the present invention, a moving image can be represented by vector data.

It is a functional block diagram which shows the structure of the moving image processing apparatus in this Embodiment. It is a flowchart which shows the flow of the process which vectorizes the still image of a start frame image. It is a flowchart which shows the flow of the process which tracks the locus | trajectory of the time-axis direction of the vectorized data. It is a flowchart which shows the flow of the process which represents the motion of the vectorized data by a parametric solid. It is a figure which shows a mode that a vertex is tracked. It is a figure which shows the change of the positional relationship of a vertex and a control point. It is a figure explaining evaluation of a control point. It is a figure which shows the example of the Bezier hexahedron which followed the square Bezier patch in the time-axis direction. It is a figure which shows the example of the Besier triangular prism which followed the triangular Bezier patch in the time-axis direction. It is a figure explaining the division | segmentation of a Bezier curve. It is a figure which shows the position of the edge of a patch, and a control point when the edge of a patch is approximated with various methods. It is a figure which shows a mode that a color value is interpolated. It is a figure which shows a mode that the image of a bitmap format is converted into the data of a vector format.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a functional block diagram showing a configuration of a moving image processing apparatus according to the present embodiment. The moving image processing apparatus shown in the figure includes a moving image reading unit 11, a still image vectorization unit 12, a vertex tracking unit 13, a control point tracking unit 14, a parametric three-dimensionalization unit 15, a color value acquisition unit 16, an output unit 17, And a storage unit 18. Each unit included in the moving image processing device may be configured by a computer including an arithmetic processing device, a storage device, and the like, and the processing of each unit may be executed by a program. This program is stored in a storage device included in the moving image processing apparatus, and can be recorded on a recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or provided through a network. Hereinafter, each part will be described.

  The moving image reading unit 11 reads a moving image file and stores moving image data in the storage unit 18.

  The still image vectorization unit 12 reads the start frame image of the moving image data from the storage unit 18 and divides the closed region having similar color values in the start frame image into triangular or quadrangular curved surface patches. Approximate the contour with a parametric curve. The divided curved surface patch is registered in the patch list, the both ends of the surface of the curved surface patch as vertices, and the points necessary for constructing the surface of the curved surface patch as control points.

  The vertex tracking unit 13 includes a data list processing unit 131, a vertex tracking determination unit 132, and a time length determination unit 133. The vertex tracking unit 13 reads frame images from the moving image data stored in the storage unit 18 in time series order, and A feature point such as a corner of the closed region is detected as a corresponding point corresponding to the vertex, and the detected corresponding point is associated with the corresponding vertex and registered in the corresponding point list. The data list processing unit 131 detects the corresponding point from the frame image, and the vertex tracking determination unit 132 determines whether the corresponding point is found and sets a presence flag indicating the presence of the corresponding point. If not found, the corresponding point is determined by referring to the previous frame, or it is determined that the corresponding point has disappeared. When the corresponding point disappears, the time length determination unit 133 sets the duration of the curved surface patch having the corresponding point and deletes it from the patch list. Surface patches deleted from the patch list are not processed in subsequent frames.

  The control point tracking unit 14 includes a linear combination approximation unit 141 and a control point evaluation correction unit 142. The vertex (corresponding point) in the previous frame, the positional relationship between the control points, and the frame in which the vertex tracking unit 13 is currently processed. From the positional relationship of the corresponding points detected from (1), the positions of the control points necessary for constituting the sides of each patch are obtained and registered in the control point list. The primary coupling approximation unit 141 obtains a coupling coefficient by expressing the positional relationship between the vertex and the control point in the previous frame as a linear coupling, and estimates the position of the control point from the corresponding point detected by the vertex tracking unit 13 and the obtained coupling coefficient. To do. The control point evaluation correction unit 142 corrects the position of the control point based on the distance between the side of the curved surface patch constituted by the control points estimated by the linear combination approximation unit 141 and the point constituting the edge extracted from the frame image. .

  The parametric three-dimensionalization unit 15 includes a curve approximation unit 151 and an error evaluation division unit 152. The vertex tracking unit 13 and the control point tracking unit 14 track the vertices and control points obtained in the time axis direction so that the moving image is parametric solid. It expresses with. The curve approximating unit 151 approximates the trajectory of the vertex and the control point in the time axis direction with a Bezier curve. The error evaluation dividing unit 152 evaluates the maximum distance between the curve approximated by the curve approximating unit 151, the apex to be approximated, and the corresponding point sequence of control points, and if the maximum distance is larger than a predetermined threshold, the approximated curve is calculated. To divide. The parametric solid in the present embodiment refers to a solid composed of a polygon whose bottom is a parametric curved patch and whose side is a parametric curved patch.

  The color value acquisition unit 16 acquires the color values of the vertices and control points of the generated parametric solid from the frame image of the moving image data. Control points for which color values cannot be directly acquired from the frame image are determined by interpolating the color values that can be acquired.

  The output unit 17 outputs the data of the vertex values, the spatio-temporal coordinate values of the control points, and the color values (x, y, t, c) at the coordinate values for all the parametric solids.

  Next, a processing flow of the moving image processing apparatus in the present embodiment will be described.

  2 to 4 are flowcharts showing the flow of processing of the moving image processing apparatus in the present embodiment. The still image of the start frame image is vectorized by the process shown in FIG. 2, the trajectory in the time axis direction of the data vectorized by the process shown in FIG. 3 is traced, and the motion of the data vectorized by the process shown in FIG. It is represented by a solid. In the following, description will be made in order from the processing shown in FIG.

  First, the vectorization process of the still image shown in FIG. 2 will be described.

  The moving image reading unit 11 reads a moving image file and accumulates moving image data in the accumulating unit 18 (step S101). At this time, designation of a time range (start frame and end frame) to be vectorized may be received from the user (step S102).

  Subsequently, the still image vectorization unit 12 reads the start frame image of the moving image data from the storage unit 18 and performs vectorization of the start frame image (step S103). The start frame image is an image in the bitmap format, and the vectorization of the bitmap format image can use the methods described in Non-Patent Documents 1, 2, and 3 or other methods. When the start frame image is vectorized, designation of a subject to be vectorized may be received from the user (step S104).

  The still image vectorization unit 12 divides the closed region in the start frame image by triangular or quadrangular curved patches, and the outline of the subject object is well approximated by the sides of the curved patches. When the image is approximated by a rectangular Bezier patch by vectorization, four vertices and 12 control points are generated for each patch. When an image is approximated by a triangular Bezier patch, three vertices and seven control points are generated for each patch. In the present embodiment, the variable corresponding to the “height” of the curved surface is replaced with “color value”. In the case of a color image, for example, in the case of an RGB display system, a curved patch may be generated for each color of R (red), G (green), and B (blue). Although the number of color components and the display system are not mentioned here, a color image or a gray image can be used depending on the application situation.

  Each list of generated vertexes and control points of each curved surface patch is stored in the storage unit 18 (step S105). At this time, it is stored which curved surface patch each vertex belongs to. For example, if an identifier is assigned to each surface patch and vertex, and the surface patch is specified, the identifier of the vertex constituting the surface patch can be acquired, or if a vertex is specified, the surface patch including the vertex (there may not be one) ) Identifier can be acquired. Similarly, vertices and control points are also associated.

  Next, processing for tracking the trajectory in the time axis direction of the vectorized data shown in FIG. 3 will be described.

  The data list processing unit 131 reads the next frame image of the moving image file from the storage unit 18 and detects corresponding points corresponding to the vertices of the curved patches in the read frame image (step S201). Also, the frame number or duration of the read frame image is assigned to each curved patch. The edge of the curved patch approximates the edge of the closed area of the frame image, and the vertices of the curved patch are the ends of the edge (curve) of the curved patch. It becomes a feature point and is easy to detect. Therefore, the data list processing unit 131 detects the corner of the closed region of the read frame image as the corresponding point of the vertex.

  FIG. 5 is a diagram illustrating a state in which vertices are tracked. It is assumed that the frame changes from T1 → T2 → T3 → T4. In the figure, the sides of the curved patch are also shown for easy understanding. In the triangular patch composed of the vertices P1, P2, and P3 in the figure, corresponding points of the vertices P1, P2, and P3 are found in the frame T2, and no corresponding point of the vertex P2 or the vertex P3 is found in the frame T3. In the frame T4, corresponding points of the vertices P1, P2, and P3 are found again.

  After reading out the frame image and detecting corresponding points, the following vertex tracking processing is performed for each curved surface patch.

  The vertex tracking determination unit 132 selects one vertex from the curved surface patch, and determines whether there is a corresponding point of the vertex (step S202).

  If a corresponding point exists, a coordinate value is added to the corresponding point list as a corresponding point for the vertex, and a presence flag indicating the presence of the corresponding point corresponding to the vertex is turned on (step S203).

  When the process for determining the existence of corresponding points for all the vertices of the curved surface patch is completed, the process proceeds to the process for the next curved surface patch, and when there is an unprocessed vertex in the curved surface patch, the process returns to step S202 ( Step S204). Since adjacent curved surface patches share vertices, it is assumed that vertices whose corresponding points already exist have been processed.

  On the other hand, if the corresponding point does not exist, it is determined whether or not the corresponding point of the vertex exists in the previous frame (step S205). By evaluating the presence flag corresponding to the vertex, it can be determined whether or not the corresponding point exists.

  If there is a corresponding point in the previous frame, the area of the curved patch including the vertex in the previous frame is calculated (step S206), and it is determined whether the calculated area is larger than a predetermined threshold (step S207). In the present embodiment, the area of a triangle connecting the vertices P1, P2, and P3 indicated by dotted lines in FIG. 5 is calculated as the area of the curved surface patch.

  If the calculated area is larger than the predetermined threshold, the coordinate value of the corresponding point in the previous frame is added to the corresponding point list as the corresponding point of the current processing target frame as the corresponding point corresponding to the vertex (step S208).

  If the calculated area is less than or equal to the predetermined threshold, it is determined that the curved patch has disappeared in the current frame. The lost surface patch is deleted from the patch list, and the time length determination unit 133 assumes that the lost surface patch lasts from the start frame to the previous frame, and assigns the number of frames or the frame number as the data of the surface patch. In subsequent frames, the process for the curved surface patch is not performed (step S209).

  On the other hand, in step S205, it is determined that the corresponding point does not exist in the previous frame, that is, the curved surface patch including the vertex for which it is determined that the corresponding point does not exist continuously for two frames has also disappeared. The time length determination unit 133 assumes that the disappeared curved patch lasts up to the previous two frames, and assigns the number of frames or the frame number as data of the curved patch. In the present embodiment, it is determined that the curved patch has disappeared when the corresponding point is not found in succession for two frames, but the number of frames is not limited to two.

  In the example shown in FIG. 5, the corresponding point of the vertices P1 and P2 is found in the frame T3, but the corresponding point of the vertex P3 is not found. Therefore, the area of the triangle P1 / P2 / P3 in the frame T2 is calculated, It is determined whether or not to continue the tracking process of the vertex P3. If the calculated area is larger than the predetermined threshold, the coordinate value of the corresponding point of the vertex P3 of the frame T2 is added to the corresponding point list as the corresponding point (P3 ') of the vertex P3 of the frame T3. In the example shown in the figure, since the corresponding point of the vertex P3 is found in the frame T4, the corresponding point of the vertex P3 of the frame T3 is regarded as P3 ', and the tracking processing of the vertex P3 is performed after the frame T4. If no corresponding point of the vertex P3 is found in the frame T4, it is determined that the curved surface patch composed of the vertices P1, P2, and P3 has disappeared in the frame T3.

  The processes from step S202 to step S209 are performed for all curved surface patches. If there is a curved patch that has not been processed yet, the process returns to step S202 (step S210), and processing is performed for each vertex of the curved patch.

  When the vertex tracking process for all the curved surface patches is completed, the control point tracking unit 14 performs the control point tracking process for generating the curve of the side of each curved surface patch. In a cubic Bezier curve, two end points and two control points are required to generate one curve segment. The two end points correspond to the two vertices of the curved patch. Further, since a curve generally generated in a cubic Bezier curve does not pass through a control point, the control point is not always a feature point such as an edge or a corner of a closed region in the frame image.

  There is a method of extracting the edge of the subject object for each frame and determining the control point so as to generate a Bezier curve that approximates the edge. In this embodiment, the vertex that forms the edge of the curved patch between the frames. Assuming that the mutual positional relationship between the (end point) and the control point does not change greatly, the control point is estimated based on the positional relationship between each vertex and each control point in the previous frame and the positional relationship between each vertex in the current frame.

First, the linear combination approximation unit 141 represents a position vector of a control point for a certain vertex by a linear combination of position vectors of two vertices constituting a side of a curved patch for a certain vertex (step S211). Specifically, as shown in the following equation (7), the control points C1 and C2 of the curve connecting the vertices P1 and P2 in the previous frame are the vertices for another vertex P3 of the same curved surface patch that does not form the curve. It is represented by position vectors P1 and P2.

As shown in FIG. 6, it is assumed that the positional relationship between the vertex and the control point in the previous frame does not change greatly even in the current frame, and the linear coupling coefficients a, b, c, and d in Equation (7) are stored. As shown in the following equation (8), the control points C1 ′ and C2 ′ in the current frame are positioned at the vertices P1 ′ and P2 ′ with respect to the vertex P3 ′ using the primary coupling coefficients a, b, c, and d. Represented as a vector.

  When the patch shape is a quadrangle, there are two vertices in addition to the vertices constituting the side, and either one may be determined as the starting point of the position vector.

  Subsequently, the control point evaluation correction unit 142 evaluates the difference between the Bezier curve generated at the vertices P1 ′ and P2 ′ and the control points C1 ′ and C2 ′ and the edge of the closed region in the currently processed frame image. Then, the positions of the control points C1 ′ and C2 ′ are corrected (step S212). If the difference between the pixels constituting the edge and the Bezier curve is calculated, the amount of calculation increases. Therefore, in this embodiment, the evaluation is performed based on the distance between the two points on the edge and the Bezier curve.

  Specifically, as shown in FIG. 7, a point on the Bezier curve where the distance between the control point C1 ′ and the Bezier curve is minimum is B1, and an intersection of the straight line C1′B1 and the edge is E1, and points B1, If the distance between E1 is smaller than a predetermined threshold, the position of the control point C1 ′ is not corrected, the distance between the points B1 and E1 is equal to or larger than the predetermined threshold, and the distance between the points C1 ′ and B1 ≦ point C1 ′, If the distance is between E1, the control point C1 ′ is moved a predetermined distance from the point C1 ′ to the point B1 to be a new control point C1 ′, and the distance between the points C1 ′ and B1 ≧ points C1 ′ and E1 If the distance is between, the control point C1 ′ is moved a predetermined distance in the direction from the point B1 to the point C1 ′ to be a new control point C1 ′. The distance for moving the control point C1 'is, for example, a value obtained by multiplying the distance between the points C1'B1 by an appropriate real number (for example, 0.5). The same processing is performed for the control point C2 ′, and the distance between the points B1 and E1 and the distance between the points B2 and E2 is determined for the Bezier curves generated at the vertices P1 ′ and P2 ′ and the new control points C1 ′ and C2 ′. The above correction process is repeated until it becomes smaller than the predetermined threshold.

  For example, Philip J. Schneider, “Solving the Nearest-point-on-curve Problem”, Graphics Gems, calculates the shortest distance between an arbitrary point and a Bezier curve. , Morgan Kaufmann Publishers (reference 1) and Philip J. Schneider, “An Algorithm for Automatically Fitting Digitized Curves”, Graphics Gems, Morgan Kaufmann Publishers (reference 2).

  The control point tracking process described above is performed for all control points (step S213). In the case of a triangular Bezier patch, there is one control point inside the triangle, and in the case of a square Bezier patch, there are four control points inside. These control points may be evaluated only by storing the primary coupling coefficient in step S211.

  When the tracking process for all the control points is completed, the process returns to step S201 to read the next frame image to obtain the vertex and control point of the frame image, and when the process is completed for all frames, or all the curved patches are lost. If so, the process proceeds to a process of expressing the motion of the vectorized data as a parametric solid (step S214).

  Next, processing for expressing the motion of vectorized data shown in FIG. 4 as a parametric solid will be described. In the process shown in FIG. 4, the trajectory of the vertex and the control point in the time axis direction is approximated by a Bezier curve, and a parametric solid is generated using the control points generated as a result.

  For each curved surface patch, the curve approximating unit 151 approximates all vertices and control point trajectories in the time axis direction of the curved surface patch with a Bezier curve (step S301). For example, the method described in Reference 2 is approximated by a cubic piecewise Bezier curve.

  FIG. 8 shows an example of a Bezier hexahedron obtained by tracking a rectangular Bezier patch in the time axis direction. FIG. 8A is a diagram illustrating the trajectory in the time axis direction of the vertex tracked by the vertex tracking unit 13, and FIG. 8B constitutes a Bezier curve that approximates the trajectory of FIG. 8A. It is the figure which illustrated the control point. FIG. 8C shows a definition area of each parameter in the Bezier solid shown in FIG. Similarly, FIG. 9 shows an example of a Besier triangular prism obtained by tracking a triangular Bezier patch in the time axis direction.

Ends (vertices, control points) in the start frame and end frame of the Bezier curve approximating the vertices and control point trajectories have the same t in the (x, y, t) space. Of the vertices and control points of each curved surface patch constituting the Bezier curve approximating the trajectory, those having the same time-axis subscript, for example, 16 points P _ij1 in FIG. 8B are (x, y, t) It is good to have the same t in space. Similarly, it is _preferable that 16 points that are P _ij2 have the same t. In addition, 10 points that become P _{ijk, 1} and 10 points that become P _{ijk, 2} in FIG. 8B may have the same t in the (x, y, t) space. Thus, in the parametric solids shown in FIGS. 8 and 9, when one value of w of the parameter (u, v, w) is determined, t is uniquely determined, and the subsequent calculation processing becomes easy.

  Subsequently, the error evaluation division unit 152 evaluates the maximum distance between the curve that approximates the trajectory of the vertex and the control point in the time axis direction and the corresponding vertex sequence of the approximation target vertex and the control point (step S302). Then, the curve having the largest error is selected from the curves that approximate the trajectory in the time axis direction of the vertex and the control point constituting one curved surface patch, and the error is compared with a predetermined threshold (step S303).

  When the maximum error is larger than the predetermined threshold, the error evaluation dividing unit 152 divides the curve having the largest error (step S304). A method of dividing a curve is described in Reference 2 for a curve on a two-dimensional plane. A three-dimensional curve can be realized by essentially the same processing. FIG. 10 shows an example of dividing a Bezier curve on a two-dimensional plane. For example, if the error increases even if the tracking point sequence from the vertex P1 to the vertex P2 indicated by the dotted line in FIG. 10 is approximated by the Bezier curve indicated by the solid line in FIG. 10, the distance between the tracking point sequence and the Bezier curve becomes the largest. Point P12 is selected as a dividing point, and a new Bezier curve is approximated before and after the dividing point. As shown in FIG. 10, when a Bezier curve is divided, a dividing point on the Bezier curve becomes a new vertex P12, and two control points C11, C12, and vertices P12, P2 of the Bezier curve connecting the vertices P1, P12 are connected. Two control points C21 and C22 of the Bezier curve are newly added.

  Further, a curve approximating the trajectory in the time axis direction of other vertices and control points constituting the same curved surface patch is also divided (step S305), and a curved surface patch corresponding to a new solid generated by the division is added to the patch list. (Step S306). When dividing another curve, a time variable T corresponding to the parameter value W used for dividing the curve is obtained in step S304, and the curve is divided by the parameter value corresponding to the time variable T. This division processing is equivalent to cutting the solid shown in FIGS. 7A and 8A along a plane perpendicular to the time axis. In some cases, vertices and control points are shared by adjacent patches, but division may be performed in one patch and division may not be performed in the other patch.

  On the other hand, if the maximum error is equal to or smaller than the predetermined threshold, the color value acquisition unit 16 determines the color values of the vertices and control points of the Bezier solid generated by tracking the patch in the time axis direction (step S307). Specifically, the color values at the start and end frames of the patch are obtained by obtaining the color values at the coordinate values of the vertices and control points of the patch from the frame. When the control point is inside the patch, the color value of the corresponding coordinate value is used. When the control point is outside the patch, an appropriate initial value, for example, the point on the side of the patch closest to the control point is used. Gives a color value. Alternatively, the value corresponding to each pixel position of P (u, v, w) when the time variable t is fixed to the initial value or the end value in Expressions (5) and (6) is calculated, and the maximum error of the color value is calculated. The color value may be obtained using a non-linear least square method, for example, the Levenberg-Marquardt method, while changing the color value so that becomes smaller than a predetermined value. At this time, in order to facilitate the calculation, the side of the patch that becomes the boundary of the object may be expressed by a curve, and the inside of the object may be expressed by a triangle whose side becomes a line segment. For example, in the start frame and the end frame, in FIG. 11A, approximation is performed using a patch in which all sides are composed of curves, and in FIG. 11B, only the contour of an object is represented by a curve, Approximation is performed using a patch that is a line segment, and in FIG. 11C, approximation is performed using a patch in which all sides including the outline of the object are line segments. In a triangle whose side is composed of a line segment, the control point is on the side. Therefore, the point that divides the side into three is regarded as the control point, and the color value at the coordinate value is used. Even when a patch whose sides are curved as shown in FIG. 11A is used, the color value at the point where the line segment connecting the vertices is divided into three as shown in FIG. 11C may be used. . Alternatively, it may be obtained by iterative calculation using the nonlinear least square method with the color value of the vertex as the initial value.

  For the color values of control points other than the start frame and end frame of the patch, and the vertex and control point color values when divided, the color values in the corresponding vertex and control point start frame may be used. The interpolation may be performed as shown in FIG. In the example shown in FIG. 12, the frames before and after the time variable T at the time of division are Tf and Tb (Tf <Tb), the color value at Tf is Cf, the color value at Tb is Cb, The color value C (T) is interpolated by the following equation (9).

C (T) = ((T−Tb) Cf + (Tf−T) Cb) / Tb−Tf (9)
The processing from step S301 to step S307 is performed for all patches. If there is a patch that has not yet been processed, the process returns to step S301 (step S308), and the movement of the patch is represented by a parametric solid.

  For all patches, the spatio-temporal coordinate values of the vertices and control points and the color values (x, y, t, c) at the coordinate values are determined, and the data is output as a file (S309). The data to be output includes, for each patch, a patch ID, a patch start time, a patch end time, and in the case of a Bezier triangular prism, 40 vertexes and x coordinate values, y coordinate values, t coordinate values, and color values of control points. Have. When a Bezier hexahedron is used, data of 64 vertices and control points are obtained.

  Next, a procedure for generating a moving image from the output vector data will be described.

  Coordinates (x, y) are specified, and when the vector data is a Bezier hexahedron, the space-time coordinate value (x, y, t) is obtained using equation (5), and when the vector data is a Bezier triangle, equation (6). The parameter (u, v, w) is obtained using Newton-Raphson, substituted into the equation (5) or (6) using the parameter (u, v, w), and the color in (x, y, t) Calculate the value. If the coordinate value (x, y) is interpreted as the center position of the pixel in the digital image, and t is the frame position in the digital image, a digital moving image can be generated from the vector data.

  As described above, according to the present embodiment, a closed region having a similar color value in the start frame image of moving image data is divided into rectangular curved surface patches, and vertexes and control points of the curved surface patches are converted into moving images. Track data in time series with reference to the frame of data, approximate the time-axis trajectory of the vertices and control points with a parametric curve, and obtain the color values corresponding to the vertices and control points of the parametric curve from the frame image of the moving image data By acquiring and outputting the color values corresponding to the spatio-temporal coordinate values of the vertices and control points of the parametric curve, it becomes possible to express moving image data as vector data, and it can be scaled with a smaller amount of data. However, it can be expressed without losing image quality. It is also possible to edit or modify the contents at the object or patch level.

  In the present embodiment, description has been made with a Bezier curve, a Bezier curved surface, and a Bezier solid in mind for parametric curves, curved surfaces, and solids, but any parametric curve / curved surface / solid may be used in essence. For example, it can be applied to a rational Bezier solid or a B-spline solid. In the conventional method, the order of the parametric curved surface patch is often expressed as a third order. In this case, as long as it is essentially second order or higher, any number of orders may be used, but those of order 3 are often used for ease of processing.

DESCRIPTION OF SYMBOLS 11 ... Moving image reading part 12 ... Still image vectorization part 13 ... Vertex tracking part 131 ... Data list processing part 132 ... Vertex tracking determination part 133 ... Time length determination part 14 ... Control point tracking part 141 ... Primary joint approximation part 142 ... Control point evaluation correction unit 15... Parametric three-dimensionalization unit 151... Curve approximation unit 152 .. error evaluation division unit 16... Color value acquisition unit 17.

Claims (7)

Reading means for reading moving image data and accumulating in the accumulating means;
Reads the specified frame image of the moving image data from the storage means, divides a closed region having a similar color value in the frame image into curved surface patches, and registers the edge of the edge of the curved surface patch as a vertex in the vertex list And vectorization means for registering the points for constituting the side as control points in the control point list,
Frame images are read from the moving image data in chronological order, and for each curved surface patch, corresponding points corresponding to vertices registered in the vertex list are detected from the read frame images, and the vertices are listed as vertices in the frame image. Vertex tracking means to register,
For each frame image read out in chronological order, the position of the control point for constituting the side of the curved surface patch when the vertex registered in the vertex list is the end of the side of the curved surface patch for each curved surface patch A control point tracking means for estimating the position based on the positional relationship between the vertices and registering it in the control point list as a control point in the frame image;
Curve approximation means for referring to the vertex list and the control point list and approximating the trajectory in the time axis direction of each vertex and control point registered by the vectorization means with a parametric curve;
Color value acquisition means for acquiring color values corresponding to vertices and control points of a parametric curve approximated by the curve approximation means from a frame image of the moving image data;
Output means for outputting color values corresponding to the spatio-temporal coordinate values of the vertices and control points of the parametric curve;
A moving image processing apparatus comprising: The control point tracking means includes
The positional relationship between the vertex and the control point in the previous frame is expressed by a linear combination of the vertex of the edge of the curved surface patch constituted by the control point and another vertex to obtain a coupling coefficient, and the position of the control point is determined. A linear combination approximation means for estimating the position of the control point from the vertex and the coupling coefficient in a frame to be obtained;
Correction means for correcting the position of the control point based on the distance between the side of the curved patch constituted by the estimated control point and the point constituting the edge extracted from the frame image;
The moving image processing apparatus according to claim 1, further comprising: The vertex tracking means includes
When the corresponding point cannot be detected, the area of the figure configured by connecting the vertices of the curved surface patch in the previous frame including the corresponding point is calculated, and if the area is smaller than a predetermined threshold, there is no corresponding point And when the area is equal to or larger than a predetermined threshold, a determination unit that sets the position of the corresponding point in the previous frame as the position of the corresponding point in the current frame;
When the determination unit determines that there is no corresponding point over a predetermined number of frames, an erasure unit that does not process the subsequent frame for the curved surface patch including the corresponding point;
The moving image processing apparatus according to claim 1, further comprising: Reading and storing moving image data in the storage means by the reading means;
The vectorization means reads out the specified frame image of the moving image data from the storage means, divides a closed region having a similar color value in the frame image into curved patches, and appoints the edge of the edge of the curved patch as a vertex Registering it in the vertex list as a control point and registering the points for constituting the side as control points in the vertex list;
A frame image is read from the moving image data in time series by the vertex tracking means, and a corresponding point corresponding to the vertex registered in the vertex list is detected from the read frame image for each curved surface patch. Registering them in the vertex list as vertices;
For each frame image read out in chronological order by the control point tracking means, for each curved surface patch, the edge of the curved surface patch is formed when the vertex registered in the vertex list is the end of the curved surface patch side. Estimating the position of the control point for the position based on the positional relationship of the vertex and registering it in the control point list as the control point in the frame image;
Referring to the vertex list and the control point list by a curve approximation means, approximating the trajectory in the time axis direction of each vertex and control point registered by the vectorization means with a parametric curve;
Obtaining color values corresponding to vertices and control points of the parametric curve approximated by the curve approximation means by the color value acquisition means from the frame image of the moving image data;
Outputting color values corresponding to spatio-temporal coordinate values of vertices and control points of the parametric curve by the output means;
A moving image processing method comprising: Registering in the control point list by the control point tracking means;
By representing the positional relationship between the vertices and the control points in the previous frame by a linear combination approximation means by using a linear combination of both vertices and another vertex of the surface of the curved surface patch constituted by the control points, a coupling coefficient is obtained, Estimating the position of the control point from the vertex in the frame for determining the position of the control point and the coupling coefficient;
Correcting the position of the control point based on the distance between the side of the curved patch constituted by the estimated control point and the point constituting the edge extracted from the frame image by the correction means;
5. The moving image processing method according to claim 4, further comprising: The step of registering in the vertex list by the vertex tracking means includes:
When the corresponding point cannot be detected by the determination unit, the area of the figure formed by connecting the vertices of the curved surface patch in the previous frame including the corresponding point is calculated, and the area is smaller than a predetermined threshold Determining that there is no corresponding point, and if the area is equal to or greater than a predetermined threshold, setting the position of the corresponding point in the previous frame as the position of the corresponding point in the current frame;
When the determination unit determines that there is no corresponding point over a predetermined number of frames by an erasure unit, the step of not performing processing on subsequent frames for the curved surface patch including the corresponding point;
The moving image processing method according to claim 4, wherein:   A moving image processing program for causing a computer to function as the moving image processing apparatus according to claim 1.