JPH08149458A - Moving image processor - Google Patents

Abstract

PURPOSE: To obtain a moving image processor coding a moving image with high compressiblity and decoding and composing the coded moving image. CONSTITUTION: Each frame of a series of moving images is analyzed in an image analysis part 11 and a feature point is extracted. A segment is extracted from the first frame of the images by an initialization part 12, depth information is added to the segment and the segment is stored as a three-dimensional model in a storage part 13. In a motion estimation/correspondence search part 15, the corresponding of the motions between the frames of each three-dimensional model and each feature point is performed. The deviation of projected image that each three-dimensional model is moved by the motion estimated value and each feature point of the model and an actual image is determined. Based on the deviation, the three-dimensional shape information on each three- dimensional model is updated by a Kalman filter 21. As for the first frame of the images, the information on each segment and the analysis value of each feature point are coded. As for the second frame or the subsequent frames, the motions between the frames of the three-dimensional model of each segment and the deviation of each feature point are coded.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention detects a movement between frames of feature points obtained by analyzing an image, and codes the movement efficiently to code a continuous moving image. The present invention relates to a moving image processing device and a moving image processing device that decodes the encoded moving image sequence.

[0002]

2. Description of the Related Art If an image is analyzed to detect an edge, and an edge position, an analysis value of each point constituting the edge and a DC component are extracted, the original image is synthesized without using any other image. it can. The amount of information at this time can be made smaller than the amount of information for all the pixels of the original image, and the information can be compressed. In the case of a moving image, the first frame is compressed by this method, and in the second and subsequent frames, it is tracked to which position each point constituting the edge of the first frame has moved, and the first frame is obtained. By transferring the analysis value of each point to each point after tracking, the image can be synthesized without obtaining the analysis value for each frame. Therefore, the position and analysis value of each point constituting the edge of the first frame, the movement amount of each point between frames, and the DC for each frame.
Moving images can be compressed by using the components as information.

As a method of encoding a moving image at a higher compression rate, a method of encoding a moving image using a three-dimensional model of each object forming the moving image has been proposed. 3 for each object
If the two-dimensional shape and its movement are known, a moving image that is exactly the same as the original moving image can be generated. As a method for extracting a three-dimensional shape model from a moving image sequence and coding a continuous moving image using the model, Hans George Mu
OB by smann, Michael Hotter, Jorn Ostermann and others
JECT-ORIENTED analysis coding of moving images. "
(Signal processing: Image Communication 1 (1989): 11
7-138, Elsvier SCIENCE PUBLISHERS BV]. According to this method, the motion vector is calculated for the edge part and the depth is calculated using it.
For a portion other than the edge, the three-dimensional shape is estimated by interpolating the depth. In addition, the luminance information is mapped as an attribute in three dimensions and is transmitted together with the interpolated three-dimensional shape data.

[0004]

However, in the above-described method of encoding the movement amount of each point forming the edge, the distribution of the two-dimensional movement amount of each point forming the edge of each frame is wide. Therefore, the encoding efficiency cannot be improved and a sufficient compression rate cannot be obtained.

Further, in the method of encoding a moving image using the above-mentioned three-dimensional model, it is difficult to obtain a motion vector even in an edge portion where the brightness changes abruptly, and it is very difficult to estimate an accurate depth from the motion vector. It's difficult.
Therefore, even if the three-dimensional shape is estimated by interpolating the depth of the edge portion, an accurate three-dimensional shape cannot be obtained.
Therefore, since there is a deviation from the actual shape, extra information increases and the compression rate cannot be increased. Further, there is a problem that the amount of initial information is large because the luminance information mapped as an attribute in three dimensions is transmitted as the initial information.

Therefore, an object of the present invention is to provide a moving image processing apparatus capable of encoding a continuous moving image at a higher compression rate, and a moving image processing apparatus for decoding the encoded moving image. Especially.

[0007]

In order to solve the above-mentioned problems, the edges obtained by analyzing a moving image are divided into categories that can be represented by one motion, the motion is estimated for each category, and the motion estimation value is calculated. The difference between the position of the feature point of the edge belonging to each category moved based on the above and the actual position of the feature point is calculated for each feature point, and the motion estimation value for each category and the feature point The position difference is used as information on the movement of each feature point between frames. Further, the three-dimensional position information of the category is acquired from the motion of the edge belonging to each category between the frames, and the motion is estimated using the three-dimensional position information.

Therefore, the moving image processing apparatus for encoding a continuous moving image according to the present invention includes an image analyzing means for extracting a feature point from each frame of the continuous moving image and a first moving image of the continuous moving image. Segmentation means for extracting a segment from the frame, shape storage means for storing three-dimensional model shape information for each segment extracted by the segmentation means based on the feature points extracted by the image analysis means, The motion between the frames of each three-dimensional model stored in the shape storage means is estimated, the feature points of the three-dimensional model of each frame are moved and projected based on the estimated motion, and the feature points of the frame. Motion estimation correspondence search means for associating the feature points of the next frame with each other, and difference calculating means for obtaining the position difference between the associated feature points. , Updating means for updating each three-dimensional model shape information stored in the shape storing means on the basis of the difference in position between the respective feature points, and the continuous moving image extracted by the segmentation means. Each segment of the first frame and an analysis value of each feature point of each segment analyzed by the image analysis unit are encoded as initial data by an initial encoding unit and the motion estimation correspondence search unit for each frame. And an encoding unit that encodes the motion estimation value between the frames of the three-dimensional model of each segment obtained above and the position difference between the feature points calculated by the difference calculating unit.

Further, the moving image processing apparatus for decoding a coded continuous moving image according to the present invention uses the encoded initial data for each segment constituting the first frame of the continuous moving image. A three-dimensional model shape for each segment based on the characteristic points and the initial decoding means for decoding the analysis value of each characteristic point, and the characteristic points for each segment decoded by the initial decoding means. A shape storage unit that stores the information, and a motion estimation value between frames of the three-dimensional model of each segment based on the encoded data of each frame, and a projection obtained by moving the three-dimensional model based on the motion estimation value. Decoding means for decoding the difference between the position of the feature point of the image and the position of the corresponding feature point of the next frame of the frame, and the shape storage means based on the difference between the positions of the corresponding feature points. 3 of each segment being 憶
It has an updating means for updating the shape information of the dimensional model, and an image synthesizing means for synthesizing the image of each frame based on the three-dimensional model shape information of each segment stored in the shape storing means.

Preferably, the shape storage means gives predetermined depth information to the respective feature points and stores three-dimensional model shape information in which each segment is a flat plate-shaped three-dimensional model as initial data. Also preferably, the updating means is
Observing the position of the characteristic point of the projected image obtained by moving the three-dimensional model based on the motion estimation value between the frames of the three-dimensional model of each segment, and the actual position of the corresponding characteristic point of the next frame of the frame. When the amount and the error between the corresponding feature points are noise, the Kalman filter obtains the least squares estimated value of the state amount to update the state amount.

[0011]

In the moving picture coding apparatus for coding continuous moving pictures of the present invention, predetermined depth direction information is added to the two-dimensional shape information of each segment of the first frame of the moving picture sequence. Using the initial value of the three-dimensional model as an estimate, the motion of the three-dimensional model is estimated, and the position of each feature point of the image obtained based on the estimation and the image of each frame to be input is compared to determine the entire three-dimensional model. Motion and the deviation of each feature point are extracted, and based on the overall movement (motion estimation value) of the three-dimensional model and the deviation of each feature point, the 3
While updating the dimensional model, it is used as information on the movement of the feature points in each frame. Therefore, the distribution of information regarding the movement of each feature point is localized, and the coding efficiency is increased.

Further, in the moving image processing apparatus for decoding a continuous encoded moving image according to the present invention, predetermined depth direction information is added to the encoded two-dimensional shape information of each segment of the first frame. Are stored as three-dimensional model shape information, and the three-dimensional model is sequentially calculated based on the movement of the encoded three-dimensional model for each frame and the deviation of each feature point forming the three-dimensional model. The image of each frame is updated, and the image of each frame is synthesized based on the updated information of the three-dimensional model. Therefore, it is possible to faithfully decode the original moving image from the information on the movement of each segment and the information on the shift of each feature point encoded at a high compression rate, and a moving image encoding the continuous moving image can be obtained. It can cooperate with an image processing device.

[0013]

DESCRIPTION OF THE PREFERRED EMBODIMENTS A moving picture coding apparatus according to an embodiment of the present invention will be described with reference to FIGS. FIG. 1 is a block diagram showing the configuration of a moving picture coding apparatus 10 according to an embodiment of the present invention. The moving image coding apparatus 10 includes an image analysis unit 11,
Initial setting unit 12, storage unit 13, motion estimation / correspondence search unit 1
5, a motion processing unit 16, a projection unit 17, a quantization unit 19, a Kalman filter 21, a correction processing unit 22, and an encoding unit 23. Moving picture coding apparatus 10 of the present embodiment
Is input with continuous image data detected by a continuous sequence detection unit (not shown) from a moving image sequence from a VTR or a video, and compresses each image data sequence,
It is a compression transmission device for sending to the transmission line 50. In addition, the moving picture coding apparatus 10 is a moving picture decoding apparatus 3 described later.
An image processing system is constructed in cooperation with 0.

The operation of each unit will be described below. The image analysis unit 11 analyzes the sequentially input image data of each frame, extracts the characteristic points, and obtains the positions of the characteristic points and the analysis values. In this embodiment, for input image data,
Image data is analyzed with multiple filters with different resolution scales, points that make up edges are detected as feature points, input image data is converted to image data of edges that are feature image data, and the edges are made up. The position of each point and the analysis value are extracted.

Further, the image analysis unit 11 extracts the DC component necessary for synthesizing images from the input image data of the first frame of each continuous sequence. The position and analysis value of the feature point obtained for the image data of the first frame are stored in the initial setting unit 12 and the encoding unit 23, and the DC component is stored in the encoding unit 2.
3 is output. Further, the position of the feature point and the analysis value for the image data of the second and subsequent frames are output to the motion estimation / correspondence search unit 15 and the correction processing unit 22.

The initial setting section 12 performs segmentation based on the positions and analysis values of the characteristic points of the characteristic image data of the first frame input from the image analysis section 11 and extracts the segments forming this image data. The information of the characteristic points of each segment is stored in the storage unit 13. In this segmentation, a total of nine types of features of red, green, blue, lightness, hue, and saturation signals, and Y signals, I signals, and Q signals corresponding to television signals are extracted from the color image, and related to the features. It is performed by recursive thresholding which performs segmentation based on the histogram.

The storage unit 13 is a storage means for storing the position information X, Y, Z, the analysis value g, the probability covariance matrix v, and the additional information a for each feature point of each segment, and is composed of a memory. It The information stored in the storage unit 13 is such that the input continuous sequence has S segments, each segment is composed of Us (s = 1 to S) edges, and each edge is Nsu ( u = 1 to Us, s =
1 to S), it is expressed by Equation 1.

[0018]

[Equation 1]

The probability covariance matrix vsun (n = 1 to N)
Since su, u = 1 to Us, and s = 1 to S) are the scattering of the points forming each edge, the same value is given to each feature point forming the same edge.

As the information stored in the storage unit 13, first, the information on the first frame is input from the initial setting unit 12, and initial data is generated. After that, every time the image data of the second and subsequent frames is input, the contents thereof are updated by the Kalman filter 21 and the correction processing unit 22, which will be described later.

The motion estimation / correspondence search unit 15 estimates the amount of movement of the segment from the information {Fsun} of each point of the edge image of the image of the previous frame, the edge position of the current frame and the analysis value, and calculates the amount of movement of the segment. The information {Fsun} of each point is associated with the feature point of the edge image of the current frame. The method will be specifically described below. First, in FIG. 2, it is assumed that the coordinate system XYZ is a camera coordinate system, the origin of the coordinate system is the center of the lens, and the optical axis is aligned with the Z axis that is the depth direction. In such a coordinate system, the point P
It can be considered that the image of is projected on a plane which is parallel to the XY plane and is separated from the origin by the focal length f of the camera. The position of the image of the point P on the projection plane becomes the position of the pixel on the image input from the camera. With respect to the plane of projection, the origin is the point of intersection of the plane with the Z axis, and the X axis and Y
A coordinate system xy parallel to the axis is set.

The coordinates of the point P in the XYZ space are p = (Xp,
Yp, Zp) and the coordinates of the point Q, which is the image of the point P on the xy plane, are q = (xq, yq), the coordinates q of the point Q are expressed by equation 2.

[0023]

[Equation 2]

A segment s (s = 1 to S) is composed of Us (s = 1 to S) edges, and each edge is Nsu (u = 1 to Us, s = 1 to S) points. The position of each of these points is represented by psun = (Xsun, Ysun, Z
sun) (n = 1 to Nsu). This segment is relatively Δωx around the X axis and Δω around the Y axis.
It rotates about the y and Z axes by Δωz, and Δt = (Δtx,
In the case of parallel translation by Δty, Δtz), the amount of movement of each point psun constituting this segment Δpsun = (ΔXsun
, ΔYsun, ΔZsun) is the rotation Δω about each axis.
Assuming that x, Δωy, Δωz, and the parallel movement amount Δt are small, Expression 3 is obtained.

[0025]

(Equation 3)

Let qsun be the projection point of the point psun on the xy plane.
= (Xsun, ysun), the amount of movement of the projection point qsun due to the movement of the segment Δqsun = (Δxsun
, Δysun) is given by equation 4.

[0027]

[Equation 4]

From Equations 2 and 4, Equation 5 is obtained.

[0029]

(Equation 5)

Equations 2 and 5 are given as follows: m = 1 out of N points
When it is applied to M pieces of ~ M, it becomes as shown in Expression 6.

[0031]

(Equation 6)

Note that Δt ^t represents a transposed matrix of the matrix Δt. Regarding Δq, since the point on the image corresponding to the virtual projection point qm ′ by the equation 2 of the three-dimensional position pm obtained before the new image is input is unknown, as shown in FIG. It is assumed that the virtual projection image Ip of the dimensional position information is the closest point to the projection feature point qm of the object image Ir. M ≧
When the value is 3, the estimated value ΔC ′ of the parameter ΔC of the amount of rotation and the amount of translation is obtained by the equation 7 by the least square method.

[0033]

(Equation 7)

The movement amount Δqsun of the three-dimensional position information by ΔC ′ obtained by the equation 7 is calculated by the equation 2 and a virtual projected image is newly created by the equation 3, and similarly similar points are assumed to be corresponding points. If the calculation of Expression 7 is repeated and Expression 8 is repeated, the virtual projected image and the object image Ir come close to each other.

[0035]

(Equation 8)

By repeating this calculation until Σ│Δqsun│ ² becomes equal to or less than the predetermined value ε, the corresponding point qsun of the new image with respect to the three-dimensional position information psun of the original image is obtained.

According to the method of motion estimation / correspondence search as described above, the object is assumed to be a rigid body, and the points constituting the three-dimensional position information are constrained by the six parameters of rotation and translation. , Motion estimation / correspondence search is performed comprehensively, not independently for each point. Therefore, as a whole, a consistent correspondence relationship is obtained for all points, and noise in the three-dimensional position information due to incorrect correspondence is reduced.

The motion processing unit 16 obtains the position information of the information {Fsun} of each segment from the three-dimensional space by the parallel movement amount t and rotation movement amount ω of each segment obtained by the motion estimation / correspondence search unit 15. In parallel and rotational movement.

The projection unit 17 converts the three-dimensional position psun = (Xsun, Ysun, Zsun) of each point of each segment into a position qsun = (xsun, ysun) on the image, and the obtained point qsun on the image. Is given an analytical value gsun. Three-dimensional position p
The conversion from sun to the projection point qsun is performed by Equation 3.

The addition section 18 obtains the difference between the position qsun 'on the image of the information {Fsun} of each feature point after projection and the corresponding edge position qsun of the input image.

The quantizer 19 quantizes the difference obtained by the adder 18 to obtain a fraction. As the quantization method, linear quantization with a certain appropriate quantization step (for example, one pixel width), non-linear quantization with some non-linear quantization steps set, quantization step not fixed, and input is performed. There is quantization in which the quantization step is appropriately changed depending on the characteristics of the image to be read, and an appropriate quantization method may be used according to the required transmission rate and image quality. For example, if a high compression rate is required, increase the quantization step, or if there are many straight lines in the image and the discontinuity of the straight lines due to quantization noise is conspicuous, perform non-linear quantization and reduce the fractionation. Try to finely quantize the part. The calculated fraction is input to the encoding unit 23 and the Kalman filter 21. The fraction input to the Kalman filter 21 is input to the addition unit 20 again in association with the position qsun ′ of each feature point after projection.

The Kalman filter 21 determines the three-dimensional position p of the information {Fsun} of each point of each segment in the previous image.
The three-dimensional position psun is updated from the sun and the edge position qsun corresponding to the input image. The Kalman filter is a filter that can sequentially obtain a least squares estimation value of a state quantity from a time series observation quantity in a system including noise. In this embodiment, the state quantity is a three-dimensional position psun and a two-dimensional position qsun which is an observed quantity. Two-dimensional position qsun
Contains noise due to the quantization of Δqsun. Also,
The motion estimation value also contains noise. 3 on the plane of the initial value
The dimensional shape {psun} is updated by the Kalman filter so as to approach the actual three-dimensional shape each time there is a motion in the segment. The probability covariance matrix vsun at the information {Fsun} of each point is the probability covariance matrix (3 ×
3) is used to update psun, and at the same time the probability covariance matrix vsun is updated.

When the edge is traced, the correction processing unit 22 does not have the edge existing in the stored image of the previous frame in the image of the current frame (disappears), or conversely, the edge of the current frame is lost. The edges present in the image are
If it does not exist (appears) in the stored image of the previous frame, such a lost edge,
Processing for newly appeared edges is performed. In the correction processing unit 22, first, the disappearance of an edge is detected and processed, and the edge that appears next is detected and processed. The processing of the edge that has appeared is to first check whether it can be added to an existing edge, then check whether it is an edge that constitutes a new segment, and whether it is held as an unknown edge. .

Each processing method will be described below. First, the disappeared point is not immediately deleted from the storage unit 13, but a mark of disappearance is added as additional information and stored. When saving, the three-dimensional position psun is moved based on the motion estimation value of the segment to which the lost portion belongs. The probability covariance matrix vsun, the analysis value gsun, and the segment to which they belong are left unchanged.

Regarding appearance, first, a point in the input image that does not correspond to a stored point is extracted and used as an appearance edge. The appearance edge is divided into a set of points, which are connected open curves or closed curves by examining the connectivity. Next, it is checked whether or not there is an edge that has disappeared in the past and has reappeared. The point with the mark of disappearance stored in the storage unit 13 is projected by moving the three-dimensional position according to the movement of the segment that belonged to in the past. Then, if a newly appearing edge exists in the vicinity of the point where the point with the mark of disappearance is projected, that point is restored as a corresponding point except the mark of disappearance. Three-dimensional position psun of revived point, probability covariance matrix vsun
Are updated by the Kalman filter 21. Through such processing, the temporarily disappeared portion is appropriately processed.

After the process of re-appearing the disappeared points, it is further judged whether or not each set of the remaining appearing edges belongs to the corresponding segment. This judgment checks whether or not the appearance edge of the target is consistently connected to the edge of the corresponding image. this is,
If the analysis values gsun of the feature points are almost continuous at the connected position, it is assumed that the characteristic values are connected without contradiction. The appearance edge added to the existing segment is given the depth value of the portion connected to the edge of the input image, which has a correspondence as the depth value (Z component), and the value of the Z component is given as the X and Y components. The value obtained from the equation 2 is given. Further, the analysis value gives the image analysis result of the points forming the set of appearance edges as it is.

Then, the appearance edge determined not to be associated with the disappearing edge or added to the existing segment and not belonging to any segment is marked as an unknown segment. It is added as information and stored in the storage unit 13. Then, by the above-described processing, the feature points marked with the unknown segment are processed as follows. Segment is S
Suppose there are individual pieces. Although the motion estimation / correspondence search unit 15 obtains the motion estimation value for each segment, assuming that the unknown segment belongs to the segment s (s = 1 to S), the three-dimensional position is the motion estimation value of the segment s. Try to move by. The three-dimensional position of the unknown feature point of the segment before movement is the Z component of the center of gravity of the segment s, or the Z component of the point belonging to the segment s to which the corresponding point was closest at the edge of the previous input image. The depth value is given in the same manner as the initial setting unit 12. When a segment unknown point is moved and then projected, it is searched whether or not an appearance edge corresponding to the point exists. It is assumed that such an operation is performed for all S segments, and as a result, the unknown segment point belongs to the segment having the corresponding corresponding appearance edge.

For an appearing edge for which a corresponding segment does not exist properly, a pixel having a number of pixels smaller than a predetermined number is considered to be noise and is ignored. The appearance edge that cannot be ignored is the (S + 1) th new segment.

The encoding unit 23 encodes the input moving picture sequence information and sends it to the transmission line 50. Encoding unit 23
Encodes the edge position qn, the analysis value, and the DC component input from the image analysis unit 11 for the image data of the first frame of each continuous image sequence. Also,
For the image data of the second and subsequent frames, the motion estimation value output from the motion estimation / correspondence search unit 15 and the fractionation Δqsun output from the quantization unit 19 are encoded and output for each segment.

When encoding the image data of the second and subsequent frames, since the segment having a small fractionation has little influence on the image quality,
Alternatively, only the motion estimation value may be transmitted. An encoding unit that performs such processing will be described with reference to FIG. FIG. 4 is a block diagram for explaining the configuration of the component of the encoding unit 23. The fractionation Δqsun for each segment input from the quantization unit 19 is input to the calculation unit 231 and the storage unit 232. Operation unit 231
In, the sum of squares Σ (| Δqsun | ² ) of each input fraction is calculated. In addition, the storage unit 232
The fractionation Δqsun input to is stored for each segment.

In the comparison unit 233, the sum of squares of the fraction calculated by the calculation unit 231 Σ (| Δq
Compare sun | ² ) with a predetermined value ε. Then, when the sum of squares Σ (| Δqsun | ² ) of the fractionation is larger than the predetermined value ε, the signal indicating that the encoding is performed is output to the fractionation encoding unit 234. If the sum of squares Σ (| Δqsun | ² ) of the fractionation is less than or equal to the predetermined value ε, the comparison unit 233 does not output the signal. The predetermined value ε is arbitrarily set according to conditions such as image quality required at the time of reproduction. Motion estimation value encoder 2
Reference numeral 35 encodes the motion estimation value for each segment input from the motion estimation / correspondence search unit 15. Further, the DC component encoding unit 236 receives the DC input from the image analysis unit 11.
Encode the components. As described above, in the coding unit including the configuration as shown in FIG. 4, the motion estimation value and the DC component are coded for each frame, but the fraction Δqsun varies more than a predetermined value. Encoding is performed. Therefore, the compression rate can be increased.

As described above, according to the moving picture coding apparatus 10, at the stage of estimating the motion of the segment for each frame, the corresponding feature points are statistically obtained by the maximum likelihood estimation method also using the image features. And can be appropriately associated.
Therefore, the fractionation of each feature point can be made sufficiently small, and encoding can be performed at a high compression rate.

Next, a moving picture decoding apparatus according to an embodiment of the present invention will be described with reference to FIG. FIG. 5 is a block diagram showing the configuration of the moving picture decoding apparatus 30 according to the embodiment of the present invention. The moving picture decoding apparatus 30 includes an initial decoding unit 3
4, an initial setting unit 32, an image synthesis unit 31, a storage unit 33, a correction processing unit 42, a motion processing unit 36, a second projection unit 39, a Kalman filter 41, and a first projection unit 37. The moving picture decoding apparatus 30 of the present embodiment expands the encoded moving picture sequence transmitted from the transmission line 50,
This is a moving image decoding device that outputs from the image combining unit 31.
Further, the moving picture decoding apparatus 30 constitutes an image processing system in cooperation with the moving picture coding apparatus 10 described above.

The operation of each section will be described below. The decoding unit 34 receives the signal transmitted from the transmission line 50,
It is a receiving unit that decrypts each piece of information, extracts it, and outputs it to each unit as appropriate. Regarding the data of the first frame of the image of the continuous sequence, the edge position {qn}, the analysis value, and the DC component are separated from the received signal, and the edge information and the analysis value are sent to the initial setting unit 32 and the edge information and the analysis value are analyzed. The value and the DC component are output to the image composition unit 31. The data of the second and subsequent frames is separated from the received signal into motion estimation values, fractions, DC components, and information regarding correction processing for each segment. The motion estimate is
The motion processing unit 36 and the Kalman filter 41 are output, the fractionation is output to the adding unit 40, the DC component is output to the image synthesizing unit 31, and the information regarding the correction process is output to the correction processing unit 42.

Based on the edge information and the analysis value input from the initial decoding section 34, the initial setting section 32 develops into information of the characteristic points constituting each segment for each segment, and stores it in the storage section 33. To do. By this processing, the content of the storage unit 33 of the moving picture decoding device 30 becomes the same as the information in the storage unit of the moving picture coding device as described above.

The storage unit 33 is a storage means for storing information on each feature point of the edge image for each segment.
The moving picture coding apparatus 10 is composed of a memory and is described above.
The storage unit 13 has the same configuration.

The motion processing unit 36 moves each segment stored in the storage unit 33 based on the motion estimation value input by the decoding unit 34. Second projection unit 39
Projects each segment moved by the motion processing unit 36 onto the image. The addition unit 40 adds the fractionation input from the decoding unit 34 to each feature point to the image information projected by the second projection unit 39, and inputs the obtained image information to the Kalman filter 41.

The Kalman filter 41 has the same function as the Kalman filter 21 of the moving picture coding apparatus 10 described above, and updates the information of each segment stored in the storage unit 33.

The first projection unit 37 uses the Kalman filter 4
Each segment is projected on the image based on the image information updated by 1. The image composition unit 31 includes a decoding unit 34.
And an image synthesizing unit that restores and outputs image data based on the edge information, the analysis value, and the DC component input from the first projection unit 37. First, with respect to the image data of the first frame input from the decoding unit 34, an image substantially equal to the input image to the moving image encoding apparatus is synthesized from the edge image, the analysis value of each point, and the DC component. As for the image data of the second and subsequent frames, the image is synthesized based on the edge information and the analysis value input from the first projection unit 37 and the DC component input from the decoding unit 34, and the image is output. To do. At that time, the image synthesis is performed except for the point that the additional information disappears.

As described above, according to the moving picture decoding apparatus 30, even a moving picture sequence encoded by three-dimensional shape data and its motion information cannot be simply represented by the three-dimensional shape data. Since the lost edge information and the appearing edge information can be corrected, even such a moving image can be decoded faithfully to the original image with good reproducibility.

[0061]

According to the moving picture coding apparatus of the present invention, movement of feature points between frames can be efficiently coded.
We were able to encode continuous video with high compression rate. Further, according to the moving picture decoding apparatus of the present invention, the original moving picture can be faithfully decoded from the continuous moving picture signal coded by the moving picture coding apparatus.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of a moving picture coding apparatus according to an embodiment of the present invention.

FIG. 2 is a diagram for explaining a method of motion estimation / correspondence search in the moving picture coding apparatus shown in FIG. 1, and is a diagram for explaining a coordinate system by showing a point P in a three-dimensional space.

FIG. 3 is a diagram for explaining a method of motion estimation / correspondence search of the moving image coding apparatus shown in FIG. 1, and is a diagram for explaining a virtual projected image.

FIG. 4 is a diagram showing another example of the configuration of the encoding unit of the moving image encoding apparatus shown in FIG.

FIG. 5 is a block diagram showing a configuration of a moving picture decoding apparatus according to an embodiment of the present invention.

[Description of Codes] 10 ... Moving image coding device 11 ... Image analysis unit 12 ... Initial setting unit 13 ... Storage unit 14 ... Initial coding unit 15 ... Motion estimation / correspondence search unit 16 ... Motion processing unit 17 ... Projection unit 18 Addition unit 19 Quantization unit 20 Addition unit 21 Kalman filter 22 Correction processing unit 23, 23b Encoding unit 231 Calculation unit 232 Storage unit 233 Comparison unit 234 Fractionation encoding unit 235 Motion estimation Value encoder 236 ... DC component encoding unit 30 ... Moving image decoding device 31 ... Image combining unit 32 ... Initial setting unit 33 ... Storage unit 34 ... Initial decoding unit 36 ... Motion processing unit 37 ... First projection unit 39 ... 2nd projection part 40 ... Addition part 41 ... Kalman filter 42 ... Correction processing part 43 ... Decoding part 50 ... Transmission path

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Office reference number FI technical display location 9061-5H G06F 15/70 410

Claims (5)

[Claims] 1. An image analysis unit that extracts a feature point from each frame of a continuous moving image, a segmentation unit that extracts a segment from the first frame of the continuous moving image, and an image analysis unit that extracts the segment. Based on the feature points
A shape storage unit that stores the three-dimensional model shape information for each segment extracted by the segmentation unit, and a motion between frames of each three-dimensional model stored in the shape storage unit is estimated, and 3 of each frame is estimated. Motion estimation correspondence search means for associating a feature point obtained by moving and projecting a feature point of a dimensional model based on the estimated motion, and a feature point of the next frame of the frame, and between the associated feature points. A difference calculating unit for obtaining a position difference, and each three-dimensional model shape stored in the shape storage unit based on a motion estimation value between frames of each of the three-dimensional models and a position difference between the feature points. Updating means for updating information, each segment of the first frame of the continuous moving image extracted by the segmentation means, and the image analysis procedure. Initial coding means for coding, as initial data, the analysis value of each feature point of each segment analyzed by, and the frame of the three-dimensional model of each segment obtained for each frame by the motion estimation correspondence searching means. A moving image processing apparatus that encodes the continuous moving image, and has an encoding unit that encodes a motion estimation value between the feature points and a position difference between the respective feature points calculated by the difference calculating unit. . 2. The encoding means is calculated by the difference calculating means only when the difference between the positions of the characteristic points calculated by the difference calculating means between the frames is equal to or more than a predetermined value. The difference between the positions of the feature points is encoded.
The moving image processing apparatus described. 3. Initial decoding means for decoding the characteristic points of each segment forming the first frame of a continuous moving image and the analysis value of each characteristic point from the encoded initial data. Shape storage means for storing three-dimensional model shape information for each segment based on the feature points for each segment decoded by the initial decoding means, and for each segment from the encoded data for each frame Difference between the motion estimation value between frames of the 3D model, the position of the feature point of the projected image obtained by moving the 3D model based on the motion estimation value, and the position of the corresponding feature point of the next frame of the frame. Decoding means for decoding and, based on the difference between the motion estimation value between the frames of the three-dimensional model of each segment and the position of the corresponding feature point,
For each frame, based on the updating means for updating the shape information of the three-dimensional model for each segment stored in the shape storage means, and the three-dimensional model shape information for each segment stored in the shape storage means And an image synthesizing means for synthesizing the images of 1., and a moving image processing apparatus for decoding encoded continuous moving images. 4. The shape storage means adds predetermined depth information to the respective feature points, and stores three-dimensional model shape information in which each segment is a flat plate-shaped three-dimensional model as initial data. 3. The moving image processing apparatus according to any one of 3 above. 5. The updating means sets the position of the feature point of the projected image obtained by moving the three-dimensional model based on the inter-frame motion estimation value of the three-dimensional model of each segment as the state quantity, and the position of the next frame of the frame. The three-dimensional model shape information is updated by obtaining the least-squares estimated value of the state quantity by a Kalman filter, using the actual position of the corresponding feature point as an observed amount, and the error between the corresponding feature points as noise. ~
4. The moving image processing apparatus according to any one of 4 above.