JPH08161505A - Dynamic image processor, dynamic image encoding device, and dynamic image decoding device - Google Patents

Abstract

PURPOSE: To provide the image processor which can encode three-dimensional shape information on respective segments constituting a dynamic image with high compressibility. CONSTITUTION: A three-dimensional model for respective segments constituting the continuous dynamic image is obtain from the dynamic image and projected on a specific reference two-dimensional plane by a projection part 21, and a connection line element extraction part 22 extracts connection line elements from the projection image to obtain the depth value from the two-dimensional projection image to the original three-dimensional model at respective feature points of the projection image. The projection image is processed through the chain coding of a chain coding part 23 and then the Huffman encoding of a 1st Huffman code generation part 24, and then sent out. Further, the depth value is processed through the ADPCM encoding of an ADPCM encoding part 25 and through the Huffman encoding of a 2nd Huffman code generation part 26, and then sent out.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving picture processing apparatus capable of extracting a three-dimensional shape model of an object in a moving picture from a moving picture and encoding and decoding the shape model at a high compression rate. And a moving picture coding apparatus and a moving picture decoding apparatus, to which the moving picture processing apparatus is applied to perform transmission / recording of a moving picture at a high compression rate.

[0002]

2. Description of the Related Art A method has been proposed in which a moving image is encoded by using a three-dimensional model of each object in the moving image sequence and the moving image sequence is compressed. If the three-dimensional shape of each object and its movement are known, a moving image sequence that is exactly the same as the original moving image sequence can be generated. Therefore, for example, in image communication, if the transmitting side and the receiving side share a three-dimensional model, the transmitting side detects the motion information of the input image, and the receiving side performs image synthesis from the motion information, the image is reproduced. it can. In this case, since only motion information needs to be transmitted, image communication at an extremely low rate can be expected. Specifically, a method has been tried in which the three-dimensional structure of the face expressed by the wire frame model of the face is shared between the transmitting side and the receiving side, and only the features such as facial expressions are transmitted to synthesize the facial image. There is.

However, when applying this encoding method to a natural image, a three-dimensional model cannot be prepared in advance,
It is necessary to extract a three-dimensional model from a given moving image sequence. As a method for extracting a three-dimensional shape model from a moving image sequence and coding a moving image using the model, Hans George Musmann, Michael Hott
er, Jorn Ostermann and others "OBJECT-ORIENTED analys
is coding of moving images. ", Signal processing: Ima
ge Communication 1 (1989): 117-138, ElsvierSCIENCE PU
There is a method disclosed in BLISHERS BV. According to this method, a motion vector is obtained for an edge portion and the depth is obtained by using the motion vector, and for a portion other than the edge, the depth is interpolated to infer a three-dimensional shape. The brightness information is mapped as an attribute in three dimensions and is transmitted together with the interpolated three-dimensional shape data.

However, in the method of estimating the three-dimensional shape and compressing the moving image sequence by the method as described above, it is difficult to obtain the motion vector even at the edge portion where the brightness changes abruptly, and the accurate depth from the motion vector is obtained. It was very difficult to guess. Therefore, even if the three-dimensional shape is estimated by interpolating the depth of the edge portion, an accurate three-dimensional shape cannot be obtained. As a result, a deviation from the actual shape occurs, extra information representing the deviation increases, and the compression rate cannot be increased. Further, since the brightness information mapped as an attribute in three dimensions is transmitted as the initial information, the initial information amount is very large.

As a similar method for extracting the three-dimensional shape model, there is the following method. Information in the predetermined depth direction is added to the two-dimensional shape information of each segment of the first frame of the moving image sequence to obtain the initial value of the three-dimensional model. The motion of the three-dimensional model is estimated, the position of each feature point of the image obtained based on the estimation and the image of each input frame is compared, and the overall motion of the three-dimensional model and each feature are compared. Extract the deviation of points. The three-dimensional model is sequentially updated based on the overall movement of the three-dimensional model (motion estimation value) and the deviation of each feature point, and a faithful three-dimensional model is acquired. According to this method, it is possible to obtain a faithful three-dimensional model and reduce the amount of initial information, as compared with the method described above.

[0006]

However, even in the moving picture coding apparatus described above, it is necessary to transmit the data of the three-dimensional shape model and the analysis value as the initial information, and the amount of the data is not sufficiently small. That is, there has been a demand to further reduce the amount of data, particularly to reduce the amount of initial data.

Therefore, an object of the present invention is to provide a moving image processing apparatus capable of encoding three-dimensional shape information of each segment forming a moving image with a smaller amount of data.

[0008]

In the moving image processing apparatus of the present invention, the three-dimensional shape information to be encoded is set to 2
The position information of the image projected on the three-dimensional plane and the depth information of the image are divided and encoded by a method suitable for the property of each information. In particular, regarding the depth information,
Focusing on the fact that the difference in the depth direction at the edge of each segment is small, encoding is performed by the ADPCM system.

Therefore, the moving image processing apparatus of the present invention is
3 of each segment that composes the input continuous moving image
A three-dimensional shape acquisition means for acquiring three-dimensional shape information;
Projection means for projecting the three-dimensional shape acquired by the three-dimensional shape acquisition means onto a predetermined two-dimensional plane; position information encoding means for encoding the position of the image projected on the two-dimensional plane by the projection means; The depth from the image projected on the two-dimensional plane by the projection means to the three-dimensional shape is ADPCM.
A depth information encoding means for encoding by a method,
The three-dimensional shape acquired by the three-dimensional shape acquisition means is encoded.

Preferably, the position of the image on the two-dimensional plane of the three-dimensional shape encoded by the position information encoding means and the depth of the three-dimensional shape encoded by the depth information encoding means. Further has entropy coding means for entropy coding.

Specifically, the three-dimensional shape acquisition means is
Modeling means for analyzing a predetermined still image related to the input continuous moving image and obtaining three-dimensional shape information of each segment forming the image, and three-dimensional shape information of each segment obtained by the modeling means And a three-dimensional shape information of each segment stored in the storage unit for storing the three-dimensional movement of each segment forming the moving image between each frame of the continuous moving image. Based on the difference between the position as a result of three-dimensionally moving each segment by the estimated motion and the actual position, and the motion estimation means for estimating each segment stored in the storage means. And an updating unit that updates the three-dimensional shape information.

Further, the moving picture coding apparatus according to the present invention is characterized in that the three-dimensional movement of each segment between the moving picture processing apparatus and each frame of the continuous moving picture is referred to as the three-dimensional movement.
The motion estimation means for estimating based on the dimensional shape information, the position of each feature point resulting from the three-dimensional movement of each segment by the motion estimated by the motion estimation means, and the actual position of each feature point A code that encodes, for each frame, a difference detection unit that obtains a difference, a motion estimation value of each segment estimated by the motion estimation unit, and a position difference of each feature point detected by the difference detection unit. Encoding means for encoding the continuous moving image.

Further, the moving picture decoding apparatus of the present invention comprises a two-dimensional image decoding means for decoding the position of the image projected on the two-dimensional plane of each segment constituting the encoded continuous moving picture, and ADPCM. Depth information decoding means for decoding the depth of the image on the two-dimensional plane coded by using the method, the position of the image on the two-dimensional plane decoded by the two-dimensional image decoding means, and the depth information Three-dimensional shape generation means for generating three-dimensional shape information of each segment forming the moving image based on the depth of the image on the two-dimensional plane decoded by the decoding means, and each of the continuous moving images. A motion decoding unit that decodes the motion estimation value of each segment and the displacement of each feature point, which is encoded for each frame.
Moving means for three-dimensionally moving the position of each segment based on the motion estimation value of each segment decoded by the motion decoding means, and two-dimensionally moving each segment at the position moved by the moving means Projection means for obtaining a projection image projected on the screen, and a modification for moving the position of each feature point of each segment in the projection image based on the displacement of each feature point of each segment decoded by the motion decoding means Means and an image synthesizing means for synthesizing images on the basis of the shape of each segment transformed by the transforming means, and decodes a coded continuous moving image.

[0014]

The moving image processing apparatus of the present invention acquires the three-dimensional shape of each segment forming an input continuous moving image, projects the three-dimensional shape on a predetermined two-dimensional plane, and two-dimensionally projects it. An image is obtained, and further depth information for the three-dimensional shape based on the two-dimensional projected image is obtained. Thereby, the information of the three-dimensional shape is separated into the position information of the two-dimensional projected image and the depth information corresponding to the image. The position information of the two-dimensional projection image is encoded by any appropriate encoding method, and the depth information is encoded by the ADPCM method. Each encoded signal is further entropy encoded and encoded at a high compression rate.

Further, the moving picture coding apparatus of the present invention uses the three-dimensional shape of each segment which forms a continuous moving picture acquired and coded by the moving picture processing apparatus, so as to obtain the continuous moving picture. Estimate the three-dimensional movement of each segment between each frame of the image. Then, the difference between the position of each feature point of the estimation result and the actual position of each feature point is obtained. The motion estimation value of each segment and the difference in position of each feature point are encoded for each frame.

Further, the moving picture decoding apparatus of the present invention is arranged such that the position of the image projected on the coded two-dimensional plane and the ADPCM
The depth information of the image coded by the method is decoded, respectively, and they are combined to generate three-dimensional shape information. Then, in each frame of the moving image, the encoded motion estimation value of each segment and the displacement of each feature point are decoded,
The position of each segment is three-dimensionally moved according to the motion estimation value, each moved segment is projected onto a two-dimensional screen, and the position of each characteristic point of each segment in the projected image is calculated for each characteristic point. Move based on displacement.
Images are combined based on the shape of each segment obtained in this way.

[0017]

DESCRIPTION OF THE PREFERRED EMBODIMENTS A moving picture coding apparatus according to an embodiment of the present invention will be described with reference to FIGS. FIG. 1 is a block diagram showing the configuration of a moving picture coding apparatus 10 according to an embodiment of the present invention. The moving image coding apparatus 10 includes an image analysis unit 11,
It has an analysis image storage unit 12, a segmentation unit 13, a storage unit 14, a motion estimation / correspondence search unit 15, a projection unit 16, an error detection unit 17, a Kalman filter 18, and an encoding unit 19.

This moving picture coding apparatus 10 constitutes an image processing system in cooperation with a moving picture decoding apparatus 30 which will be described later. The moving picture coding apparatus 10 of the present embodiment detects moving picture data from a moving picture series from a VTR or the like by a continuous sequence detecting unit (not shown), and codes and transmits the moving picture data series. It is a transmission device. In the transmission, the first step of extracting the three-dimensional shape information of the segments forming the sequence from the continuous sequence, and the second step of encoding using the extracted three-dimensional shape information. Be divided. The operation of each unit will be described below for each of the first step and the second step.

First, the operation of each part of the first step of extracting the three-dimensional shape information of each segment forming a moving image from a continuous moving image sequence will be described. VTR
A continuous sequence is detected from the moving image sequence input from the above, and is input to the image analysis unit 11. The image analysis unit 11 analyzes the sequentially input image data of each frame, extracts the characteristic points, and obtains the positions of the characteristic points and the analysis values.
In the present embodiment, with respect to the input image data, the image data is analyzed with a plurality of filters having different resolution scales, and the points forming the edges are detected as feature points,
The input image data is converted into image data of an edge, which is characteristic image data, and the position of each point constituting the edge and the analysis value are extracted.

The analysis image storage unit 12 is a memory for storing the feature point images of the continuous sequence analyzed by the image analysis unit 11. The stored information of the feature points of each frame is sequentially referred to by the motion estimation / correspondence search unit 15, and the information of the feature points of the first frame is referred to by the segmentation unit 13 and the encoding unit 19.

The segmentation unit 13 performs segmentation on the basis of the positions and analysis values of the characteristic points of the characteristic image data of the first frame input from the image analysis unit 11,
The segments making up this image data are extracted, and the information of the characteristic points for each segment is stored in the storage unit 14. This segmentation is from a color image to red,
Recursive extraction of a total of nine types of features of green, blue, lightness, hue, and saturation signals, and Y, I, and Q signals corresponding to television signals, and segmentation based on a histogram relating to the features. Performed by thresholding.

The storage unit 14 is a storage means for storing the position information X, Y, Z, the analysis value g, the probability covariance matrix v, and the additional information a for each feature point of each segment, and is composed of a memory. It The information stored in the storage unit 14 is such that the input continuous sequence has S segments, each segment is composed of Us (s = 1 to S) edges, and each edge is Nsu ( u = 1 to Us, s =
1 to S), it is expressed as in Equation 1.

[0023]

[Equation 1]

The probability covariance matrix vsun (n = 1 to N)
Since su, u = 1 to Us, and s = 1 to S) are scattered among the points forming each edge, the same value is given to each feature point forming the same edge.

As the information stored in the storage unit 14, first, the information on the first frame is input from the segmentation unit 13 to generate initial data. After that, every time the image data of the second and subsequent frames is input, the contents are updated by the Kalman filter 18 described later.

The motion estimation / correspondence search unit 15 estimates the amount of movement of each segment from the information {Fsun} of each point of the edge image of the image of the previous frame, the edge position of the current frame, and the analysis value, The information {Fsun} of each point of (1) and the feature point of the edge image of the current frame are associated with each other.

The method will be specifically described below. First, in FIG. 2, it is assumed that the coordinate system XYZ is a camera coordinate system, the origin of the coordinate system is the center of the lens, and the optical axis is aligned with the Z axis that is the depth direction. In this coordinate system, it can be considered that the image of the point P is projected on a plane that is parallel to the XY plane and is separated from the origin by the focal length f of the camera. The position of the image of the point P on the projection plane becomes the position of the pixel on the image input from the camera. With respect to the projection plane, the origin is the intersection with the Z axis of that plane and X
A coordinate system xy parallel to the axes and the Y axis is set.

The coordinates of the point P in the XYZ space are p = (Xp,
Yp, Zp) and the coordinates of the point Q, which is the image of the point P on the xy plane, are q = (xq, yq), the coordinates q of the point Q are expressed by equation 2.

[0029]

[Equation 2]

A segment s (s = 1 to S) is composed of Us (s = 1 to S) edges, and each edge is represented by Nsu (u = 1 to Us) point information. The positions of these points are psun = (Xsun, Ysun, Zsun) (n =
1 to Nsu). This segment relatively rotates by Δωx around the X axis, Δωy around the Y axis, Δωz around the Z axis, and Δt = (Δtx, Δty, Δt
z), the amount of movement of each point psun forming this segment Δpsun = (ΔXsun, ΔYsun,
ΔZsun) is the rotation around each axis Δωx, Δωy, Δ
Assuming that ωz and the parallel movement amount Δt are small, Equation 3
become that way.

[0031]

(Equation 3)

Let qsun be the projection point of the point psun on the xy plane.
= (Xsun, ysun), the amount of movement of the projection point qsun due to the movement of the segment Δqsun = (Δxsun
, Δysun) is given by equation 4.

[0033]

[Equation 4]

From Equations 2 and 4, Equation 5 is obtained.

[0035]

(Equation 5)

Equations 2 and 5 are replaced by m = 1 of N points.
When it is applied to M pieces of ~ M, it becomes as shown in Expression 6.

[0037]

(Equation 6)

Note that Δt ^t represents a transposed matrix of the matrix Δt. Regarding Δq, since the point on the image corresponding to the virtual projection point qm ′ by the equation 2 of the three-dimensional position pm obtained before the new image is input is unknown, as shown in FIG. It is assumed that the virtual projection image Ip of the dimensional position information is the closest point to the projection feature point qm of the object image Ir. M ≧
When the value is 3, the estimated value ΔC ′ of the parameter ΔC of the amount of rotation and the amount of translation is obtained by the equation 7 by the least square method.

[0039]

(Equation 7)

The movement amount Δqsun of the three-dimensional position information based on ΔC ′ obtained by the equation 7 is calculated by the equation 2 and a virtual projected image is newly created by the equation 3, and the similar points are assumed to be corresponding points. If the calculation of Expression 7 is repeated and Expression 8 is repeated, the virtual projected image and the object image Ir come close to each other.

[0041]

(Equation 8)

By repeating this calculation until Σ│Δqsun│ ² becomes equal to or less than the predetermined value ε, the corresponding point qsun of the new image with respect to the three-dimensional position information psun of the original image is obtained.

According to the method of motion estimation / correspondence search described above, the object is assumed to be a rigid body, and the points forming the three-dimensional position information are constrained by the six parameters of rotation and translation, thereby individually However, the motion estimation / correspondence search is performed comprehensively, not independently. Therefore, as a whole, a consistent correspondence relationship is obtained for all points, and noise in the three-dimensional position information due to incorrect correspondence is reduced.

The projection unit 16 includes a motion estimation / correspondence search unit 15
The position information of the information {Fsun} of each segment is moved in parallel and rotationally in the three-dimensional space by the parallel movement amount t and the rotational movement amount ω of each segment, and the three-dimensional position of each point of each segment is obtained. psun = (Xsun
, Ysun, Zsun) at position qsun = (xsun,
ysun) and the analysis value gsun is given to the point qsun on the obtained image. The conversion from the three-dimensional position psun to the projection point qsun is performed by Expression 3.

The error detector 17 finds the difference between the position qsun 'on the image of the information {Fsun} of each feature point after projection and the edge position qsun of the corresponding input image, and further quantizes the difference. Seeking fracturing. As the quantization method, linear quantization with a certain appropriate quantization step (for example, 1 pixel width), non-linear quantization with some non-linear quantization steps set, and quantization steps not fixed are input. There is quantization in which the quantization step is appropriately changed depending on the nature of the image, and an appropriate quantization method may be used according to the required transmission rate, image quality, and the like. For example, if a high compression rate is required, increase the quantization step, or if there are many straight lines in the image and the discontinuity of the straight lines due to quantization noise is conspicuous, perform non-linear quantization to reduce the fractional part. Try finer quantization. The calculated fraction is input to the Kalman filter 18.

The Kalman filter 18 uses the three-dimensional position p of the information {Fsun} of each point of each segment in the previous image.
The three-dimensional position psun is updated from the sun and the edge position qsun corresponding to the input image. The Kalman filter is a filter that can sequentially obtain a least squares estimation value of a state quantity from a time series observation quantity in a system including noise. In this embodiment, the state quantity is a three-dimensional position psun and a two-dimensional position qsun which is an observed quantity. Two-dimensional position qsun
Contains noise due to the quantization of Δqsun. Also,
The motion estimation value also contains noise. 3 on the plane of the initial value
The dimensional shape {psun} is updated by the Kalman filter so as to approach the actual three-dimensional shape each time there is a motion in the segment. The probability covariance matrix vsun at the information {Fsun} of each point is the probability covariance matrix (3 ×
3) is used to update psun, and at the same time the probability covariance matrix vsun is updated.

When the above Kalman filter update is performed for all frames of a continuous moving image, the storage unit 14 finally stores a high fidelity three-dimensional shape model for each segment.

Next, the second step of encoding this continuous moving image using the three-dimensional shape model extracted in the first step will be described. Also in the second step, the operation of each unit is the same as that in the first step. However, in the second step, the storage unit 1
The motion estimation / correspondence search is performed using the final three-dimensional shape information of each segment stored in 4, and the motion of each segment is extracted, and the difference from the actual position of each feature point is obtained.

Therefore, first, the motion estimation / correspondence search section 15 uses the three-dimensional shape information of each segment stored in the storage section 14 for the characteristics of each frame stored in the analysis image storage section 12. From the position of the points, the correspondence between the entire movement of each segment and each feature point is obtained. The method of obtaining the same is the same as in the case of the first step.
The motion obtained here is applied to the projection unit 16 and the encoding unit 19.
Is output to

Then, the projection unit 16 moves the position information of each segment in parallel and rotationally in the three-dimensional space according to the movement amount of each segment obtained by the motion estimation / correspondence search unit 15, and further The three-dimensional position of the point is converted into the position on the image. Error detector 17
Calculates the difference between the position of each feature point on the image after projection and the edge position of the corresponding input image, and further quantizes the difference to calculate the fractionation. The obtained fractionation is output to the encoding unit 19.

Finally, the operation of the encoding unit 19 will be described. The encoding unit 19 is means for encoding the information obtained in each of the first step and the second step and transmitting it to the transmission line. Therefore, the operation will be described regardless of the steps. The encoding unit 19 first encodes the three-dimensional shape information stored in the storage unit 14 and the analysis value for each continuous image sequence. For the image data of each frame, a motion estimation value output from the motion estimation / correspondence search unit 15 for each segment,
It encodes and outputs the fractionation output from the error detection unit 17.

The structure and operation of the initial coding unit 20 included in the coding unit 19 and coding the three-dimensional shape information of each continuous image sequence according to the present invention will be described with reference to FIGS.
Will be described with reference to. 4 is an initial coding unit 20 included in the coding unit 19 of the moving picture coding apparatus 10 shown in FIG.
FIG. 3 is a block diagram showing the configuration of FIG. The initial encoding unit 20 is
It has a projection unit 21, a connected line element extraction unit 22, a chain coding unit 23, a first Huffman code generation unit 24, an ADPCM coding unit 25, and a second Huffman code generation unit 26.

The projection unit 21 projects the three-dimensional shape of each segment onto a predetermined two-dimensional plane serving as a reference, and the projected two-dimensional image and each feature point on the two-dimensional image to the original three-dimensional model. And the depth information of. Specifically, FIG.
As shown in (A), the three-dimensional model 50 of the face is projected on the two-dimensional plane 51, and the two-dimensional face image 52, the feature points of the two-dimensional face image 52, and the original three-dimensional mole 50 are projected. The depth of each is obtained. The connecting line element extraction unit 22 includes a projection unit 21.
The connecting line elements of the feature points are extracted based on the two-dimensional image for each segment obtained by projection by. Figure 5
Six connecting line elements 1 to 6 shown in FIG. 5B are extracted from the two-dimensional image shown in FIG.

The positions of the extracted connecting line elements are respectively chain coded by the chain coding unit 23, and further Huffman coded by the first Huffman code generation unit 24 to obtain two-dimensional position information. It is sent to the transmission line. In addition, from each feature point of the two-dimensional image, the original 3
The depth information up to the dimensional model is obtained by the ADPCM encoding unit 2
5, ADPCM coded, and further Huffman coded by the second Huffman code generation unit 26, and sent to the transmission path as two-dimensional position information.

As described above, the depth information is converted to ADPCM.
It will be described with reference to FIGS. 6 to 8 that the three-dimensional shape data is efficiently encoded by the encoding by the method. FIG. 6 is a diagram showing the depth of each of the connecting line elements, and (A) to (F) are diagrams showing the depth of the connecting line elements 1 to 6, respectively. Generally, as shown in FIG.
The depth value of the dimensional shape model has a correlation in the connecting direction and does not change suddenly.

Therefore, when a depth difference signal is created between adjacent pixels, it becomes as shown in FIG. FIG. 7 is a diagram showing the difference in depth between the connecting line elements, and FIGS. 7A to 7F are diagrams showing the difference in depth between the connecting line elements 1 to 6, respectively. As shown in FIG. 7, the depth difference has a value near 0. Furthermore, FIG.
When the depth values of each pixel of each connecting line element shown in are accumulated,
As shown in FIG. 8, it can be seen that the concentration is near 0. Therefore, such a signal can be efficiently compressed by the encoding method that performs the primary prediction by taking the difference of the depth values.

As described above, according to the initial coding unit 20 of the moving picture coding apparatus 10 of this embodiment, the three-dimensional shape to be transmitted as the initial data is projected onto the two-dimensional plane, and the two-dimensional image and the depth value are obtained. Information, the information of the two-dimensional image is chain coded and Huffman coded, and the depth value is AD.
PCM coding, Huffman coding, and transmission are performed. Therefore, the three-dimensional shape model can be efficiently transmitted at a high compression rate.

Next, a moving picture decoding apparatus according to an embodiment of the present invention will be described with reference to FIGS. 9 and 10. Figure 9
FIG. 3 is a block diagram showing a configuration of a moving picture decoding device 30 according to an embodiment of the present invention. The moving image decoding device 30 includes the decoding unit 3
1, a storage unit 32, a motion processing unit 33, a projection unit 34, a deformation unit 35, a recomposition unit 36, and a composite image storage unit 37. The moving picture decoding apparatus 30 of the present embodiment is a moving picture decoding apparatus that expands, combines, and outputs an encoded moving picture sequence transmitted from a transmission path, and includes the above-described moving picture coding apparatus 10. Collaborate to form an image processing system.

The operation of each unit will be described below. The decoding unit 31 is a receiving unit that receives the signal transmitted from the transmission path, decodes the information, extracts the information, and outputs the information to the units as appropriate. The decoding unit 31 first receives the three-dimensional shape information of each segment forming a continuous moving image sequence, decodes it, and stores it in the storage unit 32. The position of each segment at that time is represented by the position in the first frame of the continuous moving image. Then, for the second and subsequent frames, the encoded total movement amount (global motion) of each segment and the fine movement (fractation) of each feature point are received and decoded, and the global motion is processed by the motion processing unit. 33, the fractionation is output to the transformation unit 35.

An initial decoding unit 4 included in the decoding unit 31 for receiving the three-dimensional shape information of each segment according to the present invention.
The configuration of 0 will be described with reference to FIG. Figure 10
FIG. 10 is a block diagram showing a configuration of an initial decoding unit 40 included in the decoding unit 31 of the moving image decoding device 30 shown in FIG. 9.
The initial decoding unit 40 includes a first Huffman code decoding unit 41, a chain code decoding unit 42, and a second Huffman code decoding unit 4.
3, an ADPCM decoding unit 44, and a combining unit 45.

The position of the image projected on the two-dimensional plane of each segment which is chain coded and Huffman coded is first Huffman code decoded by the first Huffman code decoding unit 41, and further chain code decoding unit. At 42, the chain code is decoded. Also, AD
The depth information of the image on the two-dimensional plane encoded by the PCM method and further Huffman-encoded is first Huffman code-decoded by the second Huffman-code decoding unit 43 and further ADPCM-coded by the ADPCM decoding unit 44. Depth information is decoded. Then, the projection image on the two-dimensional plane and the depth information for the projection image are combined by the combining unit 45, and three-dimensional shape information is generated and stored in the storage unit 32.

The storage unit 32 is a memory for storing the three-dimensional shape information of each segment forming a continuous moving image sequence and the position information of each segment. Storage unit 32
Stores the three-dimensional shape information of each segment forming the continuous moving image sequence input from the decoding unit 31 as an initial value, and thereafter, the position is updated by the motion processing unit 33 for each frame. The motion processing unit 33 includes the decoding unit 3
The storage unit 32 based on the motion estimation value input by 1
Move each segment stored in. The moved information is output to the projection unit 34 and also stored in the storage unit 32.
Position information of each segment is updated.

The projection unit 34 projects each segment moved by the motion processing unit 33 onto a two-dimensional image. The transformation unit 35 adds the fractionation input from the decoding unit 31 to each position of each feature point of the image projected by the projection unit 34, and corrects the position of each feature point. The recomposition unit 36 is based on the information of each feature point input from the transformation unit 35, the analysis value of each feature point stored in the storage unit 32, and the DC component of this continuous moving image. , Restore image data. Composite image storage unit 37
Is a memory for storing the image data restored by the re-synthesis unit 36. The moving image information stored in the combined image storage unit 37 is appropriately output to a display device or the like.

As described above, the moving picture decoding apparatus 3 according to the present embodiment.
According to 0, it is possible to properly receive and reproduce the information of the moving image coded and transmitted by the moving image coding apparatus of the present embodiment described above. Especially chain coding and ADP
It is possible to appropriately receive the three-dimensional shape information that is CM-encoded and transmitted, and generate the three-dimensional shape information.

[0065]

According to the moving picture processing apparatus of the present invention, the three-dimensional shape information of each segment forming a moving picture can be efficiently coded. Further, according to the moving picture coding apparatus of the present invention, the amount of initial data can be reduced, and a moving picture can be coded at a high compression rate. Further, according to the moving picture decoding apparatus of the present invention, it is possible to appropriately reproduce the moving picture coded at a high compression rate by the moving picture coding apparatus.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of a moving picture coding apparatus according to an embodiment of the present invention.

2 is a diagram illustrating a method of a motion estimation / correspondence search unit of the moving picture coding apparatus shown in FIG. 1, and is a point P in a three-dimensional space.
FIG. 6 is a diagram illustrating a situation in which the user desires to describe the coordinate system.

FIG. 3 is a diagram for explaining a method of motion estimation / correspondence search of the moving picture coding apparatus shown in FIG.

4 is a block diagram showing a configuration of an initial coding unit included in a coding unit of the moving picture coding apparatus shown in FIG.

5A and 5B are diagrams for explaining the operation of the projection unit of the initial encoding unit shown in FIG. 2, in which FIG.
(B) is a diagram showing connecting line elements on a two-dimensional plane as a projection result.

6 is a diagram showing the depth of each connecting line element of the projected image shown in FIG. 5, where (A) to (F) are the connecting line elements 1 to 6 respectively.
It is a figure which shows the depth of.

FIG. 7 is a diagram showing the difference in depth between the connecting line elements of the projected image shown in FIG. 5, where (A) to (F) are the connecting line elements 1 respectively.
It is a figure which shows the difference of the depth of-6.

FIG. 8 is a diagram showing a distribution of depth differences between the connecting line elements shown in FIG. 7.

FIG. 9 is a block diagram showing a configuration of a moving picture decoding apparatus according to an embodiment of the present invention.

10 is a block diagram showing a configuration of an initial decoding unit included in the decoding unit of the moving picture decoding apparatus shown in FIG.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 10 ... Moving image coding apparatus 11 ... Image analysis part 12 ... Analysis image storage part 13 ... Segmentation part 14 ... Storage part 15 ... Motion estimation / correspondence search part 16 ... Projection part 17 ... Error detection part 18 ... Kalman filter 19 ... Encoding Unit 20 ... Initial coding unit 21 ... Projection unit 22 ... Connected line element extraction unit 23 ... Chain coding unit 24 ... Huffman code generation unit 25 ... ADPCM coding unit 26 ... Huffman code generation unit 30 ... Video decoding device 31 ... Decoding unit 32 ... Storage unit 33 ... Motion processing unit 34 ... Projection unit 35 ... Deformation unit 36 ... Resynthesis unit 37 ... Synthetic image storage unit 40 ... Initial decoding unit 41 ... Huffman code decoding unit 42 ... Chain code decoding unit 43 ... Huffman Code decoding unit 44 ... ADPCM decoding unit 45 ... Compositing unit

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Internal reference number FI Technical indication H04N 7/24

Claims (5)

[Claims] 1. A three-dimensional shape acquisition means for acquiring three-dimensional shape information of each segment constituting a continuous moving image, and a three-dimensional shape acquired by the three-dimensional shape acquisition means into a predetermined two-dimensional plane. Projecting means for projecting, position information encoding means for encoding the position of the image on the two-dimensional plane projected by the projecting means, and the three-dimensional shape from the image on the two-dimensional plane projected by the projecting means The depth to the adaptive differential pulse code modulation (A
Depth information encoding means for encoding by the DPCM method, and the 3
A moving image processing apparatus for encoding a three-dimensional shape. 2. The position of the image on the two-dimensional plane of the three-dimensional shape encoded by the position information encoding means and the depth of the three-dimensional shape encoded by the depth information encoding means are described. The moving image processing apparatus according to claim 1, further comprising entropy coding means for performing entropy coding. 3. The three-dimensional shape acquisition means analyzes a predetermined still image regarding a continuous moving image and obtains three-dimensional shape information of each segment forming the image, and the modeling means obtains the three-dimensional shape information. Storage means for storing the three-dimensional shape information of each segment, and three-dimensional movement of each segment forming the moving image between each frame of the continuous moving image are stored in the storage means. The motion estimation means for estimating based on the three-dimensional shape information of each segment, and the difference between the actual position and the position resulting from the three-dimensional movement of each segment by the estimated motion. The moving image processing apparatus according to claim 1, further comprising an updating unit that updates the three-dimensional shape information of each segment stored in the storage unit. 4. The moving image processing apparatus according to claim 1, wherein the three-dimensional movement of each segment between each frame of the continuous moving image is based on the three-dimensional shape information. And a position of each feature point as a result of three-dimensionally moving each segment by the motion estimated by the motion estimating unit,
Difference detection means for obtaining the difference between the positions of the actual feature points, the motion estimation value of each segment estimated by the motion estimation means, and the difference between the positions of the feature points detected by the difference detection means, A moving picture coding apparatus, which has a coding means for coding each frame, and codes the continuous moving picture. 5. A two-dimensional image decoding means for decoding the position of an image projected on a two-dimensional plane of each segment constituting a coded continuous moving image, and an adaptive differential pulse code modulation (ADPCM) method. Depth information decoding means for decoding the depth of the image on the two-dimensional plane coded by using the position of the image on the two-dimensional plane decoded by the two-dimensional image decoding means, and the depth information decoding means Three-dimensional shape generation means for generating three-dimensional shape information of each segment forming the moving image based on the depth of the image on the two-dimensional plane decoded by the above; and for each frame of the continuous moving image. Encoded into,
A motion decoding unit that decodes the motion estimation value of each segment and the displacement of each feature point, and the position of each segment is three-dimensional based on the motion estimation value of each segment that is decoded by the motion decoding unit. A moving means for moving the segment, a projection means for obtaining a projection image obtained by projecting each segment at the position moved by the moving means on a two-dimensional screen, and a position of each feature point of each segment in the projection image. A deforming unit that moves based on the displacement of each feature point of each segment decoded by the motion decoding unit; and an image combining unit that combines images based on the shape of each segment deformed by the deforming unit, And a moving picture decoding apparatus that decodes an encoded continuous moving picture.