JP3208879B2 - MOVING IMAGE ANALYZING APPARATUS AND METHOD, AND MOVING IMAGE SYNTHESIS APPARATUS AND METHOD THEREOF - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving picture analyzing apparatus used in the fields of image recording, communication and the like, and a method thereof.
The present invention relates to a moving image synthesizing apparatus and a method thereof .

[0002]

2. Description of the Related Art In recent years, as a method of analyzing and synthesizing an image signal or the like, a method has been reported which focuses on a signal edge such as a boundary between an object and a background in the image signal and synthesizes an original signal from information at the signal edge. ing. For example, J
acques Fromment and Stephane M
Second generation by allet
compact image coding wit
h wavelet. (Wavelets-Tutor
ial in Theory and Applica
tions, II: 655-678, 1991) (Reference Document 1) is one of the above-mentioned methods, and this method will be described below.

FIG. 15 is a diagram showing a one-dimensional signal f (x). First, as shown in FIGS. 15A and 15B,
Assume a one-dimensional signal f (x). In this example, the signal f (x) is first analyzed using the multi-resolution method. Here, the multi-resolution method refers to a method of analyzing a signal using a plurality of filters having different resolutions. The characteristic of the analysis filter used for the analysis using the multi-resolution method is represented by _a function φ _a (x). The function φ _a (x) is expressed by the following equation in the frequency domain ω.

(Equation 1)

Here, a of the function φ _a (x) represents a scale at multiple resolution. Also, i in Equation 1 represents an imaginary number. ＾ represents the Fourier transform of the function with this symbol (the same applies hereinafter). This is also in the frequency domain ω
Is a differential result of the function θ (x) in the original region x of the function represented by the following equation.

(Equation 2)

The filter φ _a (x) and the signal f (x)
Is expressed as _a function Waf (x). The amplitude in the reference _{1 | W a f (x)} | of the filter output W _a position which gives the maximum value _(a x _n) (where n is a natural number)
It showed to be able to approximate the original signal using only f _(a x _n). Before describing the restoration from the maximum value, W _a f (x)
From the original signal f (x).

The method described here is called a wavelet and is a method that has been actively studied in recent years. If the analytic function is

(Equation 3) The filter φ _j ^* (x) used for the synthesis is represented by the following equation.

(Equation 4)

Here, the line on the function represents the complex conjugate. The scale a is selected so that a = 2 ^j . Where j = j1,. . . , JJ. Here, assuming that the original signal is f _o (x) in terms of analysis and synthesis,

(Equation 5)

(Equation 6) As

(Equation 7) Where j = j1,. . . , JJ are analyzed,

(Equation 8)

(Equation 9) Synthesized by Thus, W _j f (x) and s _(jJ)
The original signal f _o (x) can be restored from (x).

Accordingly, it is possible to obtain an approximation of f in inverse transform that approximates the W _j f (x) by regarding complement the previously mentioned W _j f (x _n) from the 'W _j f (x) described above it can.
The method of complementation is to convert the function obtained by complementation into W _j f (x).
Then

(Equation 10) Function

[Equation 11] By passing through two points _j x _n and _j x _{(n + 1)}

(Equation 12)

(Equation 13) Further, α and β in the above equations are obtained, these are substituted into Equation 11, and the interpolation function is added to the interpolation function to update the interpolation function.

[0009] This obtains an estimate f of the signal f (x) ^'(x) using equation 8, it W _j f using Equation 7 again
(X). By repeating this, f
^' Update (x) and restore f (x). Finally,
Using equation 9, f _o (x) is obtained.

[0010]

In addition to the image compression method described above, many image compression methods have been considered, but there is a demand that the image quality and the compression ratio must be further improved. In order to meet this demand, attempts have been reported to faithfully reproduce a two-dimensional image in consideration of the fact that the visual characteristics are sensitive to sharply changing edges.

In a moving image, two-dimensional spatial signals can be regarded as a three-dimensional spatio-temporal image by arranging them in the time direction. The human visual characteristics also detect the motion in the time direction of the spatiotemporal image in parallel with the edge detection in the normal spatial direction.

In particular, a sensitive response to moving edges has been observed in nerve cells that control visual information processing. Thus, it is suggested that moving edges are important just as spatial edge detection was important to vision.
Normally, in image compression using motion detection, motion detection is performed by some method, prediction is performed in the time direction using the method, and the error is encoded. However, regarding a moving image, an image analysis (compression) method and a synthesis (reproduction) method for faithfully reproducing the visual characteristic in consideration of the fact that the visual characteristics are sensitive to an edge where luminance changes suddenly have not yet been reported. Absent.

According to the present invention, in order to meet the above-mentioned technical demands, a moving image that faithfully reproduces a portion of moving image information in consideration of the fact that the visual characteristic is sensitive to an edge whose luminance changes rapidly. The aim is to obtain an analysis and synthesis method and its device.

[0014]

In order to achieve the above object, a moving picture analyzing apparatus according to a first aspect of the present invention provides a moving picture analyzing apparatus in which two-dimensional images are continuous in time series. This is a moving image analyzer that analyzes a positional change of image information on an image plane and a temporal change of image information on a time series, and compresses a feature point of the analysis result, and has the following means.

That is, using the first smoothing function having the two-dimensional position and time on the image plane as parameters, the convolution operation is performed with the image information of the original moving image at a predetermined resolution, and the low-frequency component of the moving image is obtained. Is calculated,
The low-frequency component is subtracted from the image information of the moving image, and the first smoothing function is differentiated once with respect to the two-dimensional position and time, respectively, by using three second smoothing functions. Analysis means for performing a convolution operation with the image information whose components have been reduced to calculate a result of the multi-resolution analysis; feature point detection means for detecting a feature point at which the analysis result is maximum or minimum; The position information of a point on the three-dimensional spatio-temporal specified by the two-dimensional position and time is classified into a set of isolated points, a set of lines, or a set of surfaces, and the feature classified as the line or the surface Position information compression means for compressing the position information of a point based on a function of predetermined position information defined corresponding to the line or the curved surface, and feature points classified as the line or the curved surface Analysis result compression means for compressing based on a function of a predetermined analysis result defined corresponding to the line or the curved surface, position information and analysis results of the feature points classified as the isolated points, And encoding means for encoding the compressed position information of the feature points classified as the line or the curved surface and the compressed analysis result.

According to the moving image analyzer according to the second aspect of the present invention, in the analyzer, the first smoothing function using the two-dimensional position and the time on the image plane as parameters is used, and The convolution operation is performed with the image information of the original moving image at the resolution of, and the low frequency component of the moving image is calculated. The calculated low frequency component is subtracted from the image information of the driving image. Also, three second smoothing functions obtained by differentiating the first smoothing function once with respect to the two-dimensional position and time, respectively, are used to obtain image information in which the low-frequency component is reduced at a plurality of resolutions. A convolution operation is performed. This allows
The result of the multi-resolution analysis is calculated.

The feature point detecting means detects a feature point at which the analysis result of the multi-resolution analysis by the analyzing means becomes maximum or minimum.

In the position information compressing means, the position information of the feature points detected by the feature point detecting means on the three-dimensional spatio-temporal space specified by the two-dimensional position and time is represented by a set of isolated points, a line, Or a set of curved surfaces. Then, the position information of the feature points classified into the line or the curved surface is compressed based on a function of predetermined position information defined corresponding to the line or the curved surface.
The analysis result compression means compresses the analysis result on the feature point classified as the line or the curved surface based on a function of a predetermined analysis result defined corresponding to the line or the curved surface.

In the encoding means, the position information and analysis result of the feature point classified as the isolated point, the compressed position information of the feature point classified as the line or the curved surface, and the compressed analysis result are obtained. The result is encoded.

The feature point detecting means uses the three second smoothing functions obtained by differentiating the first smoothing function twice with respect to the two-dimensional position and time, respectively, and uses the low-frequency component at a plurality of resolutions. The convolution operation may be performed with the image information in which is reduced, and the feature point may be detected based on the zero cross point in the result of the convolution operation.

Alternatively, the feature point detecting means may include
The feature point may be detected based on a point at which the sum of squares of the analysis results of each dimension in the dimension gives an extreme value.

Further, the position information compressing means converts the function of the position information corresponding to the line as a function of the position information of each dimension with respect to a length from a predetermined starting point on the line.
It may be defined in each dimension of the dimension space-time.

Alternatively, the position information compressing means defines a function of the position information corresponding to the line as a function of a curvature with respect to a length from a predetermined starting point on the line and a function of a torsion ratio with respect to the length. May be.

The position information compressing means may include a distance from the curved surface to a first plane defined by a time dimension and a first dimension of the image plane in the three-dimensional space within a predetermined minute range. The first curved surface to be divided is divided, and a distance from a second plane formed by a time dimension and a second dimension of the image plane in the three-dimensional space from the remaining curved surface after the division is within a predetermined minute range. The second surface included is divided, and the operation of dividing the first surface and the second surface is sequentially repeated for the remaining surfaces after the division, so that the surface is divided into a plurality of the first surfaces and the plurality of first surfaces. Dividing into a plurality of the second curved surfaces, the function of the position information corresponding to the curved surface is converted from the first plane with respect to the position information on the first plane corresponding to each of the first curved surfaces. Functions of distance and the respective second songs The corresponding, may be defined as a function of the distance from the second of said second plane relative to the position information on the plane.

Alternatively, the position information compressing means performs a multi-resolution analysis on a function of the curvature with respect to a distance from a predetermined starting point on the cutting line, which is defined for each cutting line formed by cutting the curved surface at the image plane. Detecting a singular point of the analysis result, and from the detected singular point, a plurality of curves formed by connecting singular points whose distances between adjacent singular points on the adjacent image plane are within a predetermined range. May be extracted, and the function of the position information corresponding to the curved surface may be defined as a function of a curvature with respect to a distance from a predetermined starting point on the plurality of extracted curves and a function of a torsion rate with respect to the distance.

According to a moving image analysis method according to a second aspect of the present invention, a moving image in which a two-dimensional image is continuous
A moving image analysis method for analyzing a positional change of image information on an image plane of each two-dimensional image and a temporal change of image information on a time series, and compressing characteristic points of the analysis result, comprising: Perform processing.

That is, using the first smoothing function having the two-dimensional position and time on the image plane as parameters, the convolution operation is performed on the image information of the original moving image at a predetermined resolution, and the low-frequency component of the moving image is obtained. Is calculated,
The low-frequency component is subtracted from the image information of the moving image, and the first smoothing function is differentiated once with respect to the two-dimensional position and time, respectively, by using three second smoothing functions. A convolution operation is performed with the image information whose components have been reduced, a result of the multi-resolution analysis is calculated, a feature point at which the analysis result is a maximum or a minimum is detected, and the two-dimensional position and the detected feature point are detected. The position information on the three-dimensional space-time specified by time is
Classify into a set of isolated points, a set of lines, or a set of surfaces,
The position information of the feature point classified into the line or the curved surface is compressed based on a function of predetermined position information defined corresponding to the line or the curved surface, and the feature point classified into the line or the curved surface The above analysis result is compressed based on a function of a predetermined analysis result defined corresponding to the line or the curved surface, and the position information and analysis result of the feature point classified as the isolated point, and the line or The compressed position information of the feature points classified into the curved surface and the compressed analysis result are encoded.

According to the moving image analysis method according to the second aspect of the present invention, the first moving image having a predetermined resolution is used by using the first smoothing function having the two-dimensional position on the image plane and time as parameters. And the convolution operation is performed to calculate the low-frequency component of the moving image. The calculated low frequency component is subtracted from the image information of the driving image. Further, the first smoothing function is differentiated once with respect to the two-dimensional position and time, respectively.
Using the two second smoothing functions, the convolution operation is performed on the image information in which the low-frequency components have been reduced at a plurality of resolutions, and the result of the multi-resolution analysis is calculated.

Next, a feature point at which the analysis result of the multiresolution analysis is maximum or minimum is detected.

Next, the above-mentioned 2
Position information on the three-dimensional spatio-temporal space specified by the dimensional position and time is classified into a set of isolated points, a set of lines, or a set of curved surfaces. The position information of the feature points classified into the line or the curved surface is compressed based on a function of predetermined position information defined corresponding to the line or the curved surface. The analysis result on the feature points classified into the line or the curved surface is compressed based on a function of a predetermined analysis result defined corresponding to the line or the curved surface.

Next, the position information and analysis result of the feature point classified as the isolated point, and the compressed position information and the compressed analysis result of the feature point classified as the line or the curved surface are encoded. You.

The feature points are detected by using three second smoothing functions obtained by differentiating the first smoothing function twice with respect to the two-dimensional position and time, respectively, and using the low frequency component at a plurality of resolutions. The convolution operation may be performed with the image information in which is reduced, and the feature point may be detected based on the zero cross point in the result of the convolution operation.

Alternatively, the feature point may be detected based on a point at which the sum of squares of the analysis results in each dimension of the three-dimensional space gives an extreme value.

The compression of the position information is performed by converting a function of the position information corresponding to the line into a function of the position information of each dimension with respect to a length from a predetermined starting point on the line. May be defined.

Alternatively, in the compression of the position information, a function of the position information corresponding to the line is defined as a function of a curvature with respect to a length from a predetermined starting point on the line and a function of a torsion ratio with respect to the length. May be.

In the compression of the position information, the distance from the curved surface to the first plane formed by the time dimension and the first dimension of the image plane in the three-dimensional space is included in a predetermined minute range. The first curved surface to be divided is divided, and a distance from a second plane formed by a time dimension and a second dimension of the image plane in the three-dimensional space from the remaining curved surface after the division is within a predetermined minute range. The second surface included is divided, and the operation of dividing the first surface and the second surface is sequentially repeated for the remaining surfaces after the division, so that the surface is divided into a plurality of the first surfaces and the plurality of first surfaces. Dividing into a plurality of the second curved surfaces, a function of position information corresponding to the curved surface is assigned to each of the first curved surfaces.
A function of the distance from the first plane to the position information on the first plane corresponding to the curved surface, and the position information on the second plane corresponding to each of the second curved surfaces. It may be defined as a function of the distance from the second plane.

Alternatively, the position information may be compressed by multi-resolution analysis of a curvature function with respect to a distance from a predetermined starting point on the cutting line, which is defined for each cutting line formed by cutting the curved surface at the image plane. Detecting a singular point of the analysis result, and from the detected singular point, a plurality of curves formed by connecting singular points whose distances between adjacent singular points on the adjacent image plane are within a predetermined range. Extract
The function of the position information corresponding to the curved surface may be defined as a function of a curvature with respect to a distance from a predetermined start point on the plurality of extracted curves and a function of a torsion rate with respect to the distance.

[0038] A moving image dynamics apparatus according to a third aspect of the present invention comprises a first smoothing using two-dimensional position and time of a moving image in which two-dimensional images are successive in time series on a two-dimensional image plane as parameters. Using a function, image information of the original moving image is convolved with the image information at a predetermined resolution, the low-frequency component of the moving image is calculated, and the low-frequency component is subtracted from the image information of the original moving image. Using the three second smoothing functions obtained by differentiating the one smoothing function once with respect to the two-dimensional position and time, the convolution operation is performed with the image information in which the low-frequency component is reduced at a plurality of resolutions. A result of the multi-resolution analysis is calculated, a feature point at which the analysis result is maximum or minimum is detected, and the detected feature point is designated by the two-dimensional position and time. Position information on the three-dimensional space-time is classified into a set of isolated points, a set of lines, or a set of surfaces, and the position information of the feature points classified into the line or the surface corresponds to the line or the surface. The analysis result on the feature point classified into the line or the curved surface is compressed based on the function of the predetermined position information defined as above, and the function of the predetermined analysis result defined corresponding to the line or the curved surface is obtained. The position information and the analysis result of the feature point classified as the isolated point and the compressed position information and the compressed analysis result of the feature point classified as the line or the curved surface are encoded based on A moving image synthesizing apparatus for synthesizing a moving image from a compressed signal generated and generated, comprising the following means.

That is, a function of the position information is reconstructed based on the decoding means for decoding the coded compressed signal and the compressed position information reproduced by the decoding. Position information restoring means for restoring position information of a feature point on the function; and restoring a function of the analysis result based on the compressed analysis result reproduced by the inverse encoding. An analysis result restoring means for restoring the analysis result of the point, and complementing the analysis result on the three-dimensional space-time based on the position information and the analysis result of the restored feature point; The convolution operation is performed on the analysis result to which the low frequency component has been added by using the inverse transform function of the second smoothing function, and the operation result at the plurality of resolutions is added. And an image information restoring means for restoring the image information on space-time three-dimensional above with.

According to the moving picture synthesizing apparatus according to the third aspect of the present invention, the coded compressed signal is decoded by the decoding means.

In the position information restoring means, the function of the position information is restored based on the compressed position information reproduced by the inverse encoding, and the position information of the feature point on the function is restored. You.

In the analysis result restoring means, the function of the analysis result is restored based on the compressed analysis result reproduced by the inverse encoding, and the analysis result of the feature point on the function is restored. You.

In the image information restoring means, the analysis result on the three-dimensional spatiotemporal space is complemented based on the restored position information of the characteristic points and the analysis result. The low frequency component is added to the complemented analysis result. Next, a convolution operation is performed with the analysis result to which the low frequency component has been added by using an inverse transformation function of the second smoothing function. The convolution operation results at the plurality of resolutions are combined to restore the three-dimensional spatio-temporal image information.

The position information restoring means, as a function of position information of each dimension with respect to a length from a predetermined starting point on the line, corresponds to the line defined in each dimension of the three-dimensional space-time. The function of the obtained position information may be restored.

Alternatively, the position information restoring means is defined as a function of a curvature with respect to a length from a predetermined starting point on the line and a function of a torsion rate with respect to the length.
The function of the position information corresponding to the line may be restored.

Further, the position information restoring means is configured to calculate a distance from the first plane with respect to position information on a first plane formed by the time dimension in the three-dimensional space and the first dimension of the image plane. A function defined as a function of a distance from the second plane to position information on a second plane defined by a time dimension in the three-dimensional space-time and a second dimension of the image plane; The function of the corresponding position information may be restored.

Alternatively, the position information restoring means includes a function of a curvature with respect to a distance from a predetermined starting point on a plurality of curves;
And a function of the position information corresponding to the curved surface defined as a function of the torsion rate with respect to the distance is restored, and a plurality of curves based on the restored function can be cut by the image plane in the three-dimensional space-time. A function of curvature with respect to a distance from a predetermined starting point on the cutting line may be restored from a point on the cutting line, and position information of a feature point on the function may be restored.

A moving image synthesizing method according to a fourth aspect of the present invention is a first smoothing method using a two-dimensional position and time on a two-dimensional image plane of a moving image in which two-dimensional images are successive in time series as parameters. Using a function, image information of the original moving image is convolved with the image information at a predetermined resolution, the low-frequency component of the moving image is calculated, and the low-frequency component is subtracted from the image information of the original moving image. Using the three second smoothing functions obtained by differentiating the one smoothing function once with respect to the two-dimensional position and time, the convolution operation is performed with the image information in which the low-frequency component is reduced at a plurality of resolutions. A result of the multi-resolution analysis is calculated, a feature point at which the analysis result is maximum or minimum is detected, and the detected feature point is designated by the two-dimensional position and time. Position information on the three-dimensional space-time is classified into a set of isolated points, a set of lines, or a set of surfaces, and the position information of the feature points classified into the line or the surface corresponds to the line or the surface. The analysis result on the feature point classified into the line or the curved surface is compressed based on the function of the predetermined position information defined as above, and the function of the predetermined analysis result defined corresponding to the line or the curved surface is obtained. The position information and the analysis result of the feature point classified as the isolated point and the compressed position information and the compressed analysis result of the feature point classified as the line or the curved surface are encoded based on This is a moving image synthesizing method for synthesizing a moving image from a compressed signal generated by performing the following processing.

That is, the encoded compressed signal is inversely encoded, and the function of the position information is restored based on the compressed position information reproduced by the inverse encoding.
Restore the position information of the feature points on the function, restore the function of the analysis result based on the compressed analysis result reproduced by the inverse encoding, and analyze the analysis result of the feature point on the function. Restoring, complementing the analysis result on the three-dimensional space-time based on the restored position information of the feature points and the analysis result, adding the low-frequency component to the complemented analysis result, and performing the second smoothing. Using the inverse transformation function of the function, a convolution operation is performed with the analysis result to which the low-frequency component has been added, and the operation results at the plurality of resolutions are combined to restore the image information on the three-dimensional space-time.

According to the moving picture synthesizing method according to the fourth aspect of the present invention, the encoded compressed signal is inversely encoded.

Next, the function of the position information is restored based on the compressed position information reproduced by the inverse encoding, and the position information of the feature point on the function is restored. Further, a function of the analysis result is restored based on the compressed analysis result reproduced by the inverse encoding, and an analysis result of a feature point on the function is restored.

Next, the analysis result in the three-dimensional spatiotemporal space is complemented based on the restored position information of the feature point and the analysis result, and the low-frequency component is added to the complemented analysis result. Next, using the inverse transform function of the second smoothing function, a convolution operation is performed with the analysis result to which the low frequency component has been added. The result of the convolution operation at the plurality of resolutions is synthesized, and the image information on the three-dimensional space-time is restored.

The restoration of the position information is performed as a function of the position information of each dimension with respect to the length from the predetermined starting point on the line, and the position information corresponding to the line defined in each dimension of the three-dimensional spatiotemporal space. The function of the obtained position information may be restored.

Alternatively, the position information can be restored by defining a function of the curvature with respect to the length from a predetermined starting point on the line and a function of the position information corresponding to the line, which is defined as a function of the torsion rate with respect to the length. May be restored.

Further, the position information is restored by calculating a distance from the first plane to position information on a first plane formed by a time dimension in the three-dimensional space-time and a first dimension of the image plane. A function defined as a function of a distance from the second plane to position information on a second plane defined by a time dimension in the three-dimensional space-time and a second dimension of the image plane; The function of the corresponding position information may be restored.

Alternatively, the position information can be restored by defining a function of the curvature with respect to a distance from a predetermined starting point on the plurality of curves and a function of the position information corresponding to the curved surface defined as a function of the torsion rate with respect to the distance. From a point on a cutting line that can be cut by the image plane in the three-dimensional spatiotemporal space from a point on a cutting line at which a plurality of curves based on the restored function are restored. Then, the position information of the feature point on the function may be restored.

[0057]

The change of the image information distributed in the three-dimensional space is analyzed, and the original moving image is approximated and restored using the image information at the characteristic point of the change obtained by the analysis.
An edge surface in the three-dimensional space of the image information is detected, and the image information is analyzed at points (edge points) on the edge surface,
Using this analysis result, the image information is synthesized by interpolating the image information of points other than the edge curved surface, and the original image information is restored.

The compression of the image information of the edge points of the image is performed by expressing a geometric structure such as a curved surface or a curve by a parametric function expression and analyzing the function.

[0059]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A first embodiment of a moving picture analyzing and synthesizing method and apparatus according to the present invention will be described below. The moving image analysis and combination method and apparatus of the present invention do not focus on only the image information (luminance signal, color difference signal, etc.) in a single image frame for the moving image information.
The image information is regarded as being distributed in a three-dimensional space formed by the image plane and the time axis of a continuous image frame, and the moving image information is analyzed and synthesized.

That is, the change of the image information distributed in the three-dimensional space is analyzed, and the original moving image is approximated by using the image information at the characteristic point of the change obtained by the analysis.
Restore. In the first embodiment, an edge curved surface in a three-dimensional space of image information is detected, image information is analyzed at a point (edge point) on the edge curved surface, and the analysis result is used by using the analysis result.
The image information is synthesized by interpolating the image information of points other than the edge curved surface, and the original image information is restored.

The position information of the edge point of the image is naturally encoded. The encoding (compression) of the position information of the edge points of the image is performed by expressing a geometric structure such as a curved surface or a curve in a parametric function expression and analyzing the function.

Hereinafter, the configuration of the moving picture analysis / synthesis apparatus 1 of the present invention will be described with reference to FIG. In the first embodiment, a case will be described in which moving image information for a black and white F frame (F is an integer) is encoded for convenience of description.
FIG. 1 is a diagram illustrating a configuration of the moving image analysis and synthesis device 1.
In FIG. 1, a moving image memory 11 is a frame memory for recording information (data) for F frames of a black and white moving image (hereinafter, simply referred to as a moving image). The information change analysis unit 12 performs a three-dimensional analysis of a change in luminance of the moving image information stored in the moving image memory 11, and inputs the analysis result to the feature point detection unit 13.

The feature point detecting section 13 includes the information change analyzing section 12
Is processed and analyzed, and the location of the feature point is represented as a point in a three-dimensional space, and is input to the feature point encoding unit 14 and the information encoding unit 15. The feature point encoding unit 14 further processes the analysis result of the feature point detection unit 13 and expresses the location where the feature point exists as a curved surface.
The information encoding unit 15 performs an analysis for obtaining information necessary for reproducing the input moving image at the above-described feature points, and inputs the analysis result to the general encoding unit 16.
The general encoding unit 16 encodes output information of the feature point encoding unit 14 and the subtraction circuit 15. The above components constitute the moving image analyzer 10 of the present invention. It does not matter whether each of the above parts is configured by independent hardware, or configured by software on the same computer.

The general reproducing unit 21 reproduces output information of the feature point coding unit 14 and the information coding unit 15 by inversely coding the information coded by the general coding unit 16. The information reproducing unit 22 selects a feature point in a three-dimensional space based on the information on the position of the feature point reproduced by the general reproducing unit 21, arranges output information of the information encoding unit 15 at the position, and Input to the playback unit 23. By interpolating the image components at locations outside the feature points, the visually important parts of the original moving image where the luminance changes rapidly (in three-dimensional space) are faithfully approximated, and the other parts are approximated smoothly. become. The original image reproducing unit 23 combines and reproduces the original moving image by the convex projection method based on the information reproduced by the information reproducing unit 22. Each part of the overall reproduction unit 21 to the original image reproduction unit 23 constitutes the moving image synthesizing device 20 of the present invention.

The detailed operation of each part of the moving picture analysis / synthesis apparatus 1 will be described below. The moving image memory 11 stores image information for F frames from an image signal output device such as a camera or a video on the memory. Since this image information is associated with the position (coordinates x, y) on the image frame and the time (frame t in which the information exists), Io (x, y,
t). In other words, this image information (data)
Can be considered as information distributed in the three-dimensional space of the image plane and the time axis. This data is accessed for processing from the information change analysis unit 12 as original moving image data.

The information change analysis unit 12 includes a moving image memory 11
The three-dimensional change of the original moving image stored in the memory is analyzed. Specifically, a filtering process having directionality in each direction at multiple resolutions is performed. J = j1, in this multi-resolution scale sigma _j. . . , JJ. First, the DC components are independently analyzed. This DC component is applied to the general encoding unit 1
6 is added to the image signal that has been encoded, decompressed in the moving image synthesizing device 20 and synthesized again. This analysis shows that the window or smoothing function is
(X, y, t; σ _j ), and a function G (x, y, j) corresponding to the characteristic (j = jJ) of the filter having the coarsest resolution.
t; σ _jJ ).

That is, this filtering is represented by the following equation.

[Equation 14] Here, the symbol *** in the expression represents a three-dimensional convolution.
The signal obtained by subtracting this component is expressed by the following equation.

(Equation 15) The change of the moving image is analyzed using this signal. First, by an analysis filter having the characteristics defined by the following formula,
A convolution operation is performed on the image signal I (x, y, t).

(Equation 16)

[Equation 17]

(Equation 18) That is, the result of the convolution operation is represented by the following equation.

[Equation 19]

(Equation 20)

(Equation 21)

The feature point detecting section 13 includes the information change analyzing section 12
The location of the maximum value and the minimum value (extreme value) of the analysis result of is obtained. Specifically, the results of Expressions 19, 20, and 21 are differentiated in the same direction once again, and a point (zero cross point) where the value is 0 is obtained, thereby obtaining the position of the extreme value image data in the three-dimensional space. . In other words, this is the result of the analysis by the information change analysis unit 12 in each axis direction.
The zero crossing point of the output of the analysis filter in the form of performing the differentiation is obtained.

That is, the characteristics of an analysis filter of the type that performs twice differentiation are defined by the following equations.

(Equation 22)

(Equation 23)

[Equation 24]The image signal is analyzed by the analysis filter having the above characteristics.I
Filter (x, y, t)
Find the loss point. That is, the operation represented by the following equation is
Do.

(Equation 25)

(Equation 26)

[Equation 27]

[0070] obtained by the above operation, the filter output _{W xx I (x, y,} t; σ j); expressed as = 0 and becomes zero crossing point _{Px i (σ j x xi,} y xi, t xi) . However,
j = 1,. . . , _J N _x , and _j N _x zero cross points. Similarly, the filter output W _yy I (x,
y, t; σ _j ) = 0, the zero-cross point is Py _h (x
_yh , y _yh , t _yh ; σ _j ). Where h = 1, 2,
3,. . . , It is assumed that there is a _j N _y-number of zero-cross point _j N _y.

Similarly, the filter output W _tt (x,
y, t; σ _j ) = 0, the zero-cross point is _denoted by Pt _k ( x
_tk, y _tk, t _tk; a sigma _j) represents. Where k = 1,
2,. . . , It is assumed that there is a _j N _t pieces of zero-cross point _j N _t.

[0072] Hereinafter, for simplicity of notation, the zero-crossing point _{Px i (x xi, y xi} , t xi; σ j) is expressed as the Px _i (j). Similarly, the zero cross point Py _h (x _yh , y _yh ,
t _yh; σ _j) are expressed as Py _h (j). Similarly, the zero-cross point _{Pt k (x tk, y tk} , t tk; σ j) of Pt
_k (j). Feature point coding unit 14, the zero-crossing point obtained by the feature point detection unit 13 Px _i (j), the zero-crossing point Py _h (j), and zero cross point Pt _k a (j),
They can be considered as points on a curved surface, points on a curve, or isolated points. The curved, curves, and zero-crossing point by the parameter of the isolated point Px _i (j), the zero-crossing point Py _h (j), for encoding by representing the zero-cross point Pt _k (j).

First, the zero cross point Px _i (j), the zero cross point Py _h (j), and the zero cross point Pt _k (j)
If there is a point that overlaps, these are combined into one to create a feature point set P (j). That is, the feature point set P
(J) is the filter output W _xx (x, y, t; σ _j ) =
0 or filter output W _yy (x, y, t; σ _j )
= 0 or the filter output W _tt (x, y, t;
σ _j ) = 0 in a three-dimensional space where the value is zero.

The elements of the feature point set P (j) are
(J). However, the feature point set P (j) is such that the subscripts of the feature points Pp which are its elements are p = 1, 2,. . . Np, and has a total of Np feature points.

Next, some groups are created by connecting the feature points in a three-dimensional space. Here, “connect” means
Focusing on a certain feature point Pp (j), processing is performed on a cube composed of, for example, (3 × 3 × 3) pixels around the pixel, and a feature point set P (j) is similarly included in the cube.
If a feature point that is an element is included, the point belongs to the same group.

The elements of this group can be represented as two-dimensional curved surfaces, one-dimensional curves or points in a three-dimensional space. However, they may be attached to each other. These groups are divided into groups consisting only of point elements, curve elements, and curved surface elements.

[0077] Hereinafter, the scale sigma _j in this manner, gj = 1, 2,. . . , Mj groups will be described. In the gj-th group P _gj (j), the number of elements is printed as Q (gj). For convenience of explanation, a group consisting only of isolated points is represented by gj =
1,. . . M0 _j . Also, a group consisting only of curves is represented by gj = M0 _j + 1 ,. . . , M0 _j + M1 _j . Also, a group consisting of curved surfaces is represented by gj = M0 _j + M1
_j + 1,. . . , M0 _j + M1 _j + M2 _j .

The above symbols are arranged as follows. Hereinafter, the number of elements of a certain set P is represented as | P |. Feature point set P (j)
Is a set having the feature point P _p (j) on the scale σ _j as an element, and | P (j) | = Np.

The subset P _gj (j) is the gj-th subset on the scale σ _j , and | P _gj (j) | = Q (g
j). Gj = 1 in the subset P _gj (j)
,. . . , M0 _j are sets having isolated points as elements.
Gj = M0 _j +1 in the subset P _gj (j)
,. . . , M0 _j + M1 _j are sets having characteristic points constituting the curve as elements. G in the subset P _gj (j)
j = M0 _j + M1 _j +1,. . . , M0 _j + M1 _j + M
2 _j is a set having feature points constituting a curved surface as elements.

Further, the following symbols are further defined. The feature points that are the elements of the subset P _gj (j) are
_gj (j) _{Expressed as q} . Here, q = 1, 2,. . . , Q
(Gj). Since the feature point P _gj (j) _q is a point in a three-dimensional space, it has three values as position information of (x, y, t).

The three values of this feature point are represented by (P _gj (j) (x
q), P _gj (j) (yq), P _gj (j) (tq)). The order of q is a subset P _gj (j) belonging to the curve.
, It is assumed that q = 1 is the starting point and is arranged in the following order on the curve. Elements of the subset P _gj (j) belonging to the curved surface will be described later. There are various methods for expressing this element in a compact manner. Here, the following method is used.

A group consisting only of isolated points, g _j = 1
,. . . , M0 _j have the three-dimensional position as data as they are. FIG. 2 is a diagram illustrating an example of a curve in a three-dimensional space. FIG. 3 is a diagram in which the curve in the three-dimensional space shown in FIG. 2 is graphed using the length from the starting point as a parameter.

Group consisting of curves, gj = M0 _j +1
,. . . , M0 _j + M1 _j , and [x (l), y (l), t (l)] with the length l from the start point as a parameter as shown in FIGS. 3 (A), (B) and (C), respectively. The three lines are shown using a synthesis method supplemented from the multiresolution analysis results and their extreme values. That is, the required data is the singularity obtained from the analysis.

The necessary data for the above two cases are summarized below. In the case of an isolated point, a point (P _gj
(X1), P _gj (y1), P _gj (t1)) are encoded.

For curves, the following points are coded. (1) Starting point (P _gj (x1), P _gj (y1), P _gj (t
1)) (2) End point (P _gj (xQ (gj)), P _gj (yQ (g
j)), P _gj (tQ (gj))

The graph of the parameter x (l) is represented by P _gj (j)
In case of a _{(x) = f x (l} ), (3) Position l _nx singularities x (4) value at a position l _nx singularities _x f x (l _nx)

The graph of the parameter y (l) is expressed as P _gj (j)
When (y) = _fy (l), (5) the position l _{ny of the} y singular point (6) the value f _y (l _ny ) at the position l _ny of the y singular point

The graph of the parameter t (l) is represented by P _gj (j)
When (t) = f _t (l), (7) the position l _{nt of the} t singular point, and (8) the value f _t (l _nt ) at the position l _nt of the t singular point.

FIG. 4 is a diagram showing an example of a curved surface in a three-dimensional space. FIG. 5 is a diagram for explaining the processing for the curved surface shown in FIG. Group consisting of curved surfaces, g _j = M0
_j + M1 _j + 1,. . . , M0 _j + M1 _j + M2 _j , the elements of one group are further added to the xt plane and y-
y = f ₁ (x, t) and x = f ₂ as distances from the t plane
A compression technique similar to that for a two-dimensional still image is used by dividing it into a one-valent function (y, t). However, the original curved surface cannot be represented by the single-valued function as described above, and therefore requires some contrivance.

Therefore, in the first embodiment, as shown in FIG. 5A, first, at a coordinate (x, t), a point having a small y component is used to obtain y = f _y (x, t) ( A curved surface represented by 1) is formed. The points used here are removed from the original set.

As shown in FIG. 5B, a point having a small x at the coordinates (y, t) indicates that x = f
A curved surface represented by _x (y, t) (1) is formed, and all points are represented by sequentially repeating a process of removing points used from the original set.

P _gj forming one curved surface in this way
The element of (j) is composed of several curved surfaces y = f _y (x, t) (k
k) and _{x = f x (y, t} ) constitute the (kk). Here, kk = 1, 2,. . . , Kk (gj).

Since each curved surface can be considered in the same way as a still image, the same compression processing as that for a still image can be performed on this information. The method of this compression processing will be described later. By this compression processing, data necessary for forming the starting point and the curved surface is obtained.

The information encoding unit 15 analyzes information on a curved surface, a curve, or an isolated point constituting the above-mentioned feature point, and performs compression. The reason for performing compression on these is as follows.

That is, these feature points are points that give the extreme values of the information analyzed by the information change analysis unit 12, and the extreme values change only gently on the connected feature points. .

Specifically, the subset P _gj (j)
Is divided into an isolated point, a curve, and a curved surface, and the feature point encoding unit encodes the position (x, y, t) using the subset, but this time, the analysis result W _x I (x, y, t) Σ _j ), analysis result W _y I (x, y, t; σ _j ), analysis result W _t I
(X, y, t; σ _j ) is encoded by the same method.

First, for the isolated point, the analysis result W _x I (x, y, t; σ _j ) and the analysis result W _y I (x,
y, t; σ _j ) and the analysis result W _t I (x, y, t;
Let σ _j ) be information.

Also, as for the curve, the length l from the starting point is taken as a parameter, and the analysis result W _x I (x, y, t;
σ _j ), the analysis result W _y I (x, y, t; σ _j ), and the analysis result W _t I (x, y, t; σ _j ), and consider the compression of these three functions. As with the feature point coding described above, multi-resolution analysis and its extreme value compression are one example.

For the compression of point information on a curved surface, a division method expressed as a function of the distance from the xt plane and the yt plane used in the feature point analysis can be used.

That is, the analysis result W _x I (x, y, t; σ _j ) at the point y = f ₁ (x, t), and the analysis result W
_y I (x, y, t; σ _j ) and analysis result W _t I
Compression can be performed by treating (x, y, t; σ _j ) as three monovalent functions of (x, t).

Therefore, in the compression of information on a point on a curved surface, the curved surface is treated as three planes.
This can be achieved by compressing a two-dimensional image. For example, compression using multi-resolution analysis and interpolation from its extreme values can be considered. As described above, the analysis result is compressed on the feature points.

The general encoding unit 16 includes the feature point encoding unit 14
And the output data of the information encoding unit 15 is encoded.

The general encoding unit 16 includes the feature point encoding unit 14
Further, the redundancy of the output data of the information encoding unit 15 is compressed by, for example, run-length encoding or the like, and bit allocation or the like for quantization is performed. If necessary, encoding for error correction is performed. The result is transmitted and recorded.

The general reproducing section 21 performs the reverse operation of the general encoding section 16 on the input data. That is, the signal encoded by the general encoding unit 16 is returned to the output data of the feature point encoding unit 14 and the information encoding unit 15.

The information reproducing unit 22 restores the position of the feature point from the data equivalent to the output of the feature point encoding unit 14 obtained from the general reproducing unit 21 and the equivalent of the output of the information encoding unit 15 From the information, the positions of all the feature points in the xyt three-dimensional space and the information at the positions are restored.
That is, the reverse processing of the compression performed by the feature point encoding unit 14 and the information encoding unit 15 on the original image information is performed.

The isolated point information is the position information itself in the three-dimensional space. Therefore, the position of the isolated point can be immediately specified by the information of the isolated point.

Next, of the curve information, the start point information is the same as the isolated point, and represents the position of the curve start point as it is. Therefore, the position of the start point can be immediately specified by the information of the start point. The information of the other parts of the curve is the length l
Since three graphs of x (l), y (l) and t (l) are defined as shown in FIG. 2 and each of them is compressed as a one-dimensional signal, the reverse is performed.

In this example, multi-resolution analysis is performed.
Since the extremum is encoded, it is possible to reproduce the original one-dimensional graph from the extremum. this is,
Supplementary and Convex Projection (Convex)
(Projection) method.

On the other hand, since the result of the information change analysis unit 12 on this feature point is also encoded as a function of the same length l, this is also restored in the same manner as the feature point. From these results, the positions of the feature points grouped as a curve on the three-dimensional space and the values of the information change analysis unit 12 at those positions could be restored.

Finally, the feature points grouped as a curved surface and the result of the information change analysis unit 12 thereon are restored. The position of the feature point is represented as a distance from the xt plane, and the position information is compressed.

The compressed data is represented as a graph of the distance from the original xt plane. On the other hand, the same parameter x−
The result of the information change analysis unit 12 is regarded as three two-dimensional data using t, and the two-dimensional data is compressed.
This compressed data is also returned to the original three graphs. With the above processing, the position of the feature point represented by this graph and the result of the information change analysis unit 12 thereon can be restored.

Xt, y, as repeated at the time of encoding
As a graph expression using −t as a parameter, it is possible to restore the positions in the three-dimensional space of the feature points grouped as all the curved surfaces and the result of the information change part on the feature points. By the above operation, the positions of the feature points on the xyt three-dimensional space and the analysis results of the information change analysis unit 12 at the positions are all restored.

Based on the extreme value information of the analysis result of the information change analyzing unit 12 obtained by the above-described processing, the original image reproducing unit 23 performs supplementary approximation of the extreme value information, and performs three-dimensional x-y-t The result of the information change analysis unit 12 in the entire space is restored, and the inverse transformation of Expressions 19, 20, and 21 is performed on the result to obtain the original xyt
Is restored.

Here, the convergence is performed by repeating the interpolation and the inverse transformation several times using the convex projection method. The method of the interpolation and the inverse conversion will be described later. First, the supplementation from the feature point is (x,
The case where the operation is performed independently in the directions of (y, t) will be described.

That is, it is assumed that all y and all t in the x direction are yt one-dimensional data, and the analysis result W _x I (x, y, t;
Using σ _j ), a projection is performed so that the point becomes an extreme value by a method described later.

The same applies to the y direction and the t direction.
The obtained approximate data is inversely transformed by the method defined by Expression 32, and is again projected onto the original analysis space by the method defined by Expressions 19, 20, and 21.

By repeating this operation several times, the analysis result W _x I
(X, y, t; σ _j ), analysis result W _y I (x, y, t;
σ _j ) and converge on the analysis result W _t I (x, y, t; σ _j ). Finally, inverse transformation is performed to restore I (x, y, t). The inverse transformation can be performed by convolving the next filter with the analysis result.

[Equation 28]

(Equation 29)

[Equation 30] However,

(Equation 31) It is.

Here, 'G _x (u, v, w;
σ _j ), 'G _y (u, v, w; σ _j ),' G _t (u,
v, w; σ _j ) are G _x (u, v, w;
σ _j ), G _y (u, v, w; σ _j ), G _t (u, v,
w; σ _j ) represents the Fourier transform, and the horizontal line above the symbol in the equation represents the complex conjugate.

The signal I (x, y, t) is obtained by the following equation using the inverse conversion filter defined by the above equation.

(Equation 32) Finally, a DC component is added to the original moving image Io (x, y,
t) is obtained.

[Equation 33] In this way, the original image information Io (x, y, t) can be restored.

Hereinafter, an example of a method of compressing a one-dimensional signal as in the case where the characteristic points in the information encoding unit 15 form a curve, and a method of interpolation and inverse conversion of the one-dimensional signal in the information reproducing unit 22 will be described. explain. Hereinafter, the target one-dimensional signal is defined as f ₀ (x).

FIG. 6 is a diagram showing a configuration of an apparatus for encoding information regarded as a one-dimensional signal. In FIG. 6, a low-frequency detection circuit 51 performs a convolution operation for filtering a signal S _(jJ) (x) of a low-frequency component from an input signal f _o (x).

The primary differential analysis circuit 52 includes a subtraction circuit 15
Low-frequency component signal extracted from the original signal f _o (x) in the low frequency detection circuit 51 in S _(jJ) is a result of subtracting (x) signal f (x) (f (x ) = f o ( x) -S
_(jJ) (x)) is input, and the input signal is subjected to one-time differentiation to perform (first-order differential type) multi-resolution analysis. Here, the suffix jJ is an index indicating the largest scale of multiple resolution described later.

The secondary differential analysis circuit 53 outputs the signal f
(X) is input, and a multi-resolution (secondary differential type) analysis is performed in which the input signal is differentiated twice.

The characteristic point detecting section 54 uses the output signal G 1 _j f (x) of the primary differential analysis circuit 52 and the output signal G 2 _j f (x) of the secondary differential analysis circuit 53 to obtain the characteristic. specific determining (important) point _j X _n. However, here:
j = j1,. . . , JJ are the multi-resolution scale 2 ^j
J, and n = 1, 2, 3,. . . Is an index in which important points are arranged in ascending order.

The subtraction circuit 55 digitally subtracts a signal input from a signal indicated by (−) from a signal input from an input indicated by (+) in the figure. The output data of the subtraction circuit 55 may also be compressed, and the compressed data may be expanded on the information reproducing unit 22 side and input to the addition circuit 65.

FIG. 7 is a diagram showing the configuration of an apparatus for synthesizing a one-dimensional signal from information obtained by encoding information regarded as a one-dimensional signal. In FIG. 7, an interpolation estimating unit 61 calculates a signal G1 _jf (x) and a signal G2 _jf from encoded data.
(X) is supplementarily estimated.

[0127] inverse transformation unit 62, the signal G1 _j f (x) and the signal G2 _j f Hoseki data 'G1 _j f (x) and' G of (x)
Using 2 _j f (x), the original signal f (x) is inversely transformed into a restored signal f (x).

The primary differential analysis circuit 63 is a local encoder for obtaining supplementary data 'G1 _j f (x) by using iterative calculation. The second differential analysis circuit 64
This is a local encoder for obtaining interpolation data 'G2 _j f (x) using iterative calculation. Adder circuit 65
Adds two input signals by digital operation.

Outputs of the primary differential analysis circuit 63 and the secondary differential analysis circuit 64 are signals {G1 _j f (x)}, respectively.
And the signal {G2 _j f (x)} is returned to the interpolation estimating unit 61. After this operation is repeated several times, 'f (x) is finally obtained. Finally, the low frequency detection circuit 51
The output signal _{S (jJ) f o (x} ) is the final 'recovery signal added to f (x) by _{f o (x)' (x} ) is output.

The operation of the apparatus for encoding information regarded as a one-dimensional signal will be described below. The low frequency detection circuit 51
The low frequency component of the input signal signal f _o (x) is extracted and the signal S
_(jJ) (x) is input to the subtraction circuit 55 and the addition circuit 65.

Here, in the low frequency detection circuit 51,
Among the filters used for signal analysis by the multi-resolution method, a convolution operation of a smoothing filter having the same scale as that of the coarsest resolution is performed, and the low-frequency component S
_(jJ) (x) is detected. More specifically, the smoothing function is a Gaussian function G0 _(jJ) (x).

That is, this function G0 _(jJ) (x) is expressed by the following equation:

(Equation 34) If (σ _(jJ) = 2 ^jJ ), the low frequency component S _(jJ) (x) is expressed by the following equation.

(Equation 35) In the above equation, the sign * added to the equation indicates convolution.

The subtraction circuit 55 subtracts the signal S _(jJ) (x) from the signal f _o (x), and inputs the signal S _(jJ) (x) to the primary differential analysis circuit 52 and the secondary differential analysis circuit 53 as a signal f (x). I do.

The operation of the primary differential analysis circuit 52 will be described below. The primary differential analysis circuit 52 performs digital operation on the input signal f (x),
Multi-resolution analysis using a first-order differential type analysis filter G1 _j (x) is performed.

Here, the characteristic of the analysis filter G1 _j (x) is expressed by the following equation.

[Equation 36] The following calculation is performed using the analysis filter G1 _j (x).

(37) However, the calculation of (Equation 37) is performed for each resolution j = j1
,. . . , JJ.

Hereinafter, the operation of the secondary differential analysis circuit 53 will be described. The second-order differential analysis circuit 53 performs a digital operation on the input signal f (x),
Multi-resolution analysis using a second-order differential analysis filter G2 _j (x) is performed.

Here, the characteristic of the analysis filter G2 _j (x) is expressed by the following equation.

(38) Using the analysis filter G2 _j (x), the following calculation is performed.

[Equation 39] However, the calculation of (Equation 39) is performed for each resolution j = j1
,. . . , JJ.

The characteristic point detecting section 54 outputs the output signals G1 _{jf of the} primary differential analysis circuit 52 and the secondary differential analysis circuit 53.
These characteristic (important) points are obtained using (x) and the signal G2 _j f (x).

Here, the output signal G1 _j f (x) is simply described.
The sum of the squares of the signal G2 _j f (x) is obtained, and the maximum value is used as a feature point. The calculation represented by the following equation is performed.

(Equation 40)

Further, the differential value of the sum of squares E _j (x) is calculated, and the point at which the differential value E _j (x) ′ becomes zero is a feature point.
_j X _n . Analysis results of the analysis filters in the feature point _{j X n G1 j f (j} X n) and analysis G2
seeking _j f _(j X _n) input to the overall coding unit 16.

Hereinafter, the interpolation estimating section 61, the inverse transforming sections 62, 1
Second derivative analysis circuit 63 and second derivative analysis circuit 64
Will be described in a repeated calculation for signal restoration.

First, the interpolation estimating section 61 performs
More sent come each data, i.e. feature points _j X _n, analysis result Gd _{j (j} X _n), and feature data d
_(J X _n) with the original analysis result G1 _j f (x) and the analysis result G2 _j f (x) to Hoseki estimation.

As an example of the method used for the interpolation estimation, a convex projection method (Convex) is used.
Projection Method). First,
Request G1 _j f _(j X _n) and G2 _j f _(j X _n) using the analysis result in the feature point _{j X n Gd j f (j} X n) and the feature point data d _(j X _n).

[0144] However, (the feature point data d _(j X _n) =
1), the analysis result G1 _j f (x) is the analysis result G
d _j is _{f (j X n) (G1} j f (x) = Gd j f (
_j X _n)), the value of the analysis result G2 _j f _(j X _n) is 0 (G
It is _{2 j f (j X n)} = 0).

[0145] other than the above (characteristic point data d _(j X _n) =
0), the analysis result Gd _j f ( _j X _n ) is G 2 _j f
Equal to _{(x) (G2 j f (} x) = Gd j f
A _(j X _n)), the analysis result G1 _j f (the value of _j X _n) is _{0 (G1 j f (j X} n) is a = 0). The two sequences are Hoseki the analysis result G1 _j f _(j X _n) and the analysis result G2 _j f _(j X _n) independently.

FIG. 8 is a flowchart of a process in interpolation estimation performed by the interpolation estimating unit 61 and the like. The interpolation estimating unit 61 performs the interpolation according to the steps as shown in FIG. The outline of the processing is described below. In this process, using two constraints below represents the function interpolated estimation function with 'G1 _j f (x) and the function' G2 _j f (x).

Constraint 1: Function 'G1 _j f (x) and function' G2 _j f (x) are obtained by transforming a certain function (signal) by analysis filter G 1 _j (x) and analysis filter G 2 _j (x) It is. Constraint 2: function 'G1 _j f (x) and function' G2 _j f
(X) takes the value of the analysis result G1 _j f _(j X _n) or the analysis result _{G2 j f (j X n)} ( through).

The above restriction 1 is applied to the processing in the inverse transform section 62, the first-order differential analysis circuit 63, and the second-order differential analysis circuit 64. The second constraint is applied to the processing in the interpolation estimation unit 61.

The constraint 1 will be briefly described. Analysis filter G1 _j (x) (where j = j1,..., JJ
) Using a transformation G ₁ ,
When the inverse transform is expressed as a transform G ₁ ^-1 , the first constraint is to perform a transform G ₁ G ₁ ^-1 on the function 'G1 _jf (x).

Similarly, for the function 'G2 _j f (x),
If transforming a function (signal) using the analysis filter G2 _j (x) is expressed as a transformation G ₂ and an inverse transformation G ₂ ⁻¹ , the first constraint above is that the function 'G2 _j f ( This is equivalent to performing the conversion of G ₂ G ₂ ^-1 on x). This operation is referred to as a function 'G1 _j f (x) and a function' G2 _j f (x)
Do about. Specifically, the processing shown in Expression 51 described later is performed.

The interpolation estimator 61 and the inverse converter 6 described above
The overall processing of the second and first derivative analysis circuits 63 and 64 will be described below for each part. Hereinafter, the operation of the interpolation estimating unit 61 will be described. The above-described second constraint corresponds to the processing in the interpolation estimating unit 61.

First, an explanation will be given of the supplementation of the analysis result of the first-order differential type. The error function is defined by the following equation.

[Equation 41] In this error function e1 _j (x) is the interval between the point two feature _j X _n and the feature point _j X _{(n + 1),} is expressed by the following equation.

(Equation 42) Where A (j) is determined for each scale e1
A parameter relating to the smoothness of _j (x), for example, A (j) = 2 ^j or the like is used.

An error function e1 _j (x) is obtained for the two unknowns u1 and v1 using the following two equations.

[Equation 43]

[Equation 44]

Using the unknowns u1 and v1 obtained in this way, e 1 _j (x) is divided into the respective sections [ _j X _n , _j X
_{(n + 1)} ], and a new interpolation function is obtained by updating the previous interpolation function as shown in the following equation.

[Equation 45] Here,: = indicates an update.

For the interpolation function G2 _j f (x), a new interpolation function is similarly obtained. That is, the error function is defined by the following equation.

[Equation 46] In this error function e2 _j (x) is the interval between the point two feature _j X _n and the feature point _j X _{(n + 1),} is expressed by the following equation.

[Equation 47] Where A (j) is determined for each scale e1
A parameter relating to the smoothness of _j (x), for example, A (j) = 2 ^j or the like is used.

For the two unknowns u2 and v2, an error function e2 _j (x) is obtained using the following two equations.

[Equation 48]

[Equation 49]

The error functions u2, v2 thus obtained
, The error function e 2 _j (x) is calculated for each interval [ _j
X _n , _j X _{(n + 1)} ], and a new interpolation function is obtained by updating the interpolation function up to then as shown in the following equation.

[Equation 50] The updating process of the two types of interpolation functions described above is repeated to obtain a final interpolation function. This interpolation function is used by the inverse transform unit 62
Is input to

The operation of the inverse converter 62 will be described below.
The inverse transform unit 62 supplements the signal space with G ₁ and G ₂ by performing an inverse transform of the primary differential type and the secondary differential type analysis results, that is, by an inverse transform represented by the following equation. The signal f (x) obtained here is input to the primary differential analysis circuit 63 and the secondary differential analysis circuit 64.

(Equation 51)

The operation of the primary differential analysis circuit 63 and the secondary differential analysis circuit 64 will be described below. In the first-order differential analysis circuit 63 and the second-order differential analysis circuit 64, the processing represented by the following equation achieves the operation relating to the constraint 2 by the conversions G ₁ and G ₂ .

(Equation 52)

(Equation 53)

In FIG. 6, the operation performed by the interpolation estimating unit 61, the inverse transforming unit 62, the primary differential analysis circuit 63, and the secondary differential analysis circuit 64 is repeated.
Primary differential analysis circuit 63 and secondary differential analysis circuit 64
This indicates that the analysis result {G1 _j f (x)} and the analysis result {G2 _j f (x)}, which are the outputs of the above, have returned to the interpolation estimating unit 61.

The analysis result G1 _j f (x) and the analysis result G
Regarding 2 _j f (x), the above-mentioned equations 41 to 41
The processing indicated by 53 is repeated. By repeating this operation several times, the final signal 'f
(X) is obtained. This signal 'f (x) is subtracted from the subtractor 55
Is input to

In the addition circuit 65, the signal S _(jJ) (x) of the low frequency detection circuit 51 is added to the signal 'f (x) input from the inverse conversion section 62, and the original signal is obtained as shown in the following equation. Restore the signal f _o (x).

(Equation 54) Thus, the processing for restoring the signal f _o (x) is completed.

The processing in the interpolation estimation described above will be described below with reference to FIG. In FIG. 8, step 0
1 (S01), the interpolation estimating unit 61 sets the analysis result G
1 _j f (x), the analysis result G 2 _j f (x), and a constant LP that determines the number of times of the repetition processing are set.

In step 02 (S02), the interpolation estimating unit 61 calculates the function 'G1 _j f (x) and the function' G2 _j f
(X) is calculated and input to the inverse transform unit 62. Function 'G1
_j f (x) as a function 'G2 _j f (x) is the inverse conversion in the inverse conversion unit 62, is input to the first-order differential analysis circuit 63 and the second-order differential analysis circuit 64.

A variable lp for counting the number of processes
Is cleared to zero. In step 03 (S03),
The interpolation estimating unit 61 calculates the error function e1 _j (x) and the error function e2
_j (x) is calculated.

In step 04 (S04), the interpolation estimating unit 61 calculates unknowns u1, u2, v1, and v2. In step 05 (S05), the interpolation estimation unit 61
Updates the function 'G1 _j f (x) as a function' G2 _j f (x).

In step 06 (S06), the inverse converter 62 calculates the signal 'f (x). Step 07 (S
In 07), first-order differential analysis circuit 63 and the second-order differential analysis circuit 64 calculates the function _{{ 'G1 j f (x)} } a function _{{' G2 j f (x)} }, the interpolation estimation unit 61 input.

Hereinafter, a modification of the moving image analysis / synthesis apparatus 1 according to the first embodiment of the present invention will be described. In the embodiment, three-dimensional spatio-temporal data is analyzed to obtain a set of two-dimensional feature points and information on the feature points, and the two-dimensional data is analyzed to drop one dimension at a time, and the structure is analyzed. Eventually, it is reduced to point information.

However, as described in the embodiment, if the state can be expressed as an ordinary two-dimensional function or a one-dimensional function, compression by ordinary DCT or vector quantization is possible.

Although a one-time differential system of a smoothing function is used as a filter for analyzing three-dimensional data, the present invention is not limited to this. The same operation can be performed using the singular point of the result of the analysis filter if the inverse transformation exists or the approximate transformation exists even if any analysis filter is brought. In addition, a method of reducing the number of singularities by using two analysis filters as a pair can also be used in the analysis method of the present invention.

In the first embodiment, all the isolated points, curves, and surfaces as a three-dimensional analysis result are used. However, since important information exists only as a surface, one or both of the isolated points and the curves are discarded. Things are also possible. It is also conceivable to set a threshold on the length or area of the connection or on the magnitude of the analysis signal power thereover, and to use only important feature points.

In the first embodiment, the feature points of all scales are treated independently. However, only the structure at a relatively fine scale, for example, j = 2, is used, and other scales are analyzed on this feature point. It is also conceivable to apply compression using the result.

In the first embodiment, the information encoding unit performs encoding on the curve by analyzing the analysis result W _x I (x, y, t; σ _j ) and the analysis result W _y I (x, y, t; σ _j ), Analysis result W _t I
(X, y, t; σ _j )

[Equation 55]

[Equation 56]

[Equation 57] Expressed in polar coordinate format as represented by

The fact that this angle coincides with the normal direction of the curve or curved surface formed by the characteristic points is used to make MI (x, y, t;
σ _j ) only, AI (x, y, t; σ _j ) and B
I (x, y, t; σ _j ) may be obtained from a curve or a curved surface constituting a feature point. In the first embodiment, the error signal of the analysis / synthesis result is compressed by an appropriate compression method (for example, DCT).
Or vector quantization) improves image quality.

Hereinafter, a second embodiment of the present invention will be described. In the second embodiment, the feature point detecting unit 13, the feature point encoding unit 14, the information encoding unit 15, and the information reproducing unit 22 in the first embodiment are changed.

First, in the feature point detecting section 13, the following equation (5) is used.
5 are detected. Next, in the feature point encoding unit 14, the points giving the maximum value detected by the feature point detection unit 13 are connected in the same manner as in the first embodiment, and classified into isolated points, curves, and curved surfaces. As for the isolated point, the three-dimensional coordinates are used as necessary data as in the first embodiment. Regarding the curve, the starting point of the curve group obtained using the “analysis algorithm for curves in a three-dimensional space” described later and characteristic points of curvature and torsion are used as necessary data.

Next, the information encoding unit 15 encodes an analysis result at the point on an isolated point, a curve, or a curved surface constituted by the characteristic points as described below. First, at an isolated point, the analysis result W _x I (x, y, t;
σ _j ), analysis results W _y I (x, y, t; σ _j ), and analysis results W _t I (x, y, t; σ _j ) are required data.

Next, for the curve, as in the first embodiment, the analysis result W _x I (x, y, t; σ _j ) and the analysis result W _y are taken using the length l from the starting point as a parameter. I
(X, y, t; σ _j ) and the analysis result W _t I (x, y
The three graphs of y, t; σ _j ) are compressed as one-dimensional signals, and the compression results are used as necessary data.

Next, regarding the curved surface, the value M (x, y,
t; σ _j ), a function using the length s from the start point of each curve as a parameter is considered, and this is regarded as a one-dimensional signal and compressed. This compression result is used as necessary data.

The necessary data described above is input to the information reproducing unit 22 via the general encoding unit 16 and the general reproducing unit 20 as in the first embodiment. In the information reproducing unit 22,
First, the position of the feature point is reproduced from the data on the position of the feature point. Since data relating to the coordinates of the isolated transition is sent, the position of the isolated point can be obtained as it is.

With respect to the curve, the position of the characteristic point constituting the curve in the three-dimensional space is obtained by using the "algorithm for synthesizing the curve in the three-dimensional space" described later. With respect to the curved surface, obtaining a position in 3-dimensional space of the feature points by using the "algorithm of synthesis of song plane in the three-dimensional space", which will be described later, constituting a curved surface.

Next, the analysis result on the feature point is reproduced. In the case of an isolated point, since the data on the analysis result is sent as it is, the necessary analysis result is obtained as it is.

In the case of a curve, each analysis result is converted into a one-dimensional signal as a function of the length s from the start point, and is further compressed. By associating the value of the signal with the point of the length s, an analysis result on the characteristic points constituting the curve is obtained.

In the case of a curved surface, the value M (x, y, t; σ _j ) of equation 55 is considered on the group of curves constituting the surface, and it is compressed. M on
(X, y, t; σ _j ) is reproduced.

Next, at the reproduced feature point having the value of M (x, y, t; σ _j ), the unit normal vector of the curved surface is obtained, and the x component, y component and t component of this vector are determined by M (X, y, t; σ _j ) times the analysis result W _x I
(X, y, t; σ _j ), analysis result W _y I (x, y, t;
σ _j ) and the analysis result W _t I (x, y, t; σ _j ).

An algorithm for analyzing and synthesizing a curve in a three-dimensional space and an algorithm for synthesizing a curved surface in a three-dimensional space will be described below. Note that, for the sake of explanation, the description starts with an algorithm for analyzing and combining curves on a plane.

Here, how to represent a curve on a two-dimensional plane will be considered. First, a curve is set as a parameter s and 2
The position on the dimensional plane is

[Equation 58] And

Assuming that s is a distance moved from time 0 to t, and s = s (t), || p (s) || = |
| ∂p (s) / ∂s || = 1. On the other hand, when the unit tangent vector of p (s) in the s and e _1,

[Equation 59] The unit vector perpendicular to e ₁ (s) is e ₂ (s).
However, it is assumed that e ₁ (s) is rotated 90 degrees counterclockwise.

That is,

[Equation 60] given that,

[Equation 61] Can be expressed as

On the other hand, by differentiating the inner product e ₁ (s) · e ₁ (s) = 1,

(Equation 62) Therefore, from e ′ ₁ (s) · e ₁ (s) = 0,

[Equation 63] Can be expressed as This k (s) is called the curvature of the curve p (s) at s.

Using this, the curve p (s) is

[Equation 64]

[Equation 65] Can be expressed as on the other hand,

[Equation 66]

[Equation 67] And e ₂ (s) is

[Equation 68] From e ₁ (s).

Where | _x is its vector component,
| _Y indicates the y component. From Equations 65 to 68, the initial value p (s
₁ ), e ₁ (s ₁ ) and k (s) suggest that p (s) can be determined. Specifically, it is as follows.

An algorithm for analyzing a curve on a two-dimensional plane will be described below. FIG. 9 is a flowchart showing a process of analyzing a curve on a two-dimensional plane. In FIG. 9, in step 11 (S11), the starting point p (s _i )
And the unit tangent vector e at the starting point p (s _i )
_i (s _i ) as data.

In step 12 (S12), i = 1
Set as In step 13 (S13), the unit normal vector is converted to the unit tangent vector e _i (s _i ) by the following equation.
Ask from.

[Equation 69]

In step 14 (S14), s _{i + 1}
And k (s _i ).

[Equation 70] However, k (s) is solved as being constant in the section [s _{i + 1,} s _i ]. In step 15 (S15), the next point P (s
_{i + 1)} obtaining a unit tangent vector e ₁ (s i _{+ 1)} in the.

[Equation 71]

In step 16 (S16), i = i
Increment by +1. In step 17 (S17), it is determined whether or not there is data. If there is data, the process proceeds to S13.

Thus, k (s _i ), i = 1,
.., N are obtained on the curve. Conversely, it is shown that the original curve p (s _i ) can be restored using this k (s _i ) and the initial value.
It is assumed that s _i is now known at i = 1,..., N.

An algorithm for synthesizing a curve on a two-dimensional plane will be described below. FIG. 10 is a flowchart showing a process of combining curves on a two-dimensional plane. In FIG. 10, in step 21 (S21), initial values p (s ₁ ) and e ₁
(S ₁ ). In step 22 (S22),
Set i = 1.

In step 23 (S23), e
₂ (s _{i + 1} ) is obtained by the following equation.

[Equation 72] In step 24 (S24), k (s _i ) is read.

In step 25 (S25), p (s _{i + 1} ) is obtained by the following equation.

[Equation 73] In step 26 (S26), e
₁ (s _{i + 1} ) is obtained.

[Equation 74]

In step 27 (S27), i = i
Increment by +1. In step 28 (S28), it is determined whether or not there is data. If there is data, the process proceeds to step 23 (S23).

Actually, s _i does not have data. The problem can be solved by making k (s) so that s _{i + 1} −s _i is always constant by supplementing k (s _i ). In this case, it is possible to cope with the case where p (s _i ) is limited to the sub-pixel accuracy.

The curvature analysis and synthesis will be described below.
By the way, the method of calculating the curvature k (s) of the curve p (s) on the plane and using it to obtain the original curve p (s) has been described. According to the method already reported, the curvature can be multi-resolution analyzed to find its singularity, and the original curvature k (s) can be obtained from the information at the singularity.

That is, the curvature k (s) obtained by the above-described algorithm for analyzing a curve on a two-dimensional plane is defined as 1 with respect to s.
It is sufficient to analyze the extreme value by multi-resolution as a valence function, and store the point and the analysis result there. Of course, pairs can be used for analysis.

Here, analysis and synthesis of a curve in a three-dimensional space will be considered. However, this is for considering the expression of the curve on the curve. As will be described later, this result is used for a curve formed by connecting a singular point of a curved surface to another direction in a cutting curve (this becomes a curve on the curved surface).

The surface pp (u, v) = (x (u, v), y
Curve pl (s) = (x) on (u, v), t (u, v))
(S) _,, y (s), t (s)). Where the curve p
The unit tangent vector of l (s) is

[Equation 75] Then, the unit vector e ₂ (s) can be selected perpendicularly to this.

The relationship between e ₂ (s) and e ₁ (s) is linked by the curvature described later. Let the normal unit vector of this tangent plane be e
₃ (s). However,

[Equation 76] Here, x is a vector product.

In the case of a curve in space, the curvature k (s) is defined as follows.

[Equation 77] on the other hand,

[Equation 78] Is established. Also,

[Expression 79] The following relationship also holds.

This τ (s) is called a twist ratio. The following relation between these e _1, and e _2, e ₃ its differential.

[Equation 80] Given k (s) ≧ 0 and τ (s), if the initial values e ₁ (s ₁ ), e ₂ (s ₁ ), and e ₃ (s ₁ ) are known, e ₁ (s) , E ₂ (s) and e ₃ (s) can be uniquely determined. From the starting point pl (s ₁ ) and this result,

[Equation 81] Can be obtained as

In actuality, the curve pl (s) is calculated by dividing s = s _i by 3
Taylor expansion is performed to the next, and the following equation called Bouquet's formula is used.

(Equation 82) Therefore, in the following steps, s _i, k (s _i ),
τ (s _i ) can be obtained.

Algorithm for Analyzing Curves in Three-Dimensional Space FIG. 11 is a flowchart of a process for analyzing curves in a three-dimensional space. In FIG. 11, step 51 (S51)
, The starting points pl (s ₁ ) and e ₁ (s
₁ ) and e ₂ (s ₁ ) as data. Step 52
In (S52), i = 1 is set.

In step 53 (S53), e ₃ (s _i ) is obtained by the following equation.

[Equation 83]

In step 54 (S54), p (s
_{i + 1} ), and s _{i + 1,} k (s _i ), τ (s _i )
Ask for.

[Equation 84] However, k ′ (s) = 0 in the section [s _i , s _{i + 1} ].

In step 55 (S55), e ₁ (s _{i + 1} ) and e ₂ (s _{i + 1} ) are obtained by the following equations.

[Equation 85]

[Equation 86]

In step 56 (S56), i = i
Increment by +1. In step 57 (S57), it is determined whether or not there is data, and if there is data, the process proceeds to S53. Thus, k (s _i ), τ
(S _i ), i = 1, 2,..., N are obtained.

Hereinafter, the algorithm for synthesizing the curves in the three-dimensional space will be described. Conversely, k (s _i ), τ
From the initial values (s _i ), i = 1, 2,..., N, pl (s _i ) can be restored as follows.

FIG. 12 is a flowchart showing a process of synthesizing a curve in a three-dimensional space. 12, in step 61 (S61), initial values pl (s ₁ ) and e _{1 are set.}
(S ₁ ) and e ₂ (s ₁ ). Step 62 (S6
In 2), i = 1 is set.

In step 63 (S63), e ₃ (s _i ) is obtained by the following equation.

[Equation 87] In step 64 (S64), k (s _i ), τ (s
_i ) is read, and pl (s _{i + 1} ) is obtained from s _{i + 1,} k (s _i ), and τ (s _i ) using the following equation.

[Equation 88]

In step 65 (S65), e ₁ (s _{i + 1} ) and e ₂ (s _{i + 1} ) are obtained from the following equations.

[Equation 89]

[Equation 90] In step 66 (S66), i = i + 1 is incremented. In step 67 (S67), it is determined whether or not there is data.
Proceed to processing.

[0220] s _i do not have as data in practice. Use the same facts as for a curve on a two-dimensional plane, but also use k so that s _{i + 1} −s _i = const.
The problem can be solved by re-sampling (s) and τ (s).

Here, how to represent a curved surface existing in a three-dimensional space will be considered. However, the discussion will proceed with a structure encoding of a moving image instead of an arbitrary curved surface in mind. The actually assumed target is a curved surface formed by feature points analyzed at multiple resolutions, and is a three-dimensional space xyt in spatiotemporal space.
In this space,

[Equation 91] think of.

In the following outline, a curve on the xy plane formed when this curved surface is cut at a certain t and a singular point of the curvature of the curve are connected in the t direction to form a curve. It is something to be expressed. The singularity of the curvature of the cut curve of the curved surface, which appears when the curve group is cut at an arbitrary t, can be reproduced. A curved surface can be reconstructed by connecting these.

First, one surface p (u,
Consider v). When this is cut at t = t _i , a cutting curve of one or more curved surfaces is obtained. Since each curve is a curve on the xy plane, it can be reduced to the initial value of the starting point and the singular point information of the curvature using the method described in the previous section. If it becomes a closed curve, consider an appropriate place as the starting point,
Later, one of the singularities of curvature is set as a starting point. The start point, end point, and singular point of the cut curves are noted.

Next, the same operation is performed at t = t _{i + 1} . The feature points t = t _i and t = t _{i + 1} thus formed are connected. When connecting, t = t centering on a certain feature point at t = t _i
A neighborhood is set at _{i + 1} , and if there is a feature point there, it is connected; otherwise, it is set as the end point. On the other hand, because some newly emergent singularity t = t i + _1, singular point t = t i _{+ 1} that were not bound by the above operation and starting it.

The curves are formed on the curved surface in this manner. Obviously, the singularity of the curvature on the cutting curve obtained by cutting these curves at an arbitrary t can be reproduced. it can. Hereinafter, the following steps are specifically used.

An algorithm for analyzing a curved surface in a three-dimensional space will be described below. FIG. 13 is a flowchart showing a process of analyzing a curved surface in a three-dimensional space. In FIG. 13, in step 71 (S71), the curved surface is cut at t = t1, and each of the resulting curves is subjected to the above-mentioned 2
Analysis is performed using an algorithm for analyzing a curve on a dimensional plane.

In step 72 (S72), the appearing start point and singular point are stored as the start point of the curve in the t direction. In step 73 (S73), i = 1 is set.

In step 74 (S74), t = t
Each curve formed by cutting the curved surface at _{i + 1} is analyzed using an algorithm for analyzing a curve on a two-dimensional plane.

In step 75 (S75), t = t
_The starting point or singularity appearing at _i and the starting point or singularity at t = t _{i + 1} are checked, and the following processing is performed. (A) If near, connect. (B) If there is something that newly appeared at t = t _{i + 1} instead of the neighborhood, store it as the starting point. (C) Store an existing point as an end point when t = t _i , not in the vicinity.

In step 76 (S76), i = i
Increment by +1. In step 77 (S77), it is determined whether or not there is data. If there is data, the process proceeds to S74.

The curve in the t direction thus formed is represented by 3
The analysis is performed using an algorithm for analyzing a curve in a dimensional space.

The following describes an algorithm for synthesizing a curved surface in a three-dimensional space. Conversely, when combining curved surfaces, the following steps are performed. FIG. 14 is a diagram illustrating a flowchart of a process of synthesizing a curved surface in a three-dimensional space.

In FIG. 14, step 81 (S81)
In, the curve formed in the t direction is restored using an algorithm for combining curves in a three-dimensional space. This allows
Starting point and the singularity of the cutting curve of the curved surface in all t = t _i is determined. In step 82 (S82), i
= 1.

In step 83 (S83), t = t
Restore all cut curves from the starting point and singular point of the curve cut by _i . However, this processing uses an algorithm for synthesizing a curve on a two-dimensional plane. Step 84 (S8
In 4), i = i + 1 is incremented. In step 85 (S85), it is determined whether there is data. If there is data, the process proceeds to S83.

In addition to the embodiments described above, the moving picture analyzing and synthesizing method and the apparatus of the present invention can have various configurations as shown in, for example, modifications. The embodiments described above are examples.

[0236]

As described above, according to the present invention, in consideration of the fact that the visual characteristics are sensitive to edges whose luminance changes rapidly, the moving image information is faithfully reproduced with respect to the moving image information. And a synthesis method and an apparatus therefor.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration of a moving image analysis / synthesis apparatus.

FIG. 2 is a diagram illustrating an example of a curve in a three-dimensional space.

FIG. 3 is a diagram in which a curve in the three-dimensional space shown in FIG. 2 is graphed using a length from a starting point as a parameter.

FIG. 4 is a diagram illustrating an example of a curved surface in a three-dimensional space.

FIG. 5 is a diagram for explaining processing on the curved surface shown in FIG. 4;

FIG. 6 is a diagram illustrating a configuration of an apparatus that encodes information regarded as a one-dimensional signal.

FIG. 7 is a diagram illustrating a configuration of an apparatus that synthesizes an original one-dimensional signal from information obtained by encoding information regarded as a one-dimensional signal.

FIG. 8 is a flowchart of a process in interpolation estimation performed by an interpolation estimation unit and the like.

FIG. 9 is a flowchart showing a process of analyzing a curve on a two-dimensional plane.

FIG. 10 is a flowchart illustrating a process of combining curves on a two-dimensional plane.

FIG. 11 is a flowchart of a process of analyzing a curve in a three-dimensional space.

FIG. 12 is a flowchart illustrating a process of combining curves in a three-dimensional space.

FIG. 13 is a flowchart illustrating a process of analyzing a curved surface in a three-dimensional space.

FIG. 14 is a diagram illustrating a flowchart of a process of combining curved surfaces in a three-dimensional space.

FIG. 15 is a diagram showing a one-dimensional signal f (x).

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Moving image analysis / synthesis apparatus 10 ... Moving image analysis apparatus 11 ... Moving image memory 12 ... Information change analysis unit 13 ... Feature point detection unit 14 ... Feature point encoding unit 15 ··· Information coding unit 16 · · · General coding unit 51 · · · Low frequency detection circuit 52 · · · primary differential analysis circuit 53 · · · secondary differential analysis circuit 54 · feature point detection Unit 55: subtraction circuit 20: moving image synthesizing device 21: general reproduction unit 22: information reproduction unit 23: original image reproduction unit 61: interpolation estimation unit 62: inverse conversion Section 63: Primary differential analysis circuit 64: Secondary differential analysis circuit 65: Addition circuit

Continuation of the front page (56) References IEICE Technical Report, IE88-112 (1989-2-23) p. 41-46 (On Recognition of Motion of 3D Object Using Model) IEICE Technical Report, NLC 90-28 (1990) p. 1-8 (High-precision estimation of motion information of head movement and face movement in intelligent image coding) Transactions of the Institute of Electronics, Information and Communication Engineers, BIJ 74-BI [10] (1991-10 ) P. 789 -798 (Estimation of three-dimensional motion of head in model-based coding of face images) (58) Fields investigated (Int.Cl. ⁷ , DB name) H04N ^7/ 24-7/68

Claims (24)

(57) [Claims] 1. A moving image in which two-dimensional images are successive in a time series is analyzed for a positional change of image information on an image plane of each two-dimensional image and a temporal change of image information in a time series. A moving image analyzer for compressing feature points of the analysis result, comprising: a first smoothing function having a two-dimensional position and a time on the image plane as parameters; The convolution operation is performed to calculate the low frequency component of the moving image, the low frequency component is subtracted from the image information of the original moving image, and the first smoothing function is differentiated once with respect to the two-dimensional position and time. An analysis method that performs a convolution operation on the image information in which the low-frequency components have been reduced at a plurality of resolutions using a second smoothing function, and calculates a result of the multi-resolution analysis. A step, a feature point detecting means for detecting a feature point at which the analysis result becomes a maximum or a minimum, and position information on the three-dimensional space-time specified by the two-dimensional position and time of the detected feature point, A set of isolated points,
Classify into a set of lines or a set of surfaces, and compress the position information of the feature points classified into the line or the surface based on a function of predetermined position information defined corresponding to the line or the surface. Position information compression means, and analysis result compression means for compressing the analysis result on the feature point classified into the line or the curved surface based on a function of a predetermined analysis result defined corresponding to the line or the curved surface And encoding means for encoding the position information and analysis results of the feature points classified as the isolated points, and the compressed position information and the compressed analysis results of the feature points classified as the line or the curved surface. Moving image analyzer. 2. The method according to claim 1, wherein the feature point detecting means uses three second smoothing functions obtained by differentiating the first smoothing function twice with respect to the two-dimensional position and time, respectively, and uses the low frequency component at a plurality of resolutions. The moving image analysis apparatus according to claim 1, wherein a convolution operation is performed with the image information of which the number has been reduced, and a feature point is detected based on a zero cross point in the result of the convolution operation. 3. The moving image according to claim 1, wherein the feature point detection means detects a feature point based on a point at which a sum of squares of the analysis result of each dimension of the three-dimensional space gives an extreme value. Image analyzer. 4. The three-dimensional space-time function according to claim 1, wherein the position information compression means converts a function of the position information corresponding to the line as a function of position information of each dimension with respect to a length from a predetermined starting point on the line. The moving image analyzer according to claim 1, wherein the moving image analyzer is defined by: 5. The position information compressing means defines a function of position information corresponding to the line as a function of a curvature with respect to a length from a predetermined starting point on the line and a function of a torsion ratio with respect to the length. The moving image analyzer according to claim 1. 6. The position information compressing means, wherein a distance from a first plane formed by a time dimension and a first dimension of the image plane in the three-dimensional space from the curved surface is included in a predetermined minute range. The first curved surface to be divided is divided, and a distance from a second plane formed by a time dimension and a second dimension of the image plane in the three-dimensional space from the remaining curved surface after the division is within a predetermined minute range. Dividing the included second curved surface, and dividing the first curved surface and the second curved surface,
The remaining curved surfaces after the division are sequentially repeated to divide the curved surface into a plurality of the first curved surfaces and a plurality of the second curved surfaces. The function of the position information corresponding to the curved surface is A function of a distance from the first plane to position information on the first plane corresponding to a first curved surface, and a position on the second plane corresponding to each of the second curved surfaces The moving image analysis device according to claim 1, wherein the moving image analysis device is defined as a function of a distance from the second plane to information. 7. The position information compressing means performs a multi-resolution analysis on a function of curvature, which is defined for each cutting line formed by cutting the curved surface along the image plane, with respect to a distance from a predetermined starting point on the cutting line. Detecting a singular point of the analysis result, and from the detected singular point, a plurality of curves formed by connecting singular points whose distance between the adjacent singular points on the image plane is within a predetermined range. And defining a function of position information corresponding to the curved surface as a function of a curvature with respect to a distance from a predetermined start point on the plurality of extracted curves and a function of a torsion rate with respect to the distance. The moving image analyzer according to any one of the preceding claims. 8. For a moving image in which two-dimensional images are successive in time series, a positional change of image information on an image plane of each two-dimensional image and a temporal change of image information on a time series are analyzed. A moving image analysis method for compressing feature points of the analysis result, comprising: using a first smoothing function having parameters of a two-dimensional position and time on the image plane, using a first smoothing function with image information of a moving image at a predetermined resolution. The convolution operation is performed to calculate the low frequency component of the moving image, the low frequency component is subtracted from the image information of the original moving image, and the first smoothing function is differentiated once with respect to the two-dimensional position and time. Using a second smoothing function, a convolution operation is performed on the image information in which the low-frequency components have been reduced at a plurality of resolutions, and a result of multi-resolution analysis is calculated. A feature point at which the analysis result is a maximum or a minimum is detected, and the position information of the detected feature point on the three-dimensional spatio-temporal specified by the two-dimensional position and time is defined as a set of isolated points.
Classifying into a set of lines or a set of surfaces, compressing the position information of the feature points classified into the line or the surface based on a function of predetermined position information defined corresponding to the line or the surface. Compressing the analysis result on the feature point classified into the line or the curved surface based on a function of a predetermined analysis result defined corresponding to the line or the curved surface, A moving image analysis method that encodes point position information and analysis results, and the compressed position information and the compressed analysis results of feature points classified into the line or the curved surface. 9. The method according to claim 1, further comprising the step of detecting said characteristic points by using three second smoothing functions obtained by differentiating said first smoothing function twice with respect to said two-dimensional position and time, respectively, and using said low-frequency component at a plurality of resolutions. The moving image analysis method according to claim 8, wherein a convolution operation is performed with the image information in which is reduced, and a feature point is detected based on a zero-cross point in a result of the convolution operation. 10. The moving image according to claim 8, wherein the feature point is detected based on a point at which a sum of squares of the analysis result of each dimension of the three-dimensional space gives an extreme value. Image analysis method. 11. The compression of the position information is performed by converting a function of the position information corresponding to the line into a function of the position information of each dimension with respect to a length from a predetermined starting point on the line. The moving image analysis method according to claim 8, wherein the moving image analysis method is defined by: 12. The compression of the position information defines a function of the position information corresponding to the line as a function of a curvature with respect to a length from a predetermined starting point on the line and a function of a torsion rate with respect to the length. The moving image analysis method according to claim 8. 13. The compression of the position information, wherein a distance from the curved surface to a first plane formed by a time dimension and a first dimension of the image plane in the three-dimensional space is included in a predetermined minute range. The first curved surface to be divided is divided, and a distance from a second plane formed by a time dimension and a second dimension of the image plane in the three-dimensional space from the remaining curved surface after the division is within a predetermined minute range. Dividing the included second curved surface, and dividing the first curved surface and the second curved surface,
The remaining curved surfaces after the division are sequentially repeated to divide the curved surface into a plurality of the first curved surfaces and a plurality of the second curved surfaces. The function of the position information corresponding to the curved surface is A function of a distance from the first plane to position information on the first plane corresponding to a first curved surface, and a position on the second plane corresponding to each of the second curved surfaces The moving image analysis method according to claim 8, wherein the information is defined as a function of a distance from the second plane with respect to information. 14. The method of claim 1, wherein the position information is compressed by multi-resolution analysis of a function of curvature with respect to a distance from a predetermined starting point on the cutting line, which is defined for each cutting line formed by cutting the curved surface at the image plane. Detecting a singular point of the analysis result, and from the detected singular point, a plurality of curves formed by connecting singular points whose distance between the adjacent singular points on the image plane is within a predetermined range. And defining a function of position information corresponding to the curved surface as a function of a curvature with respect to a distance from a predetermined starting point on the plurality of extracted curves and a function of a torsion rate with respect to the distance. The moving image analysis method according to the above. 15. An image of an original moving image at a predetermined resolution by using a first smoothing function having a two-dimensional position and a time on a two-dimensional image plane of a moving image in which a two-dimensional image is continuous in time series as parameters. Information and a convolution operation are performed to calculate a low frequency component of the moving image, the low frequency component is subtracted from the image information of the original moving image, and the first smoothing function is converted to the two-dimensional position and time respectively. Using the three second smoothing functions that have been differentiated once, the convolution operation is performed on the image information in which the low-frequency component has been reduced at a plurality of resolutions, and the result of multi-resolution analysis is calculated. Is detected, and the position information of the detected feature points on the three-dimensional spatiotemporal space specified by the two-dimensional position and time is a collection of isolated points. In the case, the position information of a feature point classified into a set of lines or a set of curved surfaces and classified into the line or the curved surface is based on a function of predetermined position information defined corresponding to the line or the curved surface. The analysis result on the feature points classified into the line or the curved surface is compressed based on a function of a predetermined analysis result defined corresponding to the line or the curved surface, and is classified into the isolated point. A moving image from the compressed signal generated by encoding the position information and the analysis result of the characteristic point and the compressed position information and the compressed analysis result of the characteristic point classified into the line or the curved surface. A moving image synthesizing apparatus for synthesizing, comprising: an inverse encoding unit that inversely encodes the encoded compressed signal; and a position information of the position information based on the compressed position information reproduced by the inverse encoding. A position information restoring means for restoring a function and restoring position information of a feature point on the function, and restoring a function of the analysis result based on the compressed analysis result reproduced by the inverse encoding; Analysis result restoring means for restoring the analysis result of the feature point on the function; and complementing the analysis result on the three-dimensional space-time based on the position information and the analysis result of the restored feature point. The low-frequency component is added to the result, and a convolution operation is performed on the analysis result to which the low-frequency component is added by using an inverse transform function of the second smoothing function, and the operation result at the plurality of resolutions is synthesized. An image information restoring means for restoring image information on the three-dimensional space-time. 16. The position information restoring means, as a function of position information in each dimension with respect to a length from a predetermined starting point on the line, corresponds to the line defined in each dimension of the three-dimensional space-time. The moving image synthesizing apparatus according to claim 15, wherein the function of the position information is restored. 17. The position information restoring means includes: a function of a curvature with respect to a length from a predetermined starting point on the line;
The moving image composition device according to claim 15, wherein a function of position information corresponding to the line, which is defined as a function of a torsion ratio for the length, is restored. 18. The method according to claim 18, wherein the position information restoring unit calculates a distance from the first plane to position information on a first plane formed by a time dimension in the three-dimensional space and a first dimension of the image plane. A function defined as a function of a distance from the second plane to position information on a second plane defined by a time dimension in the three-dimensional space-time and a second dimension of the image plane; The moving image synthesizing apparatus according to claim 15, wherein a function of the corresponding position information is restored. 19. The position information restoring means includes a function of a curvature with respect to a distance from a predetermined starting point on a plurality of curves, and a function of a position information corresponding to the curved surface, defined as a function of a torsion rate with respect to the distance. From a point on a cutting line that can be cut by the image plane in the three-dimensional spatiotemporal space from a point on a cutting line, the function of the curvature with respect to a distance from a predetermined starting point on the cutting line. The moving image composition device according to claim 15, wherein the position information of the feature point on the function is restored. 20. An image of an original moving image at a predetermined resolution using a first smoothing function having a two-dimensional position and a time on a two-dimensional image plane of a moving image in which a two-dimensional image is continuous in a time series. Information and a convolution operation are performed to calculate a low frequency component of the moving image, the low frequency component is subtracted from the image information of the original moving image, and the first smoothing function is converted to the two-dimensional position and time respectively. Using the three second smoothing functions that have been differentiated once, the convolution operation is performed on the image information in which the low-frequency component has been reduced at a plurality of resolutions, and the result of multi-resolution analysis is calculated. Is detected, and the position information of the detected feature points on the three-dimensional spatiotemporal space specified by the two-dimensional position and time is a collection of isolated points. In the case, the position information is classified into a set of lines or a set of curved surfaces, and the position information of the feature points classified into the line or the curved surface is based on a function of predetermined position information defined corresponding to the line or the curved surface. The analysis result on the feature points classified into the line or the curved surface is compressed based on a function of a predetermined analysis result defined corresponding to the line or the curved surface, and is classified into the isolated point. A moving image from the compressed signal generated by encoding the position information and the analysis result of the characteristic point and the compressed position information and the compressed analysis result of the characteristic point classified into the line or the curved surface. A moving image combining method for combining, wherein the encoded compressed signal is inversely encoded, and a function of the position information is restored based on the compressed position information reproduced by the inverse encoding. Restoring position information of feature points on the function, restoring a function of the analysis result based on the compressed analysis result reproduced by the inverse encoding, and analyzing an analysis result of the feature point on the function Restoring, complementing the three-dimensional spatio-temporal analysis result based on the restored feature point position information and analysis result, adding the low-frequency component to the complemented analysis result, and performing the second smoothing. A moving image for performing a convolution operation with the analysis result to which the low-frequency component has been added using the inverse transformation function of the function, and combining the operation results at the plurality of resolutions to restore the image information on the three-dimensional space-time. Image composition method. 21. The method of restoring the position information, wherein the position information is defined as a function of position information of each dimension with respect to a length from a predetermined starting point on the line, the function corresponding to the line defined in each dimension of the three-dimensional space-time. The moving image composition method according to claim 20, wherein the function of the position information obtained is restored. 22. A method for recovering the position information, comprising: a function of a curvature with respect to a length from a predetermined starting point on the line;
21. The moving image composing method according to claim 20, wherein a function of position information corresponding to the line, which is defined as a function of a torsion ratio for the length, is restored. 23. A method of restoring the position information, comprising: calculating a distance from the first plane to position information on a first plane formed by a time dimension in the three-dimensional space-time and a first dimension of the image plane. A function defined as a function of a distance from the second plane to position information on a second plane defined by a time dimension in the three-dimensional space-time and a second dimension of the image plane; The moving image composition method according to claim 20, wherein a function of the corresponding position information is restored. 24. A method for restoring position information, comprising: a function of a curvature with respect to a distance from a predetermined starting point on a plurality of curves; and a function of a position information corresponding to the curved surface, which is defined as a function of a torsion ratio with respect to the distance. From a point on a cutting line that can be cut by the image plane in the three-dimensional spatiotemporal space from a point on a cutting line, the function of the curvature with respect to a distance from a predetermined starting point on the cutting line. 21. The moving image composition method according to claim 20, wherein the position information of the feature points on the function is restored.