JP2002058034A - Apparatus and method for processing image as well as recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable rapid and accurate prediction of an image having a large motion by a smaller-scale circuit. SOLUTION: A motion vector detector 11 detects the motion. A class tap area cutting-out unit 12 selects a class tap of an interval corresponding to the detected motion. An ADRC processing circuit 14 and a class code generator 15 calculate a class based on the selected tap. A predicting arithmetic unit 17 executes the prediction by a predicting method corresponding to the calculated class.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an image processing apparatus and method, and a recording medium, and more particularly, to an image processing apparatus and method for predicting data or calculating data used for prediction, and a recording medium.

[0002]

2. Description of the Related Art Conventionally, based on a moving image composed of frames shown in FIG. 1A, a frame located between the frames of the moving image is calculated, and the frame composed of the frames shown in FIG. Various methods have been proposed for generating moving images. One of them is a method using a classification adaptive processing.

[0003] Class classification adaptive processing is to classify an input signal into several classes based on the characteristics thereof, and to apply an appropriate adaptive processing to the input signal of each class to the class. It is divided into processing and adaptive processing.

Here, the class classification processing and the adaptive processing will be briefly described.

First, the class classification processing will be described.

As shown in FIG. 2A, a certain pixel of interest and three adjacent pixels form a block of 2 × 2 pixels (class classification block).
Each pixel is represented by 1 bit (takes any level of 0 or 1). In this case, as shown in FIG. 2B, the 2 × 2 4-pixel block including the target pixel is 16 (= (2
¹ ) ⁴ ) Can be classified into patterns. Therefore, in this case, the target pixel can be classified into 16 patterns.

Here, usually, about 8 bits are assigned to each pixel. 3 × 3 blocks for class classification
If it is composed of 9 pixels, it will be classified into an enormous number of classes (2 ⁸ ) ⁹ .

[0008] The number of classes can be reduced by reducing the number of bits of pixels constituting a block for class classification.

Next, the adaptive processing will be described.

For example, a predicted value E [y] of a pixel value y of a frame of an original image (corresponding to an interpolated image (hereinafter, appropriately referred to as teacher data)) is thinned out. pixel values of the image (hereinafter, appropriately referred to as learning data) x _1, x _2, and ..., predetermined prediction coefficients w _1, w _2,
.. Are considered by a linear first-order combination model defined by the linear combination. In this case, the predicted value E [y]
Can be expressed by the following equation.

E [y] = w ₁ x ₁ + w ₂ x ₂ +... (1)

Therefore, for generalization, a matrix W composed of a set of prediction coefficients w, a matrix X composed of a set of learning data, and a matrix Y ′ composed of a set of predicted values E [y] are

(Equation 1) Defines the following observation equation.

XW = Y ′ (2)

Then, it is considered that a least square method is applied to this observation equation to obtain a predicted value E [y] close to the pixel value y of the original image. In this case, a matrix Y consisting of a set of pixel values y of the original image and a matrix E consisting of a set of residuals e of predicted values E [y] for the pixel values y of the original image are given by:

(Equation 2) From equation (2), the following residual equation is established.

XW = Y + E (3)

In this case, a prediction coefficient w _i for obtaining a prediction value E [y] close to the pixel value y of the original image is a square error.

(Equation 3) Can be obtained by minimizing.

Therefore, when the above-mentioned squared error is differentiated by the prediction coefficient w _i , the result becomes 0, that is, the prediction coefficient w _i that satisfies the following equation is a prediction value E [y ]
Is the optimum value.

[0018]

(Equation 4) ... (4)

Therefore, first, the equation (3) is calculated using the prediction coefficient w _i
By differentiating with, the following equation is established.

[0020]

(Equation 5) ... (5)

From equations (4) and (5), equation (6) is obtained.

[0022]

(Equation 6) ... (6)

Further, the learning data x, the prediction coefficient w, the teacher data y, and the residual e in the residual equation of the equation (3) are obtained.
In consideration of the relationship, the following normal equation can be obtained from Expression (6).

[0024]

(Equation 7) ... (7)

The normal equation of equation (7) can be set by the same number as the number of prediction coefficients w to be obtained. Therefore, by solving equation (7), the optimum prediction coefficient w can be obtained. . In solving equation (7), for example,
A sweeping method (Gauss-Jordan elimination method) or the like can be applied.

As described above, the optimum prediction coefficient w is obtained for each class, and further, using the prediction coefficient w, the prediction value E close to the pixel value y of the original image is obtained by the equation (1).
Finding [y] is adaptive processing.

[0027]

However, when predicting a frame from an image having a large motion, the number of pixels (hereinafter also referred to as a class tap) included in the classifying block or the number of pixels included in the block to which the adaptive processing is applied is included. If the number of pixels to be predicted (hereinafter also referred to as prediction taps) is not increased, the error of the predicted frame cannot be reduced, and therefore, the scale of a circuit required for processing increases or the time required for processing increases. There was a problem that it took.

The present invention has been made in view of such a situation, and an object of the present invention is to make it possible to quickly and accurately predict an image having a large motion with a circuit having a smaller scale.

[0029]

An image processing apparatus according to claim 1, wherein: a motion detecting means for detecting a motion; a class tap selecting means for selecting a class tap at intervals corresponding to the detected motion; A class calculating means for calculating a class based on the class taps thus obtained, and a prediction executing means for executing a prediction by a prediction method corresponding to the calculated class.

[0030] The class tap selecting means can select a class tap having a fixed total number.

The detecting means may detect the direction of the movement, and the class tap selecting means may select a class tap having a positional relationship corresponding to the direction of the movement.

The detecting means may detect the magnitude of the movement, and the class tap selecting means may select a class tap at a distance corresponding to the magnitude of the movement.

[0033] The prediction executing means can predict the pixels of the frame.

According to a sixth aspect of the present invention, there is provided an image processing method, comprising: a motion detecting step for detecting a motion; a class tap selecting step for selecting a class tap at intervals corresponding to the detected motion; And a prediction execution step of executing prediction by a prediction method corresponding to the calculated class.

According to a seventh aspect of the present invention, there is provided a recording medium program comprising: a motion detecting step for detecting a motion; a class tap selecting step for selecting a class tap at an interval corresponding to the detected motion; Basically, the method includes a class calculation step of calculating a class, and a prediction execution step of executing prediction by a prediction method corresponding to the calculated class.

According to the image processing apparatus of the present invention, the motion detecting means for detecting the motion, the class tap selecting means for selecting the class tap, and the class tap selected based on the selected class tap.
Class calculating means for calculating a class; predictive tap selecting means for selecting a predictive tap at an interval corresponding to the detected movement; predictive coefficient corresponding to the calculated class; and a predictive tap based on the selected predictive tap. And a prediction execution means for executing.

[0037] The prediction tap selecting means can select a predetermined number of prediction taps.

The detecting means may detect the direction of the movement, and the predictive tap selecting means may select a predictive tap having a positional relationship corresponding to the direction of the motion.

The detecting means may detect the magnitude of the motion, and the predictive tap selecting means may select a predictive tap at a distance corresponding to the magnitude of the motion.

The prediction executing means can predict the pixels of the frame.

According to a thirteenth aspect of the present invention, in the image processing method, a motion detecting step for detecting a motion, a class tap selecting step for selecting a class tap, and a class calculating step for calculating a class based on the selected class tap. And a prediction tap selection step of selecting a prediction tap at an interval corresponding to the detected motion; and a prediction execution step of performing prediction based on the prediction coefficient corresponding to the calculated class and the selected prediction tap. It is characterized by including.

According to a fourteenth aspect of the present invention, there is provided a program for a recording medium, comprising: a motion detecting step for detecting a motion, a class tap selecting step for selecting a class tap, and a class calculation for calculating a class based on the selected class tap. Steps, a prediction tap selection step of selecting a prediction tap at an interval corresponding to the detected motion, and a prediction coefficient corresponding to the calculated class, and based on the selected prediction tap,
And a prediction execution step of performing prediction.

According to a fifteenth aspect of the present invention, there is provided an image processing apparatus comprising: learning data generating means for generating learning data from input teacher data; motion detecting means for detecting a motion of learning data; Class tap selecting means for selecting a class tap at an interval corresponding to, class calculating means for calculating a class based on the selected class tap, and prediction coefficient calculation for calculating a prediction coefficient corresponding to the class from teacher data Means.

In the image processing method according to the present invention, a learning data generating step for generating learning data from input teacher data, a motion detecting step for detecting a motion of the learning data, and the learning data are detected. A class tap selection step of selecting a class tap at an interval corresponding to the movement, a class calculation step of calculating a class based on the selected class tap, and a prediction coefficient of calculating a prediction coefficient corresponding to the class from teacher data And a calculating step.

An image processing apparatus according to a seventeenth aspect provides a learning data generating means for generating learning data from input teacher data, a motion detecting means for detecting a motion of the learning data, and a class tap from the learning data. Class tap selecting means to select, class calculating means for calculating a class based on the selected class tap, and prediction tap selecting means for selecting a prediction tap at an interval corresponding to the detected motion from the learning data, A prediction coefficient calculation unit configured to calculate a prediction coefficient corresponding to the class from the selected prediction tap and the teacher data.

In the image processing method according to the present invention, a learning data generating step for generating learning data from input teacher data, a motion detecting step for detecting a motion of the learning data, and a class tap from the learning data are performed. A class tap selection step to select, a class calculation step of calculating a class based on the selected class tap, and a prediction tap selection step of selecting a prediction tap at an interval corresponding to the detected motion from the learning data, Calculating a prediction coefficient corresponding to the class from the selected prediction tap and the teacher data.

The image processing apparatus according to claim 1, claim 6.
In the image processing method described in the above, and the recording medium described in the claim 7, a motion is detected, a class tap at an interval corresponding to the detected motion is selected, and a class is determined based on the selected class tap. The prediction is performed using the calculated and predicted method corresponding to the calculated class.

An image processing apparatus according to claim 8, wherein
In the image processing method according to the third aspect and the recording medium according to the fourteenth aspect, a motion is detected, a class tap is selected, a class is calculated based on the selected class tap, and a motion is detected. A prediction tap at a corresponding interval is selected, and prediction is executed based on the prediction coefficient corresponding to the calculated class and the selected prediction tap.

In the image processing apparatus according to the present invention, the learning data is generated from the input teacher data, the movement of the learning data is detected, and Class taps at intervals corresponding to the detected motion are selected, a class is calculated based on the selected class taps, and from the teacher data,
A prediction coefficient corresponding to the class is calculated.

According to the image processing apparatus of the present invention, the learning data is generated from the input teacher data, the movement of the learning data is detected, and A class tap is selected, a class is calculated based on the selected class tap, a prediction tap at an interval corresponding to the detected motion is selected from the learning data, and a class is selected from the selected prediction tap and the teacher data. Is calculated.

[0051]

FIG. 3 is a diagram showing a configuration of an embodiment of an image processing apparatus according to the present invention for interpolating frames. Image data input to the image processing apparatus is supplied to the motion vector detecting unit 11, the class tap area extracting unit 12, the prediction tap area extracting unit 13, and the prediction calculating unit 17.

The motion vector detecting section 11 generates a motion vector based on, for example, two frames of the input image data, and outputs the generated motion vector to the class tap area extracting section 12 and the prediction tap area extracting section 13.
To supply.

The class tap area cutout unit 12 cuts out pixels as class taps from the image data based on the motion vector supplied from the motion vector detection unit 11, and classifies the cut out class taps into an ADRC (Adaptive D).
(Dynamic Range Coding) processing unit 14.

The prediction tap area cutout unit 13 uses the motion vector supplied from the motion vector detection unit 11
Cut out pixels as prediction taps from image data,
The extracted prediction tap is supplied to the prediction calculation unit 17.

Here, the class tap area cutout section 12
Alternatively, tap extraction by the predicted tap area extraction unit 13 will be described.

FIG. 4 shows the movement of pixels in an image obtained by panning a moving object at a speed slower than the moving speed of the moving object in order to explain the tap cutting process. This will be described with reference to a graph.

The pixels corresponding to the background in the original image before the frames are thinned are panned at a slow speed and imaged, and as shown in FIG.
The difference between the pixel positions for each frame is small.

When the frame is thinned out, the pixel corresponding to the background is compared with the original image before the frame is thinned out, as shown by the triangle in FIG. The difference increases.

The pixels corresponding to the moving object in the original image before the frames are decimated have a difference in the position of the pixels in each frame as compared with the background pixels as shown by the square in FIG. large.

As the speed of the moving object increases, the difference between the pixel positions for each frame increases.

When the frames are thinned out, the difference between the positions of the pixels corresponding to the moving object in each frame is further increased, as shown by X in FIG.

FIGS. 5 and 6 are diagrams for explaining the correspondence between the position of the pixel corresponding to the moving object and the position of the prediction tap. When the pixel corresponding to the moving object is to be predicted, when the interval between the prediction taps is narrow, the pixel corresponding to the moving object is not used as the prediction tap in the thinned image as shown in FIG. In such a case, since the pixel in the frame to be interpolated is a pixel corresponding to the moving object, the prediction is performed using pixels other than the pixel corresponding to the moving object. The prediction is not made properly.

On the other hand, by widening the interval between the prediction taps, the pixels corresponding to the moving object can be used as the prediction taps in the decimated image as shown in FIG. That is, since the prediction is performed using the pixel corresponding to the moving object, the prediction is appropriately performed.

As described above, if the interval between the prediction taps is increased in accordance with the movement of the image corresponding to the moving object,
A pixel corresponding to a moving object can be used as a prediction tap, and the image processing apparatus can perform highly accurate prediction.

The prediction tap area cutout unit 13 stores, for example, a reference tap interval which is a predetermined a as shown in FIG. Predicted tap area cutout unit 1
3, when the motion vector is supplied from the motion vector detection unit 11, for example, a horizontal motion amount α and a vertical motion amount β are extracted from the motion vector.

The prediction tap area cutout unit 13 determines whether the horizontal motion amount α is larger than a previously stored threshold value, and determines that the horizontal motion amount α is larger than the previously stored threshold value. In this case, the reference tap interval a is multiplied by the horizontal motion amount α to obtain the tap interval of the horizontal prediction tap.

The prediction tap area cutout unit 13 determines whether the vertical motion amount β is larger than a previously stored threshold value, and determines that the vertical motion amount β is larger than the previously stored threshold value. In this case, the reference tap interval a is multiplied by the vertical movement amount β to obtain the tap interval of the prediction tap in the vertical direction.

For example, as shown in FIG. 8, the reference tap interval is 1, the vertical movement amount β is 2, the horizontal movement amount α is 3, and the previously stored threshold value is 1. In the case of, the prediction tap area cutout unit 13 sets the tap interval of the prediction tap in the vertical direction to 2 (one pixel exists between the prediction taps in the vertical direction), and sets the tap interval of the prediction tap in the horizontal direction to 3 (Two pixels exist between prediction taps in the horizontal direction).

As described above, when both the horizontal motion amount α and the vertical motion amount β are larger than the threshold value, the prediction tap area cutout unit 13 multiplies the reference tap interval a by the horizontal motion amount α. The obtained interval is set as the tap interval of the prediction tap in the horizontal direction, and the interval obtained by multiplying the reference tap interval a by the movement amount β in the vertical direction is set as the tap interval of the prediction tap in the vertical direction.

On the other hand, when it is determined that the vertical movement amount β is equal to or less than the threshold value stored in advance, the prediction tap area cutout unit 13 sets the reference tap interval a to the vertical prediction tap of the prediction tap. Tap interval.

That is, when the vertical motion amount β is equal to or smaller than the threshold value stored in advance and the horizontal motion amount α is larger than the threshold value, the prediction tap area cutout unit 13 determines whether the reference tap interval a is The interval obtained by multiplying by the motion amount α is set as the tap interval of the prediction tap in the horizontal direction, and the reference tap interval a is set as the tap interval of the prediction tap in the vertical direction.

For example, as shown in FIG. 9, the reference tap interval is 1, the vertical movement amount β is 0, the horizontal movement amount α is 2, and the previously stored threshold value is 1. In the case of, the prediction tap area cutout unit 13 sets the tap interval of the vertical prediction tap to 1 (the vertical prediction tap is adjacent), and sets the tap interval of the horizontal prediction tap to 2 (horizontal direction). There is one pixel between prediction taps).

When it is determined that the horizontal movement amount α is equal to or smaller than the threshold value stored in advance, the prediction tap area cutout unit 13 sets the reference tap interval a to the tap interval of the horizontal prediction tap. I do.

That is, when the horizontal motion amount α is equal to or smaller than the threshold value stored in advance and the vertical motion amount β is larger than the threshold value, as shown in FIG. The tap interval a is set as the tap interval of the prediction tap in the horizontal direction, and the interval obtained by multiplying the reference tap interval a by the amount of movement β in the vertical direction is set as the tap interval of the prediction tap in the vertical direction.

For example, as shown in FIG. 10, the reference tap interval is 1, the vertical movement amount β is 2, the horizontal movement amount α is 0, and the previously stored threshold value is 1.
In the case of, the prediction tap region cutout unit 13 sets the tap interval of the prediction tap in the vertical direction to 2 and sets the tap interval of the prediction tap in the horizontal direction to 1 (the prediction tap in the horizontal direction is adjacent).

By doing so, the prediction tap area cutout unit 13 can set an appropriate pixel as a prediction tap even if the image includes a pixel of a moving object. As a result, the image processing apparatus Can predict a highly accurate image as a frame to be interpolated.

For example, the prediction tap area cutout unit 13 may set a pixel whose total number is constant as a prediction tap in correspondence with a numerical value stored in advance.

Note that the motion vector detecting section 11 outputs data indicating the direction of the motion vector, and the prediction tap area cutout section 13 selects a prediction tap based on the data indicating the direction of the motion vector. Is also good. In addition, for example, the prediction tap region cutout unit 13 may determine an appropriate pixel based on whether the horizontal motion amount α is 1 or more, or the vertical motion amount β is 1 or more. May be used as the prediction tap.

The motion vector detecting section 11 outputs data indicating the magnitude of the motion vector, and the prediction tap area cutout section 13 selects a prediction tap based on the data indicating the magnitude of the motion vector. It may be. Further, for example, the prediction tap region cutout unit 13 may set an appropriate pixel as a prediction tap based on the magnitude of the motion vector.

The class tap area cutout section 12 changes the interval between class taps in accordance with the motion vector in the same processing as that of the prediction tap area cutout section 13 described above, and a description thereof will be omitted.

For example, the class tap area cutout unit 12 may set a pixel having a constant total number as a class tap in correspondence with a numerical value stored in advance.

The motion vector detecting section 11 outputs data indicating the direction of the motion vector, and the class tap area extracting section 12 selects a class tap based on the data indicating the direction of the motion vector. Is also good. In addition, for example, the class tap area cutout unit 12 may determine an appropriate pixel based on whether the horizontal motion amount α is 1 or more, or the vertical motion amount β is 1 or more. May be used as a class tap.

The motion vector detecting section 11 outputs data indicating the magnitude of the motion vector, and the class tap area cutout section 12 selects a class tap based on the data indicating the magnitude of the motion vector. It may be.
Further, for example, the class tap area cutout unit 12 may set an appropriate pixel as a class tap based on the magnitude of the motion vector.

Returning to FIG. 3, the ADRC processing section 14 performs ADRC processing on the class tap supplied from the class tap area cutout section 12, thereby reducing the number of bits of the class tap. Reduce the number.

That is, for example, for simplicity, consider a class tap composed of four pixels arranged on a straight line as shown in FIG. A maximum value MAX and a minimum value MIN of the values are detected. Then, DR = MAX−MIN is set as a local dynamic range of the class tap, and based on the dynamic range DR, the pixel values of the pixels forming the class tap are requantized to K bits.

[0086] That is, from each pixel value in the class tap, and subtracts the minimum value MIN, dividing the subtracted value by DR / 2 ^K. Then, it is converted into a code (ADRC code) corresponding to the resulting division value. Specifically, for example, when K = 2, as shown in FIG. 11 (B), to which range the divided value belongs to the dynamic range DR obtained by equally dividing the dynamic range DR by 4 (= 2 ² ) If the divided value belongs to the range of the lowest level, the range of the second lowest level, the range of the third lowest level, or the range of the highest level, for example, 00
It is encoded into two bits such as B, 01B, 10B, or 11B (B represents a binary number).

Here, such ADRC processing is called non-edge matching.

The details of the ADRC processing are disclosed in, for example, Japanese Unexamined Patent Publication No. 3-53778 filed earlier by the present applicant.

The number of classes can be reduced as described above by performing the ADRC process of performing re-quantization with a smaller number of bits than the number of bits assigned to the pixels constituting the class tap. ADRC processing is performed in the ADRC processing unit 14.

The ADRC processing unit 14 sets the class tap to A
Dynamic range D generated by applying DRC processing
R and the ADRC code are supplied to the class code generator 15.

The class code generation section 15 generates a final class code based on the dynamic range DR and the ADRC code supplied from the ADRC processing section 14, and supplies the class code to the prediction coefficient ROM 16.

The prediction coefficient ROM 16 stores prediction coefficients in advance corresponding to the class codes. Prediction coefficient RO
When the class code is supplied from the class code generation unit 15, the M 16 supplies the prediction coefficient corresponding to the class indicated by the supplied class code to the prediction calculation unit 17.

The prediction operation unit 17 stores the prediction tap supplied from the prediction tap area cutout unit 13 and the prediction coefficient ROM
By linear combination with the prediction coefficients supplied from 16
A predicted value that is a frame to be interpolated is calculated. The prediction calculation unit 17 adds the calculated frame to be interpolated to the frame of the image data input to the image processing apparatus, and outputs the resultant as frame-interpolated image data.

As described above, the image processing apparatus according to the present invention appropriately changes the interval between the class taps or the prediction taps based on the motion of the input image, so that the accuracy of the accuracy can be improved without increasing the number of taps. It is possible to output image data in which a high frame is interpolated.

Next, the process of predicting a frame to be interpolated in the image processing apparatus will be described with reference to the flowchart of FIG.

In step S11, the motion vector detecting section 11 detects a motion vector from the input image data, and supplies the motion vector to the class tap area extracting section 12 and the prediction tap area extracting section 13.
In step S <b> 12, the class tap region cutout unit 12 determines a class tap interval based on the motion vector supplied from the motion vector detection unit 11. The prediction tap region cutout unit 13 determines a prediction tap interval based on the motion vector supplied from the motion vector detection unit 11.

In step S13, the class tap area cutout unit 12 selects a class tap based on the determined interval, and outputs the selected class tap to the ADRC processing unit 14. In step S14, the ADRC processing unit 14 applies an ADRC process to the class tap supplied from the class tap region cutout unit 12, generates a dynamic range DR and an ADRC code, and generates the dynamic range DR and the ADRC code. It is supplied to the class code generator 15. The class code generation unit 15 generates a class code based on the dynamic range DR and the ADRC code, and mainly stores the class code in the prediction coefficient ROM 16.

In step S15, the prediction coefficient ROM
The selector 16 selects a prediction coefficient corresponding to the class indicated by the class code supplied from the class code generator 15 and outputs the selected prediction coefficient to the prediction calculator 17.

In step S16, the prediction tap area cutout unit 13 selects a prediction tap based on the interval determined in step S12, and outputs the selected prediction tap to the prediction calculation unit 17.

In step S17, the prediction operation unit 17
Calculates a predicted value, which is a frame to be interpolated, by a linear linear combination of a prediction tap supplied from the prediction tap area cutout unit 13 and a prediction coefficient supplied from the prediction coefficient ROM 16. The prediction calculation unit 17 adds the calculated frame to be interpolated to the frame of the image data input to the image processing apparatus, and outputs the resultant data as frame-interpolated image data. The procedure returns to step S11 to perform the prediction process. repeat.

As described above, the class tap area cutout unit 12 sets the interval between class taps according to the motion vector, and the prediction tap area cutout unit 13 sets the interval between prediction taps according to the motion vector. Therefore, the image processing apparatus can quickly and accurately predict a frame with higher accuracy.

FIG. 13 shows the prediction coefficient ROM1 of FIG.
6 shows a configuration example of an image processing apparatus that performs learning for obtaining a prediction coefficient stored in No. 6.

The frame thinning filter 51 and the normal equation processing unit 57 are provided with image data (corresponding to teacher data) for obtaining a prediction coefficient applicable to any image.

The frame thinning filter 51 thins out every other frame included in the supplied teacher data, and converts the image data (corresponding to the learning data) from which the frames have been thinned out into the motion vector detecting section 52 and the class tap. Area cutout section 53 and predicted tap area cutout section 54
To supply.

The motion vector detection section 52 generates a motion vector based on the learning data, and supplies the generated motion vector to the class tap area cutout section 53 and the prediction tap area cutout section 54.

The class tap region cutout unit 53 cuts out pixels as class taps from the learning data based on the motion vector supplied from the motion vector detection unit 52, and supplies the cutout class taps to the ADRC processing unit 14. . The class tap area cutout unit 53 changes the interval between class taps according to the motion vector, similarly to the class tap area cutout unit 12.

[0107] The ADRC processing section 55 applies the ADRC processing to the class tap supplied from the class tap area cutout section 53.
Applying the processing, a dynamic range DR and an ADRC code are generated, and the generated dynamic range DR and A
The DRC code is supplied to the class code generator 56.

The class code generator 56 generates a final class code based on the dynamic range DR and the ADRC code supplied from the ADRC processor 55, and supplies the class code to the normal equation processor 57.

[0109] The prediction tap area cutout unit 54 calculates
Cut out pixels as prediction taps from the training data,
The extracted prediction tap is supplied to the normal equation processing unit 57. The prediction tap area cutout unit 54 changes the interval between prediction taps corresponding to the motion vector, similarly to the prediction tap area cutout unit 13.

Upon receiving the prediction tap and the teacher data supplied from the prediction tap region cutout unit 54, the normal equation processing unit 57 calculates a prediction coefficient that minimizes the error by using the least tap method.

[0111] That is, for example, now, the pixel values of the prediction taps included in the training _{data, x 1, x 2, x} 3, and ..., and prediction coefficients to be obtained _{w 1, w 2, w 3} , · ···, and these linear linear combinations allow the teacher data
To determine the pixel value y of a pixel, the prediction coefficients w ₁ , w ₂ ,
It is necessary that w ₃ ,... satisfy the following expression.

Y = w ₁ x ₁ + w ₂ x ₂ + w ₃ x ₃ +...

Therefore, the normal equation processing unit 57 uses the prediction taps of the same class and the pixels of the corresponding teacher data to predict values w ₁ x ₁ + w ₂ x ₂ + w ₃ x ₃ + for the true value y.
The prediction coefficients w ₁ , w ₂ ,
w _3, · · · are obtained by solving the normal equation shown in equation (7) described above. Therefore, by performing this process for each class, a prediction coefficient is generated for each class.

The prediction coefficient for each class obtained by the normal equation processing unit 57 is supplied to the memory 58 together with the class code. As a result, in the memory 58, the prediction coefficient from the normal equation processing unit 57 is stored at the address corresponding to the class indicated by the class code.

As described above, in the memory 58, at the address corresponding to each class, the optimum prediction coefficient for predicting the pixel of the class is stored.

The prediction coefficients for each class stored in the memory 58 as described above are stored in the prediction coefficient ROM 16 of FIG.

Next, the process of storing the prediction coefficients of the image processing apparatus shown in FIG. 13 will be described with reference to the flowchart of FIG. In step S31, the frame thinning filter 51 thins out frames from the input image data, and outputs the image data obtained by thinning out the frames to the motion vector detecting unit 52 and the class tap area cutting unit 5
3 and the prediction tap area cutout unit 54.

In step S 32, the motion vector detecting section 52 detects a motion vector of the image data supplied from the frame thinning filter 51. Step S3
In 3, the class tap area cutout unit 53 determines the interval between class taps based on the motion vector supplied from the motion vector detection unit 52. The prediction tap area cutout unit 54 determines an interval between prediction taps based on the motion vector supplied from the motion vector detection unit 52.

In step S34, the class tap area cutout unit 53 selects a class tap based on the determined interval, and outputs the selected class tap to the ADRC processing unit 55. In step S35, the ADRC processing section 55 applies the ADRC processing to the class tap supplied from the class tap area cutout section 53, generates a dynamic range DR and an ADRC code, and generates the dynamic range DR and the ADRC code. It is supplied to the class code generator 56. The class code generation unit 56 generates a class code based on the dynamic range DR and the ADRC code, and mainly supplies the class code to the normal equation processing unit 57.

In step S36, the prediction tap area cutout unit 54 selects a prediction tap based on the determined interval, and divides the selected prediction tap into a normal equation processing unit 57.
Output to

In step S37, the normal equation processing unit 57 calculates the prediction coefficient of the class corresponding to the class code based on the prediction tap and the teacher data, and supplies the prediction coefficient to the memory 58 together with the class code.

In step S38, the memory 58 stores
The prediction coefficient supplied from the normal equation processing unit 57 is stored in the address corresponding to the class indicated by the class code, and the procedure returns to step S31 and repeats the above processing.

As described above, the information processing apparatus shown in FIG. 13 generates, for each class, prediction taps having intervals corresponding to motion vectors and prediction coefficients used for class classification adaptive processing for selecting class taps. can do.

In the present embodiment, the class code generation unit 15 or the class code generation unit 56
Classification processing is performed based on the dynamic range DR and the ADRC code output from the RC processing unit 14 or the ADRC processing unit 55. The classification processing includes, for example, DPCM (prediction coding), BT
It is also possible to perform processing on data that has been subjected to C (Block Truncation Coding), VQ (Vector Quantization), DCT (Discrete Cosine Transform), Hadamard Transform, or the like.

The series of processes described above can be executed by hardware, but can also be executed by software. When a series of processing is executed by software, a program constituting the software can execute various functions by installing a computer built into dedicated hardware or installing various programs. It is installed from a recording medium into a possible general-purpose personal computer or the like.

FIG. 15 is a diagram for explaining an example of the configuration of a personal computer which executes the processing of prediction or the processing of storage of prediction coefficients. CPU (Central Processing Uni
t) 101 indicates various application programs and O
Actually execute S (Operating System). ROM (Rea
d-only Memory) 102 is generally a CPU 101
Stores basically fixed data of the program and the calculation parameters used by. RAM (Random-Acces
The s Memory) 103 stores a program used in the execution of the CPU 101 and parameters appropriately changed in the execution. These are interconnected by a host bus 104 composed of a CPU bus and the like.

The host bus 104 is connected via a bridge 105 to a PCI (Peripheral Component Interconnect / Int).
(Erface) bus and the like.

The keyboard 108 is operated by the user when inputting various commands to the CPU 101. The pointing device 109 is operated by the user when pointing or selecting a point on the screen of the display 110. The display 110 is composed of a liquid crystal display device or a CRT (Cathode Ray Tube), and displays various information as text or images. HDD (Hard Disk
The drive 111 drives a hard disk and records or reproduces a program or information executed by the CPU 101 on the hard disk.

The drive 112 includes the mounted magnetic disk 121, optical disk 122, and magneto-optical disk 12.
3 or the data or the program recorded in the semiconductor memory 124, and reads the data or the program into the interface 107, the external bus 106,
The data is supplied to the bridge 105 and the RAM 103 connected via the host bus 104. The keyboard 108 to the drive 112 are connected to the interface 10
7, the interface 107 includes an external bus 106, a bridge 105, and a host bus 104.
Is connected to the CPU 101 via the.

The communication board 113 is connected to, for example, a network or the like, stores image data supplied from the CPU 101 or the HDD 111 in packets of a predetermined format, transmits the packets via the network, and transmits the packets via the network. Then, the image data stored in the received packet is stored in the CPU 101, the RAM 103, or the H
Output to DD111.

The communication board 113 is connected to the CPU via the external bus 106, the bridge 105, and the host bus 104.
It is connected to 101.

The recording medium on which the program for executing the above-described series of processing is recorded is, as shown in FIG.
Apart from the computer, a magnetic disk 121 (including a floppy disk) storing the program, an optical disk 122 (CD-ROM (Compact Disc-Read Only Memory),
VD (including Digital Versatile Disc), magneto-optical disk 123 (including MD (Mini-Disc)), or a packaged medium such as semiconductor memory 124, as well as being pre-installed in a computer. It is provided with a ROM 102 provided with the program and provided to the user, a hard disk built in the HDD 111, and the like.

In this specification, the steps for describing the program stored in the recording medium are not limited to the processing performed in chronological order according to the described order, but are not necessarily performed in chronological order. This also includes processing executed in parallel or individually.

[0134]

According to the image processing apparatus according to the first aspect, the image processing method according to the sixth aspect, and the recording medium according to the seventh aspect, a motion is detected, and the motion corresponds to the detected motion. Since the class tap at the interval is selected, the class is calculated based on the selected class tap, and the prediction is performed by the prediction method corresponding to the calculated class.
It becomes possible to quickly and accurately predict an image having large motion with a circuit of a smaller scale.

An image processing apparatus according to claim 8, wherein
According to the image processing method of the third aspect and the recording medium of the fourteenth aspect, a motion is detected, a class tap is selected, a class is calculated based on the selected class tap, and the detected motion is determined. The prediction tap at the interval corresponding to is selected, and the prediction is performed based on the prediction coefficient corresponding to the calculated class and the selected prediction tap. In the circuit,
Quick, accurate and predictable.

According to the image processing apparatus of the present invention, the learning data is generated from the input teacher data, the movement of the learning data is detected, and the learning data is generated. Since a class tap at an interval corresponding to the detected motion is selected, a class is calculated based on the selected class tap, and a prediction coefficient corresponding to the class is calculated from the teacher data.
In the prediction process, an image having a large motion can be quickly and accurately predicted by a circuit of a smaller scale.

According to the image processing apparatus of the present invention, the learning data is generated from the input teacher data, the movement of the learning data is detected, and the learning data is generated. , A class tap is selected, a class is calculated based on the selected class tap, a prediction tap at an interval corresponding to the detected motion is selected from the learning data, and from the selected prediction tap and the teacher data, Since the prediction coefficient corresponding to the class is calculated, an image with a large motion can be quickly and accurately predicted by a circuit of a smaller scale in the prediction processing.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a process of interpolating a frame.

FIG. 2 is a diagram illustrating a class classification process.

FIG. 3 is a diagram illustrating a configuration of an embodiment of an image processing apparatus according to the present invention.

FIG. 4 is a diagram illustrating the movement of a pixel.

FIG. 5 is a diagram illustrating the correspondence between the position of a pixel corresponding to a moving object and the position of a prediction tap.

FIG. 6 is a diagram illustrating the correspondence between the position of a pixel corresponding to a moving object and the position of a prediction tap.

FIG. 7 is a diagram illustrating a reference tap interval.

FIG. 8 is a diagram illustrating an interval between prediction taps corresponding to a motion.

FIG. 9 is a diagram illustrating intervals of prediction taps corresponding to motion.

FIG. 10 is a diagram illustrating intervals of prediction taps corresponding to motion.

FIG. 11 is a diagram illustrating ADRC processing.

FIG. 12 is a flowchart illustrating a process of predicting a frame to be interpolated.

FIG. 13 is a diagram illustrating a configuration example of an image processing apparatus that performs learning for obtaining a prediction coefficient.

FIG. 14 is a flowchart illustrating a process of storing prediction coefficients.

FIG. 15 is a diagram illustrating a configuration example of a personal computer that executes prediction processing or prediction coefficient storage processing.

[Explanation of symbols]

11 motion vector detection unit, 12 class tap region cutout unit, 13 predicted tap region cutout unit, 14
ADRC processing unit, 15 class code generation unit, 1
6 prediction coefficient ROM, 17 prediction operation unit, 51 frame thinning filter, 52 motion vector detection unit,
53 class tap area cutout section, 54 prediction tap area cutout section, 55 ADRC processing section, 56
Class code generator, 57 normal equation processor,
58 memories, 101 CPU, 102 ROM,
103 RAM, 111 HDD, 121 magnetic disk, 122 optical disk, 123 magneto-optical disk, 124 semiconductor memory

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5C059 KK06 LB07 LB13 MA01 MA22 MA23 MA28 MA29 MC33 MC35 MD04 NN01 SS14 SS20 TA29 TA30 TB08 TC12 TD12 TD13 UA33 5C063 AC01 BA08 BA12 CA05 CA07 CA11 CA40

Claims (18)

[Claims] 1. A motion detecting means for detecting a motion, a class tap selecting means for selecting a class tap at intervals corresponding to the detected motion, and a class calculation for calculating a class based on the selected class tap. An image processing apparatus, comprising: means for performing prediction using a prediction method corresponding to the calculated class. 2. The image processing apparatus according to claim 1, wherein the class tap selecting unit selects the class tap having a constant total number. 3. The method according to claim 1, wherein the detecting means detects the direction of the movement, and the class tap selecting means selects the class tap having a positional relationship corresponding to the direction of the movement. An image processing apparatus according to claim 1. 4. The apparatus according to claim 1, wherein the detecting means detects the magnitude of the movement, and the class tap selecting means selects the class tap at a distance corresponding to the magnitude of the movement. An image processing apparatus according to claim 1. 5. The image processing apparatus according to claim 1, wherein the prediction execution unit predicts a pixel of a frame. 6. A class detection step for detecting a movement, a class tap selection step for selecting a class tap at intervals corresponding to the detected movement, and a class calculation for calculating a class based on the selected class tap. An image processing method, comprising: a step of performing a prediction using a prediction method corresponding to the calculated class. 7. A motion detecting step for detecting a motion, a class tap selecting step for selecting a class tap at an interval corresponding to the detected motion, and a class calculation for calculating a class based on the selected class tap. A recording medium in which a computer-readable program is recorded, comprising: a step; and a prediction execution step of performing prediction by a prediction method corresponding to the calculated class. 8. A motion detecting means for detecting a motion, a class tap selecting means for selecting a class tap, a class calculating means for calculating a class based on the selected class tap, and a function corresponding to the detected motion. Prediction tap selection means for selecting a prediction tap at an interval to be calculated, and prediction execution means for executing prediction based on the calculated prediction coefficient corresponding to the class and the selected prediction tap. Image processing device. 9. The prediction tap selection unit according to claim 8, wherein the total number of the prediction taps is constant.
An image processing apparatus according to claim 1. 10. The apparatus according to claim 8, wherein the detecting means detects the direction of the movement, and the prediction tap selecting means selects the prediction tap having a positional relationship corresponding to the direction of the movement. An image processing apparatus according to claim 1. 11. The apparatus according to claim 8, wherein the detecting means detects the magnitude of the motion, and the predictive tap selecting means selects the predictive tap at a distance corresponding to the magnitude of the motion. An image processing apparatus according to claim 1. 12. The image processing apparatus according to claim 8, wherein the prediction execution unit predicts a pixel of a frame. 13. A motion detecting step for detecting a motion, a class tap selecting step for selecting a class tap, a class calculating step for calculating a class based on the selected class tap, and a motion corresponding to the detected motion. A prediction tap selecting step of selecting a prediction tap at an interval to be performed; and a prediction execution step of performing prediction based on the calculated prediction coefficient corresponding to the class and the selected prediction tap. Image processing method. 14. A motion detecting step for detecting a motion, a class tap selecting step for selecting a class tap, a class calculating step for calculating a class based on the selected class tap, and a motion corresponding to the detected motion. A prediction tap selecting step of selecting a prediction tap at an interval to be performed; and a prediction execution step of performing prediction based on the calculated prediction coefficient corresponding to the class and the selected prediction tap. A recording medium on which a computer-readable program is recorded. 15. A learning data generating means for generating learning data from input teacher data, a motion detecting means for detecting a motion of the learning data, and a class having an interval corresponding to the detected motion from the learning data. Class tap selecting means for selecting a tap, class calculating means for calculating a class based on the selected class tap, and prediction coefficient calculating means for calculating a prediction coefficient corresponding to the class from the teacher data. An image processing apparatus comprising: 16. A learning data generating step of generating learning data from input teacher data, a motion detecting step of detecting a motion of the learning data, and a class having an interval corresponding to the detected motion from the learning data. A class tap selecting step of selecting a tap, a class calculating step of calculating a class based on the selected class tap, and a predictive coefficient calculating step of calculating a predictive coefficient corresponding to the class from the teacher data. An image processing method comprising: 17. Learning data generating means for generating learning data from input teacher data, motion detecting means for detecting a motion of the learning data, and class tap selecting means for selecting a class tap from the learning data. A class calculation unit that calculates a class based on the selected class tap; a prediction tap selection unit that selects a prediction tap at an interval corresponding to the detected motion from the learning data; and the selected prediction From the taps and the teacher data,
A prediction coefficient calculation unit configured to calculate a prediction coefficient corresponding to the class. 18. A learning data generating step of generating learning data from input teacher data, a motion detecting step of detecting a motion of the learning data, and a class tap selecting step of selecting a class tap from the learning data. A class calculation step of calculating a class based on the selected class tap; a prediction tap selection step of selecting a prediction tap at an interval corresponding to a detected motion from the learning data; From the taps and the teacher data,
A prediction coefficient calculating step of calculating a prediction coefficient corresponding to the class.