JP2000348019A - Data processor, data processing method and medium - Google Patents

Data processor, data processing method and medium

Info

Publication number
JP2000348019A
JP2000348019A JP16052899A JP16052899A JP2000348019A JP 2000348019 A JP2000348019 A JP 2000348019A JP 16052899 A JP16052899 A JP 16052899A JP 16052899 A JP16052899 A JP 16052899A JP 2000348019 A JP2000348019 A JP 2000348019A
Authority
JP
Japan
Prior art keywords
data
processing
prediction
tap
value
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
JP16052899A
Other languages
Japanese (ja)
Other versions
JP4135045B2 (en
Inventor
Tetsujiro Kondo
Yoshinori Watanabe
義教 渡邊
哲二郎 近藤
Original Assignee
Sony Corp
ソニー株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Corp, ソニー株式会社 filed Critical Sony Corp
Priority to JP16052899A priority Critical patent/JP4135045B2/en
Priority claimed from US09/587,865 external-priority patent/US6678405B1/en
Publication of JP2000348019A publication Critical patent/JP2000348019A/en
Application granted granted Critical
Publication of JP4135045B2 publication Critical patent/JP4135045B2/en
Anticipated expiration legal-status Critical
Expired - Fee Related legal-status Critical Current

Links

Abstract

PROBLEM TO BE SOLVED: To improve data processing capability. SOLUTION: A tap is constituted so that the standard deviation of the tap used at the time of processing a pixel is matched with a prescribed reference value. When a tap of 3×3 pixels is constituted, a tap A whose tap width is '0' is constituted on a pixel and a tap B whose tap width is '1' is constituted on the other pixel.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a data processing apparatus, a data processing method, and a medium, and more particularly to a data processing apparatus for improving the processing performance of data processing of image data and the like. The present invention relates to an apparatus, a data processing method, and a medium.

[0002]

2. Description of the Related Art The applicant of the present invention has previously proposed a class classification adaptive process as a process for improving the image quality of an image and other processes for improving an image.

The class classification adaptive processing includes class classification processing and adaptive processing. Data is classified into classes based on the nature of the data by the class classification processing, and the adaptive processing is performed for each class. Is processing of the following method.

[0004] That is, in the adaptive processing, for example, an original image (hereinafter, appropriately referred to as an input pixel) constituting an input image (an image to be processed by the class classification adaptive processing) is linearly combined with a predetermined prediction coefficient to obtain an original image. By calculating the predicted values of the pixels of an image (for example, an image that does not include noise or an image that does not have blurring), an image from which noise included in the input image has been removed or a blur that has occurred in the input image has been improved. Images and the like can be obtained.

Therefore, for example, an original image is now used as teacher data, and noise is superimposed on the original image,
Alternatively, a predicted value E [y] of a pixel value y of a pixel constituting the original image (hereinafter, appropriately referred to as an original pixel) is used as student data (pixel value) x 1 as student data using the image to which the blur is added. , X 2 ,... And a predetermined prediction coefficient w 1 ,
Let us consider a case where a linear combination model defined by a linear combination of w 2 ,... In this case, the predicted value E
[Y] can be expressed by the following equation.

E [y] = w 1 x 1 + w 2 x 2 +... (1) To generalize equation (1), a matrix W composed of a set of prediction coefficients w, A matrix X consisting of a set and a matrix Y ′ consisting of a set of predicted values E [y] are

[0007]

(Equation 1) Defines the following observation equation.

XW = Y ′ (2) where the component x ij of the matrix X is a set of i-th student data (a set of student data used for predicting the i-th teacher data y i ). In the matrix W, and the component w j of the matrix W represents a prediction coefficient by which a product with the j-th student data in the set of the student data is calculated. Y j is
represents the j-th teacher data, so E [y j ] is j
Represents the predicted value of the teacher data of the subject.

Then, it is considered that a least square method is applied to this observation equation to obtain a predicted value E [y] close to the pixel value y of the original pixel. In this case, a matrix Y consisting of a set of true pixel values y of the original pixels serving as teacher data and a matrix E consisting of a set of residuals e of predicted values E [y] with respect to the pixel values y of the original pixels are given by:

[0010]

(Equation 2) From equation (2), the following residual equation is established.

XW = Y + E (3) In this case, a predicted value E [y] close to the pixel value y of the original pixel is obtained.
Prediction coefficient w iIs the square error

[0012]

(Equation 3) Can be obtained by minimizing.

Therefore, when the above-mentioned square error is differentiated by the prediction coefficient w i , the result becomes 0, that is, the prediction coefficient w i that satisfies the following equation is the prediction value E [y] close to the pixel value y of the original pixel. Is the optimum value.

[0014]

(Equation 4) (4) Then, first, the following equation is established by differentiating equation (3) with the prediction coefficient w i .

[0015]

(Equation 5) (5) From the expressions (4) and (5), the expression (6) is obtained.

[0016]

(Equation 6) (6) Further, the student data x in the residual equation of the equation (3),
Considering the relationship between the prediction coefficient w, the teacher data y, and the residual e, the following normal equation can be obtained from Expression (6).

[0017]

(Equation 7) (7) Each of the equations constituting the normal equation of the equation (7) is represented by student data x
By preparing a certain number of training data y and teacher data y, the same number as the number of prediction coefficients w to be obtained can be obtained. Therefore, solving equation (7) (provides equation (7) In order to solve, in Expression (7), a matrix composed of coefficients related to the prediction coefficient w needs to be regular),
An optimal prediction coefficient w can be obtained. In solving equation (7), for example, the sweeping method (Gauss
s-Jordan elimination) can be used.

As described above, the optimum prediction coefficient w is obtained, and the prediction value E [y] close to the pixel value y of the original pixel is obtained by the equation (1) using the prediction coefficient w. This is the adaptive processing.

The adaptive processing differs from, for example, simple interpolation processing in that components not included in the input image but included in the original image are reproduced. That is, the adaptive processing is the same as the interpolation processing using the so-called interpolation filter as far as only the equation (1) is viewed, but the prediction coefficient w corresponding to the tap coefficient of the interpolation filter is obtained by using the teacher data y. Therefore, the components included in the original image can be reproduced. That is, easily
Images with high S / N can be obtained. From this, it can be said that the adaptive processing has a so-called image creation (resolution imagination) action. Therefore, in addition to obtaining a predicted value of an original image in which noise and blur have been removed from an input image, for example, low-resolution or It can also be used when converting a standard resolution image into a high resolution image.

[0020]

As described above, in the class classification adaptive processing, the adaptive processing is performed for each class, but in the class classification performed in the preceding stage, the original pixel whose prediction value is to be obtained is determined. (Hereinafter referred to as the original pixel of interest as appropriate)
Are extracted, and the target original pixels are classified into classes based on their properties (for example, the pattern of the pixel values of the plurality of input pixels, the inclination of the pixel values, etc.). Then, as a plurality of input pixels used for this class classification, input pixels at fixed positions when extracted from the target original pixel are extracted.

However, for example, when an input image having a blur is converted into an image in which the blur is improved by the class classification adaptive processing, regardless of the degree of the blur of the input image, the image is viewed from the original pixel of interest. If an input pixel at a fixed position is used for classifying the original pixel of interest, it may be difficult to classify the original pixel sufficiently reflecting the properties of the original pixel of interest.

That is, for example, in the case where the classification adaptive processing is performed on an input image having a small degree of blur (degree of blur), from the viewpoint of image correlation, a position relatively close to the original pixel of interest is considered. When the classification is performed by using the input pixels in (1), the classification can be performed by reflecting the properties of the original pixel of interest. In addition, when performing the classification adaptation process on an input image having a large degree of blur, from the viewpoint of the effect of the blur, the classification is performed using input pixels that are relatively far from the original pixel of interest. If you do,
Classification that reflects that property can be performed.

Therefore, if an input pixel located at a fixed position with respect to the target original pixel is used for classifying the target original pixel, it may not be possible to classify the target original pixel by reflecting the properties of the target original pixel. As a result, the processing performance of the classification adaptive processing deteriorates, that is, an image in which the input image is sufficiently improved (here, an image in which the blur is sufficiently improved) may not be obtained by the classification processing.

The present invention has been made in view of such a situation, and it is an object of the present invention to improve the processing performance of, for example, a classification adaptive processing.

[0025]

According to the present invention, there is provided a data processing apparatus comprising: determining means for determining a plurality of data to be extracted from input data based on statistics of the input data; For the output data of interest, which is the output data for which
It is characterized by including extraction means for extracting from input data, and prediction means for obtaining a predicted value of target output data based on a plurality of data extracted by the extraction means.

The data processing apparatus may further include class classification means for performing class classification on the output data of interest based on the plurality of data extracted by the extraction means and outputting a corresponding class code. The prediction means can determine the predicted value of the output data of interest using a predetermined prediction coefficient corresponding to the class code.

[0027] The determining means can determine a plurality of data to be extracted from the input data based on the standard deviation of the plurality of data. Further, the determining means can determine a plurality of data to be extracted for the output data in the block based on an average of a standard deviation of the plurality of data obtained for each of the output data in the predetermined block. Further, the determining means can determine the plurality of data to be extracted from the input data so that the statistic of the plurality of data is equal to a predetermined reference value.

The data processing apparatus of the present invention may further include a reference value storage means for storing a predetermined reference value.

The prediction means can linearly predict the output data using the prediction coefficients. Further, the prediction unit can perform linear primary prediction of the output data using the prediction coefficient and the plurality of data extracted by the extraction unit.

The data processing apparatus of the present invention may further include a prediction coefficient storage unit for storing a prediction coefficient for each class code.

The input data and the output data can be image data. Further, in this case, the extracting means can extract, from the image data as the input data, pixels spatially or temporally peripheral to the pixel as the target output data.

According to the data processing method of the present invention, a decision step for deciding a plurality of data to be extracted from the input data based on a statistic of the input data, and an output for obtaining a predicted value according to the decision in the decision step And extracting a plurality of data from the input data with respect to the output data of interest, and a prediction step of obtaining a predicted value of the output data of interest based on the plurality of data extracted in the extraction step. Features.

The program executed by the computer according to the medium of the present invention includes a determining step of determining a plurality of data to be extracted from the input data based on a statistic of the input data, and determining a predicted value according to the determination in the determining step. An extraction step of extracting a plurality of data from the input data with respect to the target output data to be output data, and a prediction step of obtaining a prediction value of the target output data based on the plurality of data extracted in the extraction step. It is characterized by including.

Another data processing apparatus according to the present invention comprises: generating means for generating student data to be a student from teacher data to be a teacher for learning a prediction coefficient; and student data based on a statistic of the student data. Determining means for determining a plurality of data to be extracted from the data, and according to the determination in the determining means,
Extraction means for extracting a plurality of data from the student data for the teacher data of interest, which is teacher data for which a prediction value is to be obtained; and calculation means for obtaining a prediction coefficient based on the plurality of data extracted by the extraction means. It is characterized by including.

The other data processing apparatus may further include class classification means for classifying the target teacher data based on the plurality of data extracted by the extraction means and outputting a corresponding class code. ,
In this case, the calculation means can determine the prediction coefficient for each class code.

The determining means can determine a plurality of data to be extracted from the student data based on the standard deviation of the plurality of data. Further, the determining means can determine a plurality of data to be extracted from the teacher data in the block based on an average of a standard deviation of the plurality of data obtained for each of the teacher data in the predetermined block. Further, the determining means can determine a plurality of data to be extracted from the student data so that the statistic of the plurality of data is equal to a predetermined reference value.

In another data processing apparatus of the present invention, reference value calculating means for obtaining a predetermined reference value can be further provided. In this case, when the student data is processed using the prediction coefficient obtained for the class code corresponding to the plurality of data, the processing result may be used as the reference value calculating means to obtain a plurality of data when the student data becomes closer to the teacher data. Statistics
It can be determined as a predetermined reference value.

The calculating means can obtain a prediction coefficient for obtaining teacher data by linear primary prediction. Further, the calculating means can determine a prediction coefficient for obtaining teacher data by linear primary prediction using the plurality of data extracted by the extracting means.

The teacher data and the student data can be image data. In this case, the extracting means can extract pixels spatially or temporally surrounding the pixel as the attention teacher data from the image data as the student data.

According to another data processing method of the present invention, a generating step of generating student data to be a student from teacher data to be a teacher for learning a prediction coefficient; A determining step of determining a plurality of data to be extracted from the extracting step, and extracting the plurality of data from the student data for the attention teacher data that is the teacher data to obtain the predicted value according to the determination in the determining step; And calculating a prediction coefficient based on the plurality of data extracted in the extraction step.

A program executed by a computer according to another medium of the present invention includes a generation step of generating student data to be a student from teacher data to be a teacher for learning a prediction coefficient; Determining a plurality of data to be extracted from the student data, and extracting a plurality of data from the student data with respect to the attention teacher data which is the teacher data for which a prediction value is to be obtained according to the determination in the determining step. It is characterized by including an extraction step and an operation step of obtaining a prediction coefficient based on a plurality of data extracted in the extraction step.

Still another data processing device of the present invention comprises:
First determining means for determining first plurality of data to be extracted from the input data based on the statistics of the input data; and output data for obtaining a predicted value in accordance with the determination by the first determining means. For a given output data, a first plurality of data is extracted from the input data, a first plurality of data, a first plurality of data extracted by the first extracting means, and a prediction coefficient. A prediction unit for obtaining a predicted value of the output data of interest; a generation unit for generating student data to be a student from teacher data to be a teacher for learning a prediction coefficient; and a student data based on a statistic of the student data. A second determination unit that determines a second plurality of data to be extracted, and attention teacher data that is teacher data for which a prediction value is to be obtained according to the determination by the second determination unit. A plurality of data extracted from the student data, and a calculating means for obtaining a prediction coefficient based on the second plurality of data extracted by the second extracting means. And

In the data processing apparatus, the data processing method, and the medium according to the present invention, a plurality of data to be extracted from the input data are determined based on the statistics of the input data, and a predicted value is obtained according to the determination. A plurality of data are extracted from the input data for the output data of interest, which is the output data to be described. Then, a predicted value of the output data of interest is obtained based on the plurality of extracted data.

In another data processing apparatus, a data processing method, and a medium according to the present invention, student data to be a student is generated from teacher data to be a teacher for learning a prediction coefficient. A plurality of data to be extracted from the student data are determined based on the student data. Then, in accordance with the determination, a plurality of data are extracted from the student data for the attention teacher data which is the teacher data for which the prediction value is to be obtained, and a prediction coefficient is obtained based on the extracted plurality of data. .

In still another data processing apparatus according to the present invention, a first plurality of data to be extracted from input data is determined based on a statistic of the input data, and a predicted value is calculated according to the determination. The first plurality of data is extracted from the input data for the output data of interest, which is the output data to be described. Then, a predicted value of the output data of interest is obtained based on the extracted first plurality of data and the prediction coefficient. On the other hand, student data to be a student is generated from teacher data to be a teacher for learning a prediction coefficient, and a second plurality of data to be extracted from the student data is determined based on the statistics of the student data. And
According to the determination, a second plurality of data is extracted from the student data for the teacher data of interest, which is teacher data for which a prediction value is to be obtained, and a prediction is made based on the extracted second plurality of data. A coefficient is determined.

[0046]

FIG. 1 shows a configuration example of an embodiment of an image processing apparatus to which the present invention is applied.

In this image processing apparatus, for example, when a blurred image is input as an input image, the input image is subjected to a class classification adaptive process, as shown in FIG. Regardless of the degree of blur of the input image, an image whose blur has been sufficiently improved (blur improved image)
Is output.

That is, this image processing apparatus comprises a frame memory 1, a class tap generation circuit 2, a prediction tap generation circuit 3, a class classification circuit 4, a coefficient RAM (Random Access Memory).
y) 5, a prediction operation circuit 6, and a tap determination circuit 7, to which an input image to be subjected to blur improvement is input.

The frame memory 1 is adapted to temporarily store an input image input to the image processing apparatus, for example, in frame units. In the present embodiment, the frame memory 1 is capable of storing a plurality of frames of input images by bank switching, whereby an input image input to the image processing apparatus is a moving image. However, such processing can be performed in real time.

The class tap generation circuit 2 pays attention to the original pixel (here, an ideal pixel without blur, which is completely blurred from the input pixels) for which a prediction value is to be obtained by the class classification adaptive processing. As an original pixel, an input pixel used for class classification of the target original pixel is extracted from the input image stored in the frame memory 1 in accordance with the tap information from the tap determination circuit 7, and is extracted as a class tap. The data is output to the classification circuit 4.

The prediction tap generation circuit 3 includes a prediction operation circuit 6
According to the tap information from the tap determination circuit 7, the input pixel used to obtain the predicted value of the target original pixel in
The input image is extracted from the input image stored in the frame memory 1 and supplied to the prediction operation circuit 6 as a prediction tap.

The class classification circuit 4 classifies the original pixel of interest based on the class tap from the class tap generation circuit 2 and assigns a class code corresponding to the resulting class to the coefficient RAM 5 as an address. It has been made to give. That is, the class classification circuit 4 performs, for example, 1-bit ADRC (Adaptive Dynamic Range Coding) processing on the class taps from the class tap generation circuit 2 and uses the ADRC code obtained as a class code as a coefficient.
The data is output to the RAM 5.

Here, in the K-bit ADRC processing, the maximum value MAX and the minimum value MIN of the pixel values of the input pixels constituting the class tap are detected, and DR = MAX-MIN is set as the local dynamic range of the set. Based on the dynamic range DR, input pixels forming the class tap are requantized to K bits. That is, from among the pixel values of the pixels constituting the class taps, the minimum value MIN is subtracted, and the subtracted value is divided (quantized) by DR / 2 K. Therefore, when the class tap is subjected to one-bit ADRC processing, the pixel value of each input pixel constituting the class tap is set to one bit. Then, in this case, a bit string obtained by arranging the 1-bit pixel values of the respective pixels constituting the class tap in a predetermined order is output as an ADRC code.

The coefficient RAM 5 stores prediction coefficients for each class obtained by performing learning in a learning device described later. When a class code is supplied from the class classification circuit 4, the coefficient RAM 5 corresponds to the class code. The prediction coefficient stored in the address is read out, and the prediction operation circuit 6
To be supplied.

The prediction operation circuit 6 supplies a prediction coefficient w, supplied from the coefficient RAM 5, for the class of the original pixel of interest.
Using the w 2 ,... and the prediction taps (pixel values of the respective pixels) x 1 , x 2 ,. By doing so, the predicted value E [y] of the target original pixel y is obtained, and this is output as the pixel value of the pixel whose blur has been improved.

The tap decision circuit 7 decides a plurality of input pixels forming a class tap and a prediction tap on the basis of the statistics of the input image stored in the frame memory 1 and forms the class tap and the prediction tap. Information about a plurality of input pixels (hereinafter, appropriately referred to as tap information) is supplied to the class tap generation circuit 2 and the prediction tap generation circuit 3.

That is, the tap determination circuit 7 basically uses, for example, a square class tap of 3 × 3 pixels in the horizontal and vertical directions with the input pixel located at the position of the target original pixel as the center pixel and the prediction. Taps (hereinafter, both are appropriately collectively referred to simply as taps) are referred to as class tap generation circuits 2.
Further, tap information to be configured by the prediction tap generation circuit 3 is output. However, the tap information indicates that the interval between pixels constituting the tap (hereinafter, appropriately referred to as tap width) is determined by the statistics of the input image.
It is different.

Specifically, for example, when the statistic of the input image is a certain value, the tap determination circuit 7
As shown in (A), tap information for forming a tap having a tap width of 0 (interval between pixels constituting the tap is 0), which is composed of 3 × 3 pixels centered on the center pixel, is output. Further, for example, when the statistic of the input image is another value, the tap determination circuit 7 determines that the tap has a 3 × 3 pixel centered on the center pixel as shown in FIG. Tap information for forming a tap having a width of 1 (an interval between pixels forming the tap is one pixel or one frame) is output.

Next, referring to the flowchart of FIG.
A blur improvement process performed by the image processing apparatus of FIG. 1 to improve blur of an input image will be described.

Input images (moving images) to be subjected to the blur improvement processing are sequentially supplied to the frame memory 1 in frame units, and the input images thus supplied in frame units are sequentially stored in the frame memory 1. Will be done.

Then, in step S1, the tap determination circuit 7 determines a plurality of input pixels forming a tap based on the statistics of the input image stored in the frame memory 1, and determines taps related to the plurality of input pixels. information,
Output to the class tap generation circuit 2 and the prediction tap generation circuit 3.

When the class tap generation circuit 2 or the prediction tap generation circuit 3 receives the tap information from the tap determination circuit 7, the class tap generation circuit 2 or the prediction tap generation circuit 3 determines in step S2 the class of the target original pixel whose prediction value is to be obtained in accordance with the tap information. A plurality of input pixels for forming a tap or a prediction tap are read from the frame memory 1 to configure a class tap or a prediction tap, respectively. The class tap or the prediction tap is supplied to the classification circuit 4 or the prediction operation circuit 6, respectively.

Upon receiving the class tap from the class tap generation circuit 2, the class classification circuit 4 classifies the target original pixel based on the class tap in step S3, and classifies the resulting class code into
Output to the coefficient RAM 5 as an address. Coefficient RAM
5 reads the prediction coefficient stored at the address corresponding to the class code from the class classification circuit 4 and supplies it to the prediction calculation circuit 6 in step S4.

In the prediction operation circuit 6, in step S5, the prediction tap from the prediction tap generation circuit 3 and the coefficient RA
The calculation shown in Expression (1) is performed using the prediction coefficient from M5 and the prediction value E of the target original pixel y is obtained.
[Y], that is, here, a pixel with improved blur is obtained, and the process proceeds to step S6. In step S6, the prediction operation circuit 6 outputs the predicted value E [y] of the target original pixel y obtained in step S5 as a pixel value in which the blur in the input pixel located at the same position as the target original pixel is improved, Proceed to step S7.

In step S7, it is determined whether or not all the original pixels in a predetermined block described later have been processed as target original pixels. If it is determined that the processing has not been performed, the process returns to step S2, and the process returns to step S2. Among the original pixels in the above, an original pixel that has not yet been set as the original pixel of interest is newly set as the original pixel of interest, and thereafter, the same processing is repeated. Therefore, in the present embodiment, taps are configured based on the same tap information for original pixels within the same block. That is, for original pixels in the same block, taps are formed from input pixels located at the same position as viewed from each original pixel.

If it is determined in step S7 that all the original pixels in the predetermined block have been processed as the original pixels of interest, the process proceeds to step S8 to determine whether there is a next block to be processed. That is, it is determined whether or not the frame memory 1 stores an input image corresponding to a block to be processed next. When it is determined in step S8 that the input image corresponding to the block to be processed next is stored in the frame memory 1,
Returning to step S1, the same processing is repeated thereafter. Therefore, tap information is newly determined for a block to be processed next, and taps are formed according to the tap information.

On the other hand, if it is determined in step S8 that the input pixel corresponding to the block to be processed next is not stored in the frame memory 1, the blur improvement processing is terminated.

FIG. 5 shows an example of the configuration of the tap determination circuit 7 of FIG.

When an original pixel in a predetermined block is determined as an original pixel of interest, the readout unit 11 stores an input pixel serving as a tap for the original pixel of interest in the frame memory 1.
(FIG. 1), and supplies the data to the standard deviation calculating unit 12.

That is, here, for example, one frame of the original image, one frame divided into several regions, or several frames (for example, from a frame immediately after a scene change to a frame immediately before the next scene change) ) Is set as one block, and tap information is determined for each block.
Reads out, from the frame memory 1, input pixels constituting a tap having a tap width according to the control of the determination unit 13 for each original pixel in a certain block as a target original pixel. , And the standard deviation calculation unit 12.

The standard deviation calculating section 12 calculates the standard deviation of the pixel value of the input pixel constituting the tap as a statistic of the tap from the reading section 11.
Further, the standard deviation calculation unit 12 calculates, for each block, the average value of the standard deviation score for taps having the same tap width.
Is calculated, and the average score is supplied to the determination unit 13 as an evaluation value of each tap width for each block. That is, the block is i, the tap width is j,
In this case, the standard deviation calculation unit 12 calculates score (i, j) represented by the following equation as an evaluation value of a tap having a tap width j configured for each original pixel in the block #i. , To the determination unit 13.

[0072]

(Equation 8) (8) Here, in the expression (8), M represents the number of original pixels in the block #i, and K represents the number of input pixels forming a tap. Further, V m, j, k represents the pixel value of the k-th input pixel in the tap having the tap width j configured for the m-th original pixel in the block #i. Mean m, j represents the average value of the taps having the tap width j (the average value of the pixel values of the input pixels constituting the taps), which is configured for the m-th original pixel in the block #i. Therefore, mean m, j is (V m, j, 1 + V m, j, 2 +...
+ V m, j, K ) / K.

The judgment section 13 compares the evaluation value score (i, j) from the standard deviation calculation section 12 with a predetermined reference value set in the reference value setting section 14, and based on the comparison result. ,
The reading unit 11 is controlled. further,
The determination unit 13 determines a tap width when forming a tap for an original pixel in a block including the target original pixel (hereinafter, appropriately referred to as a target block) based on the comparison result. It is designed to be output as information.

The reference value setting unit 14 determines that the blur of the input image is most improved (the classification result of the input pixel is
(When it comes closest to the original pixel corresponding to the input pixel)
The standard deviation of the pixel value of the input pixel (student data) constituting the tap is set as a reference value to be compared with the output of the standard deviation calculation unit 12. The method of obtaining the reference value will be described later.

Next, referring to the flowchart of FIG.
The tap determination processing for determining the structure of the tap, which is performed in the tap determination circuit 7 of FIG. 5, will be described.

In the tap determination process, when any of the original pixels in the target block is set as the target original pixel for the first time, in step S11, the reading unit 11 sets the original pixel set as the target original pixel in the target block. For example, an input pixel for forming a tap having a tap width of 0 is read from the frame memory 1 (FIG. 1), and the standard deviation
To supply.

Then, the process proceeds to a step S12, wherein the standard deviation calculating section 12 obtains the standard deviation of the tap constituted by the input pixels from the reading section 11, and obtains the standard deviation at the step S1.
Proceed to 3. In step S13, it is determined whether or not processing has been performed with all original pixels in the block of interest set as the original pixels of interest.
Returning to 11, among the original pixels in the block of interest, those that have not yet been set as the original pixel of interest are newly set as the original pixels of interest, and thereafter, the same processing is repeated.

If it is determined in step S13 that all original pixels in the block of interest have been processed as the original pixels of interest, the process proceeds to step S14, where the standard deviation calculation unit 12 determines whether each original pixel of the block of interest has been processed. The average value of the standard deviation of the obtained taps is calculated.

That is, the current block is the block #i
If the tap width of the tap formed for each original pixel of the block of interest #i is represented by j, then in step S14, score (i, j) represented by Expression (8) is calculated. You. This score (i, j) is supplied to the determination unit 13.

In step S 15, the judgment section 13 judges whether or not score (i, j) from the standard deviation calculation section 12 is close to the reference value set in the reference value setting section 14. , That is, attention block #i
If the average value of the standard deviation of the taps of the tap width j configured for each of the original pixels is not close to the reference value, step S
Proceeding to 16, the determination unit 13 controls the reading unit 11 to change the tap width j, and returns to step S11.

In this case, in step S 11, in the reading unit 11, an input pixel for forming a tap having a tap width j + 1 is set as an original pixel having a target block as a target original pixel. It is read from the memory 1 and the same processing is repeated thereafter.

On the other hand, in step S15, when it is determined that the score (i, j) from the standard deviation calculating unit 12 is close to the reference value set in the reference value setting unit 14, that is, when the target block #i If the average value of the standard deviation of the taps of the tap width j configured for each original pixel is close to the reference value, the process proceeds to step S17, where the determination unit 13 outputs the tap width j as tap information and returns. .

As described above, in the class tap generation circuit 2 and the prediction tap generation circuit 3 in FIG. 1, the original pixel of the block is determined as the original pixel of interest in accordance with the tap information output by the tap determination circuit 7 for each block. When the tap information having the tap width j is output, the tap having the tap width j is formed for the original pixels forming the target block.

Next, FIG. 7 shows an embodiment of a learning device for obtaining a prediction coefficient for each class to be stored in the coefficient RAM 5 in FIG. 1 and for obtaining a predetermined reference value set in the reference value setting section 14 in FIG. 2 shows a configuration example of a mode.

An original image (here, an image without blur) serving as teacher data y is stored in the frame memory 61, for example.
The original image is supplied on a frame basis, and the frame memory 61 temporarily stores the original image. The blur adding circuit 62 reads an original image stored in the frame memory 61 and serving as teacher data y in learning the prediction coefficient, and adds blur to the original pixels constituting the original image (for example, using a low-pass filter). Filtering) to generate a blurred image as student data (hereinafter, appropriately referred to as a blurred image). This blurred image is supplied to the frame memory 63.

The frame memory 63 includes a blur adding circuit 62
Is temporarily stored.

Note that the frame memories 61 and 63
It has the same configuration as the frame memory 1 in FIG.

The class tap generation circuit 64 or the prediction tap generation circuit 65 is a pixel constituting a blurred image stored in the frame memory 63 (hereinafter referred to as a blurred pixel as appropriate).
In the same manner as the class tap generation circuit 2 or the prediction tap generation circuit 3 of FIG. 1, a class tap or a prediction tap is formed for the target original pixel according to the tap information from the tap determination circuit 72, and the class classification circuit 66 Alternatively, they are supplied to the addition circuit 67, respectively.

The class classification circuit 66 is configured in the same manner as the class classification circuit 4 of FIG. 1, classifies the target original pixel based on the class tap from the class tap generation circuit 64, and predicts the corresponding class code. An address is given to the tap memory 68 and the teacher data memory 70.

The addition circuit 67 reads the storage value of the address corresponding to the class code output from the class classification circuit 66 from the prediction tap memory 68, and forms the storage value and the prediction tap from the prediction tap generation circuit 65. By adding (the pixel value of) the blurred pixel, an operation corresponding to a summation (Σ) that is a multiplier of the prediction coefficient w on the left side of the normal equation of Expression (7) is performed. Then, the addition circuit 67 stores the calculation result in the prediction tap memory 68 corresponding to the class code output from the class classification circuit 66.
The address is overwritten and stored.

The prediction tap memory 68 reads out the stored value of the address corresponding to the class output from the classifying circuit 66 and supplies it to the adding circuit 67, and stores the output value of the adding circuit 67 at that address. Has become.

The addition circuit 69 reads out the original pixel of interest out of the original pixels constituting the original image stored in the frame memory 61 as teacher data y, and corresponds to the class code output from the classifying circuit 66. By reading the stored value of the address from the teacher data memory 70 and adding the stored value to the teacher data (original pixel) y read from the frame memory 61, the summation on the right side of the normal equation of Expression (7) is obtained. Perform the operation corresponding to (Σ). Then, the adding circuit 69 overwrites the result of the calculation on the address of the teacher data memory 70 corresponding to the class code output from the classifying circuit 66.

The addition circuits 67 and 69 also perform the multiplication in the equation (7). That is, the adder 67 multiplies the blurred pixels x forming the prediction taps, and the adder 69 also multiplies the blurred pixels x forming the prediction taps by the teacher data y. Accordingly, the adder 69 needs the blurred pixel x, which is read from the frame memory 63.

The teacher data memory 70 reads the stored value of the address corresponding to the class code output from the class classification circuit 66 and supplies it to the addition circuit 69, and stores the output value of the addition circuit 69 at that address. It has become.

The arithmetic circuit 71 sequentially reads out the stored values stored at the addresses corresponding to the respective class codes from the prediction tap memory 68 or the teacher data memory 70, and forms the normal equation shown in the equation (7). By solving this, a prediction coefficient for each class is obtained. That is, the arithmetic circuit 71 sets the normal equation of the equation (7) from the storage value stored in the prediction tap memory 68 or the teacher data memory 70 at the address corresponding to each class code, and solves it. Thus, a prediction coefficient for each class is obtained.

The tap determination circuit 72 performs the same tap determination processing as the tap determination circuit 7 in FIG. 1 to determine tap information on taps to be generated by the class tap generation circuit 64 and the prediction tap generation circuit 65, The data is supplied to the tap generation circuit 64 and the prediction tap generation circuit 65. Further, the tap determination circuit 72
A predetermined reference value used in the tap determination processing is also calculated.

Next, referring to the flowchart of FIG.
A description will be given of a learning process performed by the learning device of FIG. 7 to obtain a prediction coefficient and a predetermined reference value for each class.

An original image (moving image) as teacher data is supplied to the learning device in frame units, and the original images are sequentially stored in the frame memory 61. Further, the original image stored in the frame memory 61 is supplied to the blur adding circuit 62, where it is converted to a blur image. The blur adding circuit 62 is configured to generate blur images having different degrees of blur, for example, for each frame.

The blurred image obtained by the blur adding circuit 62 is
The student data is sequentially supplied to the frame memory 63 and stored therein.

As described above, when the blurred images corresponding to all the original images prepared for the learning processing are stored in the frame memory 63, the tap determination circuit 72 proceeds to step S21. Is obtained as described later, and the process proceeds to step S22.

In step S22, tap determination circuit 72
In the same manner as in the tap determination circuit 7 of FIG. 1, a blur pixel that forms a tap is determined, and tap information on the blur pixel is output to the class tap generation circuit 64 and the prediction tap generation circuit 65.

In step S23, the class tap generation circuit 64 or the prediction tap generation circuit 65 forms a class tap or prediction tap for the original pixel of interest for which a prediction value is to be obtained in accordance with the tap information from the tap determination circuit 72. The blurred pixel to be blurred is the frame memory 6
3 to form class taps or prediction taps. The class tap or the prediction tap is supplied to the classifying circuit 66 or the adding circuit 67, respectively.

In the classifying circuit 66, step S24
In the same manner as in the class classification circuit 4 in FIG. 1, the target original pixel is classified using the class tap from the class tap generation circuit 64, and the class code as a result of the classification is stored in the prediction tap memory. 68
And to the teacher data memory 70 as an address.

Then, the process proceeds to a step S25, where prediction taps (student data) or teacher data are added.

That is, in step S 25, the prediction tap memory 68 reads the stored value of the address corresponding to the class code output from the class classification circuit 66 and supplies it to the addition circuit 67. The addition circuit 67 includes a storage value supplied from the prediction tap memory 68 and a prediction tap generation circuit 65.
Using the blurred pixels constituting the prediction taps supplied from, a calculation corresponding to a summation (Σ) which is a multiplier of a prediction coefficient on the left side of the normal equation of Expression (7) is performed. Then, the adding circuit 67 compares the calculation result with the class code output from the class classification circuit 66.
It is stored in the address of the prediction tap memory 68 by overwriting.

Further, at step S 25, the teacher data memory 70 reads out the stored value of the address corresponding to the class code output from the classifying circuit 66 and supplies it to the adding circuit 69. The addition circuit 69 includes a frame memory 61
, The target original pixel is read from the frame memory 63, and the necessary blurred pixel is read from the frame memory 63. The read pixel and the stored value supplied from the teacher data memory 70 are used to calculate the normal equation of the equation (7). An operation corresponding to the summation (Σ) on the right side is performed. Then, the adding circuit 69 stores the result of the operation in an overwritten form in the address of the teacher data memory 70 corresponding to the class code output from the class classification circuit 66.

Thereafter, the process proceeds to step S26, in which it is determined whether or not all the original pixels in the current block of interest have been processed as the original pixels of interest. Returning to S23, among the original pixels in the block of interest, original pixels that have not yet been set as the original pixel of interest are newly set as the original pixel of interest, and thereafter, the same processing is repeated.

If it is determined in step S26 that all the original pixels in the target block have been processed as the target original pixels, the process proceeds to step S27 to determine whether or not there is a next block to be processed. It is determined whether or not a blurred image corresponding to the block to be processed next is stored in the frame memory 63. Step S2
In step 7, if it is determined that the blurred image corresponding to the block to be processed next is stored in the frame memory 63, the process returns to step S22, and the block is set as a new target block. , One of the original pixels is newly set as the original pixel of interest, and the same processing is repeated thereafter.

On the other hand, if it is determined in step S27 that the blurred pixel corresponding to the block to be processed next is not stored in the frame memory 63, that is, if all the original pixels prepared in advance for learning have been stored. When the processing is performed using the image, the process proceeds to step S28, and the arithmetic circuit 7
1 is a prediction tap memory 68 or a teacher data memory 7
0, the stored values stored in the addresses corresponding to the respective class codes are sequentially read out, the normal equation shown in equation (7) is set, and this is solved to obtain a prediction coefficient for each class. Further, in step S29, the arithmetic circuit 71 outputs the obtained prediction coefficient for each class, and ends the learning process.

In the prediction coefficient learning process as described above, there may be a case where the number of normal equations required for obtaining the prediction coefficient cannot be obtained. For such a class, for example, For example, it is possible to output a default prediction coefficient.

FIG. 9 shows the tap decision circuit 72 of FIG.
2 shows a configuration example.

As shown in FIG.
Is a reading unit 81, a standard deviation calculating unit 82, a determining unit 8
3 and a reference value creating unit 84, of which the reading unit 81, the standard deviation calculating unit 82, or the determining unit 83 is the reading unit 11, the standard deviation calculating unit 1 in FIG.
2 or the determination unit 13. Therefore, the tap determination circuit 72 determines whether the reference value
, Except that a reference value creation unit 84 is provided.

The reference value creation unit 84 determines whether the reference value is generated in step S2 of FIG.
In 1, a reference value used for tap determination processing is obtained and supplied to the determination unit 83.

The tap determination circuit 7 configured as described above
In Step 2, as shown in the flowchart of FIG. 10, the same processing as in Steps S11 to S17 of FIG. 6 is performed in Steps S31 to S37, so that tap information is determined for each block.

Next, with reference to FIGS. 11 and 12, a description will be given of a method of calculating a reference value performed in the reference value creation unit 84 of FIG.

In the reference value creating section 84, all the original images (teacher data) prepared for the learning process are read from the frame memory 61, and the original images are
The blurred image (student data) to which the blur has been added is read from the frame memory 63. Further, the reference value creation unit 8
4 divides the original image for the blurred image into blocks as described above, and for each of the original pixels of each block,
Taps having different tap widths are formed, and a relationship between the tap width and the average value of the standard deviation of blurred pixels forming the taps is obtained.

That is, when attention is paid to a certain block, the reference value creating unit 84 sequentially converts the original pixels of the block into
A tap having a tap width of j is configured as the target original pixel.
Further, the reference value creation unit 84 obtains the standard deviation of the tap formed for each original pixel, and calculates the average value of the standard deviation in the block. Specifically, block #i
Is the total number of original pixels of M, K is the total number of blurred pixels forming the tap, and the pixel of the k-th blurred pixel in the tap having the tap width j is configured for the m-th original pixel in the block #i. The value is V m, j, k , and the average value of taps of tap width j (the average value of blurred pixels forming the taps), which is configured for the m-th original pixel in block #i, is mean
Assuming that they are represented by m and j , respectively, the reference value creation unit 84 calculates score (i, j) according to the above equation (8).

The reference value creation unit 84 changes the tap width j to several values, forms taps for each original pixel in the block #i, and sequentially averages the standard deviation of the taps of each tap width j. Calculate score (i, j).

Further, the reference value creation section 84 similarly calculates the average value of the standard deviations of the taps of each tap width j, score (i, j), for the other blocks #i.

As described above, the reference value creating section 84 sets the tap width j and the average value sc of the standard deviation of the taps of the tap width j for each block #i as shown in FIG.
The relationship with ore (i, j) is required.

FIG. 11 shows the relationship between the tap width j and the average value of the standard deviation of the tap score (i, j) for each of the four blocks # 1 to # 4. The mark for the block # 1 is for the block # 2, the mark for the block # 2 is for the block # 3, and the mark for the block # 4 is for the block # 4.

FIG. 11 shows that the tap width j is 0 to 1
The average value of the standard deviation of taps when each is 0 scor
e (i, j).

The average value of the standard deviation score (i, j) of the tap formed for each original pixel of block #i is represented by
#i represents the degree of change in the pixel value of the blurred image with respect to #i, that is, the degree of blurring. Therefore, the larger (smaller) score (i, j), the smaller (larger) the degree of blurring .

As described above, the reference value creating section 84
The average value of the standard deviation of the taps, score (i, j), is determined, and at the same time, the prediction value of the original pixel is determined by the class classification adaptive processing using the taps of each tap width j.

That is, the reference value creation unit 84 generates a normal equation of Expression (7) by fixing the tap width j to a certain value using the original pixel and the blurred image in the block #i, and By solving, the prediction coefficient for each class is obtained. further,
The reference value creating unit 84 fixes the tap width j to the above-mentioned certain value using the prediction coefficient for each class, and obtains the equation (1)
By calculating the linear prediction formula, a prediction value of each original image in the block #i is obtained.

The reference value creating section 84 performs the same processing by sequentially changing the tap width j, and further performs the same processing for the other blocks. As a result, a prediction value e (i, j) of an original pixel of each block #i when a tap having a tap width j is used is obtained. That is, each block
For the #i original pixel, the relationship between the tap width j of the tap used for the prediction of the original pixel and the predicted value e (i, j) of the original pixel is determined.

Then, the reference value creation section 84 determines how close the predicted value e (i, j) obtained as described above is to the original image. That is, the reference value creation unit 84, for example,
For block #i, the reciprocal of the sum of absolute differences between the original pixel and the predicted value e (i, j) obtained using the tap having the tap width j, etc., is calculated as the S / N of the predicted value e (i, j). (Signal to Noise Ratio). Here, FIG. 12 shows the S / N of the predicted value e (i, j) obtained for the four blocks #i shown in FIG.

In FIG. 12, for block # 1,
Predicted value e (1,0) obtained using taps with tap width j of 0
Has the best S / N, that is, its predicted value e (1,0)
Is closest to the original image, and the reference value creating unit 84 predicts the prediction value e closest to the original image in this way.
The block #i from which (i, j) is obtained and the tap width j are detected. Here, the block #i or the tap width j detected as described above is hereinafter appropriately referred to as an optimum block or an optimum tap width, respectively.

For the optimum block, the predicted value e (i, i) when the tap width j is obtained by forming the tap having the optimum tap width
The fact that j) is closest to the original image means that the standard deviation of the tap when the tap having the optimal tap width is formed for the optimal block (hereinafter referred to as the optimal standard deviation as appropriate) is the other block. It is considered that this is also the most suitable value as the standard deviation of the tap configured for.

That is, from a statistical point of view, if a tap having a tap width such that the standard deviation of the other blocks is the optimum standard deviation is formed, the predicted value closest to the original image can be obtained. Conceivable.

Therefore, the reference value creating section 84 finds the optimum standard deviation, and supplies this to the determination section 83 as a predetermined reference value.

Here, in FIG. 12, for block # 1, a predicted value e obtained using a tap having a tap width j of 0 is used.
Since the S / N of (1,0) is the best, the reference value creation unit 8
4, in FIG. 11, obtained for block # 1
tap width j of score (1, j) (shown by ○ in FIG. 11)
Is 0 when 0 is obtained as the optimum standard deviation.

Further, in this case, for the other blocks, the value indicated by the thick arrow in FIG. 11 is the tap width j at which the standard deviation matching the optimal standard deviation is obtained,
Accordingly, such a value is determined as the tap width j in the tap determination processing of FIGS. 6 and 10.

Although the tap width j takes an integer value, the tap width (a value indicated by a thick arrow in FIG. 11) at which a standard deviation coinciding with the optimum standard deviation is obtained is as shown in FIG.
As is clear from, it does not always take an integer value (rather, it often does not take an integer value). Therefore, in the tap determination processing of FIGS. 6 and 10, when the tap width at which the standard deviation matching the optimal standard deviation is not an integer value, for example, the integer value closest to the non-integer tap width is: The final tap width is determined.

That is, in the tap determination processing of FIG.
Similarly, as described above, the standard deviation calculation unit 1
It is determined whether or not score (i, j) from 2 is close to the reference value set in the reference value setting unit 14 (step S15).
If it is determined that they are close, the tap width j at that time is output as tap information (step S17), but here, "score (i, j) is close to the reference value". That is, when a tap having a tap width j that takes an integer value is configured, the standard deviation of the tap (in this embodiment, the average value of the standard deviation obtained for the block) scor
e (i, j) means that it is closest to the reference value.

In the tap determination processing of FIGS. 6 and 10, a tap width at which a standard deviation matching the optimal standard deviation is obtained is obtained. If the tap width is not an integer, the average of the tap widths is It is also possible to configure a tap having a tap width other than the integer value. That is, if the tap width at which a standard deviation matching the optimal standard deviation is obtained is, for example, 1.5, the tap width between certain pixels is set to 1 and the tap width between other pixels is set to 2
, And the average of the tap widths of the taps can be 1.5.

As described above, the optimum standard deviation is obtained, and a tap having a tap width that provides a standard deviation that matches the optimum standard deviation is configured to perform the learning process and the blur improvement process. Their processing performance can be improved.

That is, in the learning process, classification suitable for the degree of blur of the image is performed, and a prediction tap suitable for the degree of blur is formed. As a result, it is possible to obtain a prediction coefficient for each class suitable for the degree of blur of the image. Also, in the blur improvement process, a classification suitable for the degree of blur of the image is performed, and a prediction tap suitable for the degree of blur is formed.Further, using a prediction coefficient for each class suitable for the degree of blur, A predicted value of the original pixel is determined. As a result, a clear image (an image with improved blur) can be obtained regardless of the degree of blur of the image.

More specifically, for example, as described above, when performing the classification adaptive processing on an input image having a small degree of blur, an input pixel relatively close to the target original pixel is selected. If you use the classification, you can perform the classification that reflects the properties of the original pixel of interest,
In addition, when performing the classification adaptive processing on an input image having a large degree of blur, it is better to perform the classification using input pixels that are relatively far from the target original pixel. Classification can be performed, but in the case shown in FIG. 11, such classification is performed.

That is, in the case shown in FIG. 11, as described above, the larger the score (i, j) is, the smaller the degree of blur is. Therefore, the blocks # 1, # 2, # 3, and # 4 are arranged in this order. The degree of blur is increasing. Then, in the case shown in FIG. 11, from the above, taps having tap widths of 0, 1, 3, and 8 are formed for each of blocks # 1, # 2, # 3, and # 4.

Therefore, according to the method of the present invention, as the degree of blur increases, a class tap is formed using input pixels that are farther from the target original pixel. Will be performed.

Next, the above-described series of processing can be performed by hardware or software. When a series of processing is performed by software, a program constituting the software is
It is installed in a computer incorporated in an image processing device or a learning device as dedicated hardware, or a general-purpose computer that performs various processes by installing various programs.

With reference to FIG. 13, a description will now be given of a medium used to install a program for executing the above-described series of processes in a computer and to make the computer executable.

As shown in FIG. 13A, the program can be provided to the user in a state where the program is previously installed on a hard disk 102 as a recording medium built in the computer 101.

Alternatively, the program is executed as shown in FIG.
As shown in (B), a floppy (registered trademark) disk 111, a CD-ROM (Compact Disc Read Only Memory) 11
2, MO (Magneto optical) disc 113, DVD (Digital
Versatile Disc) 114, a magnetic disk 115, a semiconductor memory 116, or other such recording medium that can be temporarily or permanently stored and provided as package software.

Further, as shown in FIG. 13C, the program is transmitted from the download site 121 to the computer 1 via an artificial satellite 122 for digital satellite broadcasting.
23, wireless LAN, LAN (Local Area Network),
Via a network 111 such as the Internet,
The data is transferred to the computer 123 by wire, and
In 3, it may be stored in a built-in hard disk or the like.

The medium in the present specification means a broad concept including all these media.

In this specification, the steps of describing a program provided by a medium need not necessarily be processed in chronological order according to the order described in the flowchart, but may be performed in parallel or individually. (For example, parallel processing or object processing)
Is also included.

In the class classification application processing, learning is performed to obtain a prediction coefficient for each class using teacher data and student data, and linear primary prediction using the prediction coefficient and input data is performed to obtain a prediction coefficient from input data. Since the prediction value of the teacher data is obtained, a prediction coefficient for obtaining a desired prediction value can be obtained from the teacher data and the student data used for learning. That is, for example, a prediction coefficient for improving the resolution can be obtained by using a high-resolution image as teacher data and using an image with a reduced resolution of the image as student data. Further, for example, a prediction coefficient for removing noise can be obtained by using an image that does not include noise as teacher data and using an image obtained by adding noise to the image as student data. Therefore, the present invention can be applied to a case where noise is removed, a case where resolution is improved, and a case where, for example, waveform equalization is performed, in addition to a case where a blur is improved as described above.

In the present embodiment, the moving image is subjected to the classification application processing.
Furthermore, audio and signals reproduced from recording media (RF (R
adioFrequency) and the like.

Further, in the present embodiment, the class tap and the prediction tap are configured according to the same tap information, and therefore are configured by the same pixel. Different configurations, that is, configurations according to different tap information are also possible.

In the present embodiment, the tap width is made variable by configuring both the class tap and the prediction tap according to the tap information.
Either the class tap or the prediction tap can have a fixed tap width.

Furthermore, in the present embodiment, tap information is determined based on the standard deviation of taps.
Tap information can also be determined based on statistics other than the standard deviation of taps. That is, the tap information is based on, for example, the variance of the pixels constituting the tap, the sum of the absolute differences of the pixels, the sum of the secondary absolute differences (the sum of the absolute values of the differences between the pixels), and the like. It is possible to decide.

In the present embodiment, the tap width is changed for each block composed of a plurality of pixels.
The tap width can be changed, for example, in pixel units.

Further, in the present embodiment, the image processing apparatus and the learning apparatus for learning the prediction coefficient and the reference value for each class used in the image processing apparatus are configured as separate apparatuses. The processing device and the learning device may be integrally configured. And in this case,
The learning device can perform learning in real time, and can update the prediction coefficients used in the image processing device in real time.

In this embodiment, the coefficient RAM 5 stores
Although the prediction coefficient for each class is stored in advance, the prediction coefficient may be supplied to the image processing device together with the input image, for example. Similarly, the reference value can be supplied to the image processing device together with the input image instead of being set in the reference value setting unit 14 (FIG. 5).

Further, the class tap and the prediction tap may be configured using pixels in any of the spatial direction and the temporal direction.

In the present embodiment, the tap configuration (pixels constituting the tap) is changed by changing the tap width based on the statistics of the input image. In addition, for example, it can be changed by changing the position of a pixel forming a tap, or the like.

Further, in the present embodiment, the prediction value of the original pixel is obtained by a linear expression.
It is also possible to obtain it by a quadratic or higher equation.

In the learning process, as described above, the addition corresponding to the summation (Σ) of equation (7) is performed using the prediction tap, but the addition using the prediction tap having a different tap width is performed. The embedding is performed for the corresponding pixels of the prediction taps.

That is, as shown in FIG. 3, when the prediction tap is composed of nine pixels of 3 × 3 pixels centered on the center pixel, in the present embodiment, the prediction tap having a tap width of 0 (FIG. 3 (A)) or a prediction tap having a tap width of 1 (FIG. 3 (B)). In this case, for example, the upper left pixel of the prediction tap having the tap width of 0 is added to the upper left pixel of the prediction tap having the tap width of 1.

Further, the classification of the class taps having different tap widths is performed in the same manner. So, for example,
The pixel value of each pixel forming the prediction tap having the tap width of 0 shown in FIG. 3A is 1 and the tap width shown in FIG.
3A and 3B, the result of class classification using the class taps of FIGS. 3A and 3B becomes the same (for the same class). being classified).

[0163]

According to the data processing apparatus, the data processing method, and the medium of the present invention, a plurality of data to be extracted from the input data are determined based on the statistics of the input data, and the predicted value is determined in accordance with the determination. A plurality of data are extracted from the input data for the output data of interest, which is the output data for which is to be obtained. Then, a predicted value of the output data of interest is obtained based on the plurality of extracted data. Therefore, it is possible to obtain a predicted value close to the target output data.

According to the other data processing apparatus, data processing method, and medium of the present invention, student data to be a student is generated from teacher data to be a teacher for learning a prediction coefficient, and statistical data of the student data is generated. , A plurality of data to be extracted from the student data is determined. Then, in accordance with the determination, a plurality of data are extracted from the student data for the attention teacher data which is the teacher data for which the prediction value is to be obtained, and a prediction coefficient is obtained based on the extracted plurality of data. . Therefore, it is possible to obtain a prediction coefficient capable of obtaining a prediction value close to the teacher data.

According to still another data processing apparatus of the present invention, the first plurality of data to be extracted from the input data is determined based on the statistic of the input data, and the predicted value is calculated according to the determination. For the target output data to be output data, a first plurality of data is extracted from the input data. Then, a predicted value of the output data of interest is obtained based on the extracted first plurality of data and the prediction coefficient. On the other hand, student data to be a student is generated from teacher data to be a teacher for learning a prediction coefficient, and a second plurality of data to be extracted from the student data is determined based on the statistics of the student data. Then, according to the determination, a second plurality of data is extracted from the student data with respect to the attention teacher data which is the teacher data for which the prediction value is to be obtained, and based on the extracted second plurality of data. , A prediction coefficient is obtained. Therefore, by using the prediction coefficient, it is possible to obtain a prediction value close to the output data of interest.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration example of an embodiment of an image processing apparatus to which the present invention has been applied.

FIG. 2 is a diagram illustrating an outline of processing of the image processing apparatus of FIG. 1;

FIG. 3 is a diagram for explaining processing of a tap determination circuit 7 of FIG. 1;

FIG. 4 is a flowchart illustrating a blur improvement process performed by the image processing apparatus of FIG. 1;

FIG. 5 is a block diagram illustrating a configuration example of a tap determination circuit 7 of FIG. 1;

FIG. 6 is a flowchart for explaining tap determination processing by a tap determination circuit 7 of FIG. 5;

FIG. 7 is a block diagram illustrating a configuration example of an embodiment of a learning device to which the present invention has been applied.

FIG. 8 is a flowchart illustrating a learning process performed by the learning device of FIG. 7;

9 is a block diagram illustrating a configuration example of a tap determination circuit 72 in FIG. 7;

FIG. 10 is a flowchart illustrating a tap determination process by a tap determination circuit 72 of FIG. 9;

FIG. 11 is a diagram for explaining a process of a reference value creation unit 84 in FIG. 9;

FIG. 12 is a diagram for explaining a process of a reference value creation unit 84 in FIG. 9;

FIG. 13 is a diagram for explaining a medium to which the present invention is applied.

[Explanation of symbols]

1 frame memory, 2 class tap generation circuit,
3 prediction tap generation circuit, 4 class classification circuit, 5
Coefficient RAM, 6 prediction calculation circuit, 7 tap decision circuit, 11 reading unit, 12 standard deviation calculation unit,
13 judgment section, 14 reference value setting section, 61 frame memory, 62 blur adding circuit, 63 frame memory, 64 class tap generation circuit, 65 prediction tap generation circuit, 66 class classification circuit, 67 addition circuit, 68 prediction tap memory, 69 Adder circuit,
70 teacher data memory, 71 arithmetic circuit, 72 tap decision circuit, 81 readout section, 82 standard deviation calculation section, 83 judgment section, 84 reference value creation section, 1
01 computer, 102 hard disk, 1
03 semiconductor memory, 111 floppy disk,
112 CD-ROM, 113 MO disk, 114
DVD, 115 magnetic disk, 116 semiconductor memory, 121 download site, 122 satellite,
123 computers, 131 networks

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5B049 AA04 DD05 EE03 EE07 EE12 EE31 FF09 GG03 GG04 GG07 GG08 5B056 BB51 BB64 HH03 5B057 CA16 CB16 CE03 CG07 CH09 CH11 DC40

Claims (27)

[Claims]
1. A data processing apparatus for processing input data and predicting output data for the input data, wherein a plurality of data to be extracted from the input data is determined based on a statistic of the input data. Determining means, extracting means for extracting a plurality of data from the input data with respect to the output data of interest, which is the output data for which a prediction value is to be obtained, in accordance with the determination by the determining means; And a prediction unit for obtaining a predicted value of the output data of interest based on the plurality of data.
2. The method according to claim 1, further comprising: classifying the output data of interest based on the plurality of data extracted by the extracting unit, and outputting a corresponding class code. 2. The data processing apparatus according to claim 1, wherein a predicted value of the output data of interest is calculated using a predetermined prediction coefficient corresponding to.
3. The data processing apparatus according to claim 1, wherein the determining unit determines a plurality of data to be extracted from the input data based on a standard deviation of the plurality of data.
4. The plurality of data to be extracted for output data in a block based on an average of the standard deviations of the plurality of data obtained for each of the output data in a predetermined block. 4. The data processing device according to claim 3, wherein
5. The method according to claim 1, wherein the determining unit determines a plurality of data to be extracted from the input data such that a statistic of the plurality of data is equal to a predetermined reference value. The data processing device according to claim 1.
6. The data processing apparatus according to claim 5, further comprising a reference value storage unit that stores the predetermined reference value.
7. The data processing apparatus according to claim 2, wherein the prediction unit performs linear primary prediction on the output data using the prediction coefficient.
8. The prediction means uses the prediction coefficient and a plurality of data extracted by the extraction means,
The data processing apparatus according to claim 7, wherein the output data is subjected to linear linear prediction.
9. The data processing apparatus according to claim 2, further comprising a prediction coefficient storage unit that stores the prediction coefficient for each of the class codes.
10. The input data and the output data,
The data processing device according to claim 1, wherein the data is image data.
11. The image processing apparatus as claimed in claim 11, wherein the extracting unit extracts, from the image data as the input data, pixels that are spatially or temporally peripheral to the pixel as the target output data. The data processing device according to claim 10.
12. A data processing method for processing input data and predicting output data for the input data, wherein a plurality of data to be extracted from the input data is determined based on a statistic of the input data. A determination step; an extraction step of extracting, from the input data, a plurality of data for the output data of interest, which is the output data for which a prediction value is to be obtained, according to the determination in the determination step; A prediction step of obtaining a predicted value of the output data of interest based on the plurality of pieces of data.
13. A medium for causing a computer to execute a program for processing input data and performing data processing for predicting output data with respect to the input data, wherein the computer executes a program based on a statistic of the input data. Determining a plurality of data to be extracted from the input data; and extracting a plurality of data from the input data for the output data of interest, which is the output data for which a prediction value is to be obtained, according to the determination in the determining step. A program for causing the computer to execute a program, comprising: an extracting step of performing the following: and a predicting step of determining a predicted value of the output data of interest based on the plurality of data extracted in the extracting step.
14. A data processing device for processing input data and learning prediction coefficients used for predicting output data corresponding to the input data, comprising: Generating means for generating student data to be a student; determining means for determining a plurality of data to be extracted from the student data based on a statistic of the student data; and determining a predicted value according to the determination by the determining means. Extracting means for extracting a plurality of data from the student data for the teacher data of interest to be the teacher data to be calculated; and calculating means for obtaining the prediction coefficient based on the plurality of data extracted by the extracting means. A data processing device comprising:
15. The classifying means for classifying the teacher data of interest based on the plurality of data extracted by the extracting means and outputting a corresponding class code, wherein the calculating means comprises: 15. The data processing apparatus according to claim 14, further comprising: obtaining the prediction coefficient for each data.
16. The data processing apparatus according to claim 15, wherein said determining means determines a plurality of data to be extracted from said student data based on a standard deviation of said plurality of data.
17. The plurality of data to be extracted for teacher data in a block based on an average of the standard deviations of the plurality of data obtained for each of the teacher data in a predetermined block. 17. The data processing device according to claim 16, wherein:
18. The method according to claim 14, wherein the determining unit determines a plurality of data to be extracted from the student data so that a statistic of the plurality of data is equal to a predetermined reference value. The data processing device according to claim 1.
19. The data processing apparatus according to claim 18, further comprising a reference value calculating means for obtaining said predetermined reference value.
20. When the student data is processed by using the prediction coefficient obtained for a class code corresponding to the plurality of data, the reference value calculation unit brings the processing result closer to the teacher data. 20. The data processing apparatus according to claim 19, wherein a statistic of the plurality of data at that time is obtained as the predetermined reference value.
21. The data processing apparatus according to claim 15, wherein said calculation means obtains said prediction coefficient for obtaining said teacher data by linear primary prediction.
22. The arithmetic means for calculating the prediction coefficient for obtaining the teacher data by linear primary prediction using a plurality of data extracted by the extracting means. The data processing device according to claim 21.
23. The teacher data and student data are:
The data processing device according to claim 14, wherein the data is image data.
24. The image processing apparatus as claimed in claim 24, wherein the extraction unit extracts, from the image data as the student data, pixels spatially or temporally surrounding the pixel as the attention teacher data. 24. The data processing device according to 23.
25. A data processing method for processing input data and learning a prediction coefficient used for predicting output data corresponding to the input data, comprising: A generating step of generating student data to be a student; a determining step of determining a plurality of data to be extracted from the student data based on a statistic of the student data; and determining a predicted value according to the determination in the determining step. An extraction step of extracting a plurality of data from the student data for the teacher data of interest to be the teacher data to be obtained; and an operation step of obtaining the prediction coefficient based on the plurality of data extracted in the extraction step. A data processing method comprising:
26. A medium for causing a computer to execute a program for processing input data and performing data processing for learning a prediction coefficient used for predicting output data corresponding to the input data, the medium comprising: A generating step of generating student data to be a student from teacher data to be a teacher for learning; a determining step of determining a plurality of data to be extracted from the student data based on a statistic of the student data; According to the determination in the determining step, for the teacher data of interest, which is the teacher data for which a prediction value is to be obtained, an extraction step of extracting a plurality of data from the student data, and a plurality of data extracted in the extraction step. And calculating the prediction coefficient based on the , Medium for causing the computer to execute.
27. A data processing apparatus comprising: a first device for processing input data and predicting output data for the input data; and a second device for learning prediction coefficients used for predicting the output data. An apparatus, wherein the first apparatus is configured to determine a first plurality of data to be extracted from the input data based on a statistic of the input data, and the first determination means. According to the determination in the above, for the output data of interest, which is the output data for which the prediction value is to be obtained, a first plurality of data is extracted from the input data, A predicting unit that calculates a predicted value of the output data of interest based on the extracted first plurality of data and the prediction coefficient, wherein the second device performs learning of the prediction coefficient. Generating means for generating student data serving as a student from teacher data serving as a teacher for the second step; and determining a second plurality of data to be extracted from the student data based on a statistic of the student data. Means for extracting, from the student data, a second plurality of data for the teacher data of interest, which is the teacher data for which a prediction value is to be obtained, in accordance with the determination by the second determining means. And a calculating means for obtaining the prediction coefficient based on the second plurality of data extracted by the second extracting means.
JP16052899A 1999-06-08 1999-06-08 Data processing apparatus, data processing method, and medium Expired - Fee Related JP4135045B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP16052899A JP4135045B2 (en) 1999-06-08 1999-06-08 Data processing apparatus, data processing method, and medium

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
JP16052899A JP4135045B2 (en) 1999-06-08 1999-06-08 Data processing apparatus, data processing method, and medium
US09/587,865 US6678405B1 (en) 1999-06-08 2000-06-06 Data processing apparatus, data processing method, learning apparatus, learning method, and medium
EP00304812A EP1061473A1 (en) 1999-06-08 2000-06-07 Method and apparatus for classification-adaptive data processing
KR1020000031365A KR100746839B1 (en) 1999-06-08 2000-06-08 Data processing apparatus, data processing method, learning apparatus, learning method, and medium
CNB001264370A CN1190963C (en) 1999-06-08 2000-06-08 Data processing device and method, learning device and method and media

Publications (2)

Publication Number Publication Date
JP2000348019A true JP2000348019A (en) 2000-12-15
JP4135045B2 JP4135045B2 (en) 2008-08-20

Family

ID=15716925

Family Applications (1)

Application Number Title Priority Date Filing Date
JP16052899A Expired - Fee Related JP4135045B2 (en) 1999-06-08 1999-06-08 Data processing apparatus, data processing method, and medium

Country Status (1)

Country Link
JP (1) JP4135045B2 (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004326770A (en) * 2003-04-24 2004-11-18 Eastman Kodak Co Method and system for automatically decreasing aliasing artifact
WO2008038748A1 (en) * 2006-09-28 2008-04-03 Sony Corporation Prediction coefficient operation device and method, image data operation device and method, program, and recording medium
JP2009020844A (en) * 2007-07-13 2009-01-29 Morpho Inc Method for processing image data and imaging apparatus
JP2009151659A (en) * 2007-12-21 2009-07-09 Sony Corp Image processing apparatus, image processing method, program, and learning apparatus
US8181216B2 (en) 2001-12-27 2012-05-15 Sony Corporation Data processing apparatus, data processing method, and data processing system
US8355603B2 (en) 2002-04-26 2013-01-15 Sony Corporation Data converting apparatus and data converting method, learning device and learning method, and recording medium

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8181216B2 (en) 2001-12-27 2012-05-15 Sony Corporation Data processing apparatus, data processing method, and data processing system
US8355603B2 (en) 2002-04-26 2013-01-15 Sony Corporation Data converting apparatus and data converting method, learning device and learning method, and recording medium
JP2004326770A (en) * 2003-04-24 2004-11-18 Eastman Kodak Co Method and system for automatically decreasing aliasing artifact
JP2008109640A (en) * 2006-09-28 2008-05-08 Sony Corp Prediction coefficient operation device and method, image data operation device and method, program, and recording medium
WO2008038748A1 (en) * 2006-09-28 2008-04-03 Sony Corporation Prediction coefficient operation device and method, image data operation device and method, program, and recording medium
JP2009020844A (en) * 2007-07-13 2009-01-29 Morpho Inc Method for processing image data and imaging apparatus
JP2009151659A (en) * 2007-12-21 2009-07-09 Sony Corp Image processing apparatus, image processing method, program, and learning apparatus
US8331713B2 (en) 2007-12-21 2012-12-11 Sony Corporation Image processing apparatus, image processing method, program and learning apparatus

Also Published As

Publication number Publication date
JP4135045B2 (en) 2008-08-20

Similar Documents

Publication Publication Date Title
Rao et al. A survey of video enhancement techniques
CN104469379B (en) Generating output frames for inclusion in a video sequence
US7710498B2 (en) Image processing apparatus, image processing method and program
KR100859381B1 (en) Image processing apparatus and method, and image pickup apparatus
US6678414B1 (en) Loose-gray-scale template matching
US7336818B2 (en) Image processing device and method, and image-taking device
US6757431B2 (en) Resolution conversion for anti-aliased images using loose gray scale template matching
KR101023451B1 (en) Image processing device and method, learning device and method, recording medium, and program
JP3590996B2 (en) Hierarchical encoding and decoding apparatus for digital image signal
DE60127631T2 (en) Arrangement and method for image processing and recording carrier
JP4851690B2 (en) Method, apparatus and computer program for watermarking a sequence of video images
US7085436B2 (en) Image enhancement and data loss recovery using wavelet transforms
US7003163B2 (en) Image processing apparatus
Jeon et al. Coarse-to-fine frame interpolation for frame rate up-conversion using pyramid structure
EP1659532A2 (en) Image decoding apparatus, image decoding program, image decoding method, image encoding apparatus, image encoding program, and image encoding method
JP5628306B2 (en) Contrast improvement
US7295616B2 (en) Method and system for video filtering with joint motion and noise estimation
JP2009020879A (en) Method and system for enhancing data quality grade
US7050502B2 (en) Method and apparatus for motion vector detection and medium storing method program directed to the same
US7747069B2 (en) Data processing apparatus and data processing method
US20050175250A1 (en) Image decoding apparatus
EP1814075B1 (en) Method and apparatus for enhancing the resolution of a digital image
US8805120B2 (en) Image processing apparatus and method which generates reference data according to learning images
US7894522B2 (en) Classified filtering for temporal prediction
US9262684B2 (en) Methods of image fusion for image stabilization

Legal Events

Date Code Title Description
A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20060221

A977 Report on retrieval

Free format text: JAPANESE INTERMEDIATE CODE: A971007

Effective date: 20070413

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20070424

A521 Written amendment

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20070625

A02 Decision of refusal

Free format text: JAPANESE INTERMEDIATE CODE: A02

Effective date: 20071119

A521 Written amendment

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20080118

A911 Transfer of reconsideration by examiner before appeal (zenchi)

Free format text: JAPANESE INTERMEDIATE CODE: A911

Effective date: 20080124

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20080215

A521 Written amendment

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20080414

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20080508

A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20080521

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20110613

Year of fee payment: 3

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20120613

Year of fee payment: 4

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20130613

Year of fee payment: 5

R250 Receipt of annual fees

Free format text: JAPANESE INTERMEDIATE CODE: R250

R250 Receipt of annual fees

Free format text: JAPANESE INTERMEDIATE CODE: R250

LAPS Cancellation because of no payment of annual fees