JP2001195586A - Device and method for learning, and medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce the number of times of multiplication required for finding a covariant matrix for obtaining the coefficient of prediction for minimizing a square root error. SOLUTION: In a multiplication circuit 31, prescribed two multiplication values are operated among the SD pixels of low resolution to be used for linearly predicting the HD pixels of high resolution, and these multiplication values are stored for every relative position relation of two SD pixels used for finding these values in a product memory circuit 32. In a regular equation adding circuit 33, these stored multiplication values are integrated so that a regular equation for finding the coefficient of prediction can be generated.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a learning device, a learning method, and a medium. The present invention relates to a learning device, a learning method, and a medium that enable high-speed determination of a prediction coefficient used when performing linear prediction of a high density (High Density) image.

[0002]

2. Description of the Related Art The applicant of the present invention has previously proposed a class classification adaptive process as a process for improving the resolution of an image or for improving other images.

The class classification adaptive processing includes class classification processing and adaptive processing. Data is classified into classes based on the nature of the data by the class classification processing, and the adaptive processing is performed for each class. Is based on the following method.

That is, in the adaptive processing, for example, a linear combination of pixels constituting an SD image (hereinafter, referred to as SD pixels) and predetermined prediction coefficients is used to improve the resolution of the HD image. By calculating the predicted value of the pixel,
An image in which the resolution of the SD image is improved can be obtained.

More specifically, for example, while a certain HD image is used as teacher data, an SD image whose resolution is degraded by reducing the number of pixels of the HD image is used as student data, pixels forming the HD image are used. The prediction value E [y] of the pixel value y of the pixel value y (hereinafter, appropriately referred to as HD pixel)
Pixel values x ₁ , x ₂ ,... Of the pixels (pixels constituting the SD image)
.. And a linear combination model defined by a linear combination of predetermined prediction coefficients w ₁ , w ₂ ,... In this case, the predicted value E [y] can be expressed by the following equation.

E [y] = w ₁ x ₁ + w ₂ x ₂ +... (1)

To generalize equation (1), the prediction coefficient w
_A matrix W composed of a set of _j , a matrix X composed of a set of student data x _ij , and a matrix Y ′ composed of a set of predicted values E [y _j ]
To

(Equation 1) Defines the following observation equation.

XW = Y ′ (2) Here, the component x _ij of the matrix X is a set of i-th student data (a set of student data used for predicting the _i-th teacher data y _i ). In the matrix W, and the component w _{j of the} matrix W represents a prediction coefficient by which a product with the j-th student data in the set of the student data is calculated. Y _i is
represents the i-th teacher data, so E [y _i ] is i
Represents the predicted value of the teacher data of the subject. Note that y on the left side of the equation (1) is obtained by omitting the suffix i of the component y _i of the matrix Y, and x on the right side of the equation (1).
_1, x _2, · · · also, the component x _ij of the matrix X suffix i
Is omitted.

Then, it is considered that a least square method is applied to this observation equation to obtain a predicted value E [y] close to the pixel value y of the HD pixel. In this case, a matrix Y consisting of a set of true pixel values y of HD pixels serving as teacher data and a matrix E consisting of a set of residuals e of predicted values E [y] for the pixel values y of HD pixels are represented by:

(Equation 2) From equation (2), the following residual equation is established.

XW = Y + E (3)

In this case, a prediction coefficient w _j for obtaining a prediction value E [y] close to the pixel value y of the HD pixel is a square error.

(Equation 3) Can be obtained by minimizing.

Accordingly, if a differentiated by the prediction coefficient w _j squared error described above is zero, i.e., the prediction coefficient w _j satisfying the following equation, the predicted value close to the pixel value y of the HD pixel E [y]
Is the optimum value.

[0013]

(Equation 4) ... (4)

Therefore, first, equation (3) is calculated by using the prediction coefficient w _j
By differentiating with, the following equation is established.

[0015]

(Equation 5) ... (5)

From equations (4) and (5), equation (6) is obtained.

[0017]

(Equation 6) ... (6)

Further, considering the relationship among student data x _ij , prediction coefficient w _j , teacher data y _i , and residual e _{i in} the residual equation of equation (3), the following equation is obtained from equation (6). A normal equation can be obtained.

[0019]

(Equation 7) ... (7)

Each equation constituting the normal equation of equation (7) is
Student data x_ijAnd teacher data y _iThere is a set of
By preparing as many numbers as possible, the prediction coefficient w_j
Can be set as many as the number J of
Solving equation (7) (however, to solve equation (7),
In (7), the prediction coefficient w_jIs composed of coefficients
Matrix must be regular), the optimal prediction coefficient w_j
Can be requested. In solving equation (7),
For example, the sweeping method (Gauss-Jordan elimination method)
It is possible to use any of them.

As described above, the optimum prediction coefficient w _j is obtained in advance, and the prediction value E _j close to the pixel value y of the HD pixel is calculated by the equation (1) using the prediction coefficient w _j.
Finding [y] is adaptive processing.

The adaptive processing is performed in that components not included in the SD image but included in the HD image are reproduced.
For example, it is different from a simple interpolation process. That is, the adaptive processing is the same as the interpolation processing using the so-called interpolation filter as far as only the equation (1) is viewed, but the prediction coefficient w corresponding to the tap coefficient of the interpolation filter is obtained by using the teacher data y. Therefore, the components included in the HD image can be reproduced because they are obtained by learning. From this, it can be said that the adaptive processing has a so-called image creation (resolution imagination) action.

Also, here, the adaptive processing has been described by taking as an example the case where the resolution is improved. However, the adaptive processing is also applicable to, for example, obtaining a predicted value of an image in which noise or blur has been removed from the image. In this case, noise can be removed, blur can be improved, and the like.

FIG. 1 shows a configuration of an example of a conventional learning device that performs learning for obtaining the above-described prediction coefficient for each class.

An HD image as teacher data is supplied to the learning device on a frame basis, and the HD images are sequentially stored in the frame memory 1. The frame memory 1 stores a plurality of frames of H
The D image can be stored by bank switching or the like.
Even if the image is a moving image, the processing can be performed in real time.

The HD image as the teacher data stored in the frame memory 1 is decimated in the vertical direction or the horizontal direction by the vertical thinning filter 2 or the horizontal thinning filter 3, respectively, to obtain an SD image. That is,
In the vertical thinning filter 2, the number of pixels in the vertical direction of the HD image is thinned to, for example, 1/2, and the horizontal thinning filter 3
Is output to In the horizontal thinning filter 3, the number of pixels in the horizontal direction of the output of the vertical thinning filter 2 is, for example, 1/2.
As a result, as shown in FIG. 2, the HD image is an SD image in which the number of pixels in both the horizontal and vertical directions is halved. This SD image is supplied to and stored in a frame memory 4 configured similarly to the frame memory 1.

When the SD image is stored in the frame memory 4, the class tap configuration circuit 5 sets a predetermined HD pixel for which a predicted value is to be obtained by a class classification adaptive process as a target pixel, and further, the target pixel. Is used in a class classification for classifying into any one of several classes.
D pixels are extracted from the SD image stored in the frame memory 4. That is, the class tap configuration circuit 5 reads from the frame memory 4 some SD pixels spatially or temporally close to the position of the SD image corresponding to the position of the target pixel from the frame memory 4 and uses the tap ( Less than,
The class classification circuit 6
To supply.

The class classification circuit 6 classifies the pixel of interest based on the class tap from the class tap configuration circuit 5, and supplies a class code corresponding to the resulting class to the normal equation addition circuit 7. That is, the class tap from the class tap configuration circuit 5 is, for example, 1-bit ADRC (Adaptive Dynamic Ran
ge Coding) processing, and outputs the resulting ADRC code to the normal equation addition circuit 7 as a class code.

Here, in the K-bit ADRC processing, for example, the maximum value MAX and the minimum value MIN of the pixel values of the SD pixels constituting the class tap are detected, and DR = MAX-MIN is set to the local dynamic range of the set. Range, and based on this dynamic range DR, the SD pixels forming the class tap are K
Requantized to bits. That is, from among the pixel values of the pixels constituting the class taps, the minimum value MIN is subtracted, and the subtracted value is divided (quantized) by DR / 2 ^K. Therefore, when the class tap is subjected to 1-bit ADRC processing, the pixel value of each SD pixel constituting the class tap is set to 1 bit. Then, in this case, a bit string obtained by arranging the 1-bit pixel values of the respective pixels constituting the class tap in a predetermined order is obtained as described above.
Output as ADRC code.

When the class tap is composed of N SD pixels and the result of the K-bit ADRC processing of the class tap is a class code, the pixel of interest is (2
^N ) It will be classified into one of the ^K classes.

When the normal equation addition circuit 7 receives the class code of the pixel of interest from the class classification circuit 6, it performs several SD multiplications with prediction coefficients to linearly predict the pixel of interest.
Pixels are read from the frame memory 4 as prediction taps. That is, the normal equation adding circuit 7 uses several SD pixels spatially or temporally close to the position of the SD image corresponding to the position of the target pixel as prediction taps.
Read from the frame memory 4.

Here, the prediction tap can be composed of the same SD pixel as the class tap, or can be composed of a different SD pixel.

Further, the normal equation adding circuit 7 reads out the HD pixel as the target pixel from the frame memory 1 and performs addition for the prediction tap (student data) and the target pixel (teacher data).

That is, the normal equation adding circuit 7 uses a prediction tap (SD pixel) for each class corresponding to the class code supplied from the class classification circuit 6 to calculate the prediction on the left side of the normal equation of equation (7). Multiplication (x _in x _im ) of SD pixels (student data), which is the multiplier of the coefficient
And an operation corresponding to the summation (Σ) is performed.

Further, the normal equation adding circuit 7 also uses the prediction tap (SD pixel) and the target pixel (HD pixel) for each class corresponding to the class code supplied from the class classification circuit 6 to obtain the equation (7). ), Multiplication (x _in y _i ) of SD pixels (student data) and HD pixels (teacher data) and an operation corresponding to summation (Σ) are performed.

In the normal equation adding circuit 7, the above processing is performed for all the HD pixels stored in the frame memory 1.
This is performed as the pixel of interest, whereby the normal equation shown in equation (7) is established for each class.

After that, the prediction coefficient determination circuit 8
By solving the normal equation generated for each class in the normal equation adding circuit 7, a prediction coefficient for each class is obtained, and supplied to an address of the memory 9 corresponding to each class and stored.

In the above-described learning process of the prediction coefficients, there may be a case where a class in which the number of normal equations required for obtaining the prediction coefficient cannot be obtained may occur. For example, it is possible to output a default prediction coefficient.

Next, FIG. 3 shows an example of the configuration of the normal equation adding circuit 7 of FIG.

The normal equation adding circuit 7 includes the read circuit 1
1, and a preset normal equations equal in number to the number of H class configuring unit ₁₂₁ to 12 _H for class classifying HD pixel.

The readout circuit 11 is supplied with a class code output from the class classification circuit 6. When the readout circuit 11 receives the class code of the pixel of interest, the readout circuit 11 outputs the HD of the pixel of interest. Pixels are read from the frame memory 1 and SD pixels serving as prediction taps for the target pixel are stored in the frame memory 4.
Read from, and supplies the normal equation configuration section 12 _h which corresponds to the class code h of the pixel of interest (h = 1, 2, · ·
., H).

The normal equation constructing unit 12 _h uses the SD pixel as a prediction tap, and multiplies the SD pixel (student data), which is the multiplier of the prediction coefficient, on the left side of the normal equation of equation (7). x _in x _im ) and an operation corresponding to the summation (Σ), and using the SD pixel and the target pixel (HD pixel) which are prediction taps, the SD pixel ( Multiplication of student data) and HD pixel (teacher data) (x _in y _i )
And an operation corresponding to the summation (Σ) is performed.

[0043] In the normal equation configuration section 12 _h, class #h
The above-described multiplication and addition (operation equivalent to summation (）)) are performed on all the HD pixels classified into the following equation, whereby the normal equation shown in equation (7) for class #h is obtained. Can be set up.

Then, the normal equation is supplied to the prediction coefficient determination circuit 8, and the prediction coefficient is obtained as described above.

Next, as shown in FIG. 2, from the HD image as teacher data, when the SD image as student data are generated, for a certain target pixel y _i, as shown in FIG. 4, the It is assumed that four SD pixels are selected as constituting a prediction tap in order of spatial proximity from the position. In this case, a prediction tap having 2 × 2 pixels in the horizontal and vertical directions is configured.
Counting pixels from left to right and top to bottom, j
If the _{i th} pixel is expressed as x _ij , the upper left, upper right, lower left, and lower right SD pixels of the prediction tap are x _i1 , x _i2 ,
x _i3 and x _i4 .

[0046] When the prediction taps as described above constructed, the normal equation configuration section 12 _h of FIG. 3, for example, configured as shown in FIG.

That is, the normal equation shown in equation (7) is obtained by converting a matrix (covariance matrix) A and a vector v into

(Equation 8) If the vector W is defined as shown in Expression 1, it can be expressed by the following equation: AW = v (8)

For this reason, the normal equation constructing unit 12 _h includes a left side memory 21 composed of elements for obtaining each component of the matrix A on the left side in equation (8), and a component v of the right side vector v in equation (8). And a right side memory 22 composed of elements for obtaining

Elements constituting the left-side memory 21 and the right-side memory 22 are a multiplier for performing multiplication on SD pixels and HD pixels, an adder for adding (accumulating) a multiplication value output from the multiplier, and an adder. Is composed of a register for recording the integrated value output by the.

Then, assuming that the number of SD pixels constituting the prediction tap is J, the left side memory 21 stores J (J + 1) / 2
And the right side memory 22 stores the vector v
Has J elements equal to the dimension of That is, as shown in FIG. 4, when the prediction tap is composed of four SD pixels, the left side memory 21 is composed of ten elements, and the right side memory 22 is composed of four elements. You.

Here, since the matrix A is a J × J matrix, the left side memory 21 is simply composed of J × J elements. However, the matrix A is a symmetric matrix as shown in Expression 8, and the lower triangular component and the upper triangular component are symmetric with respect to the diagonal component. Therefore, if one of the lower triangular or upper triangular component and the diagonal component are obtained, all the components of the matrix A are obtained, and the number of elements in the left side memory 21 is J
(J + 1) / 2 pieces will suffice. In FIG. 5, the left side memory 21 is composed of an upper triangular component of the matrix A and ten elements for obtaining a diagonal component.

In the left side memory 21, the n-th matrix A
Two SD pixels x _in and x _im as prediction taps are read from the frame memory 4 (FIG. 1) and supplied to the element for performing the operation corresponding to the component in the row m-th column. And the elements are _xin and x
_im is multiplied, and the multiplied value (x _in x _im ) is integrated with the stored value already stored therein and stored. In addition,
In FIG. 5, n × m shown in a rectangle representing an element of the left side memory 21 indicates that multiplication of the SD pixel x _in and x _im is performed.

In the right side memory 22, for an element corresponding to the component of the n-th row of the vector v, an SD pixel x _in as a prediction tap and an HD pixel y as a pixel of interest
_i is read out from the frame memories 4 and 1 and supplied, respectively. In the element, x _in and y _i supplied thereto are multiplied, and the multiplied value (x _in y _i ) is obtained.
Is integrated with the stored value already stored therein and stored.

The HD as all the teacher data
By performing the above-described processing with the pixel as the target pixel,
Matrices A and v that define a normal equation for class #h are obtained. After that, the matrix A whose components are the stored values of the respective elements of the left-side memory 21 and the vector v whose components are the stored values of the respective elements of the right-side memory 22 are read and supplied to the prediction coefficient determination circuit 8. In the prediction decision circuit 8,
Based on the matrix A and the vector v, the prediction coefficient w _j (here, j = 1, 2, 3, 4) of the class #h is obtained.

Next, referring to the flowchart of FIG.
Carried out in the normal equation addition circuit 7 composed of the normal equation configuration section 12 _h as shown in FIG. 5, the normal equation configuration process for obtaining the matrix A and vector v for each class will be described.

In the normal equation construction process, first, in step S1, an array variable A [c] [n] [m] corresponding to the matrix A for each class and an array variable v corresponding to the vector v
[c] [n] are initialized. Here, the index c represents the class, and the index n or m represents the n-th row or the m-th column of the matrix A, respectively. Therefore, the array variable A [c]
[n] [m] represents the element of the n-th row and the m-th column of the matrix A for the class #c, and the array variable v [c] [n] is the element of the n-th row of the vector v for the class #c. Represents a component.

Thereafter, in step S2, a variable y representing the y coordinate of the HD pixel as the target pixel is initialized to, for example, 0, and the flow advances to step S3 to set a variable x representing the x coordinate of the HD pixel as the target pixel. , For example, is initialized to zero.
Note that, here, for simplicity of explanation, the teacher data is one frame, and the one frame is horizontal x vertical.
It is assumed that it is composed of x _max × y _max HD pixels. Furthermore, the xth from the left of such teacher data, y from the top
The coordinates of the HD pixel are represented by (x, y).

After the initialization of x and y, the process proceeds to step S4,
The variable y is incremented by 1, and the process proceeds to step S5. In step S5, it is determined whether or not the variable y is equal to or less than _ymax which is the number of pixels in the vertical direction of the teacher data.
If it is determined in step S5 that the variable y is equal to or _smaller than ymax, the process proceeds to step S6, the variable x is incremented by 1, and the process proceeds to step S7. In step S7, the variable x is x _max which is the number of pixels in the horizontal direction of the teacher data.
It is determined whether or not: If it is determined in step S7 that the variable x is not equal to or _smaller than x _max , the process returns to step S3, and the same processing is repeated.

In step S7, the variable x is set to x
_If it is determined to be less than _max , the HD at the coordinates (x, y)
The pixel is determined as the target pixel, and the process proceeds to step S8, where the class code of the class of the target pixel (supplied from the class classification circuit 6. The class code as the class classification result of the HD pixel at the coordinates (x, y)). Is set to the variable c.

Then, the process proceeds to a step S9, wherein a variable n representing a row of the matrix A is initialized to, for example, 0, and a step S1
Go to 0. In step S10, the variable n is incremented by one, and the process proceeds to step S11, where it is determined whether the variable n is equal to or less than N, which is the number of rows of the matrix A. If it is determined in step S11 that the variable n is not smaller than N, the process returns to step S6, and the same processing is repeated.

In step S11, the variable n
Is determined to be less than or equal to N, which is the number of rows of the matrix A, the process proceeds to step S12, and a variable m representing a column of the matrix A is determined.
Are initialized to, for example, 0, and the process proceeds to step S13. In step S13, the variable m is incremented by one, and the process proceeds to step S14, where it is determined whether the variable m is equal to or less than M, which is the number of columns of the matrix A. Step S1
In 4, when it is determined that the variable m is not less than M,
Returning to step S10, the same processing is repeated thereafter.

Here, the number of rows and the number of columns of the matrix A are both equal to the number of SD pixels constituting the prediction tap, and therefore, the above-mentioned M and N are the same value.

On the other hand, if it is determined in step S14 that the variable m is equal to or smaller than M, the process proceeds to step S15, and it is determined whether the variable m is equal to or larger than the variable n.
If it is determined in step S15 that the variable m is not greater than or equal to the variable n, step S16 is skipped and the process returns to step S13.

That is, when the component of the n-th row and the m-th column of the matrix A represented by the variables m and n is a lower triangular component, the component does not need to be obtained as described above. Is performed, the process returns to step S13.

In step S15, the variable m
Is determined to be greater than or equal to the variable n, that is, when the component of the n-th row and the m-th column of the matrix A represented by the variables m and n is a diagonal component or an upper triangular component, Step S
Proceeding to 16, the component A [c] [n] [m] of the n-th row and the m-th column of the matrix A for the class #c and the component v [c] [of the n-th row of the vector v for the class #c n] is calculated according to the following equation.

A [c] [n] [m] + = L [x + _Dx [n]] [y + _Dy [n]] × L [x + _Dx [m]] [y + _Dy [m]] v [c] [n] + = L [x + D _x [n]] [y + D _y [n]] × L '[x] [y] (9)

Then, the process returns to step S13, and thereafter, the same processing is repeated.

On the other hand, in step S5, the variable y
If it is determined that it is not less than _max , that is, if all HD pixels prepared as teacher data are processed as the pixel of interest, the normal equation configuration processing ends.

In equation (9), α + = β is α
The sum of β and β is set to α, that is,
It means the addition operation of β to α. Also, L [α]
[β] represents the pixel value of the SD pixel located at the coordinates (α, β), and L ′ [x] [y] represents the pixel value of the HD pixel located at the coordinates (x, y) (here, (Pixel value of the pixel of interest).

As shown in FIG. 7, D _x [t] or D _y [t] is a prediction tap viewed from the pixel of interest P (x, y) at the coordinates (x, y). The x-coordinate or the y-coordinate of the t-th SD pixel p # t that constitutes each is shown. Therefore, L [x + _Dx [n]] [y + _Dy [n]] in Expression (9) represents the pixel value of the n-th SD pixel forming the prediction tap for the target pixel, and L [ x
+ D _x [m]] [y + D _y [m]] represents the pixel value of the m-th SD pixel forming the prediction tap for the pixel of interest, so the right side of the first row of equation (9) Is the n-th matrix A
It matches x _in x _{im in} the component of the row m column.

[0071]

As described above, in the normal equation adding circuit 7, x _in x _{im in} each component of the matrix A shown _{in Expression 8} , that is, multiplication of the SD pixel x _in and x _im is performed. However, the multiplication value of two SD pixels (the multiplication value of two pixels in this specification includes the multiplication value of two pixels located at different positions in time and space, and the multiplication value of two SD pixels at the same position) The multiplication value between pixels is also included)
Since the calculation is performed when the two SD pixels become the prediction taps, the calculation may be performed twice or more times. In this case, the number of multiplications each other SD pixels, the number of pixels SD pixels constituting the prediction tap if J, the number of times proportional to the order of J ^2.

Accordingly, the number of multipliers required for multiplication between SD pixels or the multiplication time is also proportional to the order of J ² ,
As a result, when the number of SD pixels constituting the prediction tap increases, the device becomes large-scale, or a long time is required for processing.

The present invention has been made in view of such a situation, and is intended to reduce the size of an apparatus or increase the processing speed.

[0074]

A learning device according to the present invention comprises a multiplication means for calculating any two multiplication values of student data used for linearly predicting predetermined teacher data, and a multiplication value for the multiplication means. Generating a normal equation for obtaining a prediction coefficient by accumulating the multiplication values stored in the storage means for each relative positional relationship between the two student data used to obtain the prediction coefficient. And a normal equation generating means.

The learning device has an extracting means for extracting student data around a position corresponding to the position of the noted teacher data of interest, and the teacher data based on the student data extracted by the extracting means. May be further classified into any of a plurality of classes, and a class classifying unit that outputs a class code corresponding to the class may be further provided. In this case, the normal equation generating unit includes An equation is generated, and the prediction coefficient calculation means includes:
A prediction coefficient for each class can be obtained.

The storage means can store only the multiplied values that the normal equation generation means integrates twice or more.

The first and second data can be image data. Further, in this case, the second data is more S / N (Signal to Noise Rati) than the first data.
o) Image data that has deteriorated can be obtained. Further, the second data can be image data having a smaller number of pixels than the first data.

In the learning method of the present invention, a multiplication step of calculating any two multiplication values of student data used for linear prediction of predetermined teacher data, and the multiplication value is used to obtain the multiplication value. A storing step of storing for each relative positional relationship between the two student data, and a normal equation generating step of generating a normal equation for obtaining a prediction coefficient by integrating the multiplied values stored in the storing step. It is characterized by including.

The program executed by the computer according to the medium of the present invention includes a multiplication step of calculating any two multiplication values of student data used for linear prediction of predetermined teacher data, and a multiplication value of the multiplication step. 2 used to determine
A storage step of storing for each relative positional relationship between two pieces of student data, and a normal equation generation step of generating a normal equation for obtaining a prediction coefficient by integrating the multiplication values stored in the storage step It is characterized by the following.

In the learning apparatus, the learning method, and the medium of the present invention, any two multiplication values of the student data used for linear prediction of predetermined teacher data are calculated, and the multiplication value is calculated by the multiplication value. It is stored for each relative positional relationship between the two student data used for obtaining. Then, by integrating the stored multiplied values, a normal equation for obtaining the prediction coefficient is generated.

[0081]

FIG. 8 shows an example of the configuration of an embodiment of a learning apparatus to which the present invention is applied. In the figure,
Parts corresponding to those in FIG. 1 are denoted by the same reference numerals, and description thereof will be omitted below as appropriate. That is, the learning apparatus of FIG. 8 is different from that of FIG. 1 in that a multiplication circuit 31 and a product memory circuit 32 are newly provided, and a normal equation addition circuit 33 is provided instead of the normal equation addition circuit 7. Are configured in the same manner as in the above.

The multiplying circuit 31 reads out the SD pixels constituting the SD image as the student data stored in the frame memory 4, multiplies the pixel values, and supplies the multiplied values to the product memory circuit 32. I have. The product memory circuit 32 stores the multiplied value of the SD pixels output by the multiplying circuit 31 for each relative positional relationship between the two SD pixels used for obtaining the multiplied value. The normal equation addition circuit 33 appropriately reads out the stored value of the product memory circuit 32, and sets up a normal equation shown in Expression (7) for each class corresponding to the class code from the class classification circuit 6, that is, Equation 8 Is obtained for each class and supplied to the prediction coefficient determination circuit 8.

Next, referring to FIG. 9, the multiplication circuit 3 shown in FIG.
The process 1 will be described.

Now, the prediction tap is, for example, as shown in FIG.
As shown in FIG. 9, it is assumed that the SD pixels are composed of 2 × 2 SD pixels, and the four SD pixels which constitute the prediction tap are
As shown in (A), they are represented by p1, p2, p3, and p4, respectively.

The components of the matrix A are represented by S
It is composed of a multiplied value of two predetermined D pixels (including the same SD pixel). This multiplied value is obtained by selecting any two SD pixels from the prediction taps, as shown in FIG. , A vector having one as a start point and the other as an end point (hereinafter, appropriately referred to as an inter-pixel vector)
, And multiplying the SD pixels at the start point and end point of the inter-pixel vector.

That is, assuming that oneself is the start point of the inter-pixel vector, FIG. 9B shows a case where oneself is selected as the end point of the inter-pixel vector. In this case, the inter-pixel vector is 0 (( 0,0)). FIG. 9C shows a case where the right next SD pixel is selected as the end point of the inter-pixel vector. In this case, the inter-pixel vector has a magnitude of 1 in the right direction. Here, the magnitude (length) of the inter-pixel vector is expressed assuming that the distance between horizontally adjacent SD pixels and the distance between vertically adjacent SD pixels are both equal to 1.

FIG. 9 (D) shows a case where the SD pixel on the left of the user is selected as the end point of the inter-pixel vector.
In this case, the inter-pixel vector has a size corresponding to one pixel in the left direction. FIG. 9E shows a case in which an SD pixel adjacent to itself is selected as the end point of the inter-pixel vector. In this case, the inter-pixel vector has a size of 1 in the downward direction. . FIG. 9F shows a case in which an SD pixel adjacent to itself is selected as the end point of the inter-pixel vector. In this case, the inter-pixel vector has a size of 1 in the upward direction. . FIG.
(G) shows the case where the lower right adjacent SD pixel is selected as the end point of the inter-pixel vector. In this case, the inter-pixel vector has a size of √2 in the lower right direction. Become. FIG. 9 (H) shows the S
A case where the D pixel is selected as the end point of the inter-pixel vector is shown. In this case, the inter-pixel vector is √
It has a size of 2. FIG. 9 (I) shows a case where the SD pixel adjacent to the lower left is selected as the end point of the inter-pixel vector. In this case, the inter-pixel vector has a size of √2 in the lower left direction. Become. FIG.
(J) shows a case in which the SD pixel adjacent to the upper right is selected as the end point of the inter-pixel vector. In this case, the inter-pixel vector has a size of √2 in the upper right direction.

Here, considering the x-axis from left to right and the y-axis from top to bottom on an SD image as student data, the pixel-to-pixel vectors in FIGS. 9B to 9J are considered. Is
(0,0), (1,0), (-1,0), (0,1), (0, -1), (1,1), (-1, -1), (-1,
1) and (1, -1), respectively.

Then, in FIGS. 9B to 9 J, when the multiplication value of the SD pixel at the starting point and the SD pixel at the ending point of each inter-pixel vector is obtained, the multiplied value is the component of the matrix A. It constitutes.

Here, considering only the SD pixels constituting the prediction tap, according to the inter-pixel vector (0,0) (FIG. 9)
(B)), p1 × p1, p2 × p2, p3 × p3, p4
Xp4 is calculated. According to the inter-pixel vector (1,0), p1 × p3 and p2 × p4 are calculated, and according to the inter-pixel vector (-1,0), p2 × p1 and p4 × p3 are calculated. Further, according to the inter-pixel vector (0,1), p1 ×
p3, p2 × p4 are calculated, and according to the inter-pixel vector (0, −1), p3 × p1, p4 × p2 are calculated. Also,
According to the inter-pixel vector (1,1), p1 × p4 is calculated, and according to the inter-pixel vector (-1, -1), p4 × p1 is calculated. Further, according to the inter-pixel vector (-1,1), p
2 × p3 is calculated, and according to the inter-pixel vector (1, -1),
p3 × p2 is calculated.

The multiplication circuit 31 targets all the SD pixels as the student data as shown in FIGS. 9B to 9J.
A multiplication value between SD pixels having a positional relationship that matches the positional relationship between the start point and the end point of each inter-pixel vector shown in (1) is obtained and supplied to the product memory circuit 32.

As described above, the product memory circuit 32 stores the multiplied value of the SD pixel output from the multiplying circuit 31 for each relative positional relationship between the two SD pixels used for obtaining the multiplied value. It has become. Therefore, the product memory circuit 32 stores, for each of the nine types of inter-pixel vectors described with reference to FIG. 9, the multiplication value of the SD pixels at the start point and the end point.

That is, FIG. 10 shows that the prediction tap is
As shown in (A), an example of the configuration of the product memory circuit 32 in a case where the product memory circuit 32 includes four SD pixels is shown.

The product memory circuit 32, as shown in FIG.
The nine types of inter-pixel vectors (0,0), (1,0),
(0,1), (1,1), (-1,1), (-1,0), (0, -1), (-1, -1), (1, -1) , A product memory 41 _{1 to} 41 as a memory for storing a multiplication value between SD pixels at the start point and the end point thereof
_9, and a write control unit 42 for controlling the reading and writing of data the product memory 41 ₁ to 41 _9.

[0095] That is, now, in order to simplify the description, given only SD pixels p1 to p4 constituting the prediction tap, the product memory 41 _1, SD of start and end points of the inter-pixel vectors (0,0) P1 × p1, p2 × p2, which are multiplication values of pixels
p3 × p3, p4 × p4 are represented by the inter-pixel vector (0,0)
Are stored at addresses corresponding to the positions of the SD pixels p1, p2, p3, and p4 at the start point of. Also, the product memory 41
₂ is a value obtained by multiplying p1 × p2, p3 × p4, which is the product of the SD pixels at the start point and the end point of the inter-pixel vector (1,0), by the SD pixel p1, p3 at the start point of the inter-pixel vector (1,0). Each is stored at an address corresponding to the position. Further, the product memory 41
₃ is a value obtained by multiplying p1 × p3, p2 × p4, which is a multiplication value of the SD pixel at the start point and the end point of the inter-pixel vector (0, 1), by the SD pixel p1, p2 at the start point of the inter-pixel vector (0, 1). Each is stored at an address corresponding to the position. The product memory 41 ₄
Stores p1 × p4, which is the product of the SD pixel at the start point and the end point of the inter-pixel vector (1,1), at the address corresponding to the position of the SD pixel p1 at the start point of the inter-pixel vector (1,1). I do. Furthermore, the product memory 41 _5, the inter-pixel vectors (-1,1)
P2 × p3, which is the product of the SD pixels at the start and end points of
It is stored at an address corresponding to the position of the SD pixel p2 at the start point of the inter-pixel vector (-1,1). Also, the product memory 41 ₆
Calculates p2 × p1, p4 × p3, which is the product of the SD pixel at the start point and the end point of the inter-pixel vector (-1,0), as SD pixels p2, p4 at the start point of the inter-pixel vector (-1,0). Is stored in the address corresponding to the position. Product memory 41 _7, the inter-pixel vectors (0, -1) and a multiplication value of the SD pixel of the start point and the end point is p3 × p1, p4 × p2 of, between the pixel vector
They are stored at addresses corresponding to the positions of the SD pixels p3 and p4 at the start point of (0, -1). Product memory 41 _8, the inter-pixel vectors (-1, -1) the start and a multiplication value of the SD pixels of the destination p4 × p1 of the starting point of SD pixels in the inter-pixel vectors (-1, -1) p4 Is stored at the address corresponding to the position. Product memory 41 _9, the inter-pixel vectors (1, -1) and a multiplication value of the SD pixel of the start point and the end point is p3 × p2, its inter-pixel vectors (1, -1) position of the SD pixel p3 of the start point of the Is stored at an address corresponding to.

In FIG. 10, the SD pixel p # t
_The product of ₁ and p # t ₂ is indicated as t ₁ × t ₂ .

In FIG. 10, for simplicity of explanation, four SD pixels p1 to p
4 only was considered, the product memory 41 ₁ to 41 ₉ is multiplied value of each other SD pixels similarly to and stored as another student data.

Here, the starting point or the ending point of the inter-pixel vector (-x, -y) is the inter-pixel vector whose direction is only the opposite direction.
Since the end point or the start point of (x, y) respectively match, the multiplication value of the SD pixels of the start point and the end point of the inter-pixel vector (-x, -y) is the start point and the end point of the inter-pixel vector (x, y). Is equal to the multiplied value of the SD pixels. Therefore, the inter-pixel vector (-x,
-y) or the inter-pixel vector (x, y) is obtained by multiplying the SD pixel of the start point and the end point of one of them, and the other multiplied value is also obtained. No need. Therefore, the multiplication circuit 31 determines the nine types of inter-pixel vectors (0,0), (1,0), (0,1), (1,1), (-1,
From (1), (-1,0), (0, -1), (-1, -1), (1, -1), for example, only the direction is opposite to the other vector (- 1,0), (0,-
(1), (-1, -1), (0,0), (1,0), (0,1),
Only the multiplication value between the SD pixels at the start point and the end point of each of the five inter-pixel vectors (1, 1) and (-1, 1) is calculated.

From the same point of view, the product memory circuit 32 also calculates all nine types of inter-pixel vectors shown in FIG.
It is not necessary to store the multiplication value between the SD pixels at the start point and the end point. Therefore, the product memory circuit 32 also includes the multiplication circuit 31
9, the nine types of inter-pixel vectors shown in FIG.
(0,0), (1,0), (0,1), (1,1), (-1,1), (-1,0), (0, -1), (-1,-
(0,0), (1,0), (0,1), (1,1), (-1,1) of (1), (1, -1)
Only the multiplied values of the SD pixels at the start point and end point of each of the five inter-pixel vectors are stored.

Therefore, in this case, the product memory circuit 32
As shown in FIG. 11, five product memories 41 _{1 to} 41
₅ and a read / write control unit 42.

FIG. 12 shows an example of the configuration of the normal equation adding circuit 33 shown in FIG.

The normal equation adding circuit 33 includes the read circuit 11 or the normal equation constructing unit 12 ₁ shown in FIG.
To and a read circuit 51 or the normal equation configuration section 52 ₁ to 52 _H respectively corresponding to the 12 _H.

The readout circuit 51 includes a class classification circuit 6
When the readout circuit 51 receives the class code of the pixel of interest, the readout circuit 51 reads out the HD pixel, which is the pixel of interest, from the frame memory 1 and outputs the HD pixel. the SD pixels serving as prediction taps, read from the frame memory 4, and supplies the normal equation configuration section 52 _h which corresponds to the class code h of the pixel of interest.

Further, the readout circuit 51 accesses the product memory circuit 31, reads out a necessary one of the multiplied values of the SD pixels stored therein, and reads the normal equation corresponding to the class code h of the pixel of interest. supplied to the component 52 _h.

The normal equation constructing unit 52 _h uses the multiplication value of the SD pixels supplied from the readout circuit 51 and calculates a summation (Σ) which is a multiplier of the prediction coefficient on the left side of the normal equation of the equation (7). ), And the SD pixel (student data) on the right side of the normal equation of Expression (7) using the SD pixel and the target pixel (HD pixel), which are prediction taps also supplied from the readout circuit 51. And HD pixel (teacher data) multiplication (x _in
y _i ) and an operation corresponding to summation (Σ) is performed.

[0106] In the normal equation configuration section 52 _h, class #h
The above-described calculation is performed for all the HD pixels classified into the equation (2), whereby the normal equation shown in the equation (7) for the class #h, that is, the matrix A and the vector v shown in the equation (8) are obtained.

The matrix A and the vector v are
The prediction coefficient is supplied to the prediction coefficient determination circuit 8, and the prediction coefficient is obtained as described above.

Next, FIG. 13 shows a configuration example of the normal equation configuration section _52h of FIG.

The normal equation constructing unit _52h includes a left side memory 61 corresponding to the left side memory 21 in FIG.
And a right side memory 62 having the same configuration.

The left side memory 61 corresponds to the left side memory 21 shown in FIG.
Is composed of ten elements, as in the above. This element is an adder that adds (integrates) a multiplication value between SD pixels read from the product memory circuit 31, and an output of the adder. And a register for storing

The right side memory 62 is composed of four elements like the right side memory 22 of FIG. 5, and furthermore, this element is composed of the SD pixel and the HD pixel similarly to the right side memory 22.
It comprises a multiplier for performing multiplication with a pixel, an adder for adding (adding) a multiplied value output from the multiplier, and a register for temporarily storing the output of the adder.

In the left side memory 61, the n-th matrix A
A multiplied value of the SD pixels p # n and p # m constituting the prediction tap is read from the product memory circuit 32 and supplied to the element corresponding to the component in the row m-th column. In the element, the multiplied value of p # n and p # m supplied thereto is integrated with the stored value already stored therein and stored. In FIG. 13, the left side memory 61
The n × m shown in the rectangle representing the element indicates that a multiplied value of the SD pixels p # n and p # m constituting the prediction tap is supplied.

In the right-side memory 62, similarly to the right-side memory 22 in FIG. 5, an element corresponding to the component of the n-th row of the vector v has an SD pixel p as a prediction tap.
#N and the HD pixel P, which is the pixel of interest, are read from the frame memories 4 and 1, respectively, and supplied. The elements include the SD pixel p # supplied thereto.
n is multiplied by the HD pixel P, and the multiplied value is integrated with the stored value already stored therein and stored.

Then, the HD as all the teacher data
By performing the above-described processing with the pixel as the target pixel,
Matrices A and v that define a normal equation for class #h are obtained. After that, the matrix A as the storage value of each element of the left-side memory 21 and the vector v as the storage value of each element of the right-side memory 22 are read and supplied to the prediction coefficient determination circuit 8. The prediction determination circuit 8 calculates a prediction coefficient w _j (here, j = 1, 2, 3, 4) of the class #h based on the matrix A and the vector v.

[0115] Then, from the product memory circuit 32 reads out the multiplied value of the SD pixel p # n and p # m constituting the prediction taps, will be described addressing when supplying the normal equation configuration section 52 _h.

As shown in FIG. 14A, the left side memory 6
1, the multiplied value of the SD pixels p # n and p # m constituting the prediction tap is read from the product memory circuit 32 and supplied to the element corresponding to the element of the n-th row and the m-th column of the matrix A. In this case, first, the product memory circuit 3
2, a plurality of product memories (in the embodiment of FIG. 11,
It is necessary to specify, from the product memories 41 _{1 to} 41 ₅ ), the _{one in which} the multiplied value of p # n and p # m is stored.

Here, in the product memory constituting the product memory circuit 32, the SD of the start point and the end point of the inter-pixel vector (x, y) is set.
What stores a multiplication value between pixels is hereinafter referred to as a product memory (x, y) as appropriate. In this case, the product memory 41 ₁ to 41 ₉ shown in FIG. 10 each product memory (0,0), (1,0),
Describe as (0,1), (1,1), (-1,1), (-1,0), (0, -1), (-1, -1), (1, -1) be able to.

When the product memory is expressed as described above, the product memory storing the multiplied value of p # n and p # m can be specified as follows.

That is, for example, as shown in FIG. 14B, if the coordinates of the SD pixel p # n are represented by (R _x [n], R _y [n]), the SD pixel p # m Can be expressed as (R _x [m], R _y [m]). In this case, while the start point of the SD pixel p # n, inter-pixel vectors and ending the SD pixel p # m _{is, (R x [m] -R} x [n], R y [m] -R y [ n]) and can be represented, the inter-pixel vectors _{(R x [m] -R x} [n], an SD pixel of the start and end points of the _{R y [m] -R y [} n]) p # n multiplication value with p # m is the product memory _{(R x [m] -R x} [n], R y [m] -R y [n]) will be stored in the. Here, the product multiplied value of the SD pixel p # n and p # m is stored memory _{(R x [m] -R x} [n], R y [m] -R y [n])
Is represented by a variable Mp_pointer [n] [m] as appropriate.

[0120] Then, the multiplication value of the p # n and p # m is the product memory _{(R x [m] -R x} [n], R y [m] -R y [n]) which address ( At the location), which can be determined as follows.

That is, from the above, the inter-pixel vector
p # n × which is a multiplication value of SD pixels at the start point and end point of (x, y)
p # m is stored at an address corresponding to the position of the SD pixel p # n at the start point of the inter-pixel vector (x, y). Therefore, the product of p # n and p # m is the product memory (R _x [m] -R _x
[n], R _y [m] -R _y [n]), which is stored at the address (R _x [n], R _y [n]) corresponding to the position of the SD pixel p # n. .
Here, the product memory the multiplication value of the SD pixel p # n and p # m is stored _{(R x [m] -R x} [n], R y [m] -R y [n]), SD The address ( _Rx [n], _Ry [n]) corresponding to the position of the pixel p # n is appropriately represented by a variable Offset [n] [m].

From the above, for example, the multiplication value p1 × p1 of the SD pixel, which is a component of the first row and first column of the matrix A, required by the left side memory 61 shown in FIG.
It is stored in (0,0) (product memory 41 ₁ ) at an address corresponding to the position of SD pixel p1.

The multiplied value p1 × p2 of the SD pixel, which is the component of the first row and second column of the matrix A, corresponds to the position of the SD pixel p1 in the product memory (1,0) (product memory 41 ₂ ). The multiplied value p1 × p3 of the SD pixel, which is stored in the first row and the third column of the matrix A, is stored in the product memory (0, 1) (product memory 41 ₃ ). It is stored at the address corresponding to the position. Further, the multiplied value p1 × p4 of the SD pixel, which is a component of the first row and the fourth column of the matrix, is calculated by
The product p2 × p2 of the SD pixel stored in the (1,1) (product memory 41 ₄ ) at the address corresponding to the position of the SD pixel p1 and serving as the component of the second row and the second column of the matrix A is , Product memory
(0,0) (product memory 41 ₁ ) is stored at an address corresponding to the position of SD pixel p2. The multiplication value p2 × p3 of the SD pixel, which is the component of the second row and the third column of the matrix, is stored in the product memory (-1,1) (product memory 41 ₅ ) at an address corresponding to the position of the SD pixel p2. The multiplied value p2 × p4 of the SD pixel stored and serving as the component of the second row and the fourth column of the matrix A corresponds to the position of the SD pixel p2 in the product memory (0,1) (product memory 41 ₃ ). Address. Further, the multiplied value p2 × p4 of the SD pixel which is the component of the second row and the fourth column of the matrix
Is the SD pixel p2 of the product memory (0, 1) (product memory 41 ₃ ).
, And a multiplied value p3 × p3 of an SD pixel which is a component of the third row and the third column of the matrix A.
Is the SD pixel p3 in the product memory (0,0) (product memory 41 ₁ ).
Is stored at the address corresponding to the position. In addition, a multiplied value p3 × of the SD pixel which is a component of the third row and the fourth column of the matrix
p4 is stored in the product memory (1,0) (product memory 41 ₂ ) at an address corresponding to the position of the SD pixel p3, and is multiplied by the SD pixel which is the element of the fourth row and the fourth column of the matrix A. Value p4 ×
p4 is stored in the product memory (0,0) (product memory 41 ₁ ) at an address corresponding to the position of the SD pixel p4.

Next, with reference to the flowchart of FIG. 15, a learning process for obtaining a prediction coefficient for each class by the learning device of FIG. 8 will be described.

An HD image as teacher data is supplied to the learning device in frame units, and the HD images are sequentially stored in the frame memory 1.

In the HD image as the teacher data stored in the frame memory 1, the number of pixels in the vertical direction or the horizontal direction is thinned out by the vertical thinning filter 2 or the horizontal thinning filter 3, respectively.
An SD image in which the number of pixels in both the horizontal and vertical directions is halved. This SD image is supplied to the frame memory 4 and stored.

When the SD image is stored in the frame memory 4, the class tap configuration circuit 5 sets a predetermined HD pixel for which a predicted value is to be obtained by the class classification adaptive processing as a target pixel, and further, the target pixel. The SD pixels used to classify are stored in the SD memory stored in the frame memory 4.
Extract from the image as a class tap. This class tap is supplied to the classification circuit 6.

The class classification circuit 6 performs, for example, 1-bit ADRC processing on the class tap from the class tap configuration circuit 5, and uses the resulting ADRC code as the class code of the class that is the result of class classification of the pixel of interest. Output to the normal equation adding circuit 7.

In the same manner, H as teacher data
HD pixels constituting the D image are sequentially set as pixels of interest.
The class code for the target pixel is supplied from the class classification circuit 6 to the normal equation addition circuit 33.

On the other hand, in the normal equation addition circuit 33, when the supply of the teacher data to the frame memory 1 is started, in step S21, the variables Mp_pointer [n] [m] and Of are set.
The initial value is set in fset [n] [m]. That is, step S
In 21, a value (R _x [m] −) representing a product memory in which a multiplication value of SD pixels p # n and p # m is stored in a variable Mp_pointer [n] [m].
R _x [n], R _y [m] -R _y [n]) are set and the variable Offs
et [n] [m] and its product memory (R _x [m] -R _x [n], R _y [m] -R _y [n])
(R _x [n], corresponding to the position of the SD pixel p # n
R _y [n]) is set.

Then, the process proceeds to a step S22, wherein the values stored in the left side memory 61 and the right side memory 62 are initialized to, for example, 0 in the normal equation adding circuit 33. That is,
An array variable A [c] [n] [m] representing the matrix A and an array variable v [c] [n] representing the vector v are initialized to zero.

Thereafter, in step S23, the multiplication circuit 31 calculates a multiplication value between the SD pixels as the student data stored in the frame memory 4 and supplies the multiplication value to the product memory circuit 32. In the product memory circuit 32, the multiplied value from the multiplying circuit 31 is stored for each relative positional relationship between the two SD pixels used for obtaining the multiplied value. That is, in the product memory circuit 32, the SD pixels p # n and p # n
The product of #m is the product memory (R _x [m] -R _x [n], R _y [m] -R _y [n])
(R _x [n], corresponding to the position of the SD pixel p # n
R _y [n]).

Then, the process proceeds to step S24, where the normal equation adding circuit 33 performs addition processing using the multiplied value stored in the product memory circuit 32, thereby generating a normal equation for each class. That is, the normal equation adding circuit 33 obtains the matrix A and the vector v for each class. The matrix A and the vector v are supplied to the prediction coefficient determination circuit 8. In step S25, the prediction coefficient determination circuit 8 solves a normal equation for each class, which is defined by the matrix A and the vector v. Is obtained. The prediction coefficient for each class is supplied to the memory 9 and stored in the address corresponding to each class, and the processing is completed.

Next, referring to the flowchart of FIG. 16, the variable Mp_pointer in step S21 of FIG.
The details of the setting process for setting initial values to [n] [m] and the variables Offset [n] [m] will be described.

First, in step S31, a variable n representing a row of the matrix A is initialized to, for example, 0, and the flow advances to step S32. In step S32, the variable n is incremented by one, and the process proceeds to step S33, where the variable n
Is smaller than or equal to N, which is the number of rows of the matrix A.

If it is determined in step S33 that the variable n is equal to or less than the number of rows N of the matrix A, the process proceeds to step S34, where the variable m representing the column of the matrix A is set to 0, for example.
And the process proceeds to step S35. Step S35
Then, the variable m is incremented by 1 and the step S
Proceeding to 36, it is determined whether the variable m is less than or equal to M, which is the number of columns of the matrix A. If it is determined in step S36 that the variable m is not smaller than M, step S32
And the same processing is repeated thereafter.

Here, as described above, both the number of rows and the number of columns of the matrix A are equal to the number of SD pixels constituting the prediction tap, and therefore, the above M and N are the same value. .

If it is determined in step S36 that the variable m is equal to or smaller than M, the process proceeds to step S37, and it is determined whether the variable m is equal to or larger than the variable n.
If it is determined in step S37 that the variable m is not greater than or equal to the variable n, the process skips step S38 and returns to step S35.

That is, when the component of the n-th row and the m-th column of the matrix A represented by the variables m and n is a lower triangular component, the component does not need to be obtained as described above. Is performed, the process returns to step S35.

In step S37, the variable m
Is determined to be greater than or equal to the variable n, that is, when the component of the n-th row and the m-th column of the matrix A represented by the variables m and n is a diagonal component or an upper triangular component, Step S
Then, the SD pixel p # n is set to the variable Mp_pointer [n] [m].
A value representing a product memory (R
_x [m] -R _x [n], R _y [m] -R _y [n]) are set, and the product memory (R _x [m] -R _x [n], R _y [m]-
R _y [n]), an address (R _x [n], R _y [n]) corresponding to the position of the SD pixel p # n is set, and the process returns to step S35.
Hereinafter, the same processing is repeated.

On the other hand, in step S33, the variable n is set to N
If it is determined that it is not the following, the process returns.

Next, with reference to the flowchart of FIG. 17, a product memory setting process of calculating a multiplication value between SD pixels and setting the product value in each product memory constituting the product memory circuit 32 in step S23 of FIG. Will be described.

Here, for simplicity of description, it is assumed that the prediction tap is formed of rectangular SD pixels, and that only one HD image as teacher data is prepared. Therefore, S as student data
The D image is also only one frame.

First, in step S41, -1 is set as an initial value to a variable y _p representing the y coordinate of the relative coordinates of the SD pixels constituting the prediction tap. Here, the relative coordinates of the SD pixels forming the prediction tap mean the coordinates of a certain SD pixel forming the prediction tap, for example, when the position of the upper left SD pixel is set to the origin (0,0). . Thus, among the SD pixels constituting the prediction tap, with x _p +1 leftmost, relative coordinates of which from top to +1 th y _p is expressed as (x _p, y _p). Note that this (x _p , y _p )
Corresponds to the inter-pixel vector described above.

Thereafter, in step S42, the variable y _p
Is incremented by 1, and the process proceeds to step S43.
It is determined whether or not the variable y _p is smaller than the number Y _p of vertical SD pixels constituting the prediction tap. Step S4
In 3, if the variable y _p is determined to be Y _p smaller,
Proceeds to step S44, the variable x _p which represents the x-coordinate of the relative coordinates of the SD pixels constituting the prediction taps, as an initial value -
1 is set, and the process proceeds to step S45.

[0146] At step S45, the variable x _p is incremented by 1, the process proceeds to step S46, the variable x _p is whether the pixel number X _p is smaller than the lateral SD pixels constituting the prediction taps are determined. In step S46,
If the variable x _p is determined to not less than X _p, the flow returns to step S42, and similar processing is repeated.

Further, in step S46, the variable x _p
If it is determined that X _p smaller, the process proceeds to step S47, the SD image as student data stored in the frame memory 4, a variable y _s representing the y-coordinate of y _s th SD pixel from the top, for example, Initialized to 0, and the process proceeds to step S48. In step S48, the variable y _s is incremented by 1, and the process proceeds to step S49, where it is determined whether the variable y _s is equal to or less than the number of vertical pixels Y _s of the SD image as the student data. In step S49, the case where the variable y _s is determined to be not less Y _s, the flow returns to step S45, and similar processing is repeated.

In step S49, the variable y _s is set to
If it is determined that the Y _s or less, the process proceeds to step S50, the SD image as student data stored in the frame memory 4, a variable x _s representing the x-coordinate of x _s-th SD pixel from the left, For example, it is initialized to 0, and proceeds to step S51. In step S51, the variable x _s is incremented by 1, the process proceeds to step S52, the variable x _s Whether or less number of horizontal pixels of the SD image as student data X _s is determined. In step S52, if the variable x _s is determined to be not less X _s, returns to step S48, the similar processing is repeated.

In step S52, the variable _xs is
If it is determined that the X _s or less, the process proceeds to step S53, in the multiplication circuit 31, located at the coordinate (x _s, y _{s) S}
The pixel value L [x _s ] [y _s ] of the D pixel and the pixel value L [x _s + x _p ] [y _s + of the SD pixel located at the coordinates (x _s + x _p , y _s + y _p ) y _p ] is calculated. The multiplied value is supplied to the product memory circuit 32 and stored in the product memory represented by the variable Mp_pointer [y _p ] [x _p ] at the address represented by the variable offset [y _p ] [x _p ]. Is done.

Hereafter, the address of the product memory represented by the variable Mp_pointer [y _p ] [x _p ] in the product memory circuit 32 will be appropriately represented by the variable offset [y _p ] [x _p ]. The stored value is represented by a variable Val [Mp_pointer [y _p ] [x _p ]] [offset [y _p ] [x _p ]].

After the processing in step S53, step S5
Returning to 1, the same processing is repeated thereafter. As a result, the multiplication circuit 31 obtains a multiplication value of the SD pixels required to obtain each component of the matrix A, and the product memory circuit 32 calculates the multiplication value of the two SDs used for obtaining the components. Relative positional relationship between pixels (inter-pixel vector (x _p ,
y _p )).

On the other hand, in step S43, the variable y
_p is the pixel number Y of the vertical SD pixels forming the prediction tap
If it is determined that it is not smaller than _p , the routine returns.

Next, with reference to the flowchart of FIG. 18, an addition process of adding SD pixels as student data and HD pixels of teacher data for obtaining the matrix A and the vector v in step S24 in FIG. Will be described.

First, in step S61, a variable y representing the y-coordinate of the HD pixel which is the target pixel is initialized to, for example, 0, the process proceeds to step S62, the variable y is incremented by 1, and the process proceeds to step S63. . In step S63, it is determined whether or not the variable y is equal to or less than _ymax which is the number of pixels in the vertical direction of the teacher data. If it is determined in step S63 that the variable y is equal to or _smaller than ymax, the process proceeds to step S64, where the variable x representing the x coordinate of the HD pixel as the target pixel is initialized to, for example, 0, and the process proceeds to step S65. In step S65, the variable x is incremented by 1, and the process proceeds to step S66, where the variable x
But whether or not x _{ma x} below the number of pixels in the horizontal direction of the teacher data is determined. In step S65, the variable x
If is determined not less x _max, the process returns to step S62, and similar processing is repeated.

In step S66, the variable x is set to x
_If it is determined to be less than _max , the HD at the coordinates (x, y)
The pixel is determined to be the pixel of interest, and the process proceeds to step S67, where the class code of the class of the pixel of interest (the HD pixel at the coordinates (x, y) supplied from the class classification circuit 6 when the pixel of interest is the pixel of interest) Class code of the pixel of interest)
Set to variable c.

Then, the process proceeds to a step S 68, wherein a variable n representing a row of the matrix A is initialized to, for example, 0, and a step S 68
Go to 69. In step S69, the variable n is incremented by one, and the process proceeds to step S70, where it is determined whether the variable n is equal to or less than N, which is the number of rows of the matrix A.
If it is determined in step S70 that the variable n is not smaller than N, the process returns to step S65, and the same processing is repeated thereafter.

At step S70, the variable n
Is determined to be equal to or less than N, which is the number of rows of the matrix A, the process proceeds to step S71, and a variable m representing a column of the matrix A is determined.
Are initialized to, for example, 0, and the process proceeds to step S72. In step S72, the variable m is incremented by one, and the process advances to step S73 to determine whether the variable m is equal to or less than M, which is the number of columns of the matrix A. Step S7
In 3, when it is determined that the variable m is not less than M,
Returning to step S69, the same processing is repeated thereafter.

If it is determined in step S73 that the variable m is equal to or smaller than M, the flow advances to step S74 to determine whether the variable m is equal to or larger than the variable n.
If it is determined in step S74 that the variable m is not greater than or equal to the variable n, step S75 is skipped and the process returns to step S72.

That is, when the component of the n-th row and the m-th column of the matrix A represented by the variables m and n is a lower triangular component, the component need not be obtained as described above. Is performed, the process returns to step S72.

At step S74, the variable m
Is determined to be greater than or equal to the variable n, that is, when the component of the n-th row and the m-th column of the matrix A represented by the variables m and n is a diagonal component or an upper triangular component, Step S
Proceeding to 75, the component A [c] [n] [m] of the n-th row and the m-th column of the matrix A for the class #c and the component v [c] [of the n-th row of the vector v for the class #c n] is calculated according to the following equation.

A [c] [n] [m] + = Val [Mp_pointer [n] [m]] [offset [n] [m]] v [c] [n] + = L [x + D _x [ n]] [y + D _y [n]] × L '[x] [y] (10)

That is, in the normal equation adding circuit 33,
From the product memory circuit 32, the product value Val [Mp_pointer] of the SD pixels stored at the address represented by the variable offset [n] [m] in the product memory represented by the variable Mp_pointer [n] [m]
[n] [m]] [offset [n] [m]] is read out and the variable A [c] [n] [m]
Is added to (integrated). Further, in the normal equation addition circuit 33, the coordinates (x + _Dx [n], y + _Dy [n]) ( _Dx [n] or
D _y [n] represents the x-coordinate or y-coordinate of the n-th SD pixel p # n that constitutes the prediction tap viewed from the target pixel, as described with reference to FIG. Pixel value L
[x + D _x [n]] [y + D _y [n]] are read from the frame memory 4 and the pixel value of the HD pixel that is the pixel of interest
L '[x] [y] is read. Then, L [x + D _x [n]] [y + D
The multiplication value of _y [n]] and L '[x] [y] is calculated and added to the variable v [c] [n] (integrated).

Then, the process returns to step S72, and the same processing is repeated thereafter.

On the other hand, in step S63, the variable y is set to y
If it is determined that it is not less than _max , that is, if processing is performed with all HD pixels prepared as teacher data as target pixels, the process returns.

As described above, the multiplication value of the SD pixels as the student data is obtained in advance, and the multiplication value is stored for each relative positional relationship between the two SD pixels used to obtain the multiplication value. Since it is used to calculate each component of the matrix A, the number of times of multiplication between SD pixels can be reduced, and as a result, the multiplication time is reduced or the scale of the apparatus is prevented from increasing. Becomes possible.

That is, for example, as shown in FIG. 19A, assuming that a prediction tap is composed of 10 SD pixels in the spatial direction (horizontal direction, vertical direction), a matrix A is obtained. In this case, for a certain SD pixel, the range of the SD pixel for which a multiplication value with the SD pixel is calculated is as follows:
The result is as shown in FIG.

Here, any two SD pixels (including the same SD pixel) constituting the prediction tap shown in FIG. 19A are selected, and one of them is set as a start point and the other is set as an end point. By obtaining a vector (inter-pixel vector) and plotting SD pixels that can be reached starting from a certain SD pixel by using the inter-pixel vector, FIG.
9 (B) is obtained.

FIG. 19 (B) focuses on the SD pixel p indicated by a hatched circle and starts from the SD pixel p.
The range of SD pixels that can be reached by the inter-pixel vector is shown, and the number of SD pixels within this range is used by the product memory circuit 32 to calculate the product value of the SD pixels. This corresponds to the number of product memories required to store the relative positional relationship between the two SD pixels. Therefore, as is clear from FIGS. 19A and 19B, the number of necessary product memories is a value which is four times or less the number of SD pixels constituting the prediction tap, and Assuming that the number of pixels of J is J, the value is proportional to the order of J. As a result, the number of times of multiplication of the two SD pixels by the multiplication circuit 31 is also proportional to the order of J, so that the apparatus can be downsized or the processing can be shortened as compared with the conventional case. it can.

Next, the above-described series of processing can be performed by hardware or can be performed by software. When a series of processing is performed by software, a program constituting the software is
The computer is installed in a computer incorporated in the learning device as dedicated hardware, or a general-purpose computer that performs various processes by installing various programs.

With reference to FIG. 20, a description will be given of a medium used to install a program for executing the above-described series of processes in a computer and to make the computer executable.

As shown in FIG. 20A, the program can be provided to the user in a state where the program is previously installed in a hard disk 102 or a semiconductor memory 103 as a recording medium built in the computer 101.

Alternatively, the program is executed as shown in FIG.
As shown in (B), the floppy disk 111 and the CD-R
OM (Compact Disc Read Only Memory) 112, MO (Magnet
o optical) disc 113, DVD (Digital Versatile Di)
sc) 114, magnetic disk 115, semiconductor memory 116
And the like, can be temporarily or permanently stored in a recording medium such as a storage medium and provided as package software.

Further, as shown in FIG. 20C, the program is transmitted from a download site 121 to a computer 1 via an artificial satellite 122 for digital satellite broadcasting.
01 wirelessly, LAN (Local Area Network),
Via a network 131 such as the Internet,
The data is transferred to the computer 123 by wire and the computer 10
1, it may be stored in a built-in hard disk 102 or the like.

[0174] The medium in this specification means a broad concept including all these media.

In this specification, the steps of describing a program provided by a medium do not necessarily have to be processed in chronological order according to the order described in the flowchart, but may be performed in parallel or individually. (For example, parallel processing or object processing)
Is also included.

Next, FIG. 21 shows the computer 1 of FIG.
1 shows a configuration example.

As shown in FIG. 21, the computer 101 has a built-in CPU (Central Processing Unit) 142. An input / output interface 145 is connected to the CPU 142 via a bus 141, and the CPU 142 is connected to the CPU 142 by the user via the input / output interface 145.
When a command is input by operating the input unit 147 including a keyboard, a mouse, and the like, a ROM corresponding to the semiconductor memory 103 in FIG.
(Read Only Memory) The program stored in 143 is executed. Alternatively, the CPU 142 may execute a program stored in the hard disk 102,
Alternatively, the data is transferred from the network 131 and the communication unit 14
8, the program installed on the hard disk 102 or the floppy disk 111, the CD-ROM 112, and the MO disk 11 mounted on the drive 149.
3. A program read from the DVD 114 or the magnetic disk 115 and installed on the hard disk 102 is loaded into a RAM (Random Access Memory) 144 and executed. Then, the CPU 142 transmits the processing result to the LC via the input / output interface 145, for example.
Display unit 14 composed of D (Liquid CryStal Display) etc.
6 and output as necessary.

In the present embodiment, a prediction coefficient for improving the resolution of an image is obtained. However, for example, as described above, a prediction coefficient for improving S / N, blur, etc. may be obtained. It is also possible. That is, as the student data, the S / N of the teacher data is degraded,
By learning using blurred teacher data, a prediction coefficient that improves S / N and blur can be obtained.

Further, in the present embodiment, an image is a processing target, but the present invention can also be applied to, for example, a sound or the like.

Further, in this embodiment, the product memory circuit 32 stores all the multiplication values of the SD pixels output from the multiplication circuit 31. However, the product memory circuit 32 stores the matrix for each class. Of the multiplied values of the SD pixels that are the components of A, only those that are added twice or more can be stored. In this case, the multiplication value for which the addition is performed only once is calculated by a multiplication circuit 31 as shown by a dotted line in FIG.
May be directly supplied to the normal equation addition circuit 33 from In addition, the detection of the multiplied value of the SD pixels p # n and p # m in which the addition is performed twice or more is performed by, for example, the variable Mp_pointer [n] [m] and the variable Offset [n] [m described with reference to FIG. ]
Can be performed by counting the number of times the value is set for the variable Mp_pointer [n] [m] in the step S38 of the setting process for setting the initial value to. That is,
Variable whose value is set a plurality of times in step S38
The multiplication value of the SD pixels p # n and p # m specified by the indexes n and m of Mp_pointer [n] [m] is added twice or more.

Further, in the present embodiment, for simplicity of description, one frame of the HD image is used as the teacher data. However, as the teacher data, for example, a plurality of frames of the HD image are used. It is also possible to use. When using HD images of a plurality of frames as teacher data, in the flowchart shown in FIG. 15, after performing the processes of steps S21 and S22, the processes of steps S23 and S24 are repeatedly performed on the HD images of each frame, After that, the process of step S25 may be performed.

Further, in the present embodiment, one frame of the HD image and the SD image are used as teacher data and student data at a time, and steps S23 and S2 in FIG.
4 is performed, in this case, as the product memory circuit 32 for storing the multiplication value of the SD pixels, the number of pixels of the SD pixels in one frame and the number of pixels constituting the prediction tap are calculated.
A pixel having a capacity proportional to a value obtained by multiplying the number of D pixels by the number of pixels is required. That is, the capacity required for the product memory circuit 32 is proportional to the data amount of the teacher data (student data) to be subjected to the processes of steps S23 and S24 of FIG. 15 at one time. Therefore, the capacity of the product memory circuit 32 can be reduced by reducing the number of teacher data to be subjected to the processes of steps S23 and S24 in FIG. Of each block, and FIG.
(Steps S23 and S24 of Step 5 are repeated).

[0183]

As described above, according to the learning apparatus, the learning method, and the medium of the present invention, any two multiplication values of student data used for linear prediction of predetermined teacher data are calculated. The multiplied value is stored for each relative positional relationship between the two pieces of student data used for obtaining the multiplied value. Then, by integrating the stored multiplied values, a normal equation for obtaining the prediction coefficient is generated. Accordingly, the number of times of multiplication for obtaining the multiplication value to be integrated can be reduced, and as a result, the size of the apparatus can be reduced or the processing can be speeded up.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an example of a conventional learning device.

FIG. 2 is a diagram for explaining processing of a vertical thinning filter 2 and a horizontal thinning filter 3 of FIG. 1;

FIG. 3 is a block diagram illustrating a configuration example of a normal equation adding circuit 7 in FIG. 1;

FIG. 4 is a diagram showing SD pixels forming a prediction tap.

5 is a block diagram showing a configuration example of a normal equation configuration section 12 _h of FIG.

FIG. 6 is a flowchart illustrating a normal equation configuration process performed by the learning device of FIG. 1;

FIG. 7 is a diagram for explaining the process of step S16 in FIG. 6;

FIG. 8 is a block diagram illustrating a configuration example of an embodiment of a learning device to which the present invention has been applied.

FIG. 9 is a diagram for explaining processing of the multiplication circuit 31 of FIG. 8;

FIG. 10 is a block diagram illustrating a configuration example of a product memory circuit 32 of FIG. 8;

11 is a block diagram illustrating a configuration example of a product memory circuit 32 in FIG. 8;

12 is a block diagram illustrating a configuration example of a normal equation adding circuit 33 in FIG.

13 is a block diagram illustrating a configuration example of a normal equation configuration unit _12h of FIG.

FIG. 14 is a diagram for explaining addressing for the product memory circuit 32;

FIG. 15 is a flowchart for explaining processing of the learning device in FIG. 8;

FIG. 16 is a flowchart illustrating details of a process in step S21 of FIG. 15;

FIG. 17 is a flowchart illustrating details of a process in step S23 of FIG. 15;

FIG. 18 is a flowchart illustrating details of a process in step S24 of FIG. 15;

19 is a diagram for explaining the number of product memories required for the product memory circuit 32. FIG.

FIG. 20 is a diagram for explaining a medium to which the present invention is applied.

21 is a block diagram illustrating a configuration example of a computer 101 in FIG.

[Explanation of symbols]

1 frame memory, 2 vertical thinning filter, 3
Horizontal thinning filter, 4 frame memory, 5
Class tap configuration circuit, 6 Classification circuit, 8
Prediction coefficient determination circuit, 9 memories, 31 multiplication circuits,
32 product memory circuit, 33 normal equation addition circuit,
41 _{1 to} 41 ₉ product memory, 42 read / write control unit,
51 readout circuit, 52 _{1 to} 52 _H normal equation constituent part, 61 left side memory, 62 right side memory,
101 computer, 102 hard disk,
103 semiconductor memory, 111 floppy disk, 112 CD-ROM, 113 MO disk, 11
4 DVD, 115 magnetic disk, 116 semiconductor memory, 121 download site, 122 satellite, 131 network, 141 bus, 14
2 CPU, 143ROM, 144RAM, 145 input / output interface, 146 display unit, 147 input unit, 148 communication unit, 149 drive

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Kenji Tanaka 6-7-35 Kita-Shinagawa, Shinagawa-ku, Tokyo Sony Corporation F-term (reference) 5C063 AA11 BA06 BA08 CA01 5L096 EA33 GA01 GA55

Claims (8)

[Claims] 1. A learning apparatus for learning a prediction coefficient to be multiplied by a second data when linearly predicting a first data from a second data, comprising: a teacher for learning the prediction coefficient; Student data generating means for generating the student data as the second data as the student from the teacher data as the first data as the first data, among the student data used for linear prediction of predetermined teacher data Multiplying means for calculating any two multiplied values of the following; storing means for storing the multiplied value for each relative positional relationship between two student data used for obtaining the multiplied value; storing in the storing means A normal equation generating means for generating a normal equation for obtaining the prediction coefficient by integrating the multiplied values obtained, and a prediction coefficient for obtaining the prediction coefficient by solving the normal equation Learning device characterized by comprising a means out. 2. An extracting means for extracting the student data located around a position corresponding to the position of the noted teacher data of interest, based on the student data extracted by the extracting means. And classifying means for classifying the class into any of a plurality of classes, and outputting a class code corresponding to the class, wherein the normal equation generating means generates the normal equation for each of the classes. The learning device according to claim 1, wherein the prediction coefficient calculation unit calculates a prediction coefficient for each class. 3. The learning device according to claim 1, wherein the storage unit stores only the multiplied value that is integrated by the normal equation generation unit two or more times. 4. The learning device according to claim 1, wherein the first and second data are image data. 5. The learning device according to claim 4, wherein the second data is image data having a lower S / N (Signal to Noise Ratio) than the first data. 6. The learning device according to claim 4, wherein the second data is image data having a smaller number of pixels than the first data. 7. A learning method for learning a prediction coefficient multiplied by the second data when linearly predicting the first data from the second data, comprising: A student data generating step of generating student data that is the second data to be a student from the teacher data that is the first data; and a student data used for linearly predicting predetermined teacher data. A multiplication step of calculating any two multiplication values; a storage step of storing the multiplication value for each relative positional relationship between two pieces of student data used for obtaining the multiplication value; and a storage step of storing the multiplication value. A normal equation generating step of generating a normal equation for obtaining the prediction coefficient by integrating the multiplied values, and solving the normal equation, Learning method which comprises a prediction coefficient calculation step of obtaining a. 8. A medium for causing a computer to execute a program for performing a learning process of learning a prediction coefficient multiplied by the second data when linearly predicting the first data from the second data. A student data generating step of generating student data as the second data as a student from teacher data as the first data as a teacher for learning the prediction coefficient; A multiplication step for calculating any two multiplication values of the student data used for linear prediction; and for each relative positional relationship between the two student data used for obtaining the multiplication value. A storage step of storing; and a normal equation generation for generating a normal equation for obtaining the prediction coefficient by integrating the multiplied values stored in the storage step. Step and, by solving the normal equation, the program characterized by comprising a prediction coefficient calculation step of calculating the prediction coefficients, the medium for causing the computer to execute.