JP3271101B2 - Digital image signal processing apparatus and processing method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

TECHNICAL FIELD The present invention relates to spatial and / or
Also, the present invention relates to a digital image signal processing device and a processing method that require a value of a target pixel to be created using a plurality of pixels that are temporally nearby.

[0002]

2. Description of the Related Art As one of high-efficiency coding of digital image signals, there is a method of reducing the amount of transmission data by thinning out pixels by subsampling. One example is a multiple sub-Nyquist sampling encoding scheme in the MUSE scheme. In this system, it is necessary to interpolate non-transmitted pixels that have been thinned out on the receiving side. Up-conversion for converting an input standard definition video signal into high definition video has also been proposed. In this case, it is necessary to create the missing pixel from the signal of the standard definition. Furthermore, when electronically enlarging an image, interpolation of missing pixel values is required. Not limited to these, scene change detection, D
In PCM or the like, it is necessary to create an estimated value of a target pixel from values of peripheral pixels.

As described above, when creating the value of a target pixel, conventionally, an interpolation filter of a fixed tap and a fixed coefficient is usually used.

[0004]

The process of interpolating non-transmitted pixels using an interpolation filter is effective for a certain kind of image, but is not suitable for a wide variety of images such as moving images and still images. However, the interpolation process is not always effective as a whole. As a result, there is a problem that “blur”, “jerkiness” which is an unnatural motion, and the like occur in a restored image composed of transmission pixels and interpolation pixels.

As one method for solving this problem, the value of the target pixel is represented by a linear linear combination of surrounding pixels and coefficients, and the actual value of the target pixel is used so that the square of the error is minimized. A method of determining the value of this coefficient by the least square method has been proposed. Although this method is effective, it cannot be said that an interpolation value sufficiently reflecting local features of an image including a target pixel can be formed.

[0006] In order to reflect a local feature of an image, it has been proposed to perform a class classification based on a level distribution around a target pixel. As a method of generating this class, a method of directly using the level of the pixel data is considered. This method uses four pixels for class classification when pixel data is represented by 8 bits.
⁸ ) ⁴ = 232 ³² classes are required, and there is a problem that the capacity of a memory for storing data for class classification becomes too large.

Further, Japanese Patent Application Laid-Open No.
Japanese Patent Application Laid-Open No. 3-48088 discloses that when interpolating thinned pixels, an average value of surrounding reference pixels is calculated, and each pixel is represented by 1 bit according to a magnitude relationship between the average value and the value of each pixel. It has been proposed to perform classification according to a pattern of (number of reference pixels × 1 bit). However, since this method binarizes the value of each pixel, it is insufficient to accurately reflect local features of an image.
When compressing reference pixel data for class classification, there is a similar problem when the compression ratio is increased.

Therefore, one object of the present invention is to make it possible to create a value having a small error from the actual value of the pixel of interest by performing class classification.
It is an object of the present invention to provide a digital image signal processing apparatus and a processing method capable of storing data for class classification with a relatively small capacity even when the number of reference pixels is large, and performing class classification accurately. .

Another object of the present invention is to provide a digital image signal processing apparatus and a digital image signal processing apparatus capable of converting (up-conversion) a low-resolution image signal into a high-resolution image signal .
And processing methods .

[0010]

According to the first aspect of the present invention, an input digital image signal is generated based on an input digital image signal.
Digital image signal with a higher resolution than
Output to the digital image signal processor
The target pixel at the target position in the digital image signal
Multiple class prediction factors used in classifying the class,
Multiple data predictors used to generate the pixel of interest
And number, previously stored for each class table, the target position
For a given pixel in the nearby input digital image signal
Te, the operation of the class prediction coefficients from a plurality of reference pixels and the table in the input digital image signal is spatially and / or temporally near to a given pixel, included in the input digital image signal by a predetermined Pixel prediction value
Class determining means for determining the class of the pixel of interest by identifying the class predicting means to form and the class predictive coefficient that produces the predicted value closest to the true value of the predetermined pixel
Means and a data prediction coefficient corresponding to the determined class.
To predict the value of the pixel of interest using the input image signal
And the number of data prediction coefficients is in the class
A digital image signal processing apparatus characterized in that the number is smaller than the number of prediction coefficients .

According to a second aspect of the present invention, an input digital
Based on the image signal, the solution is better than the input digital image signal.
In the digital image signal processing apparatus for outputting a high image of the digital image signal, a digital image signal to be output
Used to classify the target pixel at the target position in the signal.
The multiple class prediction coefficients used and the representative values are
And the input directory near the target position.
For a given pixel in the digital image signal,
A plurality of spatially and / or temporally neighboring against
The operation of the class prediction coefficient from the reference pixel and the table in the input digital image signal, the class pre forming a prediction value of a predetermined pixel included in the input digital image signal
Means measuring, by identifying the class prediction coefficients to generate a predicted and nearest value to the true value of a predetermined pixel, and the class determining means for determining a class of the pixel of interest is determined
Output the representative value corresponding to the selected class as the value of the pixel of interest
A digital image signal processing device having a prediction unit for performing the prediction .

According to a fifth aspect of the present invention, an input digital
Based on the image signal, the solution is better than the input digital image signal.
Digital image output digital image signal with high resolution
In the image signal processing method, the output digital image signal
Used to classify the target pixel at the target position in the signal.
Generate multiple class prediction coefficients to be used and target pixel
Multiple data prediction coefficients used for each class
Steps to read from pre-stored table and attention
For a given pixel in the input digital image signal near the position,
Spatially and / or temporally for a given pixel
Reference images in multiple input digital image signals
Input and the class prediction coefficient from the table.
Predicted value of predetermined pixel contained in digital image signal
And a class prediction step to form the true value of a given pixel
Identify the class prediction coefficients that yield the closest prediction
To determine the class of the pixel of interest.
Data determination step and data corresponding to the determined class
Using the prediction coefficient and the input image signal, the value of the pixel of interest is predicted.
And a prediction step for measuring the data prediction coefficient.
Number is characterized by less than the number of class prediction coefficients
Is a digital image signal processing method .

[0013]

The classification can be performed according to the characteristics of the local image by referring to a plurality of pixels spatially and / or temporally near the pixel of interest. When a prediction value is formed by linear linear combination of the input image signal itself and a prediction coefficient for classification, a class is determined in correspondence with a prediction coefficient that produces a prediction value with a minimum error from a true value. . Even if the number of reference pixels is increased for the purpose of accurate classification, the capacity of a memory for storing data for classification is not so large.

[0014]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An example in which the present invention is applied to an up-conversion for converting the definition of a video signal from a standard one to a higher one will be described below. The present invention
In addition to this application, processing for interpolating pixels thinned out by processing such as subsampling, processing for electronically enlarging images, processing for detecting scene changes in video signals, and forming predicted values in DPCM It is possible to apply applications such as processing.

In FIG. 1, reference numeral 1 denotes an input terminal for a standard definition (for example, the current NTSC system) digital image signal (referred to as an SD signal). Specifically, transmission by broadcasting or the like, and a reproduction signal from a VTR or the like are supplied to the input terminal 1. Reference numeral 2 denotes a scan conversion circuit for converting an input signal into a signal having a block structure.

The output signal d0 of the scan conversion circuit 2 is supplied to a class predictor 3, a class determining circuit 4 and an output predictor 6. The class predictor 3 calculates a provisional or initial class prediction coefficient d2 from the table 5 in which the prediction coefficient is stored and the signal d0 to generate a prediction value d1 of the SD signal. The predicted value d1 is supplied to the class determination circuit 4.

In the class determination circuit 4, the predicted value d1
And the true value d0 of the SD signal to determine the class.
As an example, the class corresponding to the class prediction coefficient that generates the predicted value d1 with the smallest absolute prediction error from the true value d0 is determined as the class of the SD signal. The table 5 is referred to when the class is determined, and prediction coefficients of a plurality (n) of classes are sequentially supplied from the table 5 to the class determination circuit 4. The output predictor 6 generates an HD signal by calculating the data prediction coefficient d3 of the determined class and the SD signal d0. This HD signal is taken out to the output terminal 7. Table 5 stores class prediction coefficients and data prediction coefficients obtained in advance by learning.

An example of class determination and data prediction will be described with reference to FIG. In FIG. 2, 17 SD pixels included in the SD prediction tap area (each value is sd ₁
Ssd ₁₇ ), the values of 16 reference pixels other than the central pixel to be predicted (the value is sd ₈ ) are used,
The predicted value sd 'of the SD pixel is formed. That is, the predicted value sd
'Is generated by the following operation, where the class prediction coefficients are represented by k ₁ to k ₁₇ (excluding k ₈ ). _{sd' = k 1 × sd 1 +} ·· + k 7 × sd 7 + k 9 × sd 9 + ··· + k 17 × sd 17 (1)

The above-mentioned prediction formula relates to one class, and class prediction coefficients predetermined for n classes from 0 to n−1 are stored in the table 5. Although the example of FIG. 2 is a one-dimensional pixel array, the prediction may be performed using a two-dimensional pixel array. Although the number of reference pixels is not limited to 16 as a matter of course, the present invention
Even if the number of reference pixels is large, there is an advantage that the data amount of the class prediction coefficient does not extremely increase.

The HD pixel data prediction performed by the output predictor 6 is performed using three SD pixels near the position of the predicted HD pixel. In the example of FIG. 2, if the data prediction coefficients are w ₁ , w ₂ , and w ₃ , the HD pixel value hd ′ is generated by the following calculation. hd '= w ₁ × sd ₇ + w ₂ × sd ₈ + w ₃ × sd ₉ (2)

The above-mentioned prediction formula relates to one class, and data prediction coefficients predetermined for n classes 0 to n−1 are stored in the table 5. That is, as shown in FIG.
-1 stores the class prediction coefficient and the data prediction coefficient of each class. In the class determination circuit 4,
First, a prediction value sd 'is formed by the equation (1) using the class prediction coefficient of class 0, and the absolute value of the error between this and the true value is obtained. Hereinafter, for the other classes, the absolute value of the error of the predicted value is similarly obtained, and the one having the smallest absolute value is determined as the class of the SD pixel.

The class prediction coefficients in Table 5 above are:
It is determined in advance by learning. FIG. 4 is a flowchart showing a process at the time of learning. As an example, a standard picture image is used. Control of the learning process is started from step 11, and in the learning data formation in step 12, learning data corresponding to a known image is formed. Specifically, as described above, an array of 17 pixels arranged as shown in FIG. 2 is a set of learning data.

At the end of data in step 13, if processing of all input data, for example, data of one frame has been completed, the process proceeds to step 15 for determination of prediction coefficients, and if not completed, the process proceeds to normal equation generation in step 14. Control transfers.
In the normal equation generation in step 14, the normal equations of equations (8), (9), and (10) described later are created.

After the processing of all the data is completed, the control shifts to the step 15 from the end of the data in the step 13, and
In the prediction coefficient determination of No. 5, the prediction coefficients k1 to k16 are determined by solving Expression (10) described later using a matrix solution method. In the next step 16, the determined coefficients k _{1 to} k ₁₇ and S
The predicted value sd ′ is calculated by a linear linear combination (the above-described equation (1)) with the D pixel values sd _{1 to} sd _17, and the predicted sd ′ is calculated.
'The absolute value of the error between the true value sd ₈ is calculated. The calculation of the error is performed for the SD pixel used to determine the coefficient and all other SD pixels. For the SD pixels used to determine the coefficients, the error is very small.

In the next step 17, the calculated absolute value of the error is compared with the threshold value Th. If the absolute value of the error is less than the threshold value Th, the class prediction coefficient is stored in the memory as a coefficient of class i (step 1).
8). Then, it is determined in step 19 whether i = n, and if so, the learning process ends, and if not, i is incremented (step 20). Then, the process returns to step 12, and the above-described processing is repeated.

However, in step 17, the data of the SD pixel whose error is equal to or larger than the threshold value Th is determined, and in step 21 for data selection, the data used as the learning data generates an error equal to or larger than the threshold value Th. Limited to Thus, the class prediction coefficients of classes 0 to n-1 are determined.

Step 14 in FIG. 4 (normal equation generation)
And the processing of step 15 (determination of prediction coefficients) will be described in more detail. The true value sd8 of the SD pixel of interest is represented by y, its estimated value sd 'is represented by y', and the surrounding n (in FIG. 3,
n = 16) when the value of the pixel was set to x ₁ ~x _n, linear combination of n taps by coefficients k ₁ to k _n for each class _{y'= k 1 x 1 + k} 2 x 2 + ‥‥ + k n x _n (3) is set. Before learning, k _i is an undetermined coefficient.

As described above, learning is performed for each class.
If the number of data is m, according to equation _{(3), y j '=} k 1 x j1 + k 2 x j2 + ‥‥ + k n x jn (4) ( where, j = 1,2, ‥‥ m)

[0029] m> For n, since k ₁ to k _n are not uniquely determined, elements of an error vector _{E e j = y j - (} k 1 x j1 + k 2 x j2 + ‥‥ + k n x jn (5) (where j = 1, 2, ‥‥ m) and a coefficient that minimizes the following equation (6) is determined.

[0030]

(Equation 1)

This is a so-called least squares method. Here, the partial differential coefficient by k _i in Equation 4 is obtained.

[0032]

(Equation 2)

Since it is sufficient to determine each k _i so that equation (11) becomes 0,

[0034]

(Equation 3)

Using the matrix

[0036]

(Equation 4)

Is as follows. This equation is generally called a normal equation. If this equation is solved for k _i using a general matrix solution method such as a sweeping-out method, a prediction coefficient k _i is obtained.

The above-described method for determining the prediction class is merely an example, and various modifications are possible.

The data prediction coefficient wi is previously determined by learning using the HD signal and the SD signal obtained from the HD signal. FIG. 5 is a flowchart for data prediction. Control of the learning process is started from step 31, and in the learning data formation in step 32, learning data corresponding to a known image is formed. Specifically, as described above, three SD pixels and one HD pixel are a set of learning data as in the arrangement of FIG. Step 33
At the end of the data, if the processing of all the input data, for example, the data of one frame has been completed, the control is shifted to the prediction coefficient determination in step 36, and if not, the control is shifted to the class determination in step 34.

The class determination in step 34 is the same processing as that of the above-described class determination circuit 4. That is, a class prediction coefficient is formed by a linear linear combination of the class prediction coefficient determined as described above by learning and the pixel data of the SD signal, and a prediction value having a minimum error between the prediction value and the true value is generated. Discriminate the class to which the coefficient belongs. In the next normal equation generation in step 35, a normal equation is created.

After the processing of all data is completed from the end of data in step 33, the control moves to step 36, and
In the prediction coefficient determination of No. 6, the data prediction coefficient w is determined by solving using a matrix solution method. In step 37, the data prediction coefficients are stored in the memory.
Then, the control of the learning process ends. The normal equation generation in step 35 and the prediction coefficient determination in step 36 are based on the least squares method, similarly to the above-described class prediction coefficients.

In order to generate HD pixels, a representative value previously formed by learning can be used without using a data prediction coefficient. FIG. 6 shows an example of hardware at the time of learning. A digital HD signal for learning is supplied from an input terminal indicated by reference numeral 41. As this input signal, a still image signal of a different picture can be used. The HD signal is passed through the horizontal thinning circuit 42 and the vertical thinning circuit 43 to form an SD signal.

The generated SD signal is supplied to a classifying circuit 44.
Supplied to The class classification circuit 44 includes a scan conversion circuit, a class predictor, a table storing class prediction coefficients, and a class determination circuit, as in the configuration of FIG. The class code generated at the output of the class classification circuit 44 is supplied to the frequency memory 45 and the data memory 46 as addresses. These memories 45 and 46 are cleared before learning starts.

The input HD signal is supplied to a divider 49 via a delay circuit 47 and an adder circuit 48 as a dividend.
The output signal (division quotient) of the divider 49 is stored in the data memory 46.
Input data. The delay circuit 47 delays data for a time required for class classification.

The output read from the frequency memory 45 is the multiplier 50
And +1 circuit 51. The output of the +1 circuit 51 is used as the data input of the frequency memory 45.
9 is supplied as a divisor. When an address is specified by the class code, the frequency memory 45 and the data memory 46 read the contents of the address, and the frequency and data are written to the address. +
The accumulated frequency is stored in each address of the frequency memory 45 by one circuit 51.

In the configuration shown in FIG. 6, when a certain class code is generated, the cumulative frequency and the representative value of the class are read from frequency memory 45 and data memory 46, respectively, and are multiplied by multiplier 50. Therefore, a cumulative representative value is generated from the multiplier 50. The accumulated representative value and the current representative value from the delay circuit 47 are added by the adding circuit 48. The result of the addition is supplied to a divider 49 to form a representative value in consideration of the current representative value, which is written to the data memory 46.

By repeating this process for the input learning data, the accuracy of the representative value can be improved. The representative value of each class stored in the data memory 46 is used for up-conversion. FIG. 7 shows the contents of the table when the representative values are used. Class 0
Each of n-1 stores the class coefficient described above and the representative values L0 to Ln-1 determined by the configuration of FIG.

The process of determining the representative value is not limited to the hardware configuration shown in the example of FIG. 6, and can be realized by software processing.

[0049]

As described above, according to the present invention, the SD
Extract local features of the signal and define H
Since the D signal is output, upconversion for increasing the resolution can be favorably performed. In the present invention,
A prediction value is generated by a linear primary combination of a plurality of reference pixels and prediction coefficients, and a classification is performed by detecting a pixel having a minimum error between the prediction value and the true value.
Therefore, since a prediction coefficient equal to the number of reference pixels is stored,
Even if the number of reference pixels is increased, there is an advantage that the capacity of the memory for storing the class classification table does not increase so much.

[Brief description of the drawings]

FIG. 1 is a block diagram of an embodiment in which the present invention is applied to an apparatus for performing up-conversion.

FIG. 2 is a schematic diagram showing an arrangement of pixels for class classification and data prediction according to the present invention.

FIG. 3 is a schematic diagram illustrating a configuration of a table in which class prediction coefficients and data prediction coefficients are stored.

FIG. 4 is a flowchart when learning for determining a class prediction coefficient is performed by software processing.

FIG. 5 is a flowchart when learning for determining a data prediction coefficient is performed by software processing.

FIG. 6 is a block diagram of an example of a configuration at the time of learning for determining a representative value for data prediction.

FIG. 7 is a schematic diagram illustrating a configuration of a table in which class prediction coefficients and representative values are stored.

[Explanation of symbols]

3 Class predictor 4 Class determination circuit 5 Table storing class prediction coefficients and data prediction coefficients

──────────────────────────────────────────────────続 き Continued on the front page (58) Field surveyed (Int.Cl. ⁷ , DB name) H04N ^7/ 00-7/68

Claims (8)

(57) [Claims] 1. The method according to claim 1 , further comprising the steps of:
Digital with higher resolution than the input digital image signal
A digital image signal processing device that outputs a digital image signal,
A plurality of class prediction coefficients used for classifying the target pixel and a plurality of class prediction coefficients used for generating the target pixel are described.
A table in which a number of data prediction coefficients are stored in advance for each class, and a table in the input digital image signal near the noted position.
For a given pixel , a plurality of said input digitals which are spatially and / or temporally close to said given pixel
By calculating the reference pixel in the digital image signal and the class prediction coefficient from the table ,
Prediction method for forming a prediction value of a predetermined pixel included in
And the step, by identifying the class prediction coefficient generating nearest the prediction value to the true value of the predetermined pixel, and the class determining means for determining a class of the Note <br/> th pixel, the The above data prediction coefficient corresponding to the determined class and
Predict the value of the pixel of interest using the input image signal
And a predicting means for, the number of the data prediction coefficients, number of the class prediction coefficient
A digital image signal processing device characterized in that the number is less than the number . 2. The method according to claim 1 , further comprising the steps of:
Digital with higher resolution than the input digital image signal
A digital image signal processing device that outputs a digital image signal,
A table in which a plurality of class prediction coefficients used in classifying the pixel of interest and representative values are stored in advance for each class; and a table in the input digital image signal near the position of interest.
For a given pixel , a plurality of said input digitals which are spatially and / or temporally close to said given pixel
By calculating the reference pixel in the digital image signal and the class prediction coefficient from the table ,
Class prediction hand form a prediction value of a predetermined pixel included in the
And the step, by identifying the class prediction coefficient generating nearest the prediction value to the true value of the predetermined pixel, and the class determining means for determining a class of the Note <br/> th pixel, the The representative value corresponding to the determined class is
A digital image signal processing device having prediction means for outputting as a value of an eye pixel . 3. The digital image signal processing device according to claim 1, wherein the data prediction coefficient is included in an input digital image signal , and a plurality of pixels which are spatially and / or temporally close to the pixel of interest. And a coefficient that minimizes an error between the created value and the true value of the pixel of interest when the value of the pixel of interest is created by calculating the value of the pixel of interest and a plurality of coefficients. Image signal processing device. 4. The digital image signal processing device according to claim 2, wherein the representative value is a value generated in advance by learning.
And a digital image signal processing device. Claim 5.Based on the input digital image signal,
Digital with higher resolution than the input digital image signal
Output image signalsDigital image signal processing method
AndAt the position of interest in the output digital image signal
Multiple classes used when classifying the pixel of interest
A prediction coefficient and a compound used to generate the pixel of interest.
Of data prediction coefficients are stored in advance for each class.
Steps to read from the table , the aboveA place in the input digital image signal near the point of interest
For a given pixel, for the given pixelSpatial and
And / or multiple temporally nearbyThe above input digit
In the image signalClass prediction from reference pixel and above table
Coefficient of measurementWhenOf the input digital image signalDuring ~
includeThe predicted value of a given pixelClass predictions to form
Tep, True value of the predetermined pixelToGenerate the closest prediction
By identifying the class prediction coefficientsthe abovenote
Eye pixel classClass determination step to determine
When,The data prediction coefficient corresponding to the determined class and
Predict the value of the pixel of interest using the input image signal
Prediction step for performing The number of data prediction coefficients is the number of class prediction coefficients
Less than number Feature Digital image signal processing
Method. 6.Based on the input digital image signal,
Digital with higher resolution than the input digital image signal
Output image signalsDigital image signal processing method
AndAt the position of interest in the output digital image signal
When classifying the pixel of interest Multiple classes used
The prediction coefficients and the representative values are stored in advance for each class.
Steps to read from the bullA place in the input digital image signal near the point of interest
For a given pixel, for the given pixelSpatial and
And / or multiple temporally nearbyThe above input digit
In the image signalClass prediction from reference pixel and above table
Coefficient of measurementWhenOf the input digital image signalDuring ~
includeThe predicted value of a given pixelClass predictions to form
Tep, True value of the predetermined pixelToGenerate the closest prediction
By identifying the class prediction coefficientsthe abovenote
Eye pixel classClass determination step to determine
When,The representative value corresponding to the determined class is
A prediction step for outputting as an eye pixel value. Have
To Digital image signal processing method. 7. A digital image signal processor according to claim 6,
In management method, the data prediction coefficients, including in the input digital image signal
Rarely, spatially and / or temporally close to the noted pixel
By calculating the values of neighboring pixels and multiple coefficients,
When the value of the pixel of interest is created,
The coefficient must be such that the error from the true value of the eye pixel is minimized.
And a digital image signal processing method. 8. A digital image signal processor according to claim 7,
In the logical method, the representative value is a value generated in advance by learning.
And a digital image signal processing method.