JP2002237942A - Digital image signal processing unit and processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a digital image signal processing unit, that classifies an input image into classes denoting local characteristics and suppresses increase in a capacity of a memory storing data for classification, even if many numbers of reference pixels are employed. SOLUTION: A scanning conversion circuit 2 simultaneously spatially and temporally outputs reference pixel data near a target pixel. A class prediction device 3 generates a prediction value, with respect to a prescribed SD pixel through linear coupling between the class prediction coefficient from a table 5 and the reference pixel values. A class decision circuit 4, detects a minimum value in errors between the predicted values calculated for all the classes and true values of the prescribed SD pixels. The class corresponding to the minimum value is decided as the class of the target pixel. Furthermore, the coefficients are stored in the table 5 by each class. The value of the HD pixel is calculated by linear coupling between the coefficient and a plurality of the SD pixels.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

TECHNICAL FIELD The present invention relates to spatial and / or
Also, the present invention relates to a digital image signal processing device and a processing method that require a value of a target pixel to be created using a plurality of pixels that are temporally nearby.

[0002]

2. Description of the Related Art As one of high-efficiency coding of digital image signals, there is a method of reducing transmission data amount by thinning out pixels by subsampling. One example is a multiple sub-Nyquist sampling encoding scheme in the MUSE scheme. In this system, it is necessary to interpolate non-transmitted pixels that have been thinned out on the receiving side. Up-conversion for converting an input standard definition video signal into high definition video has also been proposed. In this case, it is necessary to create the missing pixel from the signal of the standard definition. Furthermore, when electronically enlarging an image, interpolation of missing pixel values is required. Not limited to these, scene change detection, D
In PCM or the like, it is necessary to create an estimated value of a target pixel from values of peripheral pixels.

As described above, when creating the value of a target pixel, conventionally, an interpolation filter of a fixed tap and a fixed coefficient is usually used.

[0004]

The process of interpolating non-transmitted pixels using an interpolation filter is effective for a certain kind of image, but is not suitable for a wide variety of images such as moving images and still images. However, the interpolation process is not always effective as a whole. As a result, there is a problem that “blur”, “jerkiness” which is an unnatural motion, and the like occur in a restored image composed of transmission pixels and interpolation pixels.

As one method for solving this problem, the value of the target pixel is represented by a linear linear combination of surrounding pixels and coefficients, and the actual value of the target pixel is used so that the square of the error is minimized. A method of determining the value of this coefficient by the least square method has been proposed. Although this method is effective, it cannot be said that an interpolation value sufficiently reflecting local features of an image including a target pixel can be formed.

[0006] In order to reflect a local feature of an image, it has been proposed to perform a class classification based on a level distribution around a target pixel. As a method of generating this class, a method of directly using the level of the pixel data is considered. This method uses four pixels for class classification when pixel data is represented by 8 bits.
8) 4 = 232 classes are required, and there is a problem that the capacity of a memory for storing data for class classification becomes too large.

Further, Japanese Patent Application Laid-Open No.
Japanese Patent Application Laid-Open No. 3-48088 discloses that when interpolating thinned pixels, an average value of surrounding reference pixels is calculated, and each pixel is represented by 1 bit according to a magnitude relationship between the average value and the value of each pixel. It has been proposed to perform classification according to a pattern of (number of reference pixels × 1 bit). However, since this method binarizes the value of each pixel, it is insufficient to accurately reflect local features of an image.
When compressing reference pixel data for class classification, there is a similar problem when the compression ratio is increased.

Therefore, one object of the present invention is to make it possible to create a value having a small error from the actual value of the pixel of interest by performing class classification.
It is an object of the present invention to provide a digital image signal processing apparatus and a processing method capable of storing data for class classification with a relatively small capacity even when the number of reference pixels is large, and performing class classification accurately. .

It is another object of the present invention to provide a digital image signal processing apparatus and a processing method capable of converting (up-conversion) an image signal having a low resolution into an image signal having a high resolution.

[0010]

According to the present invention, a class prediction coefficient used in converting a digital image signal into a higher-resolution digital image signal is compared with a standard definition digital image signal. A digital image signal processing device for generating a coefficient based on a standard definition digital image signal and a high definition digital image signal based on a high definition digital image signal having a higher resolution than the definition digital image signal Learning data forming means for forming learning data to be used for generating a predicted value of a pixel of interest in a high-definition digital image signal based on an operation of a standard definition digital image signal and a coefficient; Coefficient generation means for generating a coefficient using the learning data so that an error between the value and the true value of the pixel of interest is minimized; Using the standard definition digital image signal different from the training data and the high definition digital image signal different from the learning data, the prediction value of the target pixel is predicted, and the error between the prediction value and the true value is calculated. Calculating means, and comparing means for comparing the error with a predetermined threshold value. If the error is equal to or less than the predetermined threshold value, the coefficient is output as a class prediction coefficient. If the difference is larger than the standard definition digital image signal and the high definition digital image signal used in calculating the error, the coefficient corresponding to the new learning data is obtained as new learning data. An image signal processing device.

According to a second aspect of the present invention, a class prediction coefficient used in converting a digital image signal into a higher-resolution digital image signal is determined by using a standard definition digital image signal and a standard definition digital image signal. A digital image signal processing method based on a high-definition digital image signal having a high resolution, wherein learning data used to generate coefficients between a standard definition digital image signal and a high-definition digital image signal are provided. Generating a predicted value of the pixel of interest in the high-definition digital image signal based on the calculation of the standard definition digital image signal and the coefficient. A coefficient generation step of generating a coefficient using the training data so that an error from the value is minimized;
Using a coefficient, a standard definition digital image signal different from the learning data, and a high definition digital image signal different from the learning data, the predicted value of the pixel of interest is predicted, and the error between the predicted value and the true value is calculated. And a comparing step of comparing the error with a predetermined threshold. If the error is equal to or smaller than the predetermined threshold, the coefficient is output as a class prediction coefficient, and the error is calculated as a predetermined value. When the difference is larger than the threshold value, the standard definition digital image signal and the high definition digital image signal used in calculating the error are used as new learning data, and a coefficient corresponding to the new learning data is obtained. Is a digital image signal processing method.

The classification can be performed according to the characteristics of the local image by referring to a plurality of pixels spatially and / or temporally near the pixel of interest. When a prediction value is formed by linear linear combination of the input image signal itself and a prediction coefficient for classification, a class is determined in correspondence with a prediction coefficient that produces a prediction value with a minimum error from a true value. . Even if the number of reference pixels is increased for the purpose of accurate classification,
The capacity of the memory for storing the data for classification is not so large.

[0013]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An example in which the present invention is applied to an up-conversion for converting the definition of a video signal from a standard one to a higher one will be described below. In addition to this application, the present invention provides a process for interpolating pixels thinned out by processes such as subsampling, a process for electronically enlarging an image, a process for detecting a scene change in a video signal, and a process for DPCM. Applications such as processing for forming a predicted value are possible.

In FIG. 1, reference numeral 1 denotes an input terminal for a standard definition (for example, the current NTSC system) digital image signal (referred to as an SD signal). Specifically, transmission by broadcasting or the like, and a reproduction signal from a VTR or the like are supplied to the input terminal 1. Reference numeral 2 denotes a scan conversion circuit for converting an input signal into a signal having a block structure.

An output signal d0 of the scan conversion circuit 2 is supplied to a class predictor 3, a class determination circuit 4 and an output predictor 6. The class predictor 3 calculates a provisional or initial class prediction coefficient d2 from the table 5 in which the prediction coefficient is stored and the signal d0 to generate a prediction value d1 of the SD signal. The predicted value d1 is supplied to the class determination circuit 4.

In the class determination circuit 4, the predicted value d1
And the true value d0 of the SD signal to determine the class.
As an example, the class corresponding to the class prediction coefficient that generates the predicted value d1 with the smallest absolute prediction error from the true value d0 is determined as the class of the SD signal. The table 5 is referred to when the class is determined, and prediction coefficients of a plurality (n) of classes are sequentially supplied from the table 5 to the class determination circuit 4. The output predictor 6 generates an HD signal by calculating the data prediction coefficient d3 of the determined class and the SD signal d0. This HD signal is taken out to the output terminal 7. Table 5 stores class prediction coefficients and data prediction coefficients obtained in advance by learning.

An example of class determination and data prediction will be described with reference to FIG. In FIG. 2, 17 SD pixels included in the SD prediction tap area (each value is sd1
To sd17), the values of 16 reference pixels other than the central pixel to be predicted (the value is sd8) are used,
The predicted value sd 'of the SD pixel is formed. That is, the predicted value sd
Is generated by the following operation, where the class prediction coefficients are represented by k1 to k17 (however, excluding k8). sd '= k1 x sd1 + ... + k7 x sd7 + k9 x sd9 + ... + k17 x sd17 (1)

The above-mentioned prediction formula relates to one class, and class prediction coefficients predetermined for n classes 0 to n−1 are stored in the table 5. Although the example of FIG. 2 is a one-dimensional pixel array, the prediction may be performed using a two-dimensional pixel array. Although the number of reference pixels is not limited to 16 as a matter of course, the present invention
Even if the number of reference pixels is large, there is an advantage that the data amount of the class prediction coefficient does not extremely increase.

The HD pixel data prediction performed by the output predictor 6 is performed using three SD pixels near the position of the predicted HD pixel. In the example of FIG. 2, assuming that the data prediction coefficients are w1, w2, and w3, the HD pixel value hd 'is generated by the following operation.

Hd ′ = w1 × sd7 + w2 × sd8 + w3 × sd9 (2)

The above-mentioned prediction formula relates to one class, and data prediction coefficients predetermined for n classes 0 to n−1 are stored in the table 5. That is, as shown in FIG.
-1 stores the class prediction coefficient and the data prediction coefficient of each class. In the class determination circuit 4,
First, a prediction value sd 'is formed by the equation (1) using the class prediction coefficient of class 0, and the absolute value of the error between this and the true value is obtained. Hereinafter, for the other classes, the absolute value of the error of the predicted value is similarly obtained, and the one having the smallest absolute value is determined as the class of the SD pixel.

The class prediction coefficients in Table 5 above are:
It is determined in advance by learning. FIG. 4 is a flowchart showing a process at the time of learning. As an example, a standard picture image is used. Control of the learning process is started from step 11, and in the learning data formation in step 12, learning data corresponding to a known image is formed. Specifically, as described above, an array of 17 pixels arranged as shown in FIG. 2 is a set of learning data.

At the end of data in step 13, if processing of all input data, for example, data of one frame has been completed, the process proceeds to step 15 for determination of prediction coefficients, and if not completed, the process proceeds to normal equation generation in step 14. Control transfers.
In the normal equation generation in step 14, the normal equations of equations (8), (9), and (10) described later are created.

After the processing of all the data is completed, the control shifts to the step 15 from the end of the data in the step 13, and
In the prediction coefficient determination of No. 5, the prediction coefficients k1 to k16 are determined by solving Expression (10) described later using a matrix solution method. In the next step 16, the determined coefficients k1 to k17 and S
The predicted value sd 'is calculated by a linear linear combination (the above-described equation (1)) with the D pixel values sd1 to sd17, and the predicted sd' is calculated.
The absolute value of the error between 'and the true value sd8 is calculated. The calculation of the error is performed for the SD pixel used to determine the coefficient and all other SD pixels. For the SD pixels used to determine the coefficients, the error is very small.

In the next step 17, the calculated absolute value of the error is compared with the threshold value Th. If the absolute value of the error is less than the threshold value Th, the class prediction coefficient is stored in the memory as a coefficient of class i (step 1).
8). Then, it is determined in step 19 whether i = n, and if so, the learning process ends, and if not, i is incremented (step 20). Then, the process returns to step 12, and the above-described processing is repeated.

However, in step 17, the data of the SD pixel whose error is equal to or larger than the threshold value Th is determined, and in step 21 for data selection, the data used as the learning data generates an error equal to or larger than the threshold value Th. Limited to Thus, the class prediction coefficients of classes 0 to n-1 are determined.

Step 14 in FIG. 4 (normal equation generation)
And the processing of step 15 (determination of prediction coefficients) will be described in more detail. The true value sd8 of the SD pixel of interest is represented by y, its estimated value sd 'is represented by y', and the surrounding n (in FIG. 3,
n = 16) Assuming that the pixel values are x1 to xn, the linear primary combination of n taps by coefficients k1 to kn, y '= k1x1 + k2x2 + .SIGMA. + knxn (3), is set for each class. Before learning, ki is an undetermined coefficient.

As described above, learning is performed for each class.
When the number of data is m, yj '= k1 xj1 + k2 xj2 + ‥‥ + kn xjn (4) (where j = 1, 2,... M) according to equation (3).

When m> n, k1 to kn are not uniquely determined, and the elements of the error vector E are expressed as ej = yj- (k1 xj1 + k2 xj2 + .SIGMA. + Kn xjn) (5) (where j = 1, 2) , ‥‥ m), and a coefficient that minimizes the following equation (6) is obtained.

[0030]

(Equation 1)

This is a so-called least squares method. Here, the partial differential coefficient by ki in Equation 4 is obtained.

[0032]

(Equation 2)

Since each ki may be determined so that equation (11) is set to 0,

[0034]

(Equation 3)

Using a matrix

[0036]

(Equation 4)

## EQU3 ## This equation is generally called a normal equation. By solving this equation for ki using a general matrix solution method such as a sweeping-out method, a prediction coefficient ki can be obtained.

The above-described method for determining the prediction class is merely an example, and various modifications are possible.

The data prediction coefficient wi is previously determined by learning using the HD signal and the SD signal obtained from the HD signal. FIG. 5 is a flowchart for data prediction. Control of the learning process is started from step 31, and in the learning data formation in step 32, learning data corresponding to a known image is formed. Specifically, as described above, three SD pixels and one HD pixel are a set of learning data as in the arrangement of FIG. Step 33
At the end of the data, if the processing of all the input data, for example, the data of one frame has been completed, the control is shifted to the prediction coefficient determination in step 36, and if not, the control is shifted to the class determination in step 34.

The class determination in step 34 is the same processing as that of the above-described class determination circuit 4. That is, a class prediction coefficient is formed by a linear linear combination of the class prediction coefficient determined as described above by learning and the pixel data of the SD signal, and a prediction value having a minimum error between the prediction value and the true value is generated. Discriminate the class to which the coefficient belongs. In the next normal equation generation in step 35, a normal equation is created.

After the processing of all data is completed from the end of data in step 33, the control moves to step 36, and
In the prediction coefficient determination of No. 6, the data prediction coefficient w is determined by solving using a matrix solution method. In step 37, the data prediction coefficients are stored in the memory.
Then, the control of the learning process ends. The normal equation generation in step 35 and the prediction coefficient determination in step 36 are based on the least squares method, similarly to the above-described class prediction coefficients.

In order to generate HD pixels, a representative value formed by learning in advance can be used without using a data prediction coefficient. FIG. 6 shows an example of hardware at the time of learning. A digital HD signal for learning is supplied from an input terminal indicated by reference numeral 41. As this input signal, a still image signal of a different picture can be used. The HD signal is passed through the horizontal thinning circuit 42 and the vertical thinning circuit 43 to form an SD signal.

The generated SD signal is supplied to a classifying circuit 44.
Supplied to The class classification circuit 44 includes a scan conversion circuit, a class predictor, a table storing class prediction coefficients, and a class determination circuit, as in the configuration of FIG. The class code generated at the output of the class classification circuit 44 is supplied to the frequency memory 45 and the data memory 46 as addresses. These memories 45 and 46 are cleared before learning starts.

The input HD signal is supplied to a divider 49 via a delay circuit 47 and an adder circuit 48 as a dividend.
The output signal (division quotient) of the divider 49 is stored in the data memory 46.
Input data. The delay circuit 47 delays data for a time required for class classification.

The output read from the frequency memory 45 is the multiplier 50
And +1 circuit 51. The output of the +1 circuit 51 is used as the data input of the frequency memory 45.
9 is supplied as a divisor. When an address is specified by the class code, the frequency memory 45 and the data memory 46 read the contents of the address, and the frequency and data are written to the address. +
The accumulated frequency is stored in each address of the frequency memory 45 by one circuit 51.

In the configuration shown in FIG. 6, when a certain class code is generated, the cumulative frequency and the representative value of the class are read from frequency memory 45 and data memory 46, respectively, and are multiplied by multiplier 50. Therefore, a cumulative representative value is generated from the multiplier 50. The accumulated representative value and the current representative value from the delay circuit 47 are added by the adding circuit 48. The result of the addition is supplied to a divider 49 to form a representative value in consideration of the current representative value, which is written to the data memory 46.

By repeating this process for the input learning data, the accuracy of the representative value can be improved. The representative value of each class stored in the data memory 46 is used for up-conversion. FIG. 7 shows the contents of the table when the representative values are used. Class 0
Each of n-1 stores the class coefficient described above and the representative values L0 to Ln-1 determined by the configuration of FIG.

The process of determining the representative value is not limited to the hardware configuration shown in the example of FIG. 6, and can be realized by software processing.

[0049]

As described above, according to the present invention, the SD
Extract local features of the signal and define H
Since the D signal is output, upconversion for increasing the resolution can be favorably performed. In the present invention,
A prediction value is generated by a linear primary combination of a plurality of reference pixels and prediction coefficients, and a classification is performed by detecting a pixel having a minimum error between the prediction value and the true value.
Therefore, since a prediction coefficient equal to the number of reference pixels is stored,
Even if the number of reference pixels is increased, there is an advantage that the capacity of the memory for storing the class classification table does not increase so much.

[Brief description of the drawings]

FIG. 1 is a block diagram of an embodiment in which the present invention is applied to an apparatus for performing up-conversion.

FIG. 2 is a schematic diagram showing an arrangement of pixels for class classification and data prediction according to the present invention.

FIG. 3 is a schematic diagram illustrating a configuration of a table in which class prediction coefficients and data prediction coefficients are stored.

FIG. 4 is a flowchart when learning for determining a class prediction coefficient is performed by software processing.

FIG. 5 is a flowchart when learning for determining a data prediction coefficient is performed by software processing.

FIG. 6 is a block diagram of an example of a configuration at the time of learning for determining a representative value for data prediction.

FIG. 7 is a schematic diagram illustrating a configuration of a table in which class prediction coefficients and representative values are stored.

[Explanation of symbols]

3 ・ ・ ・ Class predictor, 4 ・ ・ ・ Class decision circuit, 5 ・
..Tables storing class prediction coefficients and data prediction coefficients

 ──────────────────────────────────────────────────続 き Continued on the front page F-term (reference) AA21 BA07 BB04 CB04

Claims (2)

[Claims] A class prediction coefficient used in converting a digital image signal into a higher-resolution digital image signal includes a standard definition digital image signal and a high-resolution digital image signal having a higher resolution than the standard definition digital image signal. A digital image signal processing device for generating a digital image signal based on a high-definition digital image signal, comprising: a learning unit for forming learning data used to generate coefficients in the standard-definition digital image signal and the high-definition digital image signal. A data forming means for generating a predicted value of a pixel of interest in the high-definition digital image signal based on the calculation of the standard definition digital image signal and the coefficient; Coefficient generating means for generating the coefficient using the learning data so that an error with the value is minimized; Using the standard definition digital image signal different from the learning data and the high definition digital image signal different from the learning data to predict the predicted value of the pixel of interest, and Calculating means for calculating an error with the value; and comparing means for comparing the error with a predetermined threshold. If the error is equal to or less than a predetermined threshold, the coefficient is used as a class prediction coefficient. Output, and when the error is larger than a predetermined threshold, the standard definition digital image signal and the high definition digital image signal used when calculating the error are used as new learning data to generate the new learning data. A digital image signal processing device for obtaining coefficients corresponding to various learning data. 2. The method according to claim 1, wherein the class prediction coefficients used in converting the digital image signal into a higher resolution digital image signal are a standard definition digital image signal and a high definition digital image signal having a higher resolution than the standard definition digital image signal. A digital image signal processing method for generating a digital image signal based on a high-definition digital image signal, the method comprising the steps of: Forming a predicted value of the pixel of interest in the high-definition digital image signal based on the data forming step and the calculation of the standard definition digital image signal and the coefficient; A coefficient generation step of generating the coefficient using the learning data so that an error with the value is minimized; The prediction value of the pixel of interest is predicted using the coefficient, the standard definition digital image signal different from the learning data, and the high definition digital image signal different from the learning data, and A calculating step of calculating an error between the value and the true value; and a comparing step of comparing the error with a predetermined threshold. If the error is equal to or less than a predetermined threshold, the coefficient is classified into a class. Output as a prediction coefficient, and when the error is larger than a predetermined threshold, the standard definition digital image signal and the high definition digital image signal used in calculating the error are used as new learning data. And calculating a coefficient corresponding to the new learning data.