JPH07154642A - Digital picture signal processor - Google Patents

Abstract

PURPOSE:To classifies picture elements in an excellent way with a few number of classes when a picture element not existing such as an interleaved picture element in subsampling is generated by linear coupling between peripheral picture elements and a coefficient obtained in advance through learning. CONSTITUTION:A block including a picture element as an interpolation object and plural reference picture elements around the picture element is formed. Two picture elements each in the reference picture elements are subtracted and the difference is fed to a sorting circuit 4. The sorting circuit 4 rearranges the picture elements based on the magnitude of the difference and a class code corresponding to the resulting sequence is generated. The class code is fed to a memory 6 as an address. Coefficients each minimizing an error between a predicted value and a true value are stored in advance to the memory 6. An interpolated value is generated by linear coupling between the coefficient and the surrounding picture element.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a digital image signal processing device, for example, a sub-image signal processing device, which needs to generate a value of a pixel of interest using a plurality of pixels which are spatially and / or temporally close to each other. Receiving the sampling signal,
The present invention relates to a digital image signal processing device for interpolating thinned pixels.

[0002]

2. Description of the Related Art As one method for band compression or information amount reduction when recording or transmitting a digital image signal, there is a method of reducing the amount of transmission data by thinning out pixels by subsampling. One example is the multiple sub-Nyquist sampling encoding method in the MUSE method. In this system, it is necessary to interpolate non-transmitted pixels that have been decimated on the receiving side. In addition, up-conversion that converts an input standard definition video signal into a high definition video has also been proposed. In this case, it is necessary to create the missing pixels from the standard definition signal. Furthermore, when the image is electronically magnified, interpolation of missing pixel values is required. Not limited to these, scene change detection, D
In PCM or the like, a technique of creating a predicted value of a target pixel from values of surrounding pixels can be applied.

Offset subsampling is known as an example of subsampling. An example of two-dimensional offset subsampling is shown in FIG. The sampling interval (T) in the horizontal direction (x direction) and the vertical direction (y direction)
x, Ty) is set to twice the pixel interval (Hx, Hy) in the original signal, and every other pixel is thinned out (thinned pixel is indicated by x), and adjacent transmission pixels (indicated by ○) in the vertical direction are set. The offset is half the sampling interval (Tx / 2). As shown in FIG. 8, the transmission band obtained by performing such offset sub-sampling can widen the spatial frequency component in the horizontal or vertical direction with respect to the spatial frequency in the oblique direction.

When the offset sub-sampled image signal is displayed on the monitor or printed out, it is necessary to interpolate the thinned pixel from the adjacent pixel as shown in FIG. This interpolation processing functions as a spatial filter that allows the frequency components in the shaded area shown in FIG. 8 to pass through and blocks the frequency components in the area including the folding point A. Is positioned as a post filter.

By the way, the offset sub-sampling as described above is a very effective method when the pre-filter before sampling correctly performs the filtering process. If the pre-filter cannot be applied sufficiently, or if the pre-filter is not applied sufficiently to widen the transmission band, the problem of image quality deterioration due to aliasing distortion occurs.

In order to reduce the occurrence of the above-mentioned folding distortion,
Adaptive interpolation methods have been proposed. This is a method in which the optimum interpolation method is determined in advance during subsampling, and the determination result is transmitted or recorded as auxiliary information. For example, which one of the horizontal half average value interpolation and the vertical half average value interpolation is closer to the true value is detected at the time of sub-sampling and transmitted as 1 bit of auxiliary information per pixel. However, at the time of interpolation, interpolation processing is performed according to this auxiliary information.

[0007]

In the above-described adaptive interpolation method using the auxiliary information, it is necessary to transmit the auxiliary information in addition to the transmission pixels, which causes a problem that the compression rate of the data amount decreases. Further, when an error occurs in the auxiliary information in the process of transmission or recording / reproduction, there is a drawback that the reproduced image is likely to be deteriorated because incorrect interpolation is performed.

As one method for solving this problem, Japanese Patent Laid-Open No. 63-48088 proposed by the applicant of the present application describes the value of a pixel of interest by a linear linear combination of the peripheral pixel and a coefficient, and the error It has been proposed to determine the value of this coefficient by the method of least squares using the actual value of the pixel of interest so that the sum of squares of is minimized. Here, the coefficient of the linear linear combination is previously determined by learning, and the coefficient of determination is stored in the memory. Furthermore, when interpolating the pixel of interest,
The average value of the surrounding reference pixels is calculated, and each pixel is represented by 1 bit according to the magnitude relationship between the average value and the value of each pixel,
The classification is performed according to the pattern of (the number of reference pixels × 1 bit), and the interpolated value reflecting the local feature of the image including the target pixel is formed.

However, in this method, when the number of reference pixels is small, it becomes insufficient to reflect local features,
On the contrary, when the number of reference pixels is large, the number of classes is large, and there is a problem that the scale of generation of a memory or the like becomes large.

Therefore, an object of the present invention is to provide a digital image signal processing apparatus which can perform class classification with high accuracy by using a relatively small number of classes.

[0011]

According to a first aspect of the present invention, a value of a target pixel is created by using values of a plurality of pixels existing spatially and / or temporally near the target pixel. In the digital image signal processing device requiring the above, a class classification for determining the class of the pixel of interest using a plurality of reference pixels included in the input digital image signal and spatially and / or temporally neighboring the pixel of interest. Means and
When the value of the pixel of interest is created by a linear linear combination of the values and the coefficients of a plurality of pixels spatially and / or temporally adjacent to the pixel of interest included in the input digital image signal,
A coefficient generating unit that generates a coefficient for each class so as to minimize the error between the created value and the true value of the pixel of interest,
In the digital image signal processing device, the class classification means detects level differences between reference pixels in a plurality of directions and determines an order when the detected level differences are sorted as a class of the pixel of interest. is there.

According to a second aspect of the present invention, it is necessary to create the value of the target pixel by using the values of a plurality of pixels existing spatially and / or temporally in the vicinity of the target pixel. In a signal processing device, a class classification means for determining a class of a pixel of interest using a plurality of reference pixels included in an input digital image signal and spatially and / or temporally neighboring the pixel of interest, and an input digital image When the value of the pixel of interest is created by the linear linear combination of the values and the coefficients of a plurality of pixels included in the signal and spatially and / or temporally neighboring the pixel of interest, the created value and the pixel of interest are The class classification means has a coefficient generation means for generating a coefficient for each class so as to minimize the error from the true value.
A digital image signal characterized by detecting a level difference between reference pixels in a plurality of directions and determining a level distribution pattern defined by a value obtained by nonlinearly quantizing the detected level differences as a class of the pixel of interest. It is a processing device.

[0013]

The difference between the values of a plurality of pixels spatially and / or temporally adjacent to the target pixel is obtained. This difference value represents the degree of correlation in the direction connecting the positions of the two pixels. Difference values are obtained for a plurality of directions, and the difference values are sorted. The order of the sorted results is used as class information. In addition, a level distribution pattern defined by a value obtained by compressing the difference value by nonlinear quantization is used as class information. These pieces of class information can express the characteristics of the partial image including the pixel of interest well with a relatively small number of classes, and can improve the prediction accuracy.

[0014]

DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment in which the present invention is applied to a subsampling signal interpolating device will be described below. INDUSTRIAL APPLICABILITY The present invention can be applied to a signal conversion device or the like that converts a standard resolution image signal into a high definition image signal. In FIG.
Reference numeral 1 is an input terminal for an offset sub-sampled digital image signal. Specifically, a transmission signal by broadcasting or the like, a reproduction signal from a VTR or the like is supplied to the input terminal 1. Reference numeral 2 is a blocking circuit for converting an input signal into a block structure signal.

The output signal of the blocking circuit 2 is supplied to the difference calculation circuit 5 and the interpolation value generation circuit 7. The interpolation value generation circuit 7 is connected to a memory 6 in which prediction coefficients acquired in advance by learning are stored as described later. This representative value is a value for accurately predicting the interpolation value of the thinned pixels.

The difference calculation circuit 3 calculates the difference in value between two pixels using a plurality of reference pixels spatially close to the interpolation pixel. Explaining with reference to FIG. 2, it is assumed that the pixel of X at the center position is the pixel to be interpolated. The twelve pixels to which the values of a to l are given are peripheral transmission pixels used as reference pixels. The difference calculation circuit 3 generates difference values α _{1 to} α ₆ by the following subtraction.

Α ₁ = a−c α ₂ = b−d α ₃ = e−i α ₄ = f−j α ₅ = g−k α ₆ = h−1

As can be seen from FIG. 2, the above-mentioned subtraction is to calculate the difference value with respect to the six directions consisting of the horizontal direction, the vertical direction and the four oblique directions with the pixel of interest as the center. The difference value represents the direction of correlation of partial images in the block centered on the pixel of interest. That is, the direction in which the difference value is small is the direction in which the correlation is strong.

The subtraction for detecting the direction in which the image of the block has a strong correlation is not limited to the above. For example, to check the degree of correlation in the horizontal direction, the difference values of g and l and the difference values of h and k that do not pass the pixel of interest are used, and the difference values of e and j and f and The difference value of i can be used.

The output signal of the difference calculation circuit 3 is supplied to the sorting circuit 4. The sorting circuit 4 determines the difference value α
Order _{1 to} α ₆ according to their size. The difference value is
Either a value with polarity or an absolute value may be used. A class code generation circuit 5 is connected to the sorting circuit 4, and a class code is generated from this. The class code generates a class code corresponding to the order of the result of the sorting performed by the sorting circuit 4. The class code is
The class corresponding to the order when the degree of correlation of the block in plural directions is sorted is designated.

The sorting process of the sorting circuit 4 will be described with reference to the flowchart of FIG. Here, the difference values are arranged in descending order. In addition, in FIG.
T means that the result of the judgment is positive, and F means that this is negative. First, in the first step 11, the number n of data is read (n = 6 in the above example). Next, in step 12, the data α _i (difference value) is read. First, the size of the data α ₁ and other data is checked.

In step 13, it is determined whether i≤n-1. If so, control transfers to step 14, otherwise control transfers to step 17.
After step 13, in step 14, j ≦ n
If yes, then control is passed to step 15. Step 15 is followed by step 16 of replacement. When the condition of step 15 is not satisfied, and after step 16, the process returns to step 14.

From step 13 to step 16,
The process of selecting the maximum value in the data and replacing the selected maximum value with the previously selected maximum value is performed. First, a value larger than the data α ₁ is searched from α _{2 to} α _n , and it is replaced with α ₁ . The maximum value becomes the first rank instead of α ₁ . Next, look for alpha ₂ greater than the alpha ₃ to? _N, replacing them alpha ₂ and. By repeating this process, data sorted in a large order can be obtained. In the present invention, a process of rearranging in ascending order may be used.

When using six difference values, there are 720 different orders. The sorting circuit 4 outputs the information on the order of the result of rearranging the data at the time of the above-mentioned sort processing. The class code generation circuit 5 receives this information from the sorting circuit 5 and generates a 10-bit class code representing 720 classes. At the time of sorting in the sorting circuit 4, it is also possible to multiply the difference values by a weighting coefficient in advance and sort the weighted values. Further, not only the difference value in the space but also the difference value in the time direction may be obtained.

The class code is supplied to the memory 6 as an address, and the prediction coefficient corresponding to the class is read from the memory 6. The interpolation value generation circuit 7 generates an interpolation value by linear linear combination of the prediction coefficient and the values of surrounding pixels.
The interpolation value generation circuit 7 outputs the interpolation value of the thinned pixel to the output terminal 8. As an example, 12 used for classification
Values a to l of the peripheral pixels and the prediction coefficient w ₁ from the memory 6
~ W ₁₂ is used to generate the interpolated value y ′ of the pixel of interest by the following linear linear combination.

[0026] _{y'= a · w 1 + b} · w 2 + c · w 3 + d · w 4 + e · w 5 + f · w 6 + g · w 7 + h · w 8 + i · w 9 + j · w 10 + k · w 11 ＋ l ・ w ₁₂ (1)

Here, the pixel referred to for class classification is the same as the pixel used for the interpolation calculation, but it is not always necessary.

Prediction coefficient w ₁ -stored in the memory 6
w ₁₂ is a value determined in advance by learning. Figure 4
It is a flow chart which shows the operation when learning is performed by software processing. The control of the learning process is started from step 21, and in the learning data formation of step 22, learning data corresponding to a known image is formed. In particular,
As mentioned above, multiple pixels in the arrangement of FIG. 2 can be used. At the data end of step 23, if processing of all input data, for example, one frame of data has been completed,
If the prediction coefficient determination in step 26 is not completed,
Control is transferred to the class determination in step 24.

As described above, the class determination in step 24 is a step of determining the difference value with respect to the peripheral pixels of the target pixel and determining the class corresponding to the order of the result of sorting the difference values. In the normal equation generation in the next step 25, a normal equation described later is created.

After the processing of all the data is completed from the end of the data of step 23, the control proceeds to step 26, and step 2
In the determination of the prediction coefficient of No. 6, the coefficient is determined by solving the equation (9) described later using the matrix solution method. The predictive coefficient store of step 27 stores the predictive coefficient in the memory, and step 28
Then, the control of the learning process ends.

Step 25 in FIG. 4 (normal equation generation)
The process of step 26 (determination of prediction coefficient) will be described in more detail. At the time of learning, the true value y of the pixel of interest is known. The predicted value of the pixel of interest is y ', and the values of the surrounding pixels are x.
₁ when the ~x _n, sets the linear 1 of n tap according to the coefficient w ₁ to w _n for each class linear combination _{y'= w 1 x 1 + w} 2 x 2 + ‥‥ + w n x n (2). Before learning, w _i is an undetermined coefficient.

As mentioned above, learning is done for each class,
When the number of data is m, according to the equation (2), y _j ′ = w ₁ x _j1 + w ₂ x _j2 + ... + w _n x _jn (3) (where j = 1, 2, ...

When m> n, w _{1 to} w _n are not uniquely determined, so the elements of the error vector E are e _j = y _j − (w ₁ x _j1 + w ₂ x _j2 + ... + w _n x _jn (4) (However, j = 1, 2, ..., M) is defined, and the coefficient that minimizes the following equation (5) is obtained.

[0034]

[Equation 1]

This is a so-called least squares method. Here, the partial differential coefficient by w _i of the equation (5) is obtained.

[0036]

[Equation 2]

Since each w _i may be determined so that the equation (6) becomes 0,

[0038]

[Equation 3]

If a matrix is used as

[0040]

[Equation 4]

It becomes This equation is generally called a normal equation. If this equation is solved for w _i using a general matrix solution method such as a sweeping method, the prediction coefficient w _i is obtained, and this prediction coefficient w _i is stored in the memory using the class code indicating the sort order as an address. Keep it.

Although FIG. 4 shows a software configuration for learning, learning can also be performed by a hardware configuration. Further, the interpolation value is not limited to the linear linear combination using the prediction coefficient, but the data value itself may be created in advance by learning, and this value may be used as the interpolation value.

FIG. 5 shows another embodiment of the present invention. Similar to the above-described embodiment, when the thinned pixels are interpolated, the reference pixels around the target pixel are used for the class classification.
A non-linear quantization circuit 9 is provided instead of the sorting circuit 4. As described above, the difference value between the values of the peripheral pixels is obtained, and this difference value is supplied to the non-linear quantization circuit 9.

An example of the quantization characteristic of the non-linear quantization circuit 9 is shown in FIG. 0, 1, 2, 3 for the level of the difference value
, But the level near 0 is quantized more finely than other levels. originally,
Since the image signal has a spatial correlation, the difference value is usually a small value. Therefore, it is preferable to finely quantize the level near 0, rather than linear quantization in which quantization is performed uniformly.

The output signal of the non-linear quantization circuit 9 is 2-bit data for each difference value. The class code generation circuit 5 generates a 6 × 2 = 12-bit class code when there are six difference values. Coefficients are stored in the memory 6 as previously obtained by learning using class classification using non-linear quantization. An interpolated value of the thinned pixel is generated by linear linear combination of the prediction coefficient from the memory 6 and the value of the peripheral pixel.

[0046]

According to the present invention, since the order of sorted results is used for classification, classification can be performed with a small number of classes, in other words, with a small hardware scale. For example, when the number of reference pixels is 12, even if each pixel is compressed to 1 bit, a 12-bit class code results in 2 ¹² classes, which can be reduced to 720 classes. The classification is
Since it is based on the order of the degree of correlation obtained for each of a plurality of directions of the local image, the classification can be performed well, and the interpolation accuracy does not decrease.

If nonlinear quantization is used,
The class code is 2 bits × 6 = 12 bits. Although the number of classes does not decrease, there is an advantage that the classification accuracy is higher because the difference value is calculated and 2 bits are allocated to each difference value as compared with the case where 1 bit is allocated to each pixel.

[Brief description of drawings]

FIG. 1 is a block diagram of an embodiment in which the present invention is applied to a sub-sampling signal interpolation device.

FIG. 2 is a schematic diagram showing positions of pixels to be referred to for classification.

FIG. 3 is a flowchart showing a sorting process for classifying.

FIG. 4 is a flowchart when learning for obtaining a prediction coefficient is performed by software processing.

FIG. 5 is a block diagram of another embodiment of the present invention.

FIG. 6 is a schematic diagram for explaining another embodiment of the present invention.

FIG. 7 is a schematic diagram showing a structure of two-dimensional offset subsampling.

FIG. 8 is a schematic diagram showing a spatial frequency spectrum of a band that can be transmitted by two-dimensional offset subsampling.

FIG. 9 is a schematic diagram showing an interpolation process.

[Explanation of symbols]

 3 Difference calculation circuit 4 Sorting circuit 6 Memory storing prediction coefficient 7 Interpolation value generation circuit

Claims (3)

[Claims] 1. A digital image signal processing apparatus which requires that a value of a pixel of interest is created by using values of a plurality of pixels existing spatially and / or temporally near the pixel of interest. Class classification means included in the digital image signal for determining the class of the pixel of interest using a plurality of reference pixels spatially and / or temporally adjacent to the pixel of interest; When the value of the pixel of interest is created by linearly combining the values of a plurality of pixels spatially and / or temporally neighboring the pixel of interest and the coefficient, the created value and the pixel of interest are There is a coefficient generation means for generating a coefficient for each class so as to minimize the error from the true value, and the class classification means detects and detects the level difference between the reference pixels in a plurality of directions. A digital image signal processing device, characterized in that the order in which the sorted level differences are sorted is determined as the class of the pixel of interest. 2. A digital image signal processing apparatus which requires that a value of a pixel of interest is created by using values of a plurality of pixels existing spatially and / or temporally near the pixel of interest. Class classification means included in the digital image signal for determining the class of the pixel of interest using a plurality of reference pixels spatially and / or temporally adjacent to the pixel of interest; When the value of the pixel of interest is created by linearly combining the values of a plurality of pixels spatially and / or temporally neighboring the pixel of interest and the coefficient, the created value and the pixel of interest are There is a coefficient generation means for generating a coefficient for each class so as to minimize the error from the true value, and the class classification means detects and detects the level difference between the reference pixels in a plurality of directions. A digital image signal processing device, characterized in that a level distribution pattern defined by a value obtained by nonlinearly quantizing the level difference is determined as a class of the pixel of interest. 3. The digital image signal processing device according to claim 1, wherein the pixel of interest is a pixel thinned by sub-sampling, and the coefficient from the coefficient generating means and the spatial value of the pixel of interest are / Or linear 1 with the values of multiple pixels that are temporally close
The digital image signal processing device further comprising an interpolation value generating means for generating an interpolation value of the thinned pixels by the next combination.