WO2001097510A1

WO2001097510A1 - Image processing system, image processing method, program, and recording medium

Info

Publication number: WO2001097510A1
Application number: PCT/JP2001/005117
Authority: WO
Inventors: Tetsujiro Kondo; Yasunobu Node; Katsuhisa Shinmei
Original assignee: Sony Corporation; Shiraki, Hisakazu
Priority date: 2000-06-15
Filing date: 2001-06-15
Publication date: 2001-12-20
Also published as: KR100816593B1; KR20020062274A

Abstract

An image processing system for receiving an input image signal to generate an output image signal of a higher quality than that of the input image signal, comprising first and second signal processing means. The first signal processing means performs a processing of storage type and includes storage means for storing an image signal of a quality identical to that of the output image signal. The first signal processing means adds the input image signal and the image, as stored in the storage means, to generate a first image signal of a higher quality than that of the input signal and to store the first image signal in the storage means. The second signal processing means performs classifying and adapting operations, and generates a second image signal of a quality higher than that of the input image by extracting based on the input image signal in accordance with a pixel position in the output image signal, by classifying the target pixel into one of a plurality of classes according to the features, and by operating the input image signal by an operation system preset in conforming with the classified class. The image processing system further comprises output selecting means for making a decision on the basis of the first image signal and the second image signal to select one of these first and second image signals as the output image signal.

Description

Specification

Image processing apparatus, image processing method, program, and recording medium

The present invention is applicable to, for example, a noise elimination device and a noise elimination method for removing noise of an image signal, and an image conversion device and an image conversion method for converting an input image signal into an image signal having a higher resolution. The present invention relates to an image processing device, an image processing method, a program, and a recording medium. Background art

As the image signal processing device, there are a device that accumulates image signals with time and a classification adaptive process proposed by the present applicant. For example, taking the noise removal process as an example, FIG. 1 shows a configuration in which image signals are accumulated with time, which is a configuration known as a motion adaptive recursive filter.

The input image signal is supplied to an addition circuit 2 through an amplifier 1 that performs amplitude adjustment for each pixel. The frame memory 3 stores the output image signal of the frame immediately before (current frame) the current frame (current frame of the input image signal (hereinafter, current frame)). I have. The image signal stored in the frame memory 3 is sequentially read out for each pixel corresponding to each pixel position of the input image signal, and is supplied to the addition circuit 2 through the amplifier 4 that performs amplitude adjustment. The addition circuit 1 adds the pixel of the current frame passed through the pump 1 and the pixel of the previous frame passed through the amplifier 4, outputs the added output as an output image signal, and supplies it to the frame memory 3. In the frame memory 3, the stored image signal is rewritten to the output image signal of the addition circuit 1. The input image signal of the current frame is also supplied to the subtraction circuit 5 for each pixel. Further, the image signal of the previous frame stored in the frame memory 3 is sequentially read out for each pixel corresponding to each pixel position of the input image signal and supplied to the subtraction circuit 5. The subtraction circuit 5 outputs the difference between the pixel value of the current frame at the same pixel position on the image and the pixel value of the previous frame.

The difference output from the subtraction circuit 5 is supplied to an absolute value conversion circuit 6, converted into an absolute value, and then supplied to a threshold value processing circuit 7. The threshold value processing circuit 7 compares the absolute value of the pixel difference supplied thereto with a predetermined threshold value, and determines whether the pixel is a moving part or a stationary part for each pixel. That is, when the absolute value of the pixel difference is smaller than the threshold value, the threshold value processing circuit 7 determines that the input pixel is a stationary portion, and when the absolute value of the pixel difference is larger than the threshold value, the input pixel is Judge as a moving part.

The result of the static motion determination in the threshold value processing circuit Ί is supplied to the weight coefficient generation circuit 8. The weighting coefficient generation circuit 8 sets the value of the weighting coefficient k (0≤k≤1) according to the result of the static / dynamic determination in the threshold processing circuit 7, supplies the coefficient k to the amplifier 1, and The coefficient 1 k is supplied to the amplifier 4. The amplifier 1 multiplies its input signal by k, and the amplifier 4 multiplies its input signal by 11 k.

In this case, when the threshold value processing circuit 7 determines that the pixel of the current frame is stationary, a fixed value between k = 0 to 0.5 is set as the value of the coefficient k. Therefore, the output of the addition circuit 2 is a value obtained by weighting and adding the pixel value of the current frame and the pixel value of the previous frame from the frame memory 3.

-, The threshold processing circuit 7 determines that the pixel of the current frame is a moving part , K = 1 is set as the value of the coefficient k. Therefore, the pixel value of the current frame (the pixel value of the input image signal) is output from the adding circuit 2 as it is.

Since the stored signal in the frame memory 3 is rewritten every frame by the output image signal from the adder circuit 2, the pixel values of a plurality of frames are integrated in the still portion in the image signal stored in the frame memory 3. It becomes something. Therefore, assuming that the noise changes randomly for each frame, the noise is gradually reduced and removed by the weighted addition, and the noise of the image signal (same as the output image signal) stored in the frame memory 3 is reduced. The stationary part is the one after noise removal.

However, noise removal by the above-described motion adaptive recursive fill has the following problems.

For example, when the noise level is high, a moving part may be mistaken for a still part, and in such a case, image quality deterioration such as blur may be observed. In addition, noise cannot be removed from moving parts.

On the other hand, a noise eliminator using the classification adaptive processing has been proposed by the present applicant. In the classification adaptive processing, noise can be removed from both stationary and moving parts. However, with respect to the complete still image portion, the above-described motion-adaptive recursive fill has better noise removal performance.

The present invention is also effective when applied to a resolution conversion device for increasing the resolution of an input image signal in addition to the noise removal processing.

In other words, the current television system is a so-called standard system in which the number of scanning lines per frame is 525 or 625, and a high-definition system in which the number of scanning lines per frame is larger than that. There are various types such as a multi-level system, for example, a high-definition system with 1 1 2 5 lines. In this case, for example, in order for a device compatible with the high-definition method to be able to handle the image signal of the standard method, an image signal having the resolution of the standard method must be converted into an image signal having a resolution matching the high-definition method. Resolution conversion (referred to as up conversion as appropriate). Therefore, conventionally, various resolution conversion devices for image signals using methods such as linear interpolation have been proposed. For example, up conversion by accumulation type processing and up conversion by class classification adaptive processing have been proposed.

However, the resolution conversion device based on the accumulation type processing can output the converted output image with little deterioration for the still image portion, but the image degradation occurs for the image portion with large motion. In addition, the resolution conversion device using the classification adaptive processing can obtain a converted output image with little deterioration in the case of a moving image portion, but cannot obtain a very good image in a stationary portion. There was a problem.

That is, in the past, it has been difficult to realize a resolution conversion device capable of forming an image without deterioration by accurately responding to both stationary and moving portions of the image.

Therefore, an object of the present invention is to provide an image processing apparatus and an image processing method capable of performing good processing as a whole by taking advantage of the configuration of accumulating image signals over time and the configuration of adaptive classification processing. , A program and a recording medium.

Disclosure of the invention

The invention according to claim 1 is an image processing apparatus that receives an input image signal and generates an output image signal having higher quality than the input image signal.

A storage unit for storing an image signal of the same quality as the output image signal, and by adding the input image signal and the image stored in the storage unit, First signal processing means for generating a first image signal of higher quality than the image and storing the first image signal in the storage means;

The feature based on the input image signal is extracted according to the position of the pixel of interest in the output image signal, the pixel of interest is classified into one of a plurality of classes according to the characteristic, and a predetermined value is determined corresponding to the classified class. A second signal processing means for generating a second image signal of higher quality than the input image by calculating the input image signal by a calculation method;

Output selection means for making a determination based on the first image signal and the second image signal, and selecting one of the first and second image signals as an output image signal;

An image processing apparatus having

The invention according to claim 26 is an image processing method for receiving an input image signal and generating an output image signal with higher quality than the input image signal, wherein the image signal having the same quality as the output image signal is stored in storage means, A first signal processing step of generating a higher quality first image signal than the input image by adding the signal and the stored image, and storing the first image signal in storage means;

The feature based on the input image signal is extracted according to the position of the pixel of interest in the output image signal, the pixel of interest is classified into one of a plurality of classes according to the characteristic, and a predetermined value is determined corresponding to the classified class. A second signal processing step of generating a second image signal of higher quality than the input image by calculating the input image signal by an arithmetic method;

An output selection step of performing a determination based on the first image signal and the second image signal, and selecting one of the first and second image signals as an output image signal;

Is an image processing method. The invention according to claim 51 is a program for causing a computer to execute image processing for generating an output image signal having higher quality than an input image signal.

An image signal of the same quality as the output image signal is stored in the storage means, and the input image signal and the stored image are added to generate a first image signal of higher quality than the input image, and the first image A first signal processing step for storing the signal in the storage means;

Is a program for executing.

The invention according to claim 52 is a computer-readable recording medium which records a program for causing a computer to execute image processing for generating an output image signal having higher quality than an input image signal. An image signal of the same quality as the image signal is stored in the storage means, and the input image signal and the stored image are added to generate a first image signal of higher quality than the input image, and the first image signal A first signal processing step of storing

The feature based on the input image signal is extracted according to the position of the pixel of interest in the output image signal, the pixel of interest is classified into one of a plurality of classes according to the characteristic, and a predetermined value is determined corresponding to the classified class. The input image signal A second signal processing step of generating a second image signal of higher quality than the input image by calculating the

Is a computer-readable recording medium on which a program for executing the program is recorded.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram showing an example of a conventional motion adaptive recovery filter.

FIG. 2 is a block diagram showing a basic configuration of the present invention.

FIG. 3 is a block diagram showing an embodiment of the present invention.

FIG. 4 is a block diagram of an example of a noise elimination circuit by accumulation type processing according to an embodiment of the present invention.

FIG. 5 is a flowchart showing an example of processing of a noise elimination circuit by accumulation type processing according to an embodiment of the present invention.

FIG. 6 is a block diagram showing an example of a class classification adaptive noise elimination circuit according to one embodiment.

FIG. 7 is a schematic diagram illustrating an example of a cluster map and a prediction map.

FIG. 8 is a block diagram showing an example of a feature detection circuit constituting a part of the classification adaptive noise elimination circuit.

FIG. 9 is a schematic diagram for explaining an example of a feature detection circuit. FIG. 10 is a block diagram showing a configuration at the time of learning for generating the coefficient data used in the classifying adaptive noise elimination circuit.

FIG. 11 shows a case where an embodiment of the present invention is processed by software. Is a flowchart for explaining the processing of FIG.

FIG. 12 is a flowchart showing a processing flow of the motion adaptive recursive filter.

FIG. 13 is a flowchart showing the flow of noise removal processing by the classification adaptive processing.

FIG. 14 is a flowchart showing the flow of processing at the time of learning for generating coefficient data used in the classifying adaptive noise elimination circuit.

FIG. 15 is a block diagram of another embodiment of the present invention.

FIG. 16 is a schematic diagram for explaining a resolution conversion process performed by another embodiment.

FIG. 17 is a block diagram showing an example of a configuration of a resolution conversion section by a storage process in another embodiment.

FIG. 18 is a schematic diagram for explaining the conversion processing of the resolution conversion unit by the accumulation processing.

FIG. 19 is a schematic diagram for explaining the conversion processing of the resolution conversion unit by the accumulation processing.

FIG. 20 is a block diagram showing an example of the configuration of a resolution conversion unit based on the classification adaptive processing.

FIG. 21 is a schematic diagram for explaining the processing operation of the resolution conversion unit by the class classification adaptive processing.

FIG. 22 is a block diagram showing an example of a feature detection circuit in the resolution conversion unit based on the classification adaptive processing.

FIG. 23 is a schematic diagram for explaining the operation of the feature detection circuit. FIG. 24 is a block diagram showing a configuration at the time of learning for generating coefficient data used in the resolution conversion unit by the classification adaptive processing.

FIG. 25 is a diagram illustrating a process of selecting an output image signal according to another embodiment. FIG.

FIG. 26 is a flowchart for explaining a process of selecting an output image signal in another embodiment.

FIG. 27 is a flowchart for explaining a process when another embodiment of the present invention is processed by software.

FIG. 28 is a flowchart showing the flow of the conversion process of the resolution conversion unit by the accumulation process.

FIG. 29 is a flowchart showing the flow of the resolution conversion processing by the classification adaptive processing.

FIG. 30 is a flowchart showing a flow of a learning process for generating coefficient data used in the resolution conversion process by the class classification adaptive process.

BEST MODE FOR CARRYING OUT THE INVENTION

FIG. 2 shows the overall configuration of the present invention. The input image signal is supplied to an accumulation processing unit 100 and a class classification adaptive processing unit 00. The accumulation processing section 100 is a processing section having a configuration for accumulating image signals as time passes. The class classification adaptive processing unit 200 detects a feature based on the input image signal according to the position of the pixel of interest in the output image signal, classifies the pixel of interest into one of a plurality of classes according to the characteristic, and performs classification. An output image signal is generated by calculating an input image signal by a predetermined calculation method corresponding to the class.

The output image signal of the accumulation type processing unit 100 and the output image signal of the class classification adaptive processing unit 200 are supplied to the selection circuit 301 and the output judgment circuit 302 of the output selection unit 300. . The output determination circuit 302 determines which output image signal is appropriate to output based on the output image signal of each processing unit. A selection signal corresponding to the result of this determination is generated. The selection signal is The signal is supplied to the selection circuit 301, and one of the two output image signals is selected.

When the present invention is applied to noise elimination, the accumulation processing unit 100 has the same configuration as the above-described motion adaptive recursive filter. Then, by repeating the weighted addition of the current frame and the previous frame, noise removal is satisfactorily performed on the pixels in the stationary portion.

On the other hand, the classification adaptive processing unit 200 is a noise removing unit based on the classification adaptive processing. The noise elimination unit using the classification adaptive processing extracts pixels of each frame at the same position among a plurality of frames, classifies noise components of the pixels based on a change between the frames of the pixels, and classifies the noise components. Since the noise component is removed from the input image signal by the arithmetic processing set in advance corresponding to the class, the noise is removed regardless of the moving part and the stationary part. However, with regard to the completely stationary part, the noise elimination unit of the accumulation type, which can accumulate information of long frames, has a larger noise elimination effect than the noise elimination unit by the classification adaptive processing.

The output selection unit 300 determines the stillness of the image in units of a predetermined number of pixels, and selects an output image signal from the noise removal unit based on the accumulation type processing in the stationary part according to the determination result. By selecting an output image signal from the noise removing unit based on the classification adaptive processing in the moving part, an output image signal from which the noise is removed in both the stationary part and the moving part can be obtained.

Further, when the present invention is applied to a resolution conversion device that performs up-conversion, the storage type processing unit 100 stores image information in a frame memory over a long period in the time direction, High resolution images It is configured to form an image signal. According to this configuration, a converted output image signal with little deterioration can be obtained for a still image or an image in which panning or tilting is performed simply on the entire screen.

On the other hand, the classification adaptive processing unit 200 is a resolution conversion unit based on the classification adaptive processing. The resolution conversion unit classifies a feature of a pixel of interest in an image based on an input image signal based on a feature of a plurality of pixels including the pixel of interest and its temporal and spatial surrounding pixels. Generates a high-resolution output image signal by generating a plurality of pixels in a high-resolution image corresponding to the target pixel by performing image conversion calculation processing that is set in advance corresponding to the classified class. I do. Therefore, the resolution conversion unit based on the classification adaptive processing can obtain a converted output image signal with little deterioration even in the moving part. However, for the stationary part, the storage-type resolution converter that handles image information longer in the time direction can perform better resolution conversion.

In consideration of the characteristics of each resolution conversion unit, the output selection unit 300 outputs the image signal from one resolution conversion unit and the image signal from the other resolution conversion unit for each pixel or every predetermined number of pixels. Since either one of the image signals can be selected and output, a high-quality converted output image with little deterioration can be obtained.

Hereinafter, an embodiment in which the present invention is applied to a noise removing device will be described with reference to FIG. The input image signal is supplied to the motion adaptive recursive filter 11 forming an example of the accumulation type processing unit 100 for each pixel, and the class classification forming the example of the classification adaptive processing unit 200. It is supplied to the adaptive noise elimination circuit 12.

As the motion adaptive recursive filter 11, a configuration similar to that of the example of FIG. 1 described above can be used. This motion adaptive recursive filter 1 1 These output image signals are supplied to the output selection unit 13 corresponding to the output selection unit 300.

Further, the classifying adaptive noise elimination circuit 12 extracts pixels of each frame at the same position among a plurality of frames, classifies the noise components of the pixels based on a change between the frames of the pixels, and An output image signal from which a noise component has been removed is generated from an input image signal by an arithmetic processing set in advance corresponding to the classified class, and its detailed configuration will be described later. The output image signal from the classification adaptive noise elimination circuit 12 is also supplied to the output selection unit 13.

The output selection unit 13 includes a static / movement determination circuit 14, a delay circuit 15 for timing adjustment, and a selection circuit 16, and the output image signal from the motion adaptive recursive filter 11 is delayed. The image signal supplied to the selection circuit 16 through the circuit 15 and output from the adaptive noise removal circuit 12 is supplied to the selection circuit 16 as it is.

The output image signal from the motion adaptive recursive filter 11 and the output image signal from the classification adaptive noise elimination circuit 12 are supplied to a static / motion determination circuit 14. In this example, the static / movement determining circuit 14 determines whether each pixel is a stationary portion or a moving portion from the two output image signals, and supplies the determination output as a selection control signal to the selection circuit 16. You.

In the output image signal from the motion adaptive recursive filter 11, as described above, the pixels in the still part of the image are noise-removed, but the pixels in the moving part of the image are output as they are without noise removal. You. On the other hand, in the output image signal from the classification adaptive noise elimination circuit 12, noise elimination is performed irrespective of the still part and the moving part of the image.

Therefore, the output image signal of the motion adaptive recursive filter When compared with the output image signal from the classifying adaptive noise elimination circuit 12, both pixel values of the stationary portion are almost equal because the noise is removed, but the motion adaptive recursive Although noise remains in the output image signal of the filter 11, the noise is removed in the output image signal from the adaptive noise removal circuit 12, so that the pixel values of both pixels are noise. Will be different by the minute.

In this example, the static / movement determining circuit 14 determines whether each pixel is a static portion or a moving portion of the image by using the above-described properties. That is, the still / moving judgment circuit 14 calculates the difference between the pixel value of the output image signal from the motion adaptive recursive filter 11 and the pixel value of the output image signal from the classification adaptive noise elimination circuit 12. It has a difference value calculation circuit 14 1, an absolute value conversion circuit 14 2 for converting a difference value from the difference value calculation circuit 14 1 into an absolute value, and a comparison judgment circuit 14 3.

When the absolute value of the difference value from the absolute value conversion circuit 144 is larger than a predetermined value, the comparison determination circuit 144 determines that the moving portion is present, and determines the difference value from the absolute value conversion circuit 144. When the absolute value of is smaller than a predetermined value, it is determined that the portion is a stationary portion. Then, the comparison and determination circuit 144 controls the selection circuit 16 so as to select the output image signal from the motion adaptive recursive filter 11 for the pixel determined to be a still part of the image, The selection circuit 16 is controlled so as to select an image signal output from the classifying adaptive noise elimination circuit 12 for the pixel determined to be the moving part of the image.

Therefore, from the selection circuit 16, that is, from the output selection unit 13, for the stationary part, information of a long frame can be accumulated, and the output image from the motion adaptive recursive filter that can remove noises well can be stored. The signal is output, and the motion part is An output image signal from the class classification adaptive noise elimination circuit 12 is output instead of the output image signal from the adaptive recursive filter. Therefore, an output image signal from which noise has been removed is obtained from the output selection unit 13 over the stationary part and the moving part.

The motion adaptive recursive filter 11 is not limited to the configuration shown in FIG. 1 but may be a configuration shown in FIG. In FIG. 4, reference numeral 101 denotes a delay circuit for time alignment, and reference numeral 104 denotes a motion vector detection circuit. The input image signal passed through the delay circuit 101 is supplied to the synthesizing circuit 102. The image stored in the storage memory 103 is supplied to the synthesizing circuit 102 via the shift circuit 105. The combined output of the combining circuit 102 is stored in the storage memory 103. The stored image in the storage memory 103 is taken out as an output and supplied to the motion vector detection circuit 104.

The motion vector detection circuit 104 detects a motion vector between the input image signal and the image stored in the storage memory 103. The shift circuit 105 shifts the position of the image read from the storage memory 103 horizontally and / or vertically according to the motion vector detected by the motion vector detection circuit 104. . Since the motion compensation is performed by the shift circuit 105, in the synthesizing circuit 102, the pixels spatially located at the same position are added as described below.

The composite value of the output of the composite circuit 102 = (pixel value of input image X N + pixel value of accumulated image X M) / (N + M) (N and M are predetermined coefficients)

Therefore, the accumulation memory 103 accumulates the result of adding the pixel data over a plurality of frame periods. By this processing, noise components having no correlation can be removed.

Fig. 5 shows the case where the processing of the configuration shown in Fig. 4 is performed by software processing. This is a flowchart showing the flow of the operation. In the first step S51, a motion vector is detected between an image region on the accumulated image and an image region on the input image corresponding to the image region. In the next step S52, the position of the stored image is shifted based on the detected motion vector. Then, the input image and the stored image whose position has been shifted are synthesized and stored (step S53). In step S54, the stored image is read from the storage memory and output.

[Explanation of Classification Adaptive Noise Removal Circuit]

Next, the class classification adaptive noise elimination circuit used in this embodiment will be described in detail. In the example described below, as the class classification adaptive processing, class classification is performed according to the three-dimensional (spatio-temporal) distribution of the signal level of the input image signal, and the prediction coefficients obtained by learning in advance for each class are stored in a memory. Then, a process of outputting an optimum estimated value (that is, a pixel value after noise removal) by an arithmetic process according to a weighted addition formula using such a prediction coefficient is described.

In this example, noise removal is performed by performing the classification adaptive processing in consideration of the motion of the image. That is, a pixel region to be referred to for detecting a noise component and a pixel region to be used for an arithmetic process for removing noise are cut out according to the motion estimated from the input image signal, Based on these, an image from which noise has been removed by the classification adaptive processing is output. FIG. 6 shows an overall configuration of a class adaptive noise removal circuit used in this embodiment.

The input image signal to be processed is supplied to a frame memory 21. The frame memory 1 stores the supplied image of the current frame and supplies the image of the previous frame to the frame memory 22. Frey The memory 22 stores the supplied one-frame image, and supplies the image one frame before to the frame memory 23. In this way, images of newer frames are stored in the frame memories 21, 22, and 23 in this order.

In the following description, the frame memory 2 stores the current frame, the frame memory 21 stores the frame one frame after the current frame, and the frame memory 23 stores the frame one frame before the current frame. The case is taken as an example.

Note that the storage contents of the frame memories 21, 11, 23 are not limited to this. For example, images at a time interval of two frames may be stored. Also, not limited to three consecutive frames, five frame memories may be provided to store images of five consecutive frames. Furthermore, a field memory can be used instead of the frame memory.

The image data of the rear frame, the current frame, and the previous frame stored in the frame memories 21, 1, and 23 are respectively stored in the motion vector detector 24, the motion vector detector 25, and the first frame. It is supplied to the area cutout section 26 and the second area cutout section 27.

The motion vector detection unit 24 detects a motion vector for a pixel of interest between the current frame image stored in the frame memory 12 and the previous frame image stored in the frame memory 23. I do. Further, the motion vector detection unit 25 calculates the motion vector of the pixel of interest between the image of the current frame stored in the frame memory 22 and the image of the subsequent frame stored in the frame memory 21. Detects torque.

The motion vector (movement direction and motion amount) of the pixel of interest detected by each of the motion vector detection units 24 and 25 is determined by the first area segmentation. It is supplied to the output section 26 and the second area cutout section 27. As a method of detecting a motion vector, a block matching method, an estimation using a correlation coefficient, a gradient method, or the like can be used.

The first area cutout unit 26 starts the image data of each frame supplied thereto from the evening, referring to the motion vectors detected by the motion vector detection units 24 and 25, and The extracted pixel value is supplied to the feature detection unit 28.

The feature detection unit 28 generates a class code representing information related to the noise component based on the output of the first area cutout unit 26, as described later, and stores the generated class code in the coefficient ROM 29. Supply. in this way

Since the pixels extracted by the first area cutout unit 26 are used for generating a class code, they are called class taps.

The coefficient R OM 29 stores a prediction coefficient determined by learning as described later in advance in Class I, more specifically, along with an address associated with the class code. The coefficient R OM 29 is calculated by the feature detector

28 Receives the class code supplied from 8 as an address and outputs the corresponding prediction coefficient.

On the other hand, the first area cutout unit 27 is a frame memory 21, 1, 2

3 extracts pixels for prediction from the data of three consecutive frames stored in each of the three, and supplies the values of the extracted pixels to the estimation calculation unit 30.

. The estimation operation unit 30 outputs the output of the second region extraction unit 27 and the coefficient R OM

Based on the prediction coefficient read out from the reference numeral 29, a weighting operation as shown in the following equation (1) is performed to generate a predicted image signal from which noise has been removed. As described above, the pixel values extracted by the second region cutout unit 27 are used in the weighted addition for generating the predicted image signal.

, Called prediction taps. = W 1 XX 1 + W 2 X 2 + '---+ Wn X n (1) where Χ,, X 2, ···, X _η are each prediction taps, and W i, 2, , W _n are the prediction coefficients.

FIG. 7 shows the tap structures of the class taps and the prediction taps cut out by the first area cut-out unit 26 and the second area cut-out unit 27, respectively. In FIG. 7, a target pixel to be predicted is indicated by a black circle, and a pixel cut out as a class tap or a prediction tap is indicated by a circle with a shadow. Fig. 7A shows an example of the structure of a basic class tap. From the current frame f [0] including the pixel of interest and the frames temporally before and after the current frame, that is, f [1-1] and f [+1], the pixel at the same spatial position as the pixel of interest is Cut out as a class tap.

That is, in this example, the class tap has a tap structure in which only one pixel is extracted in each of the previous frame f [-1], the current frame f [0], and the subsequent frame f [+1].

In the first area cutout unit 26, if the motion vector of the pixel of interest detected by the motion vector detection units 24 and 25 is sufficiently small and determined to be a static part, Frame f [_ 1], current frame f

[0], the pixel at the same pixel position in each frame of the subsequent frame f [+1] is extracted as a class tap for noise detection. Therefore, the pixel position of the class tap in each frame to be processed is constant, and the tap structure does not change.

On the other hand, if it is determined that the movement of the target pixel is larger than a certain level and it is a moving part, the first area cutout unit 26 sets the previous frame f [-1], the current frame f [0], and the subsequent frame f From each frame of [+ 1], the pixel at the position corresponding to the Extract. In other words, the pixel at the position corresponding to the motion vector is extracted

. The position of the pixel to be extracted from the image data of the subsequent frame f [+1] is determined by the motion vector detected by the motion vector detection unit 24, and the image data of the previous frame f [1-1] is determined. The position of the pixel extracted from the evening is determined by the motion vector detected by the motion vector detection unit 25. FIG. 7B shows an example of a basic prediction tap structure extracted by the second region cutout unit 27. From the pixel data of the frame of interest and the image data of the frames temporally located before and after the frame of interest, a total of 13 pixels including the pixel of interest and, for example, 12 pixels surrounding the pixel of interest are obtained. It is cut out as a prediction tap.

Further, FIGS. 7C and 7D show the case where the cut-out position is temporally shifted according to the motion vector output from the motion vector detection units 24 and 25. FIG. As shown in Fig. 7E, the motion vector in the frame of interest is (0, 0), the motion vector in the previous frame is (11, 11), and the motion vector in the subsequent frame is (0, 0). In the case of 1, 1), the cutout positions of the class taps and prediction taps in the entire frame are translated in accordance with the motion vector.

Note that a tap structure similar to the above-described class tap may be used as the prediction tap.

As a result of the pixel extraction based on such a motion vector, the cluster tap extracted by the first region cutout unit 26 becomes a corresponding pixel on the image between a plurality of frames. Similarly, the prediction tap extracted by the second area cutout unit 27 also becomes a corresponding pixel on an image between a plurality of frames due to the motion correction.

It should be noted that the number of frame memories is increased and, for example, five instead of three, for example, by storing the current frame and two frames before and after the current frame. Alternatively, a class tap structure may be used in which only the pixel of interest is extracted from the current frame, and a pixel corresponding to the pixel of interest is extracted from each of the preceding and succeeding two frames. In such a case, the pixel region to be extracted is expanded temporally, so that more effective noise removal can be performed.

As will be described later, the feature detection unit 28 uses the variation in the pixel value of the pixel of the three frames extracted as a class tap in the first area extraction unit 26 as To detect. Then, a class code corresponding to the level change of the noise component is output to the coefficient ROM 29. That is, the feature detecting unit 28 classifies the level variation of the noise component of the target pixel into classes, and outputs a class code indicating which of the classified classes.

In this embodiment, the feature detection unit 28 performs ADRC (Adaptive Dynamic Range Coding) on the output of the first region cutout unit 26, and performs level fluctuation of the corresponding pixel of the pixel of interest over a plurality of frames. Generates a class code consisting of the ADRC output.

FIG. 8 shows an example of the feature detecting section 28. Fig. 8 shows the generation of class code by 1-bit ADRC.

As described above, the dynamic range detection circuit 281 includes, from each of the frame memories 21, 1, and 23, two pixels corresponding to the pixel of interest of the current frame and the pixel of interest before and after the current frame. A total of three pixels are supplied. The value of each pixel is represented by, for example, 8 bits. The dynamic range detection circuit 281 detects the maximum value MAX and the minimum value MIN among the three pixels, and calculates the dynamic range DR by an operation of MAX_MIN = DR.

And the dynamic range detection circuit 281, as its output, The calculated dynamic range DR, the minimum value MIN, and the pixel value p X of each of the three input pixels are output.

The pixel values P of the three pixels from the dynamic range detection circuit 281 are sequentially supplied to a subtraction circuit 282, and the minimum value M IN is subtracted from each pixel value PX. By removing the minimum value M IN from each pixel value P X, the normalized pixel value is supplied to the comparison circuit 283.

The output (DR / 2) of the bit shift circuit 284 that reduces the dynamic range DR to 1/2 is supplied to the comparison circuit 283, and the magnitude relationship between the pixel value PX and DR / 2 is detected. . When the pixel value PX is larger than DR / 2, the 1-bit comparison output of the comparison circuit 283 is set to “Γ”, otherwise, the comparison output is set to “0”. The 283 generates the 3-bit ADRC output by parallelizing the comparison outputs of the three pixels obtained sequentially.

The dynamic range DR is supplied to a bit number conversion circuit 285, and the number of bits is converted from 8 bits to, for example, 5 bits by quantization. Then, the dynamic range converted into the number of bits and the 3-bit ADRC output are supplied to the coefficient ROM 29 as a class code.

Under the cluster-top structure described above, the pixel value should not fluctuate or be small between the target pixel of the current frame and the corresponding pixels of the previous and subsequent frames. Therefore, when a change in pixel value is detected, it can be determined that the change is caused by noise.

To explain one example, in the case of the example shown in Fig. 9, the pixel value of the class tap extracted from each temporally continuous frame of t-1, t, t + 1 is 1-bit ADRC processing. This generates a 3-bit [0 10] ADRC output. And the dynamic range DR The one converted to 5 bits is output. The 3-bit ADRC output represents the noise level fluctuation for the pixel of interest.

In this case, by performing multi-bit ADRC instead of one bit, it is possible to more accurately express the noise level fluctuation. The noise level is represented by a code obtained by converting the dynamic range DR into 5 bits. The purpose of converting 8 bits to 5 bits is to avoid having too many classes.

Thus, in this example, the class code generated by the feature detection unit 28 is, for example, a 3-bit code related to the noise level fluctuation in the time direction obtained as a result of ADRC, and the result of the dynamic range DR. And a code consisting of, for example, 5 bits related to the noise level obtained as By using the dynamic range DR for class classification, motion and noise can be distinguished, and differences in noise level can be distinguished.

Next, a learning process, that is, a process of obtaining a prediction coefficient to be stored in the coefficient ROM 29 will be described with reference to FIG. Here, the same components as those in FIG. 6 are denoted by the same reference numerals.

A noise-free input image signal (referred to as a teacher signal) used for learning is supplied to a noise adding unit 31 and a normal equation adding unit 32. The noise adding unit 31 adds a noise component to the input image signal to generate a noise-added image (referred to as a student signal), and supplies the generated student signal to the frame memory 21. As described above with reference to FIG. 6, the frame memories 21, 23 store the images of the student signal of three frames that are temporally continuous, respectively.

In the following description, the frame memory 22 stores the image of the current frame, and the frame memories 11 and 23 respectively store the image after the current frame. And the case where the image of the previous frame is recorded. However, as described above, the storage contents of the frame memories 21, 1, and 23 are not limited to this.

In the subsequent stage of the frame memories 21, 1, and 23, almost the same processing as that described above with reference to FIG. 6 is performed. The same reference numerals as those in FIG. 6 denote the same processes. However, the class code generated by the feature detecting unit 28 and the prediction tap extracted by the second region extracting unit 27 are supplied to the normal equation adding unit 32. The teacher signal is further supplied to the normal equation addition unit 32. The normal equation adding unit 32 performs a process of generating a normal equation in order to generate a coefficient based on these three types of inputs, and the prediction coefficient determination unit 33 executes a prediction coefficient for each class code from the normal equation. To determine. Then, the prediction coefficient determination unit 33 supplies the determined prediction coefficient to the memory 34. The memory 34 stores the supplied prediction coefficients for each class. The prediction coefficients stored in the memory 34 and the prediction coefficients stored in the coefficient ROM 29 (FIG. 6) are the same.

Next, the normal equation will be described. In the above equation (1), before learning, the prediction coefficients,..., Wn are undetermined coefficients. Learning is performed by inputting multiple teacher signals for each class. When the number of classes of the teacher signal 信号 is denoted by m, the following equation (2) is set from the equation (1).

yk = W l XX kl + W ₂ XX k2 +---+ Wn X kn (2)

(k = 1, ■ · ·, m)

For m> n, prediction coefficients w !,.. · ·, than w _n are not uniquely determined, elements e _k error base-vector e, it is defined by the following equation (3).

ek = yk-{wi X xki + w ₂ x k2 +-·-+ w _n xx _kn } (3) (k = 1, 2, · · ·, m)

Then, the prediction coefficient is determined so as to minimize the error vector e defined by the following equation (4). That is, the prediction coefficient is uniquely determined by the so-called least square method. e = sigma e, (4) k = o formula e ² (4) As a practical calculation method for determining the prediction coefficients to minimize, predict e ² coefficients Wi (〖= 1, 2, · )) (Equation (5) below), and each prediction coefficient wi should be determined so that the partial differential value becomes 0 for each value of i. 2½ e. (5)

The specific procedure for determining each prediction coefficient wi from equation (5) will be described. If Χ and Υι are defined as in equations (6) and (7), equation (5) can be written in the form of the determinant of equation (8) below.

xpi ' ^x pj (6)

/ J = 0

^X \ 2 '

21 ^X 22 .w ₂

(8)

• · · «· • · ■

^X n2 ··· ^X nn — one Y n set (8) is generally called a normal equation. The prediction coefficient determination unit 33 calculates each parameter in the normal equation (8) based on the above three types of inputs, and further converts the normal equation (8) according to a general matrix solution such as a sweeping-out method. Perform the calculation process to solve to calculate the prediction coefficient.

Next, in order to add noise in the noise adding unit 31, for example, any of the following four methods can be used.

1. Like in computer simulation, random noise is generated and added to the input image signal.

2. Add noise to the input image signal via the RF system.

3. Extract a noise component as a difference between a flat image signal with a small level change and a signal obtained by processing the image signal through an RF system, and use the extracted noise component as an input image. Add to signal.

4. The noise component is extracted as the difference between the signal obtained by performing the processing using the RF system on the flat image signal and the image signal component from which the noise is removed by frame addition of the signal. The extracted noise component is added to the input image signal.

The noise elimination circuit 12 using the above-described class classification adaptive processing performs, for example, a pixel of interest and a pixel corresponding to the pixel of interest as a class tap when performing the class classification adaptive processing to remove noise from the image signal. The noise level between frames based on the data of the class It detects the fluctuation of the noise level and generates a class code corresponding to the detected fluctuation of the noise level.

Then, pixels to be used for noise component detection processing (class taps) and pixels to be used for prediction calculation processing (prediction taps) are extracted so as to estimate the motion between frames and correct the estimated motion. I do. Then, for each class information reflecting the noise component, an image signal from which noise has been removed is calculated by linear linear combination of the prediction tap and the prediction coefficient.

Therefore, it is possible to select a prediction coefficient that accurately corresponds to the inter-frame variation of the noise component.Thus, by performing the estimation operation using such a prediction coefficient, it is possible to effectively remove the noise component. Yes And even if there is movement, the noise level can be detected correctly and noise can be removed. In particular, it is possible to prevent the image from being blurred due to the erroneous determination that the moving part is a stationary part as in the case of the motion adaptive recursive fill described with reference to FIG. Furthermore, a cluster-top structure having no spatial spread in a frame, for example, only a pixel of interest is extracted from the current frame, and a pixel corresponding to the pixel of interest is extracted from a frame temporally before / after the current frame. When such a tap structure is used as a class tap and / or a prediction tap, the influence of a spatial blur factor on processing can be reduced. That is, it is possible to reduce the occurrence of blurring in the output image signal due to, for example, the influence of edges and the like.

Thus, the classification adaptive noise elimination circuit 1 2

Denoising is performed independently of motion. However, the perfectly stationary part is inferior to a moving adaptive recursive filter that can store information of long frames. In the present invention, as described above, in the stationary part, the output of the motion adaptive recursive filter as shown in FIG. 1 or FIG. 4 is selected and output, and in the moving part, the class classification as shown in FIG. Since the output of the adaptive noise elimination circuit is selectively output, an image signal output with good noise elimination can be obtained in both the moving part and the stationary part of the image. It should be noted that the class taps and prediction taps in the first area cutout section 26 and the second area cutout section 27 in the description of the class classification adaptive elimination circuit are merely examples, and it goes without saying that they are not limited to these. . Also, in the above description, the feature detection unit 28 uses a one-bit ADRC encoding circuit. However, as described above, a multi-bit ADRC encoding circuit may be used. An encoding circuit may be used.

Furthermore, in the above description, the selection between the output of the motion adaptive recursive filter 11 and the output of the classification adaptive noise elimination circuit 12 is performed on a pixel-by-pixel basis. The selection may be performed in units of pixel blocks or objects each including a number of pixels, or in units of frames. In those cases, the static / movement determination circuit performs the static / movement determination in selected units.

In the above example, the output of one motion adaptive recursive filter and the output of one class classification adaptive elimination circuit are selected as alternatives. It is also possible to provide a plurality of noise removing circuits by processing and select an output image signal from them.

In one embodiment of the present invention, the processing is not limited to being performed by hardware, but may be performed by software. The processing by soft-to-air will be described below. FIG. 11 shows an embodiment of the present invention. 9 is a flowchart showing a flow of a noise removal process. As shown in steps S 1 and S 2, the classification adaptive noise elimination process and the motion adaptive recursive fill process are performed in parallel. The difference between the outputs obtained in each process is calculated (step S3).

In step S4, the difference is converted into an absolute value, and in step S5 of the determination, it is determined whether or not the absolute value of the difference is large. If it is determined that the absolute value of the difference is large, the output of the adaptive noise removal for classification is selected (step S6). Otherwise, the output of the motion adaptive recursive filter is selected (step S7). This completes the processing for one pixel.

FIG. 12 is a flowchart showing details of the processing S2 of the motion adaptive recursive filter. In a first step S11, an initial input image is stored in a frame memory. In the next step S12, the difference between the image in the frame memory and the next input image (frame difference) is calculated. This difference is converted into an absolute value in step S13.

The absolute difference is compared with a threshold value in step S14. When the difference is equal to or larger than the threshold, the weight coefficient k by which the input image signal is multiplied is set to 1 (step S15). That is, the weighting factor (11 k) multiplied to the output signal of the frame memory is set to 0 because it is a moving part. On the other hand, if the difference is smaller than the threshold value, k is set to a value within the range (0 to 0.5) in step S16.

Then, the pixel in the frame memory and the pixel at the same position of the next input image are weighted and added (step S17). The addition result is stored in the frame memory (step S18). At the same time, the process returns to step S12. Then, the addition result is output (step S 1 9) _σ

FIG. 13 is a flowchart showing details of the classification adaptive noise elimination process S1. In the first step S21, a motion vector is detected between the current frame and the previous frame. In the next step S22, a motion vector is detected between the current frame and the next frame. In step S23, the first region is cut out. That is, the class setup is extracted. In step S24, the extracted class tap is subjected to feature detection processing. The coefficient corresponding to the detected feature is read out of the coefficients obtained by the learning processing in advance (step S25).

In step S26, a second area (prediction tap) is cut out. In step S27, an estimation operation is performed using the coefficient and the prediction tap, and an output from which noise has been removed is obtained. In the first region cutout processing (step S23) and the second region cutout processing (step S26), the motion vectors detected in steps S21 and S22 are used. The cutout position is changed.

FIG. 14 is a flowchart showing a flow of a learning process for obtaining a coefficient used in the classification adaptive noise elimination process. In step S31, a student signal is generated by adding noise to an image signal (teacher signal) without noise. Regarding the student signal, in step S32, a motion vector is detected between the current frame and the previous frame. In the next step S33, a motion vector is detected between the current frame and the next frame. The region cutout position is changed by these detected motion vectors.

In step S34, the first area (class tap) is cut out. Feature detection is performed based on the extracted class taps (step Step S35). Next, in step S36, a second region (prediction tap) is cut out. Then, in step S37, based on the teacher image signal, the data of the prediction gap, and the detected features, data necessary for solving a normal equation having a prediction coefficient as a solution is calculated.

Then, in step S38, it is determined whether the addition of the normal equations has been completed. If not, the process returns to step S31. If it is determined that the processing has been completed, a prediction coefficient is determined in step S39. The obtained prediction coefficients are stored in the memory and used in the noise removal processing.

As described above, according to the noise elimination circuit of the present invention, in the stationary part, the output of the noise elimination circuit having a large noise elimination effect for the stationary part such as the motion adaptive recovery filter is selectively output. The output of the noise elimination circuit that can remove noise in the moving part, such as the adaptive noise elimination circuit for classification, is selected and output, so that the noise can be removed well in both the moving part and the stationary part of the image. The resulting image signal output is obtained.

Next, another embodiment in which the present invention is applied to a resolution conversion device that performs up-conversion will be described with reference to FIGS. In another embodiment described below, an image signal of the above-described standard television system (hereinafter, referred to as SD) is used as an input image signal, and is converted into an output image signal of a high-definition system (hereinafter, referred to as HD). This is the case. In another embodiment described below, as shown in FIG. 16, for each pixel of interest of the SD image, four pixels of the HD image are created and the resolution is converted. is there.

FIG. 15 is a block diagram showing a configuration of another embodiment. As shown in FIG. 15, in this example, the input image signal is stored in a storage type Supply to the high-density storage resolution conversion circuit 111 that constitutes an example of the resolution conversion unit based on logic, and to the class classification adaptive processing resolution conversion circuit 112 that constitutes an example of the resolution conversion unit that performs the classification adaptive processing. Is done.

The high-density storage resolution conversion circuit 1 1 1 includes a frame memory for storing image signals of HD-equivalent images, and converts between an image based on the image signals stored in the frame memory and an image based on the SD input image signal. By accumulating the SD input image signal in the frame memory while correcting the pixel position with reference to the movement of the image, an HD-equivalent output image signal is generated in the frame memory. The detailed configuration will be described later. The converted image signal equivalent to HD from the high-density storage resolution conversion circuit 111 is supplied to the output selection unit 113.

Further, the class classification adaptive processing resolution conversion circuit 112 includes a plurality of features including a target pixel in the image based on the SD input image signal and its temporal and spatial surrounding pixels. Detect from pixel. Then, the target pixel is classified into classes based on the detected features, and a plurality of pixels in the HD image corresponding to the target pixel are determined by an image conversion operation set in advance corresponding to the classified class. By generating it, a high-resolution output image signal is generated, and its detailed configuration will be described later. The converted image signal corresponding to HD from the class classification adaptive processing resolution conversion circuit 112 is also supplied to the output selection unit 113.

The output selection unit 113 includes a judgment circuit 114, which will be described in detail later, and a selection circuit 115, and converts the converted image signal from the high-density storage resolution conversion circuit 111, and class classification adaptive processing. The converted image signals from the resolution conversion circuits 112 are supplied to the selection circuits 115, respectively.

Also, the converted image signal from the high-density storage resolution conversion circuit 111 The converted image signal from the Lass classification adaptive processing resolution conversion circuit 112 is supplied to the judgment circuit 114. The determination circuit 114 determines, from the two converted image signals, the motion and the activity of the image based on the image signals in units of a predetermined number of pixels, and selects a selection circuit 111 according to the determination result. 5 is used to select either the converted image signal from the high-density storage resolution conversion circuit 111 or the converted image signal from the class classification adaptive processing resolution conversion circuit 112 in units of a predetermined number of pixels. A selection control signal for performing selection control is generated. In this example, it is determined which conversion image signal to select for each pixel, and the determination output is supplied to the selection circuit 115 as a selection control signal.

[Configuration example of high-density storage resolution conversion circuit]

FIG. 17 shows a configuration example of the high-density storage resolution conversion circuit 111 used in this embodiment. This high-density storage resolution conversion circuit 1 1 1 is effective for resolution conversion of an image having a static or full-screen simple pan / tilt movement, excluding scene change zoom.

The high-density storage resolution conversion circuit 111 includes a frame memory 110 as shown in FIG. This frame memory 110 stores each pixel value of an image signal of one frame having a resolution equivalent to an HD image (see FIG. 16).

The SD input image signal is first supplied to the linear interpolation unit 211. The linear interpolation unit 211 generates an image signal having the number of pixels equivalent to the HD image from the SD input image signal by linear interpolation, and outputs the generated image signal to the motion vector detection unit 212. The processing in the linear interpolation unit 211 is performed in order to perform matching with the same image size when detecting a motion vector between an SD input image and an HD equivalent image in the frame memory 210. It is.

The motion vector detection unit 2 1 2 outputs the output image of the linear interpolation unit 2 1 1 The motion vector is detected between the image stored in the frame memory 210 and the image equivalent to the HD image. As a method of detecting a motion vector, for example, representative point matching on the entire screen is performed. In this case, the accuracy of the detected motion vector is one pixel unit in an image equivalent to HD. That is, the input image signal of the SD image has an accuracy of one pixel or less.

The motion vector detected by the motion vector detector 2 12 is supplied to the phase shifter 2 13. The phase shift unit 211 shifts the phase of the SD input image signal in accordance with the motion vector supplied thereto, and supplies it to the image storage processing unit 214. The image storage processing section 214 stores the image signal stored in the frame memory 210 and the SD input image signal that has been phase-shifted by the phase shift section 213. The signal stored in the frame memory 210 is rewritten by the signal.

FIGS. 18 and 19 show conceptual diagrams of the processing in the image accumulation processing section 2 14. FIGS. 18 and 19 show the accumulation processing only in the vertical direction for the sake of simplicity, but the accumulation processing is similarly performed in the horizontal direction.

FIGS. 18A and 19A show SD input image signals. In FIG. 19, black circles indicate pixels actually existing on the SD image, and white circles indicate non-existent pixels. In the example shown in Fig. 19, the motion vector detector 2 12 detects 3 pixels of motion in the vertical direction in an HD-equivalent image, so the phase shifter 2 13 detects the SD input image signal. The phase shift of the three pixels in the vertical direction is shown. In this case, the accuracy of the detected motion vector is one pixel equivalent to HD as described above, and the pixel position in the SD input image signal after the phase shift is shown in FIG. 19B. As described above, the one corresponding to any pixel position in the image signal equivalent to the HD image stored in the frame memory 10 It has become.

Then, in the image storage process, each pixel after the phase shift and the corresponding pixel in the image signal (FIG. 18B, FIG. 19C) corresponding to the HD image of the frame memory 110 are stored. After adding and averaging each other, the corresponding pixel of the frame memory 210 is rewritten by the added output pixel. In other words, motion compensation is performed for the motion of the SD image, and the pixels of the HD accumulated image and the pixels of the SD input image at the same position are added. In addition, regarding this addition, weighting may be performed between the HD accumulated image and the SD input image.

As a result of this image storage processing, the original SD image is shifted according to the motion vector with an accuracy of one pixel unit of the HD image, and is stored in the frame memory 210. For the SD image shown, the image stored in the frame memory 210 is an HD equivalent image as shown in FIG. 18B or FIG. 19C. FIG. 18 and FIG. 19 are explanatory diagrams of only the vertical direction, but the horizontal image is similarly converted from the SD image to the HD equivalent image.

The image signal stored in the frame memory 210 by the above-described storage processing is supplied to the output selection unit 113 as an HD output image signal as an output of the high-density storage resolution conversion circuit 111. Since the HD output image signal from the high-density storage resolution conversion circuit 111 is generated by the above-described high-density storage processing of the image in the time direction, the scene change is performed as described above. In the case of an SD input image having a static portion or a simple pan and tilt movement excluding zoom and the like, it is possible to obtain an HD output image without deterioration and without aliasing.

However, in the case of other moving parts such as a scene change part and a zoom part, one or more predetermined number of pixels described below are used. Class-adaptive resolution conversion circuit that performs SD-to-HD conversion can obtain high-quality HD output images.

[Example of configuration of class classification adaptive processing resolution conversion circuit]

Next, a class classification adaptive processing resolution conversion circuit 112 used in another embodiment will be described in detail. In the example described below, as the class classification adaptive processing, the target pixel of the SD input image signal is subjected to class classification according to the characteristics of the target pixel, and the prediction coefficients obtained by learning in advance for each class are stored in the memory. In addition, a process of outputting optimal estimated pixel values of a plurality of HD pixels corresponding to a pixel of interest by an arithmetic process according to a weighted addition formula using such prediction coefficients is described.

FIG. 20 shows an example of the overall configuration of the class classification adaptive processing resolution conversion circuit 112 used in this embodiment.

The SD input image signal to be processed is supplied to a field memory 122. This field memory 1 2 1 always stores the SD image signal one field before. Then, the SD input image signal and the SD image signal one field before stored in the field memory 221 are supplied to the first area cutout section 222 and the second area cutout section 223. You.

The first area cutout unit 222 looks at a plurality of pixels (hereinafter referred to as class taps) from the SD input image signal or the SD image signal in order to extract the feature of the pixel of interest in the SD input image signal. Perform the following processing.

The first area cutout unit 222 supplies the extracted pixel values of the plurality of images to the feature detection unit 222. The feature detection unit 222 generates a class code representing the feature of the pixel of interest from the pixel of interest in the first area and its temporal and spatial surrounding pixels, and generates the generated class code as a coefficient ROM 2. Supply 2 to 5. As described above, since the plurality of pixels cut out by the first area cutout unit 222 are used for generating a class code, they are called class taps as described above.

The coefficient ROM 225 stores in advance a prediction coefficient determined by learning as will be described later, in class I, more specifically, along with an address associated with the class code. Then, the coefficient ROM 225 receives the class code supplied from the feature detecting unit 224 as an address, and outputs a prediction coefficient corresponding to the address.

On the other hand, the second area cutout unit 223 converts the SD input image signal and the SD image signal of the previous field stored in the field memory 221 into a pixel area for prediction (second area). A plurality of prediction pixels including the target pixel included therein are extracted, and the values of the extracted pixels are supplied to the estimation calculation unit 226.

Based on the pixel values of the plurality of prediction pixels from the second region cutout unit 223 and the prediction coefficients read from the coefficient ROM 225, the estimation calculation unit 226 uses the following equation (1). By performing the operation shown in 1), a plurality of pixel values of the HD image corresponding to the target pixel of the SD image are obtained, and a predicted HD image signal is generated. As described above, the pixel values extracted by the second area cutout unit 223 are used in weighted addition for generating a predicted HD image signal, and are therefore referred to as prediction taps. Equation (11) is similar to equation (1) in the above-described embodiment.

y = w 1 XX 1 + w 2 XX 2 + · · · + Wn X n (11) where X,, X 2, · · ·, X η are each prediction tap, and W , · · · · ·, W _n are the prediction coefficients.

Next, with reference to FIG. 11, an example of a class tap extracted by the first area cutout unit 222 will be described. In this example, it is extracted as a class tap. The plurality of pixels output are assumed to be as shown in FIG. 21.It is assumed that the field including the pixel of interest and the field before it are included.In FIG. 21, the pixels indicated by black circles are: Pixels in the n-th field (for example, odd-numbered fields) indicate pixels in the white circle, pixels in the n-th field (for example, even-numbered fields) indicate pixels, and the cluster type indicates the pixel of interest and its time. And a plurality of pixels spatially and spatially adjacent to each other.

When the pixel of interest is a pixel in the n-th field, it has a class tap structure as shown in Fig. 21A. From the n-field, the pixel of interest and the upper and lower Seven pixels, that is, a pixel and two pixels on each side of the pixel, are extracted as class taps, and six pixels spatially adjacent to the pixel of interest are extracted as class taps from the previous field. Therefore, a total of 13 pixels are cut out as clusters.

When the pixel of interest is a pixel in the n + 1st field, it has a class tap structure as shown in FIG. 21B. From the n + 1 field, the pixel of interest and the left and right 1 Three pixels with each pixel are extracted as class taps, and six pixels spatially adjacent to the pixel of interest are extracted as class taps from the previous field. Therefore, a total of nine pixels are cut out as class taps. In this example, a tap structure similar to the above-described class tap is also used for the prediction tap cut out by the second region cutout unit 27. Next, a configuration example of the feature detection unit 224 will be described. In another embodiment, a plurality of pixel value patterns that are cut out as class taps in the first area cutout 222 are characterized as the target pixel. This pixel value There are a plurality of patterns corresponding to the cluster type, and each of the pixel value patterns is regarded as one class.

The feature detection unit 224 classifies the feature of the pixel of interest into classes using the plurality of pixel values cut out as class taps by the first region cutout unit 222, and responds to the class tap in advance. It outputs a class code indicating which of a plurality of classes is assumed.

In this embodiment, the feature detection unit 224 performs ADRC (Adactive Dynamic Range Coding) on the output of the first region cutout unit 222, and uses the ADRC output as the feature of the pixel of interest. Generated as a class code that represents

FIG. 22 shows an example of the feature detecting section 224. Fig. 22 shows the generation of class code by 1-bit ADRC.

As described above, 13 or 9 pixels are supplied as class taps to the dynamic range detection circuit 122 from the first area cutout section 222. The value of each pixel is represented by, for example, 8 bits. The dynamic range detection circuit 1 2 1 detects the maximum value MAX and the minimum value MIN of the plurality of pixels as the class tap, and calculates the dynamic range DR by the operation of MAX-MIN = DR .

Then, the dynamic range detection circuit 11 outputs the calculated dynamic range DR, the minimum value MIN, and the pixel value PX of each of the plurality of input pixels as its output.

The pixel values PX of the plurality of pixels from the dynamic range detection circuit 122 are sequentially supplied to a subtraction circuit 222, and the minimum value MIN is subtracted from each pixel value PX. By removing the minimum value MIN from each pixel value Px, the normalized pixel value is supplied to the comparison circuit 123. The output (DR / 2) of the bit shift circuit 124 that reduces the dynamic range DR to 1/2 is supplied to the comparison circuit 123, and the magnitude relationship between the pixel value PX and DR / 2 is detected. . Then, as shown in FIG. 13, when the pixel value PX is larger than DR / 2, the 1-bit comparison output of the comparison circuit 12 3 is set to “、”. Otherwise, the comparison circuit 1 2 3 The comparison output of one bit is set to “0.” Then, the comparison circuit 13 parallelizes the comparison outputs of a plurality of pixels as the class taps obtained in order to make a 13-bit or 9-bit comparison output. Generates ADRC output.

The dynamic range DR is supplied to a bit number conversion circuit 125, and the number of bits is converted from 8 bits to, for example, 5 bits by quantization. Then, the dynamic range obtained by converting the number of bits and the ADRC output are supplied to the coefficient ROM 225 as a class code. Note that if multi-bit ADRC is performed instead of one bit, Of course, the feature of the target pixel can be classified in more detail.

Next, learning, that is, a process of obtaining a prediction coefficient to be stored in the coefficient RM225 will be described with reference to FIG. Here, the same reference numerals are given to the same components as those in the class classification adaptive processing resolution conversion circuit 112 of FIG.

An HD image signal (referred to as a teacher signal) used for learning is supplied to a thinning processing unit 13 1 and a normal equation adding unit 13 2. The thinning processing unit 1331 performs a thinning process on the HD image signal to generate an SD image signal (referred to as a student signal), and supplies the generated student signal to the field memory 121. As described with reference to FIG. 20, the field memory 221 stores the student signal one field earlier in time. One field of the issue is stored.

In the subsequent stage of the field memory 21, almost the same processing as described above with reference to FIG. 20 is performed. However, the class code generated by the feature detection unit 222 and the prediction tap extracted by the second region extraction unit 222 are supplied to the normal equation addition unit 132. A teacher signal is further supplied to the normal equation adding unit 1 32. The normal equation adding unit 1 3 2 performs a process of generating a normal equation in order to generate a coefficient based on these three types of inputs, and the prediction coefficient determining unit 1 3 3 performs prediction for each class code from the normal equation. Determine the coefficient. Then, the prediction coefficient determination unit 133 supplies the determined prediction coefficient to the memory 134. The memory 134 stores the supplied prediction coefficients. The prediction coefficient stored in the memory 13 and the prediction coefficient stored in the coefficient R OM 25 (FIG. 20) are the same.

The process of determining the prediction coefficients,..., W _n in the above equation (11) for each class is the same as that described with reference to equations (2) to (8) in the above-described embodiment. The description is omitted because it is similar. As described above, the class classification adaptive processing resolution conversion circuit 112 classifies the feature of the target pixel of the SD image into a class, and performs an estimation operation using a prediction coefficient prepared in advance based on the classified class. By creating a plurality of pixels of the HD image corresponding to the pixel of interest.

Therefore, it is possible to accurately select a prediction coefficient corresponding to the feature of the target pixel of the SD image. By performing an estimation operation using such a prediction coefficient, a plurality of HD images corresponding to the target pixel can be obtained. Pixels can be created well. Then, even if there is movement, a converted image signal with little deterioration can be obtained.

Thus, the class classification adaptive processing resolution conversion circuit 1 1 2 A converted image signal with little deterioration can be obtained without depending on the stillness and motion of the image.However, long frames are required for the complete still part as described above and simple motion of the entire image such as pan and tilt. It is inferior to the converted image signal from the high-density storage resolution conversion circuit 111 that can store the information of the image.

In another embodiment, by taking advantage of the features of the two resolution conversion circuits 111 and 112 as described above, a resolution-converted output image signal with less deterioration is appropriately output from the output selection unit 113. To get to. That is, in the output selection section 113, the determination circuit 114 determines which resolution conversion output is to be selected, and, based on the determination output, outputs an appropriate Control to Obtain Resolution-Converted Output Image Signal Next, returning to FIG. 15, the details of the determination circuit 114 will be described, and the selection operation thereby will be described.

In the decision circuit 1 14, the converted image signal from the high-density accumulated resolution converting circuit 1 11 and the converted image signal from the class classification adaptive processing resolution converting circuit 1 1 2 are sent to the difference value calculating circuit 2 4 1 The difference value between the two is calculated. Then, the difference value is converted to an absolute value by the absolute value conversion circuit 242 and supplied to the comparison determination circuit 243.

The comparison judgment circuit 243 judges whether the absolute value of the difference value from the absolute value generation circuit 242 is larger than a predetermined value, and supplies the judgment result to the selection signal generation circuit 249. I do.

When the selection signal generation circuit 249 receives from the comparison determination circuit 243 a judgment result that the absolute value of the difference value from the absolute value conversion circuit 242 is larger than a predetermined value, Classification adaptive processing The resolution conversion image signal from the resolution conversion circuit 1 1 2 is selected by the selection circuit 1 1 5 A selection control signal is generated and supplied to the selection circuit 1 15.

This choice is made for the following reasons. That is, as described above, in the case of the high-density accumulation resolution conversion circuit 111, signal deterioration is small in still or simple pan / tilt images, but movements such as rotation and deformation, and observables in images are not affected. Degradation is seen in the image signal with respect to movement. Therefore, if the level of output pixels of the converted image signal from the high-density storage resolution conversion circuit 111 and the level of the converted image signal from the classification adaptive processing resolution conversion circuit 112 are extremely different, This is thought to be due to the deterioration of

Therefore, when the absolute value of the difference value calculated by the difference value calculation circuit 241 is larger than a predetermined threshold value, the class classification adaptive processing resolution conversion circuit 213 capable of responding to motion is also used. It is better to use the converted image signal. As can be seen from the above, the difference value calculation circuit 241, the absolute value conversion circuit 242, and the comparison judgment circuit 243 constitute a static motion judgment circuit for an image.

Next, when the comparison / judgment circuit 243 determines that the absolute value of the difference value from the absolute value generation circuit 242 is smaller than a predetermined value, the selection signal generation circuit 249 becomes As described below, the converted image signal from the high-density storage resolution conversion circuit 111 and the converted image signal from the class classification adaptive processing resolution conversion circuit 112 that have the higher activity are used. A selection control signal to be output from the selection circuit 115 is generated and supplied to the selection circuit 115. By outputting the pixel with the higher activity, it is possible to output a more active image without blurring.

Note that, in this example, a plurality of pixels before and after the target pixel of the SD image in the resolution conversion output signal equivalent to HD are used as the activity standard. Is used. .

For this reason, in the judgment circuit 114, the converted image signal from the high-density accumulation resolution converting circuit 111 and the converted image signal from the class classification adaptive processing resolution converting circuit 112 are used for the activity calculation. Are supplied to the region cutout portions 244 and 245, respectively, for cutting out the region of FIG.

The region cutout sections 2 4 4 and 2 4 5 are used to output resolution conversion signals equivalent to HD from the high-density storage resolution conversion circuit 1 11 and the class classification adaptive processing resolution conversion circuit 1 12, for example, as shown in Fig. 25B As shown by a broken line in FIG. 25C, a plurality of pixels before and after the target pixel of the SD image are cut out as pixels in the activity calculation area.

The plurality of pixels cut out as an area for the activity calculation are supplied to detectors 246 and 247, respectively, which detect a dynamic range as an activity, and each of the activities within the area (in this example, dynamic Range) is detected. Then, those detection outputs are supplied to a comparison circuit 248, the magnitudes of the two dynamic ranges are compared, and the comparison output is supplied to a selection signal generation circuit 249.

The selection signal generation circuit 24 9 is used when the judgment output of the comparison judgment circuit 24 3 indicates that the absolute value of the difference value is smaller than a predetermined threshold value. Based on the output of the above, a selection control signal for selecting and outputting a resolution conversion output having a larger dynamic range of a plurality of pixels cut out as an activity calculation area is output to the selection circuit 1. Supply 1 to 5.

The operation of the decision circuit 114 and the selection circuit 115 will be further described with reference to the flowchart of FIG. Figure 26 The operation of the flowchart corresponds to a case where the determination circuit 114 is realized by software processing. In the following description, an example will be described in which an appropriate one of the output of the high-density accumulation resolution conversion circuit 111 and the output of the class classification adaptive processing resolution conversion circuit 112 is selected for each pixel.

First, a difference value between the two pixels is calculated (step S101), and it is determined whether or not the absolute value of the difference value is larger than a threshold value (step S102). The converted output image signal from the adaptive processing resolution conversion circuit 112 is selected and output (step S107).

Next, when the absolute value of the difference value is small, the two activities (in this example, the dynamic range) are calculated in units of the activity calculation area described above (steps S103 and S104), and the calculated values are calculated. The two activities are compared (step S105), and the pixel with the larger activity is output (steps S106, S108). As a result, an image without blurring with higher activity is selected and output. In addition, as a criterion of the activity, in the above example, the dynamic range in a specific area surrounded by a dotted line as shown in FIG. 25 is used, but the present invention is not limited to this. Alternatively, for example, the variance in a specific area, the sum of absolute differences between a target pixel and pixels on both sides thereof, or the like can be used.

Further, in the above description of the selection processing, the case where selection is performed in pixel units has been described. However, the selection is not limited to selection in pixel units, and may be performed in block units, object units, frame units, or the like. Good.

Also, in the above example, the output of one high-density storage resolution conversion circuit and the output of one class classification adaptive processing resolution conversion circuit have been selected as an alternative. Classification adaptive processing It is also possible to provide a plurality of resolution conversion circuits and select an output image signal from them.

Furthermore, the class taps and prediction taps in the first area cutout section 222 and the second area cutout section 223 in the description of the classification adaptive processing are merely examples, and it goes without saying that they are not limited thereto. No. Also, in the above description, the structure of the class tap and the prediction tap are the same, but they need not be the same structure.

Further, in the above-described embodiment, the conversion from the SD image to the HD image has been described as an example. In addition, the classification adaptive processing and the high-density accumulation are not limited to the above-described embodiments.

In another embodiment of the present invention, the processing is not limited to being performed by hardware, but may be performed by software. The processing by software will be described below. FIG. 27 is a flowchart showing the flow of the resolution conversion processing of one embodiment. As shown in steps S111 and S112, the resolution conversion processing by the class classification adaptive processing and the resolution conversion processing by the high-density accumulation processing are performed in parallel. The output obtained in each process is processed by the output determination process (step S113). Then, in step S114, an output is selected according to the determination result in step S113. Thus, the processing for one pixel is completed.

FIG. 28 is a flowchart showing details of the resolution conversion processing S112 by the high-density accumulation processing. In the first step S1221, an initial input frame image is linearly interpolated to form an image having the number of pixels of HD. The image after this interpolation is stored in the frame memory (step S122). In step S 1 2 3, similarly linear for the next frame Interpolation is performed. Then, in step S14, a motion vector is detected using the two-frame image obtained by the linear interpolation.

In step S125, the input SD image is phase-shifted by the detected motion vector. The phase-shifted image undergoes an image accumulation process (step S126). In the next step S127, the accumulation result is stored in the frame memory. Then, an image is output from the frame memory (step S128).

FIG. 29 is a flowchart showing details of the resolution conversion processing S111 by the class classification adaptive processing. In the first step S131, a first region is cut out. That is, class taps are extracted. In step S132, feature detection is performed on the extracted class tap. Among the coefficients obtained by the learning processing in advance, the coefficient corresponding to the detected feature is read out (step S133). In step S134, the second region (predicted tap) is cut out. In step S135, an estimation operation is performed using the coefficients and the prediction taps, and an up-converted output (HD image) is obtained.

FIG. 30 is a flowchart showing a flow of a learning process for obtaining a coefficient used for a resolution conversion process by the class classification adaptation. In step S141, a high-resolution HD signal (teacher signal) is thinned out to generate a student signal. Regarding the student signal, in step S142, the first area (class tap) is cut out. Feature detection is performed based on the extracted class taps (step S144). Next, in step S144, a second region (prediction tap) is cut out. Then, in step S145, based on the teacher image signal, the data of the prediction taps, and the detected features, data necessary for solving a normal equation in which the prediction coefficient is solved is calculated. Then, in step S146, it is determined whether the addition of the normal equations has been completed. If not, the process returns to step S142 (first region cutout process). When it is determined that the processing has been completed, in step S147, a prediction coefficient is determined. The obtained prediction coefficients are stored in the memory and used in the resolution conversion processing.

As described above, according to another embodiment of the present invention, a high-density storage structure capable of handling information in the time direction for a long time and a result of the classification adaptive processing can be selected for each pixel. Images can be output.

The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

Claims

The scope of the claims

1. An image processing apparatus that receives an input image signal and generates an output image signal with higher quality than the input image signal

A storage unit for storing an image signal having the same quality as the output image signal, and adding the input image signal and the image stored in the storage unit to obtain a first image having higher quality than the input image. A first signal processing unit that generates an image signal and stores the first image signal in the storage unit; and a feature based on the input image signal according to a pixel position of interest in the output image signal. Extracting, classifying the pixel of interest into one of a plurality of classes according to the characteristics, and calculating the input image signal by a predetermined calculation method corresponding to the classified class, First signal processing means for generating a first image signal of higher quality than the input image;

Output selection means for performing a determination based on the first image signal and the second image signal, and selecting one of the first and second image signals as an output image signal;

An image processing apparatus having:

2. In Claim 1,

The first signal processing means,

An image processing apparatus for generating the first image signal by accumulating a large number of temporally continuous image signals.

3. In Claim 1,

The second signal processing means,

First extraction means for extracting first pixel data from the input image signal according to a pixel position of interest in the second image signal;

Feature detecting means for detecting a feature based on the first pixel data, and classifying the noted pixel into one of a plurality of classes according to the feature; Second extraction means for extracting a second pixel data from the input image signal according to the target pixel position;

Storage means for storing method information for specifying a method of generating the pixel data at the target pixel position using the second pixel data for each class;

A pixel generation unit configured to generate data of the target pixel position based on the method information and the second pixel data;

An image processing apparatus having:

4. In Claim 1,

An image processing apparatus that generates determination information based on the first image signal and the second image signal, and controls the output selection unit based on the determination information.

5. In Claim 1,

An image processing apparatus, wherein the output image signal has a smaller noise component than the input image signal.

6. In Claim 5,

The first signal processing means,

The image signal stored in the storage means and the input image signal are added by performing weighting according to the static movement of the image based on the input image signal, and the added output rewrites the image signal of the storage means. As a result, the first image signal from which noise has been removed is generated as the addition output, and the second signal processing means includes:

The corresponding pixels on the image are extracted among a plurality of frames, the noise component of the pixels based on the change between the frames of the pixels is classified as the feature, and the noise component is set in advance according to the classified class. The second image from which the noise component has been removed from the input image signal Generate an image signal,

The output selection means determines a static motion of an image based on the first and second image signals in a predetermined number of pixels, and according to the determination result, the predetermined number of pixels An image processing device for selecting and outputting one of the first image signal and the second image signal.

7. In Claim 6,

The output selection unit,

A determining unit that determines whether the predetermined number of pixels is a still portion or a moving portion of the image;

Based on the determination result of the determination unit, a selection is made to select and output the above-described first image signal for pixels in a stationary portion, and to select and output the above-described second image signal for pixels in a moving portion. Department and

An image processing apparatus having:

8. In Claim 7,

The determination unit is

A difference value calculation unit that calculates a difference value between the first image signal and the second image signal for each of the predetermined number of pixels;

If the absolute value of the difference value is equal to or greater than the threshold value based on the comparison result between the absolute value of the difference value and a preset threshold value, it indicates that the pixel is the moving part. A comparing unit that outputs a judgment value and outputs a judgment value indicating that the pixel is a stationary part when the absolute value of the difference value is smaller than the threshold value;

An image processing apparatus having:

9. In Claim 6,

The first signal processing means,

A motion determining unit that determines whether the image is still or not based on the input image signal, A weighting unit that weights the input image signal and the image signal stored in the storage unit in accordance with the still / movement determination in the motion determination unit.

An adding unit that adds the weighted input image signal and the image signal from the storage unit.

Has,

An image processing device wherein the image signal of the storage means is rewritten with the image signal from the adding unit.

10. In Claim 6,

The second signal processing means includes:

A motion information detector for detecting motion information about a pixel of interest in the image based on the input image signal;

A class tap extracting unit for extracting, as a cluster, a plurality of pixels at positions corresponding to the target pixel for a plurality of frames using the motion information detected by the motion information detecting unit;

A class classification unit that classifies the noise component of the pixel of interest based on the characteristics of the class tap extracted by the class tap extraction unit;

An arithmetic processing unit that determines an arithmetic processing corresponding to the class based on the class classified by the class classification unit, and generates an image signal from which the noise component of the pixel of interest is removed by the determined arithmetic processing. When,

An image processing apparatus having:

1 1. In Claim 10,

The feature of the class tap used in the class classification unit is that the image processing device is a noise component distribution of the plurality of pixels as the class tap.

1 2. In Claim 10,

In the arithmetic processing unit,

By calculating the pixel values of a plurality of pixels at the position corresponding to the target pixel and calculation coefficients for the plurality of pixels set in advance according to the class classified by the class classification unit, An image processing apparatus for generating an image signal from which the noise component of the pixel of interest has been removed.

1 3. In Claim 10,

The calculation coefficient used in the calculation processing unit is a prediction coefficient obtained in advance,

Extracting a pixel of interest from the teacher image data having less noise than the input image signal;

Detecting motion information about the pixel of interest from student image data having noise equivalent to the input image signal;

Extracting, as a class tap, a plurality of pixels at positions corresponding to the pixel of interest from the student image data of the plurality of frames according to the motion information detected for the pixel of interest;

Classifying the noise component of the pixel of interest based on the characteristics of the class tap;

Deriving a prediction coefficient for generating, from the student signal, an output signal having the same quality as the pixel corresponding to the pixel of interest extracted from the teacher signal for each of the classes.

An image processing device obtained as the above prediction coefficient.

1 4. In Claim 1,

The output image signal has a higher resolution than the input image signal. An image processing apparatus characterized by the above-mentioned.

1 5. In Claims 14,

The first signal processing means,

The input image signal is stored in the storage unit while correcting the pixel position by referring to the movement between the image based on the image signal stored in the storage unit and the image based on the input image signal. By accumulating, the storage means generates the high-resolution first image signal,

The second signal processing means includes:

The feature is detected based on a plurality of pixels including a target pixel and its temporal and spatial surrounding pixels, and the high-resolution second image signal is generated by classifying the feature based on the feature. Image processing device.

1 6. In Claim 15,

The output selection unit,

A determining unit that determines the image motion and the activity based on the first and second image signals in units of a predetermined number of pixels,

A selection unit that selects one of the first and second image signals in units of a predetermined number of pixels according to a determination result of the determination unit;

An image processing apparatus comprising:

1 7. In claim 16,

The determination unit is

When the absolute value of the difference value is equal to or greater than the threshold value based on the result of comparison between the absolute value of the difference value and a preset threshold value, the predetermined number of pixel portions is a motion portion. And outputting a determination value indicating that the absolute value of the difference value is smaller than the threshold value. A comparison unit that outputs a judgment value indicating that the minute is a stationary part;

An image processing apparatus having:

1 8. In Claim 16,

The determination unit is

A static motion determining unit that determines the static motion of the predetermined number of pixels 、,

When the static / movement determining unit determines that the predetermined number of pixel portions is a moving portion, a signal for selecting and outputting the second image signal is supplied to the selecting unit. Selection signal generator

An image processing apparatus having:

1 9. In claim 16,

The determination unit is

A static motion determining unit that determines static motion for each of the predetermined number of pixels,

An activity determining unit that determines which of the first image signal and the second image signal has higher activity;

When the predetermined number of pixel portions is determined to be a static portion by the still / movement determining unit, based on the determination result by the activity determining unit, the first and second image signals A selection signal generation unit that supplies a signal for selecting and outputting a higher activity of the image to the selection unit;

An image processing apparatus having:

20. In claim 19,

The activity determination unit includes:

For each of the first and second image signals, the dynamic range of the pixel values of a plurality of pixels within a predetermined area is calculated, and the calculated two dynamic ranges are compared to determine the level of activity. Image processing device to determine.

2 1. In claim 15,

The first signal processing means includes: (a motion detection unit that detects a motion between an image based on the image signal stored in the storage unit and an image based on the input image signal;

An image accumulation processing unit that corrects a pixel position based on the motion detected by the motion detection unit, adds the input image signal to an image signal stored in the storage unit, and accumulates the image signal;

An image processing apparatus comprising:

2 1. In claim 15,

The second signal processing means,

A cluster tap extraction unit that extracts a plurality of pixels including a target pixel in an image based on the input image signal and temporally and spatially surrounding pixels of the target pixel as a class tap,

A class classifying unit for classifying based on the characteristics of the cluster map extracted by the cluster map extracting unit;

An image conversion operation corresponding to the class is determined based on the class classified by the class classification unit, and a plurality of pixels in the high-resolution image corresponding to the pixel of interest are generated by the determined operation. By doing so, an arithmetic processing unit that generates the second image signal

An image processing apparatus having:

2 3. In Claims 1 and 2,

The class classification unit,

An image processing apparatus characterized in that the features of the cluster tap are classified into classes by the pixel value pattern of the plurality of pixels as the class tap.

2 4 · In Claim 2, In the arithmetic processing unit,

Calculation of a plurality of pixels in a predetermined region of the input image signal corresponding to the cluster map and an operation of the plurality of pixels set in advance according to the class classified by the class classification unit An image processing apparatus that generates a plurality of pixels in the high-resolution image corresponding to the target pixel by performing an operation with a coefficient.

2 5. In claim 14,

Extracting a plurality of pixels corresponding to the pixel of interest from a teacher signal having the same quality as the output image signal;

Extracting a plurality of pixels including the target pixel and its temporal and spatial surrounding pixels as class taps from a student signal having the same quality as the input image signal;

Classifying the feature of the pixel of interest based on the feature of the class tap;

A step of deriving, for each of the classes, a prediction coefficient for generating an output signal of the same quality as the pixel corresponding to the pixel of interest extracted from the teacher signal from the student signal;

An image processing device obtained as the above prediction coefficient.

26. In an image processing method for receiving an input image signal and generating an output image signal with higher quality than the input plane image signal,

An image signal of the same quality as the output image signal is stored in a storage unit, and the input image signal and the stored image are added to generate a first image signal of higher quality than the input image. A first signal processing step of storing one image signal in the storage means; According to the position of the pixel of interest in the output image signal, a feature based on the input image signal is extracted, and the pixel of interest is classified into one of a plurality of classes according to the feature. A first signal processing step of generating a second image signal of higher quality than the input image by calculating the input image signal by a predetermined arithmetic method;

An output selection step of making a determination based on the first image signal and the second image signal, and selecting one of the first and second image signals as an output image signal;

An image processing method comprising:

2 7. In Claim 26,

The first signal processing step is

An image processing method for generating the first image signal by accumulating a large number of temporally continuous image signals.

2 8. In Claim 26,

The second signal processing step includes:

A first extraction step of extracting first pixel data from the input image signal according to a pixel position of interest in the second image signal;

A feature detecting step of detecting a feature based on the first pixel data and classifying the noted pixel into one of a plurality of classes according to the feature; A second extraction step of extracting pixel data of

A storage step for storing method information for specifying a method of generating the pixel data at the pixel position of interest using the second pixel data for each class;

A pixel generating step of generating data of the target pixel position based on the method information and the second pixel data; An image processing method comprising:

2 9. In Claim 6,

An image processing method for generating determination information based on the first image signal and the second image signal, and selecting one of the first and second image signals as an output based on the determination information.

30. In claim 16,

An image processing method, wherein the output image signal has less noise components than the input image signal.

3 1. In Claim 30,

In the first signal processing step, the image signal stored in the storage unit and the input image signal are added by performing weighting according to the static movement of the image by the input image signal, and the added output is obtained. By rewriting the image signal in the storage means, a first image signal from which noise has been removed is generated as the addition output,

In the second signal processing step, corresponding pixels on an image are extracted from a plurality of frames, and the noise components of the pixels based on the change between the frames of the pixels are classified into classes, and the classified pixels are classified. A second image signal from which a noise component has been removed is generated from the input image signal by an arithmetic process preset for the class,

The output selection step includes determining a static motion of an image in a predetermined number of pixels, and according to the determination result, the first image signal and the second image signal in the predetermined number of pixels. An image processing method that selects and outputs one of the following.

3 2. In Claim 31,

The output selection step includes:

Judgment to determine whether the predetermined number of pixels is a still portion or a moving portion of an image Steps and

Based on the determination result in the determination step, selection is made to select and output the first image signal for pixels in a stationary portion and to select and output the second image signal for pixels in a moving portion. Steps and

An image processing method comprising:

3 3. In Claim 3 2,

The above determination step includes:

A difference value calculating step of calculating a difference value between the first image signal and the second image signal for each of the predetermined number of pixels;

If the absolute value of the difference value is equal to or greater than the threshold value based on a comparison result between the absolute value of the difference value calculated in the difference value calculation step and a preset threshold value, A comparing step of outputting a judgment value indicating that the pixel is a moving part, and outputting a judgment value indicating that the pixel is a stationary part when the absolute value of the difference value is smaller than the threshold value;

An image processing method comprising:

3 4. In Claim 31,

The first signal processing step includes:

A motion determining step of determining whether the image is still or not based on the input image signal;

A weighting step of weighting the input image signal and the image signal stored in the storage means according to the still / movement determination in the motion determination step;

An adding step of adding the weighted input image signal and the image signal from the storage means.

Has, An image processing method in which the image signal in the storage means is rewritten with the pixel signal from the adding step.

3 5. In Claim 31,

The second signal processing step includes:

A motion information detection step for detecting motion information on a pixel of interest in an image based on the input image signal;

A class tap extracting step of extracting, as a class tap, a plurality of pixels at a position corresponding to the target pixel for a plurality of frames using the motion information detected in the motion information detecting step;

A class classification step of classifying a noise component of the pixel of interest based on characteristics of the class tap extracted in the class tap extraction step;

An operation processing step of determining an operation processing corresponding to the class based on the class classified in the class classification step, and generating an image signal from which the noise component of the pixel of interest has been removed by the determined operation processing When

An image processing method comprising:

3 6. In claim 35,

A feature of the cluster tap used in the class classification step is an image processing method in which a noise component distribution of the plurality of pixels as the class tap is used.

3 7. In claim 35,

In the calculation processing step,

By calculating the pixel values of a plurality of pixels at the position corresponding to the target pixel and the calculation coefficients for the plurality of pixels set in advance according to the class classified in the class classification step, Featured image above Image processing method for generating image signal from which noise component of element is removed

3 8. In Claim 3 7,

Extracting the target pixel from the teacher image data having less noise than the input image signal, wherein the operation coefficient is a prediction coefficient obtained in advance;

Detecting, from the student image data having noise equivalent to the input image signal, motion information about the pixel of interest;

Extracting a plurality of pixels at a position corresponding to the pixel of interest from the student image data of the plurality of frames as Crustal degrees according to the motion information detected for the pixel of interest;

Classifying the noise component of the pixel of interest based on the characteristics of the cluster map;

Deriving, from the student signal, a prediction coefficient for generating an output signal having the same quality as a plurality of pixels at a position corresponding to the pixel of interest extracted from the teacher signal, into the class 每.

An image processing method that calculates the above as the prediction coefficient.

3 9. In claim I 6,

An image processing method, wherein the output image signal has a higher resolution than the input image signal.

40. In Claims 39,

The first signal processing step refers to a motion between an image having the same resolution as the image of the output image signal and an image based on the input image signal, based on the image signal stored in the storage unit. By accumulating the input image signal in the storage means while correcting the pixel position, the first image signal of high resolution is generated in the storage means. The second signal processing step includes detecting the feature based on a plurality of pixels including a pixel of interest and its temporal and spatial surrounding pixels, and classifying the feature based on the feature. Generate a second image signal,

The output selection step is an image processing method for selectively outputting one of the first and second image signals.

4 1. In claim 40,

The output selection step includes:

A judging step of judging the motion and the activity of the image based on the first and first image signals in units of a predetermined number of pixels, respectively, and according to the judgment results in the judging step, the first and second A selection step of selecting one of the image signals in units of a predetermined number of pixels;

An image processing method comprising:

4 2. In Claim 41,

The above determination step includes:

A difference value calculating step of calculating a difference value between the first and second image signals for each of the predetermined number of pixels;

When the absolute value of the difference value is equal to or greater than the threshold value based on the result of comparison between the absolute value of the difference value and a preset threshold value, the predetermined number of pixel portions is a motion portion. And a comparison step of outputting a judgment value indicating that the predetermined number of pixel portions is a stationary portion when the absolute value of the difference value is smaller than the threshold value. An image processing method comprising:

4 3. In Claim 41, The above determination step includes:

A static motion determining step of determining static motion for each of the predetermined number of pixels; and when the static motion determining unit determines that the predetermined number of pixel portions is a moving portion, A selection signal generating step of supplying a signal for selecting and outputting the image signal of (2);

An image processing method comprising:

4 4. In Claim 41,

The above determination step includes:

A static motion determining step of determining static motion for each of the predetermined number of pixels; and an activity determining step of determining which of the first and second image signals has a higher image activity.

When the predetermined number of pixel portions is determined to be a static portion in the static motion determination step, based on a determination result in the activity determination step, the first and second image signals of the first and second image signals are determined. A selection signal generation step of supplying a signal for selecting and outputting a higher image activity to the selection unit;

An image processing method comprising:

4 5. In Claim 4 4,

The above activity determination step includes:

For each of the first and second image signals, the dynamic range of the pixel values of a plurality of pixels within a predetermined area is calculated, and the two dynamic ranges are compared to determine the level of activity. Image processing method.

4 6. In Claim 40,

The first signal processing step includes:

An image based on the image signal stored in the storage means; A motion detection step for detecting a motion between the image signal and the image, and a pixel position corrected by the motion detected in the motion detection step, and the input image signal is stored in the storage means. An image accumulation processing step of accumulating by adding to

An image processing method comprising:

4 7. In claim 40,

The second signal processing step includes:

A class tap extracting step of extracting, as class taps, a plurality of pixels including a pixel of interest in an image based on the input image signal and temporally and spatially surrounding pixels thereof;

A class classification step of classifying according to the characteristics of the class tap extracted in the class tap extraction step;

An image conversion operation process corresponding to the class is determined based on the class classified by the class classification step, and the operation process step of generating the high-resolution image signal is performed by the determined operation process. Image processing method.

4 8. In Claim 4 7,

The above classifying step includes:

An image processing method for classifying a feature of the class tap according to a pattern of pixel values of the plurality of pixels as the class tap. 4 9. In Claims 4 7,

In the calculation processing step,

A plurality of pixels in a predetermined region of the input image signal corresponding to the class tap and an operation on the plurality of pixels preset according to the class classified by the class classification unit By calculating with a coefficient, the output of the pixel of interest is output An image processing method for generating an image signal.

5 0. In Claims 4 7,

The calculation coefficient used in the calculation processing step is a prediction coefficient obtained in advance,

Extracting a plurality of pixels at positions corresponding to the target pixel as class taps from a student signal having the same quality as the input image signal; and classifying the characteristic of the target pixel based on the characteristic of the cluster tap. Steps and

An image processing method for calculating as the above prediction coefficient.

51. In a program for causing a computer to execute image processing for generating an output image signal higher in quality than the input image signal, the image signal having the same quality as the output image signal is stored in a storage unit, and A first signal processing step of generating a higher quality first image signal than the input image by adding the signal and the stored image, and storing the first image signal in the storage unit; When,

The feature based on the input image signal is extracted according to the position of the pixel of interest in the output image signal, and the pixel of interest is classified into one of a plurality of classes according to the characteristic, and the classified pixel corresponds to the class. A first signal processing step of generating the first image signal of higher quality than the input image by calculating the input image signal by a predetermined arithmetic method;

A determination is made based on the first image signal and the second image signal.