JP2000278699A - Image pickup unit and image pickup method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To eliminate a noise in a moving picture caused by deficiency in a luminous quantity in the case of image pickup or the like. SOLUTION: Imaging light is made incident onto a CCD 2 via a lens 1. The CCD 2 of this image pickup unit converts the incident imaging light into an electric signal. A classification adaptive processing section 5 receives a picture signal generated on the basis of the electric signal. The classification adaptive processing section 5 classifies the picture signal on the basis of a class tap segmented from the received picture signal and applies an arithmetic operation of linear combination to the picture signal by using a prediction coefficient decided in advance for each class and a prediction tap segmented separated from the picture signal. The prediction coefficient is decided for each class by using the picture signal not including a noise as a teacher signal yk and using a signal adding the noise to the teacher signal yk as a pupil signal. Through the processing above, the noise can effectively be eliminated even from a moving picture.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image pickup apparatus such as a video camera and an image pickup method.

[0002]

2. Description of the Related Art In a conventional video camera, a recursive filter including a frame memory is used to remove random noise caused by a shortage of light quantity at the time of image pickup. An example of a video camera having a recursive filter will be described with reference to FIG. The imaging light is the lens 1
The light is incident on the CCD 102 via the line 01. CCD102
Converts incoming imaging light into electrical signals, samples and holds the electrical signals, and AGC (Auto Gain Controller).
The signal is supplied to the circuit 103. Sample hold and AGC
The circuit 103 samples and holds the electric signal output from the CCD 102, and converts the signal obtained thereby into an A signal.
By inputting the signal to the GC amplifier, the signal is controlled so as to maintain an appropriate value. Sample hold and AGC circuit 1
The output of 03 is supplied to the processing unit 104.

[0005] The synchronization signal generation unit 107 generates a horizontal / vertical synchronization signal, and supplies the generated horizontal / vertical synchronization signal to the process unit 104 and the timing generation unit 108. The timing generator 108 generates various timing pulses for driving the CCD 102 based on the supplied synchronization signal. Further, the processing unit 104 performs Y / Y based on the output of the sample hold and AGC circuit 103.
A C signal is generated, and the generated Y / C signal is supplied to the recursive filter 106. The recursive filter 106 removes, from the supplied signal, random noise generated due to a shortage of light amount during imaging.

[0004] An example of the configuration of the recursive filter 106 is shown in FIG.
Shown in The input signal from the A / D converter 105 is the multiplier 1
21. The multiplier 121 multiplies the supplied signal by (1−k) and supplies the result of the multiplication to the adder 122. The adder 122 adds the signal supplied from the multiplier 121 and the signal supplied from the multiplier 124, outputs the addition result to the outside as an output signal, and supplies the result to the frame memory 123. The frame memory 123 supplies the supplied signal to the multiplier 124 after holding the supplied signal for a period corresponding to, for example, one frame. The multiplier 121 is
The supplied signal is multiplied by k, and the multiplication result is added to an adder 122.
To supply.

Here, the value of the variable k may be set appropriately to remove noise. For the still image part, k
Is set to a value close to 1, it is possible to remove, for example, random noise caused by insufficient light quantity at the time of imaging.

[0006]

However, if the value of the variable k is set to a large value for a moving image portion, addition is performed between images temporally separated by, for example, one frame, and the entire image is blurred. There is a possibility that it will. Also, if the value of the variable k is set to a small value (around 0), the input signal is used as it is as an output signal, and random noise generated due to insufficient light quantity at the time of imaging cannot be removed.

It is therefore an object of the present invention to provide an image pickup apparatus and an image pickup method capable of removing noise caused by a shortage of light quantity at the time of image pickup even in a moving image portion.

[0008]

According to a first aspect of the present invention, in an image pickup apparatus, an image pickup device for converting input light into an electric signal, a signal processing unit for generating an image signal based on an output of the image pickup device, An imaging apparatus comprising: a classification adaptive processing unit that performs a classification adaptive processing as a processing for removing noise based on a signal.

According to a fifth aspect of the present invention, in the imaging method, an imaging step of converting input light into an electric signal, a signal processing step of generating an image signal based on an output of the imaging device, and a method of reducing noise based on the image signal. A classification adaptive processing step of performing a classification adaptive processing as a removal processing.

According to the invention described above, by performing the classification adaptive processing, noise generated due to, for example, the fact that the light input to the image pickup device does not have a sufficient light amount with respect to the sensitivity of the image pickup device, etc. Can be removed.

[0011]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A configuration of an embodiment of the present invention in which the present invention is applied to a video camera will be described with reference to FIG. The imaging light is transmitted through a CCD 2 through a lens 1.
Incident on. The CCD 2 converts the incident imaging light into an electric signal, and samples and holds the electric signal and an AGC (Aut).
o Gain Controller) Supply to circuit 3. The sample and hold and AGC circuit 3 samples and holds the electric signal output from the CCD 2 and inputs the signal obtained by this to an AGC amplifier, thereby controlling the signal to keep an appropriate value. Sample hold and AG
The output of the C circuit 3 is supplied to the processing unit 4.

The synchronizing signal generator 7 generates horizontal and vertical synchronizing signals, and outputs the generated horizontal and vertical synchronizing signals to the processing unit 4.
And the timing generator 8. The timing generator 8 generates various timing pulses for driving the CCD 2 based on the supplied synchronization signal. Also,
The process unit 4 includes a sample hold and AGC circuit 3
A Y / C signal is generated based on the output of
The signal is supplied to the A / D converter 5. The A / D converter 5
The supplied signal is A / D converted and supplied to the classification adaptive processing unit 6. The classification adaptive processing unit 6 removes noise by performing the classification adaptive processing on the supplied signal.

Here, the classification adaptive processing will be described. First, as shown in FIG. 2A, the classification will be described by taking a 2 × 2 pixel block (class classification block) composed of a pixel of interest and three pixels adjacent thereto as an example. When the four pixels in such a block are each represented by one bit, that is, when two levels of '1' or '0' are taken, such a block of 2 × 2 pixels has (2 ¹ ) ⁴ = It can be classified into 16 patterns (see FIG. 2B). Such a pattern classification is a class classification.

When such a classification is performed on an actual image signal, there is a problem that the number of classes involved in the classification becomes enormous. That is, as a practical example, considering that a case where 8 bits are assigned to one pixel as pixel data and the class classification block is composed of 3 × 3 = 9 pixels, the number of classes is as large as 8 ⁹ = 2 ^27. It becomes something. In order to handle such a class number, a large-scale circuit configuration is required.

Therefore, in the actual class classification adaptive processing, ADRC (Adaptive Dynamic Range) is added to the pixel data.
Coding) process to generate a code value with a reduced number of bits compared to the original pixel data, and classify using this code value to reduce the number of classes and reduce the number of classes that can be handled easily Is performed. ADRC will be described using a class classification block including four pixels as shown in FIG. 3A as an example. First, the maximum value MAX and the minimum value MIN of the pixel values are detected for these four pixels. Then, DR = MAX−MIN is calculated by these 4
A local dynamic range of a class classification block composed of pixels is defined as a dynamic range DR.
, A process of requantizing the pixel values of the pixels constituting the classification block into K bits.

[0016] That is, the minimum value MIN is subtracted from the pixel values of the pixels constituting the class classifying block, dividing the subtracted value by DR / 2 ^K. Then, a code (ADRC code) corresponding to the division value is generated. For example, when K = 2, as shown in FIG.
It is determined which range is obtained by dividing R by 2 ² = 4, and the divided value is in the range of the lowest level, the range of the second level from the bottom, and the range of the third level from the bottom. , And if they belong to the highest level range, they are coded into 2-bit ADRC codes such as 00B, 01B, 10B, and 11B, respectively. Here, "B" indicates a binary number. ADRC for each classifying block generated in this way
The image signal can be transmitted by transmitting the code together with the dynamic range DR, the minimum value MIN, and the like.

In this case, on the decoding side, the ADRC code 0
0B, 01B, 10B, and 11B, respectively, the center value of the lowermost level range obtained by equally divided into four, the dynamic range DR L _00, the center value L ₀₁ of the second level in the range from below, from the bottom The pixel value is converted into a center value L ₁₀ of the third level range and a center value L ₁₁ of the highest level range, and the minimum value MIN is added to the converted value to decode the pixel value. Here, such ADRC processing is called non-edge matching.

Here, the case where the pixel values are classified into classes has been described. However, in the classification adaptive processing, data other than pixel values, for example, DPCM (prediction coding), BTC (Bl
ockTruncation Coding), VQ (vector quantization), D
It can also be performed on data on which CT (discrete cosine transform), Hadamard transform, or the like has been performed.

In addition, the class classification adaptive processing utilizes the fact that an image pattern in a class classification block can be expressed with a relatively small amount of information using an ADRC code in addition to the transmission of an image signal. It is also possible to apply to processing for the purpose of improvement or the like. In this case, for example, between an image signal (noted as a teacher signal) having a property matching the purpose of the processing, such as not including noise, and an image signal to be processed (noted as a student signal) Is determined in advance for each class represented by the ADRC code. Then, a pixel at a predetermined position (denoted as a prediction tap) in the student signal,
An operation as shown in the following expression (1) is performed between the prediction coefficient and the prediction coefficient, and pixel data subjected to processing such as improvement of image quality is obtained as an aggregate of pixel values y as operation results.

Y = w ₁ × x ₁ + w ₂ × x ₂ + ‥‥ + w _n × x _n (1) where x ₁ , ‥‥, x _n are each prediction tap,
w ₁ , ‥‥, and w _n are respective prediction coefficients.

According to the present invention, random noise generated due to, for example, a shortage of light quantity at the time of imaging is removed by performing the classification adaptive processing. Classification adaptive processing unit 16 in one embodiment of the present invention
Will be described with reference to FIG. Process part 1
The image signals supplied from the area extraction units 11 and 12
Supplied to The area cutout unit 11 cuts out class taps and outputs the cutout class taps to the feature amount extraction unit 13.
To supply. The feature amount extraction unit 13 is necessary for generating a class code such as a maximum value MAX, a minimum value MIN, and a dynamic range DR of pixel values of pixels (hereinafter, referred to as class taps) constituting a supplied class classification block. The characteristic amount is extracted, and the extracted characteristic amount is supplied to the class code generating unit 14 together with the pixel data in the class classification block.

The class code generator 14 generates a class code for each class classification block by performing the above-described operation including ADRC based on the supplied feature amount, and stores the generated class code in a memory. 15 The memory 15 stores a prediction coefficient for each predetermined class, that is, a prediction coefficient corresponding to a class code. These prediction coefficients are stored, for example, using a class code as an address, and the memory 15 outputs the prediction coefficient according to the class code supplied from the class code generation unit 14. This prediction coefficient is supplied to the estimation calculation unit 16.

On the other hand, the region cutout unit 12 cuts out prediction taps from the supplied image signal, and supplies the cutout prediction taps to the estimation calculation unit 16. The estimation calculation unit 16
Based on the prediction coefficient supplied from the memory 15 and the prediction tap supplied from the region cutout unit 12, the calculation as in the above-described equation (1) is performed. Pixel value y as operation result
, An image signal from which random noise generated due to a shortage of light quantity at the time of imaging is removed is obtained.

Next, learning, that is, a process of calculating a prediction coefficient will be described. The learning generally includes a teacher signal y _k (k = 1, 2, ‥‥, n) as an image signal to be generated by the classification adaptive processing and an image signal to be subjected to the classification adaptive processing. This is performed based on the student signal. That is, based on the prediction taps x _k1 , x _k2 } cut out from the student signal and the teacher signal y _k , the prediction coefficient w ₁ , ‥‥, w _n are determined.

Y _k = w ₁ × x _k1 + w ₂ × x _k2 + ‥‥ + w _n × x _kn (2) (k = 1, 2, ‥‥, m) A specific calculation method will be described. When the number m of types of teacher signals is larger than the total number n of prediction coefficients, the prediction coefficients w ₁ ,
Since ‥‥ and w _n are not uniquely determined, the element e _k of the error vector e is defined by the following equation (3), and the prediction coefficient is set so as to minimize the error vector e defined by the equation (4). To be determined. That is, the prediction coefficient is uniquely determined by the so-called least square method.

E _k = y _k − {w ₁ × x _k1 + w ₂ × x _k2 + Δ + w _n × x _kn } (3) (k = 1, 2, Δm)

[0027]

(Equation 1)

As a practical calculation method for obtaining the prediction coefficient minimizing e ² in Equation (4), partial differentiation of e ² with the prediction coefficient w _i (i = 1, 2 ‥‥) is performed (Equation 4). (5)) Each prediction coefficient w _i may be determined so that the partial differential value becomes 0 for each value of _i .

[0029]

(Equation 2)

The procedure for determining each prediction coefficient w _i from equation (5) will be described. X as in equations (6) and (7)
_{When ji} and Y _i are defined, equation (5) can be written in the form of a determinant of equation (8).

[0031]

(Equation 3)

[0032]

(Equation 4)

[0033]

(Equation 5)

Equation (8) is generally called a normal equation. Equation (8) can be solved by a general matrix solution such as a sweeping-out method.

A configuration related to learning in one embodiment of the present invention will be described with reference to FIG. Here, a description will be given of an example of learning related to the classification adaptive processing in order to remove random noise generated due to a shortage of light amount at the time of imaging. In this case, using image signals including no noise as the teacher signal y _k, also using a signal obtained by adding a random noise caused by the shortage of the light amount at the time of imaging the teacher signal y _k as a student signal. The teacher signal is the noise adding unit 2
1. Area cutout unit 22 and normal equation addition unit 26
Supplied to

The noise adding unit 21 converts the supplied signal into
A student signal is generated by adding a noise pattern corresponding to random noise generated due to a shortage of light quantity at the time of imaging. The noise adding unit 21 can be configured using, for example, a low-pass filter or the like. The student signal generated by the noise adding unit 21 is supplied to the area cutout unit 23. The region cutout unit 23 cuts out class taps from the supplied student signal, and supplies the cutout class taps to the feature amount extraction unit 24.

The feature value extracting unit 24 determines the maximum value MAX and the minimum value MI of the pixel values based on the supplied class taps.
The feature amount such as N and dynamic range DR is extracted, and the extracted feature amount is supplied to the class code generating unit 25 together with the pixel data of the class tap. Class code generator 25
Generates a class code based on the output of the feature extracting unit 24 and combines the generated class code with the normal equation adding unit 2.
6

On the other hand, the region cutout unit 22 cuts out prediction taps from the supplied teacher signal, and supplies the cutout prediction taps to the normal equation adding unit 26. The teacher signal is further supplied to the normal equation adding unit 26. The normal equation addition unit 26 generates data related to the normal equation (8) based on the prediction tap supplied from the region cutout unit 22, the class tap supplied from the class code generation unit 25, and the teacher signal. The obtained data is supplied to the prediction coefficient determination unit 27.

The prediction coefficient determination unit 27 calculates a prediction coefficient by performing a calculation process for solving the normal equation (8) based on the supplied data, and supplies the calculated prediction coefficient to the memory 28. The memory 28 stores the supplied prediction coefficients.
By loading the thus-determined prediction coefficients into the memory 15 in FIG. 1, noise such as random noise caused by insufficient light quantity at the time of imaging is removed by the class classification adaptive processing. be able to.

A tap structure according to an embodiment of the present invention will be described with reference to FIG. FIG. 6A shows an example of a tap structure of a class tap, and FIG. 6B shows an example of a tap structure of a prediction tap. 6A and 6B,
Each numerical value indicates a frame number. That is, 0 indicates the current frame, -1 and -2 indicate one frame before, 2
Indicates the frame before the frame. Further, +1 and +2 indicate the frame after one frame and the frame after two frames, respectively. Each tap is indicated by a circle (a class tap in FIG. 6A and a prediction tap in FIG. 6B). The target pixel is indicated by a double circle. In this example, the pixel of interest is also used as a class tap and a prediction tap.

In one example of such a tap structure, pixels having the same spatial pixel position as the target pixel are cut out from each frame as a class tap and a prediction tap. For this reason, illustration of the spatial tap structure is omitted in FIG. The tap structure that can be used in one embodiment of the present invention is not limited to the one described above with reference to FIG. For example, the tap structure of the class tap and the tap structure of the prediction tap may be different. Also, the frame from which the class tap or the prediction tap should be cut out is not limited to the current frame and the two frames before and after the current frame. Further, a plurality of pixels may be cut out from one frame as a class tap or a prediction tap. That is, an appropriate tap structure may be used in accordance with conditions such as the properties of the captured image signal and applications.

In the above-described embodiment of the present invention, the present invention is applied to a video camera. However, the present invention can be applied to other image pickup apparatuses such as a still camera.

The present invention is not limited to the above-described embodiment of the present invention, and various modifications and applications can be made without departing from the gist of the present invention.

[0044]

According to the present invention, by performing the classification adaptive processing, it is possible to remove noise such as random noise generated when the input light does not have a sufficient light amount with respect to the sensitivity of the image sensor. it can.

Therefore, it is possible to realize effective noise removal for a moving image, in which noise removal by the recursive filter is not particularly effective.

Therefore, for both still images and moving images,
An imaging device capable of removing blur or the like due to noise can be realized.

[Brief description of the drawings]

FIG. 1 is a block diagram for describing a configuration of an embodiment of the present invention.

FIG. 2 is a schematic diagram for explaining a classification adaptive process;

FIG. 3 is a schematic diagram for explaining an ADRC process;

FIG. 4 is a block diagram for explaining a partial configuration of an embodiment of the present invention.

FIG. 5 is a block diagram for explaining a process of calculating a prediction coefficient.

FIG. 6 is a schematic diagram for explaining a tap structure.

FIG. 7 is a block diagram illustrating an example of a configuration of a conventional video camera.

8 is a block diagram showing a part of an example of the configuration shown in FIG. 7 in detail.

[Explanation of symbols]

 6 Classification adaptive processing unit

 ────────────────────────────────────────────────── ─── Continued on the front page (72) Hideo Nakaya, Inventor 6-7-35 Kita-Shinagawa, Shinagawa-ku, Tokyo Inside Sony Corporation (72) Inventor Juichi Shiraki 6-35, Kita-Shinagawa, Shinagawa-ku, Tokyo No. Sony Corporation F term (reference) 5C022 AA00 AC42 AC54 AC69 5C059 KK00 LC01 MA05 MA14 MA23 MA28 MA29 MC18 PP04 PP14 SS14

Claims (5)

[Claims] 1. An image pickup apparatus, comprising: an image pickup device that converts input light into an electric signal; a signal processing unit that generates an image signal based on an output of the image pickup device; and a process that removes noise based on the image signal. And a class classification adaptive processing means for performing class classification adaptive processing. 2. The method according to claim 1, wherein the class classification adaptive processing means includes: a first area cutout means for cutting out a first pixel area from the image signal; and an output from the first area cutout means. A feature amount extracting unit that extracts a feature amount indicating a feature of one pixel region; a class code generating unit that generates a class code indicating a class to which the first pixel region belongs from an output of the feature amount extracting unit; Prediction coefficient output means for outputting a prediction coefficient corresponding to the class code; second area cutout means for cutting out a second pixel area from the image signal; output of the second area cutout means; An image processing apparatus comprising: an arithmetic processing unit that performs an arithmetic process based on an output of the output unit. 3. The prediction coefficient output means according to claim 2, wherein said prediction coefficient output means includes storage means for storing a coefficient calculated in advance as a prediction coefficient corresponding to each class code generated by said class code generation means. Imaging device. 4. The method according to claim 2, wherein the prediction coefficient output unit cuts out a first pixel region from a plurality of image signals having no noise by a cutout unit similar to the second region cutout unit. From the image signal generated by adding noise to be removed by the classification adaptive processing to a plurality of image signals having no noise, a second pixel is extracted by the same cutout means as the first area cutout means. A region is cut out, a class code is generated based on the second pixel region, and a plurality of image signals without noise are generated based on the first pixel region and the first image signal. An imaging apparatus comprising: storing the prediction coefficients generated for each of the class codes obtained for each of the class codes. 5. An imaging method, wherein: an imaging step of converting input light into an electric signal; a signal processing step of generating an image signal based on an output of the imaging device; and a process of removing noise based on the image signal And a class classification adaptive processing step of performing a class classification adaptive processing as the above.