JP2003333610A - Image processor, image processing method, recording medium, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To carry out thinning processing utilizing all pixels. <P>SOLUTION: When a horizontal component low pass filter subjects pixels to (1,0,1,0,1)/3 filtering with a green pixel in the first line as a noticed pixel, a simple average of the green pixel as the noticed pixel and adjacent green pixels is calculated because a sum total of pixel values of green pixels multiplied by one and red pixels multiplied by 0 is divided by 3. When a red pixel in the first line is taken as a noticed pixel, a simple average of the red pixel as the noticed pixel and adjacent red pixels is calculated in the same manner. Similarly, green pixels and blue pixels in the second line are filtered. Since every fourth pixel of a horizontal component is thinned in a horizontal 1/3 thinning processing part, a thinned pixel group is generated which is constituted of pixels obtained by taking every fourth pixel as a noticed pixel to calculate a simple average value of the noticed pixel and adjacent pixels of the same color. This method is applicable for a digital camera. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus, an image processing method, a recording medium, and a program, and particularly to an image processing apparatus, an image processing method, and a recording medium suitable for use in thinning processing. , And the program.

[0002]

2. Description of the Related Art For example, it is used in digital cameras (digital still cameras, digital video cameras, and digital cameras capable of capturing still images and moving images).
An image sensor, which is a device that converts light into an electric signal, has a structure in which fine photodiodes are arranged on the surface of a chip. Photodiodes are semiconductors that generate electric charges according to the intensity of the applied light, and because they do not have the ability to capture color, it is necessary to capture incident light by color element in order to capture a color image. .

A single plate type color image pickup device used in a general digital camera or a video camera has three or four color filters arranged in a mosaic pattern on a photodiode to form an individual photo sensor. It is designed so that the diode responds to individual color elements. For example,
For RGB (red, green, blue, that is, three primary colors of light) each has an image pickup device, that is, for a three-plate camera using three image pickup devices, RBG color filters are arranged in a mosaic pattern. That is, a camera that captures images using a single-plate color image sensor that uses one image sensor is called a single-panel camera.

For example, a CCD (Charge C
oupled Device) and CMOS (Complementary Metal-O)
xide Semiconductor: A complementary metal oxide semiconductor) sensor is often used.

Originally, CCD refers to all elements having a function of transferring charges in a bucket brigade type, but at present, C
In general, the CD generally means a CCD image pickup device using this CCD for reading out electric charges. CCD is
It is used in general image devices such as digital cameras and image scanners that input images.

The CCD image pickup device includes a photodiode and
It is composed of a circuit for successively taking out and transferring the charges generated by the photodiode. The concrete structure of the CCD image pickup device is such that photodiodes and electrodes are arranged on a silicon substrate like a grid (this number is the number of pixels of the CCD), and this grid is the same as the film. When the optical image passing through the lens is formed as described above, each photodiode generates an electric charge according to the intensity of the received light. The accumulated charges move to the adjacent electrodes just like being pushed out by the pump by alternately applying voltage to the electrodes, and finally output one by one. Two-dimensional image data is generated by amplifying this or converting it into a digital value. In the case of a moving image, this process is repeated endlessly to generate a moving image as a continuous still image.

A CMOS sensor is not a CCD but a
This is an image sensor using a CMOS switch for reading out electric charges, and each pixel is composed of one photodiode and a switch using a CMOS transistor. That is, it has a structure in which a switch is attached to each of the photodiodes arranged in a grid pattern, and the switches are sequentially switched to read out the charges one pixel at a time. Due to such a mechanism, for example, smear peculiar to CCD does not occur in principle, and power consumption is low (1/10 of CCD).
Degree), can be driven by a single low voltage, and
Since the same manufacturing line as general chips can be used and it can be configured in one chip together with peripheral functions, various advantages such as small size and low cost can be expected. However, in order to put the CMOS sensor into practical use as an image sensor, it is necessary to take measures against noise caused by switching noise and the like.

The number of sensors of a solid-state image pickup device, that is, the number of pixels of an image to be picked up is as follows:
In most cases, it was 1 million pixels or less. On the other hand, in the digital still camera, the number of pixels is 1 million pixels in order to improve the semiconductor technology in recent years and increase the resolution.
The number tends to increase to, 000,000 pixels and 3 million pixels.

Further, in recent years, digital cameras capable of picking up still images and moving images have been developed and have become widespread.

However, as the number of pixels in the captured image increases, the time required to read all the pixels becomes longer. Unlike a still image, in a moving image, a smooth moving image cannot be obtained if the image reading time is long.

In order to solve this, if resolution is not required, there is a method of thinning out a part of pixels and reading out.

Conventionally, particularly when a CCD is used as an image pickup device, when displaying a moving image, for example, 2
Pixels composed of × 2 (here, Gr, r, B, Gb
4 pixels), as shown in FIG. 1, a method of thinning out only 2 pixels out of 8 pixels and not using other pixels, or 2 pixels out of 12 pixels as shown in FIG. A method of thinning out only the pixel and not using other pixels has been used.

[0013]

However, as shown in FIG.
Alternatively, in the thinning method shown in FIG. 2, not only is it impossible to perform thinning at equal intervals, but since all pixels are not used, there is a problem in that a displayed or recorded moving image is folded back. Will occur.

The present invention has been made in view of such a situation, and it is possible to realize a thinning process capable of obtaining a good thinned image without folding back.

[0015]

An image processing apparatus according to the present invention comprises an acquisition means for acquiring an image signal composed of a plurality of color components and having a 2 × 2 color coding, and an image signal acquired by the acquisition means. Of these, a filter means for calculating an arithmetic mean value of (2n-1) pixel values of the pixel of interest and pixels of the same color component in the vicinity of the pixel of interest, and the pixel value of the pixel of interest is calculated by the filter means. And a thinning-out unit for thinning out pixels having the added average value as a pixel value at a thinning rate of 1 / (2n-1).

The filter means may use the pixel values of all the pixels of the image signal of one frame to calculate the arithmetic mean value.

The filter means selects pixels of the same color component which are arranged in the horizontal direction with respect to the pixel of interest,
Calculating an average value of pixel values of (2n-1) pixels,
The pixel value of the pixel of interest can be set.

The filter means selects pixels of the same color component which are arranged in the vertical direction with respect to the pixel of interest,
Calculating an average value of pixel values of (2n-1) pixels,
The pixel value of the pixel of interest can be set.

With respect to the pixel of interest, the filter means selects pixels of the same color component in a predetermined range in the horizontal direction and the vertical direction, and adds and averages the pixel values of (2n-1) pixels. The value can be calculated and used as the pixel value of the pixel of interest.

The filter means has a plurality of color components among the plurality of color components.
For at least one color component of the pixel of interest, pixels of the same color component are selected in order from those arranged in the position close to the pixel of interest, and the pixel values of (2n-1) pixels are added. The average value may be calculated and used as the pixel value of the target pixel.

For the pixel of interest of all color components of the plurality of color components, the filter means selects pixels of the same color component in order from those arranged at positions close to the pixel of interest, and ( It is possible to calculate the arithmetic mean value of the pixel values of 2n-1) pixels and set it as the pixel value of the target pixel.

The acquisition means may further include an image pickup means for picking up an image signal.

The image pickup means may be composed of a CMOS IC.

The image pickup means, the filter means, and the thinning-out means can be constituted by the same CMOS IC.

It is possible to further include display means for displaying an image corresponding to the image signal acquired by the acquisition means, and when the image displayed by the display means is a moving image, it is acquired by the acquisition means. The processed image signal can be processed by the filter means and the thinning means to be displayed by the display means, and when the image displayed by the display means is a still image, the image acquired by the acquisition means The signal may be displayed by the display means without being processed by the filter means and the decimation means.

Recording means for recording the image signal obtained by the obtaining means may be further provided.
When the image signal recorded by the recording means is a signal corresponding to a moving image, the image signal acquired by the acquisition means may be processed by the filter means and the thinning means and recorded by the recording means. You can
When the image signal recorded by the recording unit is a signal corresponding to a still image, the image signal acquired by the acquisition unit is recorded by the recording unit without being processed by the filter unit and the thinning unit. You can

Output means for outputting the image signal acquired by the acquisition means can be further provided.
When the image signal output by the output unit is a signal corresponding to a moving image, the image signal acquired by the acquisition unit may be processed by the filter unit and the thinning unit, and output by the output unit. You can
When the image signal output by the output unit is a signal corresponding to a still image, the image signal acquired by the acquisition unit is output by the output unit without being processed by the filter unit and the thinning unit. You can

The image processing method of the present invention comprises an acquisition control step for controlling acquisition of an image signal having a plurality of color components and having 2 × 2 color coding, and an image whose acquisition is controlled by the processing of the acquisition control step. Of the signal, the pixel of interest and the pixel of the same color component in the vicinity of the pixel of interest (2n-1)
The filter calculation step of calculating the addition average value of the pixel values of the number of pixels and setting it as the pixel value of the target pixel, and the pixel having the addition average value calculated by the processing of the filter calculation step as the pixel value are 1 / (2n− 1) a thinning step of thinning at a thinning rate.

The program recorded on the recording medium of the present invention comprises an acquisition control step for controlling acquisition of an image signal composed of a plurality of color components and having a 2 × 2 color coding, and an acquisition control step. A filter that calculates an arithmetic mean value of (2n-1) pixel values of the target pixel and pixels of the same color component in the vicinity of the target pixel among the image signals controlled to be the pixel value of the target pixel. The present invention is characterized by including a calculation step and a thinning step of thinning out pixels whose pixel value is the addition average value calculated by the processing of the filter calculation step at a thinning rate of 1 / (2n-1).

The program of the present invention comprises an acquisition control step for controlling the acquisition of an image signal having a color coding of 2 × 2, which is composed of a plurality of color components, and an image signal whose acquisition is controlled by the processing of the acquisition control step. Of these, a filter calculation step of calculating an arithmetic mean value of (2n-1) pixel values of the pixel of interest and pixels of the same color component in the vicinity of the pixel of interest and setting the pixel value of the pixel of interest, and a filter calculation step A thinning-out step of thinning out pixels whose pixel value is the addition average value calculated by the processing of 1) at a thinning rate of 1 / (2n-1).

In the image processing apparatus, the image processing method, and the program of the present invention, an image signal composed of a plurality of color components and having a 2 × 2 color coding is acquired, and the pixel of interest is included in the acquired image signal. And an average value of (2n-1) pixel values of pixels of the same color component in the vicinity of the target pixel is calculated and set as the pixel value of the target pixel, and the calculated average value is set as the pixel value. Pixel is 1 / (2n
-1) Thinning rate is used.

[0032]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

FIG. 3 is a block diagram showing the structure of a digital camera to which the present invention is applied.

The operation input unit 11 is composed of, for example, buttons, keys, a jog dial, etc., and allows the user to input an operation input for image pickup such as still image pickup, moving image recording start and end, or exposure and focus adjustment. receive.

The control unit 12 has a CPU (Central Processi).
ng Unit), ROM (Read Only Memory), and RA
The digital camera 1 is configured by an M (Random Access Memory) and controls the operation of the digital camera 1. Based on the user's operation input from the operation input unit 11, the imaging unit 13, the thinning processing unit 14, the camera signal processing unit. 15 and drive 18
A control signal for controlling is generated and output to each unit.

The image pickup section 13 converts, for example, an optical lens, a drive section for driving the optical lens, an image pickup element such as a CCD sensor or a CMOS sensor, and an analog electric signal acquired by these image pickup elements into a digital signal. A still image or a moving image is imaged under the control of the control unit 12.

Under the control of the control unit 12, the thinning processing unit 14 responds to the image signal input from the image pickup unit 13 by
Thinning processing is performed as necessary.

Here, the image pickup section 13 and the thinning processing section 14
May be configured as different processing blocks, or may be configured as the same chip (for example, the image pickup unit 13 and the thinning processing unit 14 are one CMOS IC).
May be configured).

Under the control of the control unit 12, the camera signal processing unit 15 responds to the input image data by, for example,
Performs γ processing, white balance, demosaicing, and other processing.

The display unit 16 is composed of, for example, a liquid crystal panel or the like, and displays the input image data. Display 1
6 functions as a finder for confirming an image to be captured, and displays a captured still image or a recorded moving image in accordance with a user operation input from the operation input unit 11. The output unit 17 is connected to another device or a network by wire or wirelessly and outputs a captured still image or a recorded moving image.

A magnetic disk 21, an optical disk 22, a magneto-optical disk 23, a semiconductor memory 24, or the like is appropriately mounted on the drive 18, and the image signal supplied from the camera signal processing section 15 is transferred to the magnetic disk 21, the optical disk 22, or the like. Magneto-optical disk 23 or semiconductor memory 2
4 for output and recording, or for outputting still image data or moving image data recorded on the magnetic disk 21, the optical disk 22, the magneto-optical disk 23, the semiconductor memory 24 or the like to the display unit 16 for display. Alternatively, the output unit 17 outputs the output to another device.

A magnetic tape may be attached to the drive 18. Furthermore, by providing a recording medium such as a semiconductor memory or a hard disk inside the digital camera 1 in addition to the drive 18, captured image data can be stored inside the digital camera 1. You may do it.

The magnetic disk 21, the optical disk 22, and the magneto-optical disk 2 mounted on the drive 18
3 or the program read from the recording medium such as the semiconductor memory 24 is supplied to the control unit 12 and installed as necessary.

When the user operates the operation input unit 11 to display an image which can be captured by the image capturing unit 13 on the display unit 16 (that is, when the display unit 16 is used as a finder). ), The control unit 12 controls the imaging unit 13 to continuously acquire image data and output the image data to the thinning processing unit 14. The thinning processing unit 14 executes the thinning processing according to the control of the control unit 12, and outputs the thinned image data to the camera signal processing unit 15. Details of the thinning process will be described later. The camera signal processing unit 15 performs a predetermined process on the thinned-out image data supplied from the thinning-out processing unit 14, and outputs it to the display unit 16 to display it.

Since the time required to read out the image can be shortened by performing the thinning-out, the moving image can be displayed on the display unit 16 in a smooth manner.

When the user refers to the image displayed on the display unit 16 and gives an instruction to capture a still image using the operation input unit 11, the control unit 12 controls the image capturing unit 13, Image data for one frame is acquired and output to the thinning processing unit 14. The thinning processing unit 14 includes the control unit 12
According to the control of 1, the image data composed of all the pixels not thinned out is output to the camera signal processing unit 15 without executing the thinning-out process. The camera signal processing unit 15 is supplied from the thinning processing unit 14 and is not thinned out.
The frame image data is subjected to a predetermined process, output to the display unit 16 for display, and output to a predetermined recording medium mounted on the drive 18 for recording.

Since thinning is not performed, a high resolution image is recorded on a predetermined recording medium mounted on the drive 18.

When the user refers to the image displayed on the display unit 16 and uses the operation input unit 11 to instruct to capture a moving image, the control unit 12 controls the image capturing unit 13 to The image data is continuously acquired, and the thinning processing unit 14
To output. The thinning processing unit 14 executes the thinning processing according to the control of the control unit 12, and outputs the thinned image data to the camera signal processing unit 15. The camera signal processing unit 15 performs a predetermined process on the thinned-out image data supplied from the thinning-out processing unit 14, outputs the processed data to the display unit 16 for display, and also performs a predetermined process mounted on the drive 18. Output to a recording medium for recording.

Since the time taken to read out the image can be shortened by performing the thinning-out, the moving image can be recorded on a predetermined recording medium mounted on the drive 18.

Further, the user operates the operation input unit 11 to output the still image data or the moving image data recorded on a predetermined recording medium mounted on the drive 18 to the display unit 16. It can be displayed as an output or can be output from the output unit 17 to another device or the like.

The digital camera 1 may further be provided with a microphone for acquiring audio data, an audio data processing unit for processing the acquired audio data, a speaker for outputting the audio data, and the like.

FIG. 4 is a block diagram showing a more detailed structure of the thinning processing section 14.

The horizontal component low-pass filter 31 horizontally (1, 0, 1, 0,
1) / 3 filter is applied. The horizontal 1/3 thinning-out processing unit 32 performs resampling (thinning-out) every three pixels in the horizontal direction.

For example, as shown in FIG. 5, so-called 2 ×, where green and red color filters are arranged in the first line and blue and green color filters are arranged in the second line, respectively. Bayer array of 2 (G (green)
Filtering and resampling when coding is performed in a checkered pattern will be described with reference to FIG.

In the first line, a pixel corresponding to the arrangement position of the green color filter (hereinafter referred to as a green pixel) and a pixel corresponding to the arrangement position of the red color filter (hereinafter referred to as the red pixel). Pixels) are alternately arranged one by one. Horizontal component low-pass filter 3
1 is (1, 0,
When the filter of 1, 0, 1) / 3 is applied, the part corresponding to the three green pixels including the target pixel is multiplied by 1, and the part corresponding to the red pixel is multiplied by 0, Since the sum of the pixel values of 1 is divided by 3, the simple average of the green pixel that is the pixel of interest and the green pixel adjacent to that pixel is calculated.

On the other hand, the horizontal component low pass filter 31
With one red pixel as the pixel of interest, (1, 0, 1, 0,
When the filter of 1) / 3 is applied, the part corresponding to the three red pixels including the pixel of interest is multiplied by 1, and the part corresponding to the green pixel is multiplied by 0 to obtain those pixel values. Since the sum is divided by 3, the simple average of the red pixel that is the pixel of interest and the red pixel adjacent to that pixel is calculated.

Similarly, the horizontal pixel low-pass filter 31 also filters the green pixel in the second line and the pixel corresponding to the arrangement position of the blue color filter (hereinafter referred to as the blue pixel). .

Then, in the horizontal 1/3 thinning-out processing section 32, each pixel is thinned out every 3 pixels of the horizontal component, so that finally every 3 pixels indicated by the arrow in the drawing of FIG. The pixel of interest is used as the pixel of interest, and a thinned pixel group composed of pixels obtained by calculating a simple average value with pixels of the same color adjacent to the pixel is generated.

In FIG. 6, the green pixel is the target pixel, the red pixel is the target pixel, and the blue pixel is the target pixel and the green pixel is the target pixel. Although the pixels filtered in the above are separately illustrated in the upper and lower stages, the horizontal component low-pass filter 31 can simultaneously execute the processing of the same line.

FIG. 7 is a diagram showing a band of an image before being input to the thinning processing section 14. The Nyquist limit frequency fn in the image before thinning is 1/2 of the sampling frequency fs of the image before thinning.

FIG. 8 shows that the horizontal component low-pass filter 31 has (1, 0, 1, 0,
FIG. 9 is a diagram showing a band after a filter of 1) / 3 has been applied, and FIG. 9 is a diagram showing a band after being thinned out in the horizontal 1/3 thinning processing unit 32.

In FIGS. 8 and 9, the Nyquist limit frequency fn 'and the sampling frequency fs' are both appropriate compared with the image before being input to the thinning processing unit 14 described with reference to FIG. Is a limited value,
Furthermore, these sampling points are evenly spaced.

That is, in the conventional thinned-image generating method, the thinned-out image is generated by using only some pixels without using most of the pixels. Since the filtering process is executed using all the pixels and the thinning process is performed at equal intervals to generate the thinned image, it is possible to obtain the thinned image having no image phase shift and few aliasing.

In the thinning-out processing unit 14, the horizontal component low-pass filter 31 applies a filter of (1, 0, 1, 0, 1) / 3 to the image before thinning out, and the horizontal 1/3 thinning-out processing is performed. Although it has been described that the sampling frequency fs ′ is thinned to 1/3 in the unit 32, a trap point is included in the sampling frequency fs ′ after thinning, the filter is properly band-limited at the Nyquist limit frequency fn ′, and Any filter may be used as long as there is no image phase shift later.

For example, in such a thinning method, even if the thinning rate is other than 1/3, the thinning rate is 1 / (2n-1).
(However, n is a positive integer). For example, in the digital camera 1 of FIG. 3, instead of the thinning processing unit 14, the thinning processing unit 41 shown in FIG.
May be provided.

The horizontal component low-pass filter 51 horizontally (1, 0, 1, 0,
A filter of 1, 0, 1, 0, 1) / 5 is applied. The horizontal 1/5 thinning processing unit 52 performs resampling (thinning) every 5 pixels in the horizontal direction.

For example, filtering and resampling when coding is performed with a 2 × 2 Bayer array described with reference to FIG. 5 will be described with reference to FIG.

The horizontal component low-pass filter 51 uses one green pixel as the pixel of interest in the first line,
When the filter of (1, 0, 1, 0, 1, 0, 1, 0, 1) / 5 is applied, the part corresponding to the five green pixels including the pixel of interest is multiplied by 1, and The portion corresponding to the pixel is multiplied by 0, and the sum of those pixel values is divided by 5. Therefore, a simple average of the green pixel as the target pixel and the green pixel adjacent to the pixel is calculated. It

On the other hand, the horizontal component low pass filter 51
With one red pixel as the pixel of interest, (1, 0, 1, 0,
When a filter of 1, 0, 1, 0, 1) / 5 is applied, the part corresponding to the five red pixels including the target pixel is multiplied by 1, and the part corresponding to the green pixel is multiplied by 0. Then, since the sum of those pixel values is divided by 5, the simple average of the red pixel that is the target pixel and the red pixel adjacent to the pixel is calculated.

Similarly, the green pixels and the blue pixels in the second line are also filtered by the horizontal component low-pass filter 51. Then, in the horizontal ⅕ thinning processing unit 52, each pixel is thinned out every 5 pixels of the horizontal component, so that finally every 5 pixels shown by the arrow in the drawing, As the pixel of interest, a thinned pixel group composed of pixels obtained by calculating a simple average value of pixels of the same color adjacent to the pixel is generated.

In FIG. 11, the case where the green pixel is the target pixel, the case where the red pixel is the target pixel, and
The pixels filtered when the blue pixel is the target pixel and when the green pixel is the target pixel are shown separately in the upper and lower stages, respectively, but the thinning processing unit 41 uses the same line. It is possible to simultaneously perform the processing of.

For example, as described with reference to FIG. 7, the Nyquist limit frequency fn in the image before thinning is
When an image having a sampling frequency fs of the image before thinning is 1/2 is input to the thinning processing unit 41,
This will be described with reference to FIGS. 12 and 13.

FIG. 12 shows the horizontal component low-pass filter 51.
In (1, 0, 1,
FIG. 13 is a diagram showing a band after the filter of 0, 1, 0, 1, 0, 1) / 5 is applied, and FIG. 13 is a band after the thinning processing in the horizontal 1/5 thinning processing unit 52. FIG.

In the band after the filtering and the band after the decimation processing shown in FIGS. 12 and 13, both the Nyquist limit frequency fn 'and the sampling frequency fs' are shown in FIG. Compared with the image before the thinning processing described by using, the values are appropriately limited, and these sampling points are evenly spaced.

That is, in the same way as in the above-described thinning processing unit 14, the filtering processing is performed by using all the pixels by the processing of the thinning processing unit 41, and the thinning processing is performed at equal intervals. It is possible to obtain a thinned image with little folding back.

In this way, the thinning rate 1 / (2n-
1) (where n is a positive integer), pixels of the same color (2n-1) around the pixel of interest are added and averaged to perform thinning, thereby providing a trap point at the sampling frequency fs' after thinning. Therefore, it is possible to perform the decimation process that is appropriately band-limited at the Nyquist limit frequency fn ′ and that has no image phase shift after the decimation. Further, in this thinning-out process, a thinned-out image is generated by using all pixels, so that it is possible to obtain a thinned-out image with few aliasing.

Such thinning processing is performed not only in the horizontal direction but also in the vertical direction by providing the thinning processing unit 61 shown in FIG. 14 in place of the thinning processing unit 14 in the digital camera 1 of FIG. It is possible to carry out.

The delay line 71 is a delay line for four lines, and simultaneously synchronizes the image signals of five lines in the vertical direction. Vertical component low-pass filter 72
In the vertical direction with respect to the input image signal (1,
A filter of 0, 1, 0, 1) / 3 is applied. Vertical 1
The / 3 decimation processing unit 73 resamples (decimates) every three pixels in the vertical direction.

For example, filtering and resampling when coding is performed with a 2 × 2 Bayer array described with reference to FIG. 5 will be described with reference to FIG.

In the first vertical line, one green pixel and one blue pixel are alternately arranged, and in the second vertical line, one red pixel and one green pixel are alternately arranged. There is. The vertical component low-pass filter 72 sets one green pixel as a pixel of interest in the first vertical line,
When the filter of (1, 0, 1, 0, 1) / 3 is applied, the part corresponding to the green pixel is multiplied by 1, and the part corresponding to the red pixel is multiplied by 0 to obtain the pixel of interest. Including 3
Since the sum of the pixel values of one pixel is divided by 3, the simple average of the green pixel that is the target pixel and the green pixel that is adjacent to that pixel is calculated.

On the other hand, the vertical component low pass filter 72
With one red pixel as the pixel of interest, (1, 0, 1, 0,
When a filter of 1) / 3 is applied, a portion corresponding to a red pixel is multiplied by 1, and a portion corresponding to a green pixel is multiplied by 0, so that a total of three pixel values including a target pixel is obtained. Since it is divided by 3, the simple average of the red pixel that is the pixel of interest and the red pixel adjacent to that pixel is calculated.

Similarly, the green and blue pixels in the second vertical line are also filtered by the vertical component low-pass filter 72.

Then, in the vertical 1/3 thinning-out processing unit 73, each pixel is thinned out every three pixels of the vertical component, so that finally every three pixels indicated by the arrow in the drawing of FIG. The pixel of interest is used as the pixel of interest, and a thinned pixel group composed of pixels obtained by calculating a simple average value with pixels of the same color adjacent to the pixel is generated.

In FIG. 15, the green pixel is the target pixel, the red pixel is the target pixel, and the blue pixel is the target pixel and the green pixel is the target pixel. Although the filtered pixels are shown by dividing them into a right line and a left line, respectively, the thinning-out processing unit 61 can simultaneously execute the same vertical line processing.

Also in the thinning processing of the vertical component described with reference to FIGS. 14 and 15, a trap point can be put in the sampling frequency fs ′ after thinning, and the band is appropriately band-limited at the Nyquist limit frequency fn ′. Moreover, it is possible to perform the thinning-out processing in which the image phase is not shifted after the thinning-out. In addition, even in this thinning-out process, a thinned-out image is generated using all pixels, so that it is possible to obtain a thinned-out image with few aliasing.

Further, in the digital camera 1 shown in FIG.
When the image capturing unit 13 can read out the image in block units, the thinning processing unit 81 shown in FIG. 16 is provided in place of the thinning processing unit 14 to perform the horizontal and vertical thinning processing simultaneously.

The horizontal / vertical component low-pass filter 91 horizontally (1, 0, 1,
0,1) / 3, vertically (1,0,1,0,1) /
3, that is, in the horizontal and vertical directions (3, 0, 3, 0, 3)
This filter has a filter coefficient of / 9 and performs a process of adding and averaging adjacent 9 pixels of the same color in the horizontal and vertical directions.
The horizontal / vertical 1/3 thinning-out processing unit 92 resamples (thinning) every 3 pixels in the horizontal direction and the vertical direction, that is, 9 pixels into 1 pixel.

For example, when coding is performed with a 2 × 2 Bayer array, the horizontal / vertical component low-pass filter 91 for the input pixel as shown in FIG.
As shown in FIG. 8, in each of the pixels of the blue component, the green component, and the red component, (3, 0, 3, 0, 3) /
Since the filtering process of 9 is performed, the pixel of interest other than the corresponding color component is multiplied by 0, and the pixel of the corresponding color component is multiplied by 1 for each pixel of interest, so that nine pixels including the pixel of interest are included. Since the total pixel value is divided by 9, nine pixels of the same color that are adjacent to the target pixel in the horizontal and vertical directions are added and averaged.

Then, the horizontal / vertical 1/3 decimation processing section 92
In, since nine adjacent pixels in the horizontal and vertical directions are thinned out to one pixel, a thinned image as shown in FIG. 19 can be obtained.

Here, among three pixels of the same color in the horizontal and vertical directions, a square pixel group (n, m) in which the pixel of interest is the central pixel of 5 × 5 pixels in the horizontal and vertical directions (where n is 1 to 5, m takes a value of 1 to 5), (1,1)
(1,3) (1,5) (3,1) (3,3) (that is, target pixel) (3,5) (5,1) (5,3) (5,5)
The above nine pixels are added and averaged. However, as long as the phase position of the pixels after thinning does not change, any shape may be used for the addition processing.

For example, the horizontal / vertical component low-pass filter 9
As shown in FIG. 20, 1 represents (3, 0, 3, 0, 3) / in the horizontal and vertical directions in the pixels of the blue component and the red component.
9 filter processing is performed, and the green component pixel is subjected to filter processing in the horizontal and vertical directions with a filter coefficient of (1, 2, 3, 2, 1) / 9, and is adjacent to the target pixel in a diamond shape. You may make it perform the arithmetic mean of 9 pixels.

As described above, with respect to the green component pixels arranged in a checkered pattern, (1, 2, 3, 2,
The filter of (3), (0, 3, 0, 3) / 9 described with reference to FIG. 18 is performed by performing the arithmetic mean of 9 pixels adjacent to the rhombus using the (1) / 9 filter. Compared to the case where the processing is used, it is possible to calculate the arithmetic mean of the 9 pixels that are closest to the pixel of interest in the green pixel.

As described above, when the thinning processing is performed in the horizontal and vertical directions at the same time, particularly when the filter coefficient differs depending on the pixel as described with reference to FIG. 20, the image pickup unit 13 of the digital camera 1 shown in FIG. Although it can be realized even if the image pickup device included in is a CCD, the image pickup device is
CMOS is preferable to CD.

Even when the thinning-out processing is simultaneously performed in the horizontal and vertical directions, the thinning-out processing that can achieve the same effect is a thinning-out rate 1 / (2n-1) even if the thinning-out rate is other than 1/3. (However, n is a positive integer).

As described with reference to FIGS. 16 to 20, even in the horizontal and vertical component thinning processing to which the present invention is applied, a trap point can be put in the sampling frequency fs ′ after thinning, and the Nyquist limit frequency is reduced. With fn ′, it is possible to perform thinning processing that is appropriately band-limited and that has no image phase shift after thinning. Further, even in the thinning processing of the horizontal and vertical components, the thinned image is generated by using all the pixels, so that the thinned image with less aliasing can be obtained.

In the processing described above, since the pixel ranges of the respective color components to be subjected to the addition processing spatially overlap, the color coding of the image before and after the thinning is the same. Therefore, the present invention can be applied to processing using any color coding as long as it is 2 × 2 color coding.

The case where the Bayer array using the three primary colors of light is used has been described here.
The present invention is also applicable when color coding using complementary colors is used. For example, in the complementary color filter, there are many cases where a 2 × 2 pattern is created by “CMYG” in which G (green) is added in addition to CMY (cyan, magenta, yellow) which are the three primary colors of colors. It goes without saying that the present invention can be applied even when such a complementary color filter is used.

The series of processes described above can also be executed by software. The software is, for example, a general-purpose personal computer, in which a program that constitutes the software is installed in dedicated hardware, or various functions can be executed by installing various programs. Installed from the recording medium.

As shown in FIG. 3, this recording medium is a magnetic disk 21 (including a flexible disk) and an optical disk 2 on which a program is recorded, which is distributed in order to provide the program to the user, separately from the computer.
2 (CD-ROM (Compact Disk-Read Only Memory), DVD
(Including a Digital Versatile Disk), a magneto-optical disk 23 (including an MD (Mini-Disk) (trademark)), a package medium including a semiconductor memory 24, or the like.

Further, in the present specification, the steps for writing the program recorded on the recording medium are not limited to the processing performed in time series according to the order described, but are not necessarily performed in time series. It also includes processing executed in parallel or individually.

[0101]

According to the present invention, picked-up images can be thinned out and displayed, recorded, or output.
Further, according to the present invention, (2n-1) (n is a positive integer) pixels of the same color are simply averaged using all the pixels in the captured image, if necessary, and then 1 / ( 2n-
By thinning out to 1), it is possible to obtain a thinned image with little folding back and no phase shift of the image after thinning.

[Brief description of drawings]

FIG. 1 is a diagram illustrating conventional vertical 1/8 thinning.

FIG. 2 is a diagram illustrating conventional vertical 2/12 thinning.

FIG. 3 is a block diagram showing a configuration of a video camera to which the present invention is applied.

FIG. 4 is a block diagram showing a more detailed configuration of a thinning-out processing unit in FIG.

FIG. 5 is a diagram illustrating a Bayer array.

FIG. 6 is a diagram illustrating a process executed by a thinning processing unit in FIG.

7 is a diagram illustrating a band of an image input to the thinning processing unit in FIG.

FIG. 8 is a diagram illustrating a band of an image filtered by a thinning processing unit in FIG.

9 is a diagram illustrating a band of an image output from the thinning processing unit in FIG.

FIG. 10 is a block diagram showing a different configuration of a thinning processing unit.

11 is a diagram illustrating a process executed by the thinning-out processing unit in FIG.

12 is a diagram illustrating a band of an image filtered by a thinning processing unit in FIG.

13 is a diagram illustrating a band of an image output from the thinning processing unit in FIG.

FIG. 14 is a block diagram showing a different configuration of a thinning processing unit.

FIG. 15 is a diagram illustrating a process executed by the thinning-out processing unit in FIG.

FIG. 16 is a block diagram showing a different configuration of a thinning processing unit.

FIG. 17 is a diagram for explaining an image signal input to the thinning-out processing unit in FIG. 16;

FIG. 18 is a diagram for explaining a process executed by the thinning-out processing unit in FIG. 16;

19 is a diagram for explaining an image signal output from the thinning-out processing unit in FIG.

20 is a diagram for explaining a different example of the process executed by the thinning-out processing unit in FIG.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 digital camera, 11 operation input section, 12 control section, 13 imaging section, 14 thinning processing section, 15
Camera signal processing unit, 16 display unit, 17 output unit,
18 drive, 31 horizontal component low-pass filter, 32 horizontal 1/3 decimation processing unit, 41 decimation processing unit, 51 horizontal component low-pass filter, 52
Horizontal 1/5 decimation processing unit, 61 decimation processing unit, 7
1 delay line, 72 vertical component low pass filter, 73 vertical 1/3 decimation processing unit, 81 decimation processing unit, 91 horizontal and vertical low pass filter, 92
Horizontal vertical 1/3 thinning processing unit

Continued front page    (72) Inventor Katsuaki Hirota             6-735 Kita-Shinagawa, Shinagawa-ku, Tokyo Soni             -Inside the corporation F term (reference) 5C024 CX14 CY12 EX52 GY31 HX05                       HX21                 5C065 AA01 AA03 BB13 CC01 DD15                       EE06 GG02 GG17 GG23 GG24

Claims (16)

[Claims] 1. An image processing apparatus for processing an image, comprising: an acquisition unit configured to acquire an image signal composed of a plurality of color components and having a 2 × 2 color coding; and the image signal acquired by the acquisition unit. A filter means for calculating an arithmetic mean value of (2n-1) pixel values of the pixel of interest and pixels of the same color component in the vicinity of the pixel of interest to obtain the pixel value of the pixel of interest; An image processing apparatus, comprising: a thinning-out unit that thins out the pixels whose pixel value is the addition average value calculated by the above, at a thinning rate of 1 / (2n-1). 2. The image processing apparatus according to claim 1, wherein the filter means calculates the arithmetic mean value using pixel values of all the pixels of the image signal of one frame. . 3. The filter means selects pixels of the same color component which are lined up in the horizontal direction with respect to the pixel of interest, and calculates an average value of pixel values of (2n−1) pixels. The image processing device according to claim 1, wherein the image value is calculated and used as the pixel value of the pixel of interest. 4. The filter means selects pixels of the same color component arranged in the vertical direction with respect to the pixel of interest, and calculates an average value of pixel values of (2n−1) pixels. The image processing device according to claim 1, wherein the image value is calculated and used as the pixel value of the pixel of interest. 5. The filter means selects pixels of the same color component within a predetermined range in the horizontal direction and the vertical direction with respect to the pixel of interest, and sets pixel values of (2n−1) pixels. The image processing apparatus according to claim 1, wherein an arithmetic mean value of the above is calculated and set as a pixel value of the target pixel. 6. The same color component is arranged in order from the one arranged at a position closer to the target pixel with respect to the target pixel of at least one color component among a plurality of the color components. 2. The image processing apparatus according to claim 1, wherein the pixel is selected, the arithmetic mean value of the pixel values of the (2n−1) pixels is calculated, and the arithmetic mean value is set as the pixel value of the target pixel. 7. The filter means selects pixels of the same color component in order from the pixels arranged at positions closer to the pixel of interest for the pixels of interest of all color components of the plurality of color components. 2. The image processing apparatus according to claim 1, wherein the average value of pixel values of the selected (2n-1) pixels is calculated and used as the pixel value of the pixel of interest. 8. The image processing apparatus according to claim 1, wherein the acquisition unit further includes an image capturing unit that captures the image signal. 9. The image processing apparatus according to claim 8, wherein the imaging unit is composed of a CMOS IC. 10. The image processing apparatus according to claim 8, wherein the image pickup unit, the filter unit, and the thinning unit are composed of the same CMOS IC. 11. The display device further comprises a display unit for displaying an image corresponding to the image signal acquired by the acquisition unit, and when the image displayed by the display unit is a moving image, the image is acquired by the acquisition unit. The image signal is
When the image processed by the filter unit and the thinning unit and displayed by the display unit and the image displayed by the display unit is a still image, the image signal acquired by the acquisition unit is the filter The image processing apparatus according to claim 1, wherein the image is displayed by the display unit without being processed by the unit and the thinning unit. 12. The recording means for recording the image signal acquired by the acquisition means is further provided, and when the image signal recorded by the recording means is a signal corresponding to a moving image, the acquisition means acquires the image signal. The obtained image signal is processed by the filter means and the thinning means, recorded by the recording means, and when the image signal recorded by the recording means is a signal corresponding to a still image, the acquisition The image processing apparatus according to claim 1, wherein the image signal acquired by the means is recorded by the recording means without being processed by the filter means and the thinning means. 13. The apparatus further comprises output means for outputting the image signal acquired by the acquisition means, and when the image signal output by the output means is a signal corresponding to a moving image, acquired by the acquisition means. The obtained image signal is processed by the filter means and the thinning-out means, output by the output means, and when the image signal output by the output means is a signal corresponding to a still image, the acquisition The image processing apparatus according to claim 1, wherein the image signal acquired by the means is output by the output means without being processed by the filter means and the thinning means. 14. An image processing method for an image processing apparatus for processing an image, comprising: an acquisition control step of controlling acquisition of an image signal composed of a plurality of color components and having 2 × 2 color coding; Of the image signals whose acquisition is controlled by the processing, the arithmetic mean of (2n-1) pixel values of the target pixel and pixels of the same color component in the vicinity of the target pixel is calculated, and the target pixel is calculated. And a pixel having the pixel value of the addition average value calculated by the process of the filter calculation step is 1 / (2n−
1) A thinning step of thinning at a thinning rate. 15. A program for an image processing apparatus for processing an image, comprising: an acquisition control step for controlling acquisition of an image signal composed of a plurality of color components and having 2 × 2 color coding; and the acquisition control step. Among the image signals whose acquisition is controlled by the processing of step S1, the addition average value of (2n-1) pixel values of the pixel of interest and pixels of the same color component in the vicinity of the pixel of interest is calculated, 1 / (2n−) is defined as the pixel calculation value of the pixel and the pixel having the addition average value calculated by the process of the filter calculation step
1) A thinning step of thinning at a thinning rate, and a recording medium having a computer-readable program recorded thereon. 16. An acquisition control step, which is a computer-executable program for controlling an image processing apparatus for processing an image, for controlling acquisition of an image signal having a plurality of color components and having a 2 × 2 color coding. And an addition average value of (2n-1) pixel values of the pixel of interest and the pixel of the same color component in the vicinity of the pixel of interest in the image signal whose acquisition is controlled by the process of the acquisition control step. Is calculated as the pixel value of the pixel of interest and the pixel having the addition average value calculated by the process of the filter calculation step as the pixel value
1) A thinning step of thinning at a thinning rate.