JP5572940B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus that can capture high-quality moving images and the like without reducing the frame rate.

  Recent electronic cameras, electronic video cameras, etc. can not only capture and generate high-resolution still images by reading out the pixel values of all pixels from the image sensor, but also by thinning out or adding such pixel values and reading them out, It can shoot and generate high-frame-rate moving images and live-view images. However, in moving image shooting or the like, a phenomenon such as moire that causes a reduction in image quality inevitably occurs because the pixel values are thinned out and read out.

Thus, for example, Patent Document 1 is a factor of image quality degradation by avoiding regular sampling Nyquist frequency aliasing while temporally or spatially changing an area where pixel values are thinned and read out from an image sensor. A technique for reducing color moire is disclosed.
JP 2003-338988 A

  However, even if the pixel values are read out from the image sensor, read out by addition, or read out while changing the method of thinning out in terms of time or space, as in the prior art of Patent Document 1, etc. It is difficult to completely avoid the effects of sampling Nyquist frequency aliasing.

  In view of the above-described problems of the conventional technology, an object of the present invention is to provide a technology capable of capturing a moving image or the like without reducing the frame rate and without causing image quality degradation due to moire or the like.

In order to solve the above-described problems, an aspect of an imaging apparatus illustrating the present invention includes a plurality of pixels arranged two-dimensionally and a color filter that separates light incident on each pixel in a predetermined pattern. Among the plurality of first scanning lines including the image sensor having the color filter array formed on the light receiving surface and the pixel array having the first combination of color filters in the color filter array, the first interval is provided. In each of a plurality of second scanning lines including a first scanning line extraction unit that reads out all pixel values on one scanning line and a pixel column having a second combination of color filters different from the first combination. , to be the same sequence as the second combination color filters, an extraction unit and a second scan line extraction unit for reading one pixel values within a second interval for each second interval, A first filter unit that removes high-frequency components in a direction of each first scan line and a second filter unit for pixel value columns on each of the plurality of first scan lines read by one scan line extraction unit; A second filter unit that removes high-frequency components in a direction intersecting each second scanning line direction with respect to a column of pixel values on a plurality of second scanning lines read by the scanning line extraction unit; comprising a filter unit for closed and an image generating unit that generates an image using a pixel value deleted the high frequency components, a.

Another aspect of the imaging apparatus illustrating the present invention receives a plurality of pixels arranged two-dimensionally and a color filter array in which color filters for color-separating incident light on each pixel are arranged in a predetermined pattern. Among the plurality of first scanning lines composed of the pixel array having the image sensor on the surface and the first combination of color filters in the color filter array, they are on the plurality of first scanning lines within the first interval. A first scanning line extraction unit that reads out all pixel values and adds and outputs a plurality of pixel values at the same position on the plurality of first scanning lines, and a second combination color different from the first combination In each of the plurality of second scanning lines composed of pixel columns having filters, the second scanning lines are arranged at the same position on the second scanning lines at every second interval so as to have the same arrangement as the second combination of color filters. It reads one pixel value, respectively, extractor and a second scan line extraction unit for adding and outputting a plurality of one pixel value at the same position in the plurality of second scan lines within a given distance A first filter unit that removes high-frequency components in a direction of each first scanning line with respect to a column of pixel values on each of the plurality of first scanning lines read by the first scanning line extraction unit ; A second component that removes high frequency components in a direction intersecting each second scanning line direction with respect to a column of pixel values on a plurality of second scanning lines read by the second scanning line extraction unit. A filter unit having a filter unit, and an image generation unit that generates an image using a pixel value from which a high-frequency component has been removed .

In addition , the image generation unit extracts and extracts the high-frequency components from the pixel values on each first scanning line from which the high-frequency components have been removed by thinning out the pixel values so as to have the same array as the first combination of color filters. An image may be generated by thinning out and extracting the column of pixel values on each removed second scanning line.

Moreover, you may further provide the display part which displays the image produced | generated by the image generation part .

  According to the present invention, it is possible to capture a moving image or the like without reducing the frame rate and without causing image quality degradation due to moire or the like.

  FIG. 1 is a configuration diagram of an imaging apparatus 100 according to an embodiment of the present invention.

  The imaging device 100 includes an imaging lens 1, an imaging device 2, an A / D conversion unit 3, a buffer memory 4, a CPU 5, a timing generator (TG) 6, a card interface (card I / F) 7, an operation member 9, and a storage unit 10. The display unit 11 and the image processing unit 12 are configured. The TG 6 is connected to the CPU 5, and the image sensor 2 and the A / D converter 3 are connected to the TG 6, respectively. The buffer memory 4, the CPU 5, the card I / F 7, the operation member 9, the storage unit 10, the display unit 11, and the image processing unit 12 are connected via a bus 13 so that information can be transmitted. FIG. 1 shows only the main part of the imaging apparatus 100. For example, in FIG. 1, an optical low-pass filter or the like installed between the imaging lens 1 and the imaging element 2 is omitted.

  The imaging lens 1 is composed of a plurality of optical lenses, and forms a subject image on the light receiving surface of the imaging element 2.

  The imaging element 2 operates in accordance with a timing pulse generated by the TG 6 based on a command from the CPU 5 and images a subject imaged by the imaging lens 1 provided in front. Further, each of the plurality of imaging pixels arranged on the light receiving surface of the imaging device 2 is provided with a color filter of any of R (red), G (green), and B (blue) in a Bayer array type. Yes. As the image sensor 2, a CCD or CMOS semiconductor image sensor or the like can be appropriately selected and used. FIG. 2 shows a part of the pixels arranged on the light receiving surface of the image sensor 2. Each cell represents one pixel. Symbols R, G, and B in each cell indicate a pixel having each color filter. In order to express the position of each cell in FIG. 2, the horizontal scanning direction is expressed as X1, X2, etc., and the vertical scanning direction is expressed as Y, Y1, Y2, etc. from the top. Therefore, for example, the pixel value of the cell of the color filter G at the X coordinate X3 in the horizontal scanning direction and the Y coordinate Y2 in the vertical scanning direction is expressed as G (X3, Y2).

  An image signal that is a pixel value of image data of a subject imaged by the image sensor 2 is converted into a digital signal by the A / D converter 4. The digital image signal is temporarily recorded in a frame memory (not shown) and then recorded in the buffer memory 4. In the present embodiment, the image sensor 2 performs all pixel readout in the case of still image capturing, for example, at 10 fps in accordance with instructions from the CPU 5 and TG 6 based on the mode setting of the operation member 9 by the user. In the case of imaging a moving image or a live view display through image displayed on the display unit 11, it is assumed that thinning-out reading is performed at 30 fps.

  As the buffer memory 4, an arbitrary nonvolatile memory among semiconductor memories can be appropriately selected and used.

  When the imaging device 100 is turned on by the user operating the power button on the operation member 9, the CPU 5 reads the control program and the image processing program stored in the storage unit 10 and initializes the imaging device 100. When the CPU 5 receives an instruction from the user via the operation member 9, the CPU 5 outputs a subject imaging command to the TG 6 based on the control program or the image captured by the image processing unit 12 based on the image processing program. Image processing is performed, and control such as recording in the card memory 7 and display on the display unit 11 is performed. As the CPU 5, a CPU of a general computer can be used.

  A card memory 8 is detachably attached to the card I / F 7. The image held in the buffer memory 4 is subjected to image processing in the image processing unit 12 based on an instruction from the CPU 5 and recorded in the card memory 8 in a file format such as JPEG format or YUV format.

  The operation member 9 transmits an operation signal corresponding to the content of the member operation by the user to the CPU 5. The operation member 9 includes operation members such as a power button, a mode setting button such as a shooting mode, and a release button.

  The storage unit 10 holds a control program for controlling the imaging device 100 used by the CPU 5, a processing program for reducing moire generated in a moving image or a through image, and an image processing program such as color correction and white balance correction. The program held in the storage unit 10 can be referred to as appropriate from the CPU 5 via the bus 13. The storage unit 10 can select and use a storage device such as a general hard disk device, a magneto-optical disk device, or a removable memory card.

  The display unit 11 displays a captured image, a live view display, a mode setting screen, and the like. A liquid crystal monitor or the like can be appropriately selected and used for the display unit 11.

  The image processing unit 12 performs not only image processing such as contour enhancement processing and white balance correction based on an instruction from the CPU 5 but also thinning and reading out pixel values from the image sensor 2 at the time of image capturing such as a moving image or a through image. This is a digital front-end circuit that performs image processing for reducing image quality reduction such as moire described later. In the present embodiment, image processing for reducing image quality reduction due to moire or the like is performed only when the pixel values are read out from the image sensor 2 such as a moving image or a through image. On the other hand, the reduction of moire and the like in the case of still image capturing is a low-pass that is optimal for the pixel pitch determined by the optical format of the image sensor 2 and the number of pixels of the image sensor 2 so that the suppression of moire during still image capture is optimal. It is assumed that the measurement is performed using an optical low-pass filter (not shown) set so as to obtain an effect.

  The image processing for reducing image quality reduction due to moire or the like when a live view display through image is displayed on the display unit 11 of the imaging apparatus 100 according to the present embodiment will be described below with reference to the flowchart of FIG. The details will be described with reference to FIG. The same applies to the case of moving image capturing by the image capturing apparatus 100.

  When the user presses the power button on the operation member 9, the CPU 5 reads the control program held in the storage unit 14 of the imaging device 100 and initializes the imaging device 100. The CPU 5 waits until receiving an imaging instruction of the subject from the user, and reads out pixel values from the imaging device 2 on the display unit 11 to reduce and generate moire based on the processing from step S10 to step S15. The live view display live image is displayed at 30 fps. In the present embodiment, while the user is using the mode setting button on the operation member 9 to set the shooting mode setting, the image recording resolution, or whether to use the electronic zoom, etc. The live view need not be displayed.

  Step S10: The CPU 5 issues a live view display through image capture command to the TG 6. The TG 6 emits a timing pulse to the image sensor 2, and the image sensor 2 images the subject imaged by the imaging lens 1. The TG 6 is a horizontal scanning line whose Y coordinate value in the vertical scanning direction is an odd number (for example, Y1, Y3, Y5, etc.) among the plurality of pixel values of the imaging element 2 shown in FIG. (Hereinafter referred to as an odd field), all pixel values in the horizontal scanning line direction of the odd fields Y9, Y15 and the like starting from the odd field Y3 at every three intervals are sequentially read out (FIG. 4A), and A / The data is transferred to the image processing unit 12 via the D conversion unit 3, the buffer memory 4 and the bus 13.

  Step S11: The image processing unit 12 removes high-frequency components that cause image quality degradation due to moire from the pixel values of each odd field that are sequentially transferred. As a result, it is possible to prevent deterioration in image quality due to moire in the horizontal scanning direction. Next, the image processing unit 12 thins out and extracts pixel values used for generating the through image from the pixel values of each odd field from which the high-frequency component is removed. Specifically, the image processing unit 12 in the present embodiment extracts pixel values having adjacent R and G color filters every five intervals. That is, among the pixel values in the odd fields shown in FIG. 4A, pixels such as R (X3, Ym) and G (X4, Ym), R (X9, Ym) and G (X10, Ym) are used. Extract values (m = 3, 9, 15, etc.). The image processing unit 12 transfers the data to the buffer memory 4 via the bus 13 and records it. Such processing is performed on the pixel values of all odd fields at every three intervals starting from the odd field Y3, and the pixel values having R and G color filters as shown in FIG. Generate data. Note that the X and Y coordinate values of each cell in FIGS. 4A and 4B indicate the respective coordinate values of FIG. In the present embodiment, since the processing in the image processing unit 12 is performed by pipeline processing, the processing in step S10 and step S11 is preferably performed in parallel. When the reading of the pixel values of the odd field of the image sensor 2 in step S10 is completed, the reading process of the pixel values of the even field in the next step S12 is started. However, the parallel processing by the pipeline of the image processing unit 12 in step S11 may be performed.

  Step S12: The CPU 5 uses the TG 6 to the image sensor 2 and among the plurality of pixel values of the captured image shown in FIG. 2, the Y coordinate value in the vertical scanning direction is an even number (for example, Y2, Y4, Y6). Etc.) out of all the pixel values in each of the horizontal scanning lines (hereinafter referred to as even fields), pixel values having adjacent G and B color filters are extracted every five intervals. Specifically, the CPU 5 sets pixel values such as G (X3, Yn) and B (X4, Yn), G (X9, Yn) and B (X10, Yn) among the pixel values on each even field. Extracted (n = 2, 4, 6, etc.) and sequentially recorded in the buffer memory 4 as one pixel column via the A / D converter 3 (FIG. 5A). The CPU 5 extracts all the predetermined pixel values of all the even fields, and then transfers the pixel value data of the even fields recorded in the buffer memory 4 to the image processing unit 12 via the bus 13.

  Step S13: When the image processing unit 12 receives the transferred pixel value data of the even field, the image processing unit 12 arranges high-frequency components that cause image quality degradation such as moire in the vertical scanning directions shown in FIG. Are removed from the column of pixel values. As a result, it is possible to prevent deterioration in image quality due to moire in the vertical scanning direction. Next, the image processing unit 12 thins out and extracts even fields used for generating a through image. Specifically, the image processing unit 12 of the present embodiment extracts the pixel values of the even field at every three intervals starting from the even field Y4, and the G and B color filters as shown in FIG. Data consisting of pixel values having The image processing unit 12 transfers the data to the buffer memory 4 via the bus 13 and records it. Note that the X and Y coordinate values of each cell in FIGS. 5A and 5B indicate the respective coordinate values in FIG. In the present embodiment, these processes by the image processing unit 12 are preferably performed in parallel by pipeline processing.

  Step S14: The CPU 5 reads the pixel value data shown in FIGS. 4B and 5B from the buffer memory 4, respectively. The CPU 5 generates an image of a single through image to be displayed on the display unit 11 by alternately arranging odd and even fields as shown in FIG.

  Step S15: In order to display the image generated in step S14 in accordance with the size of the liquid crystal screen of the display unit 11, for example, the CPU 5 performs a process of setting the size to 640 × 400 in a display format such as VGA. And display on the display unit 11 in an interlaced or progressive manner.

  The processing from step S10 to step S15 as described above is performed at 30 fps for live view display on the display unit 11 until the user fully presses the release button of the operation member 9 to perform actual shooting of a still image. Repeated at a rate of In addition, even when moving images are captured by the user, the same processing in steps S10 to S15 is performed. In the case of moving image capturing, it is preferable that in step S15, recording on the memory card 8 or the storage unit 10 is performed simultaneously with display on the display unit 11.

  As described above, in the present embodiment, in order to avoid moiré or the like that causes a reduction in the image quality of the image by being thinned and read out from the image sensor 2 at the time of moving image capturing or live view display, By removing the high-frequency component in the horizontal scanning line direction and the high-frequency component in the vertical scanning line direction for the pixel values in the even field, respectively, it is possible to reduce the deterioration of the image quality of the moving image or the through image.

In addition, since high-frequency components in the horizontal scanning line direction are removed from the pixel values in the odd field and high-frequency components in the vertical scanning line direction are removed from the pixel values in the even field, high-speed processing is possible. It can be maintained without reducing the frame rate required for recording or displaying a through image.
≪Supplementary items of embodiment≫
In the present embodiment, the pixel value of the image sensor 2 is read in the horizontal scanning line direction. However, the present invention is not limited to this, and can be applied to an image sensor that reads the pixel value in the vertical scanning direction.

  In this embodiment, in step S10, the odd field to be thinned and read out from the image sensor 2 is started from Y3 and used to generate a through image using the odd field at every three intervals. It is not limited to this. Starting from an arbitrary odd field such as Y1, pixel values of an odd field at an arbitrary interval may be read and used to generate a moving image or a through image.

  Also read out the pixel values of odd fields within a predetermined interval, for example, odd fields Y1, Y3, Y5 within three intervals, and add pixel values located at the same X coordinate or calculate an average value This may be read as the pixel value of the odd field Y3, and the pixel values of the other odd fields Y9, Y15, etc. may be read out by the same procedure. In addition, although the width | variety of the space | interval was set to three, you may carry out by other arbitrary widths.

  In the present embodiment, among the even fields in FIG. 5B in step S13, the even fields starting from the even field Y4 and extracted every three intervals are used to generate a through image. The invention is not limited to this, and the pixel value of the even field at every arbitrary interval starting from any even field such as Y2 may be read and used to generate a through image.

  Also, the pixel values of even fields within a predetermined interval, for example, even fields Y2, Y4, and Y6 within three intervals are read out, and pixel values located at the same X coordinate are added or an average value is calculated. The pixel values of the even field Y4 may be read out, and other pixel values of the even field Y10, Y16, etc. may be read out in the same procedure, and used to generate a moving image or a through image. . In addition, although the width | variety of the space | interval was set to three, you may carry out by other arbitrary widths.

  In this embodiment, in step S11, the image processing unit 12 extracts pixel values having adjacent R and G color filters from the odd field pixel values from which high-frequency components have been removed at every five intervals. However, the present invention is not limited to this. Since the pixel values to be extracted need only be an array of R and G, R (X3, Ym) and G (X6, Ym), R (X9, Ym), G (X12, Ym), and the like It may be extracted.

  In step S12 of the present embodiment, pixel values having adjacent color filters of G and B are extracted every five intervals even when pixel values are read from each even field of the image sensor 2. The invention is not limited to this, and the pixel values to be extracted may be an array of G and B. Therefore, G (X3, Yn) and B (X6, Yn), R (X9, Yn) and G ( X12, Yn), etc. (n = 2, 4, 6, etc.). Note that the X coordinate values of the pixel values to be thinned and read out are preferably matched to the odd and even fields, respectively.

  In addition, in the storage unit 10, data of the address of the pixel of the image sensor 2 that indicates which pixel value of the pixel at which position in which field is used to generate a moving image or a through image is stored in advance in the storage unit 10. Is preferred.

  In the present embodiment, the TG 6 sequentially reads pixel values of a predetermined odd or even field from the image sensor 2 in accordance with an instruction from the CPU 5, but the present invention is not limited to this. For example, the TG 6 reads all pixel values of the image sensor 2 in advance and temporarily stores them in the buffer memory 4. The CPU 5 thins out and reads out the odd and even fields according to the processing program, and the image processing unit 12. May be transferred to and processed.

  In the present embodiment, each pixel value is read and processed as it is in each of the odd and even fields, but the present invention is not limited to this. For example, the pixel values of each odd and even field are thinned out in advance at an arbitrary interval in the horizontal scanning line direction, or a value obtained by adding or averaging a plurality of pixel values having the same color filter within the interval is calculated. Then, processing may be performed after generating new odd and even fields with a reduced number of pixels. Thereby, processing can be performed at higher speed.

  It should be noted that the present invention can be implemented in various other forms without departing from the spirit or main features thereof. For this reason, the above-described embodiment is merely an example in all respects and should not be construed in a limited manner. Further, all modifications and changes belonging to the equivalent scope of the claims are within the scope of the present invention.

The figure which shows the structure of the imaging device 100 which concerns on one Embodiment of this invention. The figure which shows typically a part of pixel arranged in the light-receiving surface of the image pick-up element 2 The flowchart which shows operation | movement of the imaging device 10 which concerns on one Embodiment of this invention. The figure which shows the pixel value data which (a) read out the pixel value of the predetermined odd field from the image pick-up element 2, and was thinned out (b). The figure which shows the pixel value data which read out the predetermined pixel value among the (a) even field from the image pick-up element 2, and then thinned out the predetermined even field. The figure which shows the image produced | generated using FIG.4 (b) and FIG.5 (b).

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image pick-up lens, 2 Image pick-up element, 3 A / D conversion part, 4 Buffer memory, 5 CPU, 6 TG, 7 Card I / F, 8 card memory, 9 Operation member, 10, 11 Insertion / extraction drive part, 10 Storage part, DESCRIPTION OF SYMBOLS 11 Display part, 12 Image processing part, 13 Bus | bath, 100 Imaging device

Claims (4)

An image pickup device having a plurality of pixels arranged two-dimensionally and a color filter array in which a color filter for color-separating incident light to each pixel is arranged in a predetermined pattern on a light receiving surface;
Read out all the pixel values on the first scanning line at a first interval among a plurality of first scanning lines made up of pixel rows having a first combination of color filters in the color filter array. In each of a plurality of second scanning lines composed of pixel columns having one scanning line extraction unit and a second combination of color filters different from the first combination, the same arrangement as the color filters of the second combination An extraction unit having a second scanning line extraction unit that reads out one pixel value within the second interval for each second interval;
A first filter that removes a high-frequency component in the direction of each first scan line from the column of pixel values on each of the plurality of first scan lines read by the first scan line extraction unit. And the pixel value column on the plurality of second scanning lines read by the second scanning line extraction unit and the second scanning line extraction unit, the high frequency in a direction intersecting each second scanning line direction A filter unit having a second filter unit for removing components;
An image generation unit that generates an image using the pixel value from which the high-frequency component has been removed;
An imaging apparatus comprising: An image pickup device having a plurality of pixels arranged two-dimensionally and a color filter array in which a color filter for color-separating incident light to each pixel is arranged in a predetermined pattern on a light receiving surface;
All the pixel values on the plurality of first scanning lines within the first interval among the plurality of first scanning lines formed of the pixel columns having the first combination of color filters in the color filter array are obtained. A color filter having a second combination different from the first scanning line extraction unit that reads and adds and outputs the plurality of pixel values at the same position on the plurality of first scanning lines. In each of the plurality of second scanning lines made up of the pixel columns having the same, the ones at the same position on the second scanning line at every second interval so as to have the same arrangement as the color filters of the second combination. And a second scanning line extraction unit that adds and outputs the plurality of the first pixel values at the same position on the plurality of second scanning lines within a predetermined interval. And,
A first filter that removes a high-frequency component in the direction of each first scan line from the column of pixel values on each of the plurality of first scan lines read by the first scan line extraction unit. And the pixel value column on the plurality of second scanning lines read by the second scanning line extraction unit and the second scanning line extraction unit, the high frequency in a direction intersecting each second scanning line direction A filter unit having a second filter unit for removing components;
An image generation unit that generates an image using the pixel value from which the high-frequency component has been removed;
An imaging apparatus comprising: In the imaging device according to claim 1 or 2,
The image generation unit
Among the pixel values on each of the first scanning lines from which the high-frequency components have been removed, the pixel values are thinned out and extracted so as to be in the same array as the color filter of the first combination, and the high-frequency components are extracted. An imaging apparatus, wherein the image is generated by thinning out and extracting the column of the pixel values on each removed second scanning line. The imaging apparatus according to any one of claims 1 to 3,
An imaging apparatus, further comprising a display unit that displays the image generated by the image generation unit.

Images