JP2014072877A - Imaging apparatus and imaging method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To produce an image whose distortion component is sufficiently reduced, while suppressing the degradation of image quality of the peripheral part of the image, caused by optical distortion, in an imaging apparatus.SOLUTION: The imaging apparatus produces the image by sampling the output of the effective pixel of an imaging element 102 and performs distortion correction processing for reducing a component corresponding to the distortion aberration of an optical system 101, included in the image. When an output image having a predetermined number of pixels smaller than the total number of effective pixels of the imaging element is produced, a correction target image is produced by sampling the output of the number of effective pixels larger than the predetermined number of pixels and by using a sampling pitch different between the central part and peripheral part of the imaging element or so as to be equivalent to the case of using the sampling pitch. The distortion correction processing is performed to the correction target image and the output image is produced from the resulting image after the distortion correction processing.

Description

  The present invention relates to an image pickup apparatus having an image pickup element such as a digital still camera or a digital video camera, and more particularly to an image pickup apparatus capable of reducing a component corresponding to distortion aberration of an image pickup optical system included in an image generated by image pickup. About.

  In the imaging apparatus as described above, an optical distortion is corrected by correcting a component (hereinafter also referred to as a distortion component) in an image corresponding to a distortion aberration (hereinafter also referred to as an optical distortion) of the imaging optical system by electrical processing. There is a case where the design freedom of the imaging optical system is allowed and the size of the imaging optical system is reduced.

  Patent Document 1 discloses an imaging system that electrically corrects a distortion component in an image by performing enlargement / reduction processing corresponding to the angle of view on a signal (image) from an imaging element. Patent Document 2 discloses an aberration correction apparatus that electrically corrects a distortion component in an image by controlling a sampling point of an image sensor.

  In general, the resolution required for moving images is lower than the resolution required for still images. Therefore, when capturing a still image, the charge (data) is read from all effective pixels of the image sensor to generate a high-quality still image, while during moving image capturing, “reading skip” is performed to select a pixel from which data is read. In many cases, the calculation load is reduced and a moving image is generated at a high frame rate.

Japanese Patent Laid-Open No. 06-205242 JP-A-10-224695

  In the case where the negative optical distortion of the imaging optical system is allowed and the imaging optical system is downsized, the spatial frequency on the imaging element side is the screen (effective pixel area of the imaging element) in consideration of the spatial compression effect due to the optical distortion. Shift to the high frequency side at the periphery.

Here, the spatial compression effect due to optical distortion will be described with reference to FIGS. 11 and 12. FIG. 11A shows the characteristics of the amount of optical distortion (hereinafter referred to as distortion amount) with respect to the image height. The horizontal axis indicates the standardized image height, and the vertical axis indicates the amount of distortion. The “distortion amount” here is a ratio between the ideal image height in the adopted projection method and the real image height obtained through the optical system. For example, in the central projection method, when the focal length of the optical system is f and the angle of view is ω, the distortion amount V is the ratio between the ideal image height h (= f × tan ω) and the real image height h ′.
V = [{h ′ / (f × tan ω)} − 1] × 100 [%]
It is represented by

  When the amount of distortion is negative, as shown in FIG. 11B, the space is compressed in the peripheral portion with respect to the center portion of the screen, and the rectangular object is distorted into a barrel shape.

Here, when the function of the distortion amount V with respect to the ideal image height h is V (h), the image height (real image height) L in consideration of optical distortion is
L (h) = h ′ = h + h × V (h)
It can be expressed as.

At this time, considering the minute section dh in the radial direction (meridional direction) from the optical axis on the image plane, the space extension ratio km to the ideal image height is
km = dL (h) / dh
= 1 + V (h) + dV (h) / dh
It can be seen that this depends on the amount of distortion and the differential value of the distortion curve.

Similarly, when considering a minute angle dθ in the angular direction (sagittal direction) from the optical axis on the image plane, the space extension ratio ks to the ideal image height is
ks = L (h) dθ / hdθ
= 1 + V (h)
It turns out that it depends on the amount of distortion.

  So far, the spatial stretching ratio to the ideal image height of the image in which the optical distortion remains is considered, but the spatial compression ratio at which the object is compressed by the optical distortion can be defined as the reciprocal of the spatial stretching ratio.

In FIG. 12, the distortion curve V (h) with respect to the ideal image height h is
V (h) = − 0.25 × h ³
The calculation result of the space stretching ratio with respect to the image height is shown. The horizontal axis in FIG. 12 indicates the standardized image height, and the vertical axis indicates the spatial stretching ratio. meri indicates the meridional direction, and sagi indicates the sagittal direction.

  As can be seen from FIG. 12, in the case of an optical system in which an optical distortion of −25% is generated in a cubic function at the maximum image height, a radial space is provided at the outermost part with respect to the center part of the screen. Is compressed to about ½.

Here, when the Nyquist frequency determined by the sampling pitch p of the image sensor is fn,
fn = p / 2
It is. When the optical system having the above optical distortion is used, the Nyquist signal fn at the center of the screen is
fn ′ = fn × 2
Until the high frequency side.

  In other words, if the sampling pitch of the image sensor is constant over the entire area of the screen, the spatial frequency band that can be resolved at the periphery is narrower than the center of the screen, and the spatial frequency corresponding to the Nyquist signal at the center is reduced. The object cannot be resolved at the periphery.

  Therefore, when using an imaging optical system in which large optical distortion remains, the peripheral portion (especially in the radial direction) with respect to the center portion of the image (screen) regardless of the presence or absence of distortion correction processing, which is electrical correction of distortion components. It is difficult to avoid image quality degradation.

  In the imaging system disclosed in Patent Document 1 described above, distortion correction processing can be performed by image enlargement / reduction processing. However, since the sampling points of the imaging device have an equal pitch, the image distortion is caused by optical distortion. There is a problem that the image quality of the central part and the peripheral part becomes non-uniform.

  In the aberration correction apparatus disclosed in Patent Document 2 described above, the spatial sampling point on the image sensor is randomly controlled in accordance with the distortion aberration characteristic. However, since a method of rearranging and reading out pixels is employed, there is a concern that noise such as jaggy occurs at the edge of the image after the distortion correction processing. When processing such as pixel interpolation is performed in order to suppress the occurrence of such noise, the calculation load for the interpolation processing increases particularly when sampling all pixels of the image sensor.

  According to the present invention, it is possible to generate an image in which distortion components are sufficiently reduced while suppressing deterioration in image quality at the periphery of the image due to optical distortion, and it is also possible to suppress an increase in calculation load. An imaging apparatus and an imaging method are provided.

  An image pickup apparatus according to one aspect of the present invention generates an image by sampling an image pickup element that photoelectrically converts an optical image formed by an optical system, and an output of an effective pixel of the image pickup element, and is included in the image Image generation means for performing distortion correction processing for reducing a component corresponding to distortion aberration of the optical system. When generating an output image having a predetermined number of pixels smaller than the total number of effective pixels of the image sensor, the image generation unit samples the outputs of the effective pixels larger than the predetermined number of pixels, and the center of the image sensor The correction target image is generated so as to be equivalent to the case where different sampling pitches are used in the peripheral portion or different sampling pitches are used. Then, distortion correction processing is performed on the correction target image, and an output image is generated from the result image after the distortion correction processing.

  According to another aspect of the present invention, an imaging method generates an image by sampling an output of an effective pixel of an imaging device in an imaging device having an imaging device that photoelectrically converts an optical image formed by an optical system. And a distortion correction process for reducing a component corresponding to the distortion aberration of the optical system included in the image is performed. In this imaging method, an image to be corrected is sampled on a correction target image having a predetermined number of pixels smaller than the total number of effective pixels of the imaging device, and the output of a number of pre-effective pixels larger than the predetermined number of pixels is obtained. Generating a sample using a different sampling pitch in the central part and the peripheral part of the image or equivalent to the case where a different sampling pitch is used, a step of performing a distortion correction process on the correction target image, and the distortion And a step of generating an output image from the result image after the correction processing.

  Furthermore, an imaging program as another aspect of the present invention causes an imaging device having an imaging device that photoelectrically converts an optical image formed by an optical system to generate an image by sampling the output of effective pixels of the imaging device. A computer program executable by the computer of the imaging apparatus for performing distortion correction processing for reducing components corresponding to distortion aberration of the optical system included in the image. The imaging program samples an image to be corrected having a predetermined number of pixels smaller than the total number of effective pixels of the imaging device, outputs outputs of effective pixels larger than the predetermined number of pixels, and A step of generating a sample using a different sampling pitch in the central part and the peripheral part or equivalent to using a different sampling pitch, a step of performing a distortion correction process on the correction target image, and the distortion correction Generating an output image from the processed result image.

  According to the present invention, since the distortion correction processing is performed on the correction target image in which the deterioration of the image quality in the peripheral portion of the image due to the optical distortion is suppressed, the distortion component is sufficiently reduced and the center portion to the peripheral portion is An output image having good image quality can be generated. In addition, an increase in calculation load during image generation can be suppressed.

1 is a block diagram illustrating a configuration of an imaging apparatus that is Embodiment 1 of the present invention. 3 is a flowchart illustrating processing performed by the imaging apparatus according to the first embodiment. 3 is a diagram illustrating a Bayer array of pixels of an image sensor used in the image pickup apparatus according to Embodiment 1. FIG. 3 is a diagram illustrating non-equal pitch pixel sampling from the image sensor in Embodiment 1. FIG. The figure which shows the Nyquist frequency in the imaging device of Example 1 and Example 2 of this invention mentioned later. FIG. 6 is a diagram illustrating a maximum imaging frequency in Examples 1 and 2. 9 is a flowchart illustrating processing performed by the imaging apparatus according to the second embodiment. FIG. 6 is a diagram illustrating pixel mixture sampling from the image sensor in the second embodiment. FIG. 6 is a diagram illustrating non-equal pitch pixel sampling from an image sensor in an imaging apparatus that is Embodiment 3 of the present invention. FIG. 10 is a diagram illustrating another non-equal pitch pixel sampling from the image sensor in the third embodiment. The figure explaining the space compression by a distortion aberration. The figure explaining the space extending | stretching ratio resulting from a distortion aberration.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 shows a basic configuration of an image pickup apparatus that is Embodiment 1 of the present invention. The imaging optical system 101 forms a subject image (optical image) on light from a subject (not shown). The image sensor 102 photoelectrically converts the subject image and outputs an analog signal. The aperture 101a included in the imaging optical system 101 adjusts the amount of light reaching the imaging element 102 by changing the aperture diameter. The position of the focus lens 101b included in the imaging optical system 101 is adjusted by an unillustrated autofocus (AF) mechanism or manual focus mechanism in order to focus the imaging optical system 101 in accordance with the subject distance. . The imaging optical system 101 may include optical elements such as a low-pass filter and an infrared cut filter. The imaging optical system 101 may be provided integrally with the imaging apparatus, or may be provided interchangeably.

  The A / D converter 103 converts an analog signal from the image sensor 102 into a digital signal and inputs the digital signal to the image processing unit 104. The reading position control unit 120 selects a pixel from which charges (data) are read out from all effective pixels of the image sensor 102 at an equal pitch or an unequal pitch. The effective pixel of the image sensor 102 is a pixel that can read data used to generate an image among all the pixels of the image sensor 102.

  The image processing unit 104 performs various types of image processing, and performs distortion correction processing described later to generate an output image.

  Specifically, the image processing unit 104 first obtains state information indicating the state of the imaging apparatus (focal length, focus position and aperture value of the imaging optical system 101, imaging mode described later, etc.) from the state detection unit 108. The state detection unit 108 may obtain state information directly from the system controller 110, or may obtain state information related to the imaging optical system 101 from the imaging optical system control unit 106. Next, the image processing unit 104 selectively reads out distortion correction information corresponding to the state information from the storage unit 107, and performs distortion correction processing on the image (an image including a distortion component) generated by the image processing unit 104. By doing so, the distortion component in the image is reduced.

  As described above, the distortion component is a component (image component) in the image corresponding to the distortion aberration (optical distortion) of the imaging optical system 101. The image processing unit 104 and the reading position control unit 120 correspond to an image generation unit.

  Hereinafter, imaging processing (image generation processing) performed by the imaging apparatus when the imaging optical system 101 is a zoom lens will be described with reference to the flowchart of FIG. This processing is executed by a system controller (computer including a CPU or the like) 110 according to an imaging program that is a computer program installed via a recording medium such as a semiconductor memory or an optical disc, or a communication network such as the Internet or a LAN. The imaging program itself that can be executed by the computer and a recording medium on which the imaging program is recorded are also included in the embodiments of the present invention.

  In step S <b> 101, the system controller 110 detects the currently set imaging mode via the state detection unit 108. The imaging mode is set by a user operation on an operation unit (not shown). In the imaging mode, the first imaging mode for generating a high-resolution image using all the effective pixels of the image sensor 102, and using some effective pixels among all effective pixels, that is, thinning out the effective pixels. There is a second imaging mode that reads and generates a low-resolution image.

  Note that the output image generated in the second imaging mode is an image having a smaller number of pixels than the output image generated in the first imaging mode, and the high-resolution generated in the first imaging mode. It is called a low-resolution image in the sense that the resolution is lower than the image.

  In step S102, the system controller 110 determines whether or not the imaging mode detected in step S101 is the first imaging mode (whether it is the second imaging mode).

  When the first imaging mode is set, the system controller 110 proceeds to step S103, and via the imaging optical system control unit 106 according to the input of a zoom control signal by the user's zoom operation in the operation unit. The imaging optical system 101 is caused to perform a zoom operation. Along with the zoom operation, the focus operation of the imaging optical system 101 is also performed to maintain the in-focus state, and the aperture operation is also performed to appropriately set the brightness of the subject image. Then, the system controller 110 acquires state information (zoom position and focus position) of the imaging optical system 101 after the zoom operation from the state detection unit 108.

  Next, in step S104, the system controller 110 determines whether or not to execute the distortion correction process based on the state information of the imaging optical system 101 acquired in step S103. Specifically, it is determined whether or not distortion correction information corresponding to the zoom position and focus position indicated by the state information is stored in the storage unit 107. The distortion correction information here is, for example, the ratio between the real image height and the ideal image height for each position in the image (image including the distortion component) obtained from the distortion aberration characteristic of the imaging optical system 101 (the “distortion described above”). Amount V ").

  If it is determined that the distortion correction process is to be executed, the system controller 110 causes the image processing unit 104 to execute the distortion correction process in step S105. The image processing unit 104 performs a coordinate conversion process on the image including the distortion component using the distortion correction information read from the storage unit 107 according to the state information. A generally known algorithm may be used for the distortion correction processing (coordinate conversion processing) algorithm. For example, the enlargement / reduction ratio (“extraction ratio km, ks” described above) for converting the real image height at each position of the image including the distortion component into the ideal image height is calculated from the distortion correction information described above. Then, enlargement / reduction processing is performed using the enlargement / reduction ratio. In addition, pixel interpolation processing is performed as necessary.

  When the result image after the distortion correction processing is obtained in this way, the system controller 110 causes the image processing unit 104 to generate an output image from the result image, and thereafter ends this processing.

  On the other hand, if it is determined in step S104 that the distortion correction process is not to be executed (not required), the system controller 110 generates an output image without causing the image processing unit 104 to execute the distortion correction process, and performs this process. finish.

  If the imaging mode is set to the second imaging mode in step S102, the system controller 110 proceeds to step S106 and acquires the state information of the imaging optical system 101 after the zoom operation, as in step S103. To do.

  In step S107, the system controller 110 determines whether or not to execute the distortion correction process as in step S104. If it is determined to execute the distortion correction process, the system controller 110 proceeds to step S108.

  In step S <b> 108, the system controller 110 performs data reading from effective pixels selected at an unequal pitch by the reading position control unit 120 among all effective pixels of the image sensor 102, i.e., pixel thinning-out reading at an unequal pitch. . As a result, a correction target image as a low resolution image is generated. For example, when the imaging device 102 is an XY address readout type imaging device such as a CMOS sensor, the period of clock pulses for driving the horizontal and vertical shift registers for pixel readout is arbitrary. It can be realized by controlling to.

  Here, pixel thinning readout at an unequal pitch from the image sensor 102 (hereinafter also referred to as unequal pitch pixel sampling) will be described with reference to FIGS. It is assumed that a total of 2304 effective pixels of 48 pixels in the horizontal direction and 48 pixels in the vertical direction are arranged at an equal pitch on the image sensor 102. Here, the imaging optical system 101 will be described as having a rotational symmetry with respect to the optical axis in the range of the quadrant of the imaging element. Further, in the actual imaging device 102, as shown in FIG. 3, four (RGGB) pixels having primary color filters of R (red), G (green) and B (blue) are arranged horizontally and vertically by a Bayer arrangement. It is assumed that they are periodically arranged in the direction. However, in the description here, these four pixels are treated as one pixel, and the concept of the color filter is omitted.

  FIG. 4 shows an outline of non-equal pitch pixel sampling in the case of the final pixel decimation rate in the second imaging mode (the ratio of the number of recorded pixels to the total number of effective pixels of the image sensor 102 is 1/4). . FIG. 4 shows 24 × 24 effective pixels included in the ¼ quadrant, but the actual image sensor 102 has 24 × 24 × 4 = 2304 effective pixels. Of the effective pixels shown in FIG. 4, the hatched pixels are pixels from which data is read (sampled).

  A pixel thinning rate of 1/4 (2304/4 = 576 pixels) can be realized by sampling at equal pitch pixels every other pixel in the horizontal direction and every other pixel in the vertical direction. However, this causes the Nyquist frequency of the image sensor 102 to shift uniformly throughout the effective pixel region (hereinafter referred to as the screen) of the image sensor 102. When the imaging optical system 101 has a large negative optical distortion, the spatial frequency that can be resolved at the center and the periphery of the screen changes due to the optical distortion as described above. The Nyquist signal cannot be resolved at the periphery.

  Therefore, in this embodiment, when the distortion correction process is executed in the second imaging mode, a correction target image as a low resolution image is generated in consideration of the spatial frequency shift caused by the optical distortion. Specifically, the number of effective pixels sampled from the image sensor 102 when generating an output image as a low resolution image is larger than the number of pixels of the output image (predetermined number of pixels = 576 pixels), and the image sensor 102. The sampling pitch is made different between the central portion and the peripheral portion of the (screen). That is, by performing oversampling on the output image and generating a correction target image in which the sampling pitch is changed, an effect equivalent to changing the Nyquist frequency of the image sensor according to the position in the screen is obtained.

Here, when the effective pixel number for sampling data (output) when generating the correction target image is S1, the total effective pixel number of the image sensor 102 is Se, and the minimum value of the sampling pitch is Lm, It is desirable to set Sl so as to satisfy the conditions.
Sl = Se / Lm ² (1)
By oversampling so as to satisfy the condition (1), it is possible to prevent a step from being generated in the image at a position where the sampling pitch changes.

  In this embodiment, as shown in FIG. 5, the pixel decimation rate is set so that non-equal pitch pixel sampling is performed in which the radial Nyquist frequency in the peripheral portion is doubled with respect to the central portion of the screen. Different in the center and the periphery. The horizontal axis in FIG. 5 represents the standardized image height, and the vertical axis represents the Nyquist frequency as a ratio with respect to the axis.

Specifically, at the center from the center of the screen (on the axis) to 80% image height, pixel thinning readout is performed by setting a pixel thinning rate (sampling pitch is 2 pixels) every other pixel in the horizontal and vertical directions. I do. On the other hand, in the peripheral portion where the image height is 80% or higher, the pixel thinning rate is set to zero (sampling pitch is 1 pixel) and all pixels are read out. At this time, a pixel to be read from the image sensor corresponds to the hatched pixel in FIG. Further, the effective pixel number Sl for sampling data when generating the correction target image is 2304 pixels, which corresponds to the region surrounded by the double line in FIG. That is, the total number of pixels read from the image sensor through pixel thinning is smaller than the effective number of pixels for sampling data when generating the correction target image.
Here, regarding the pixel value of a pixel that is not read from the image sensor and does not have data to be sampled (for example, the pixel in the first row and the second column, and hereinafter referred to as a non-data pixel), the data of the surrounding pixels It is good to calculate by interpolation. Since the non-data pixel is originally arranged in a region where the Nyquist frequency is low, a simple process may be performed in which the data of the pixel closest to the non-data pixel is used as the pixel value of the non-data pixel. For example, the data of the pixel in the first row and the first column may be used as the pixel value of the non-data pixel in the first row and the second column.

  As described above, in this embodiment, oversampling is performed twice the vertical and horizontal directions with respect to the number of pixels (predetermined number of pixels) of the output image, and a correction target image is generated using different sampling pitches in the central portion and the peripheral portion of the screen. Thus, the Nyquist frequency is controlled.

  FIG. 6 shows a case where pixel sampling of this embodiment is performed in an imaging apparatus using an imaging optical system in which an optical distortion of −25% is generated in a cubic function at the maximum image height shown in FIG. Nyquist frequency (maximum resolution frequency) is shown as a ratio to the axis. The horizontal axis in FIG. 6 indicates the standardized image height, and the vertical axis indicates the Nyquist frequency (ratio to the axis).

  By setting the pixel readout pitch from the image sensor 102 to an unequal pitch, it is possible to control the Nyquist frequency, which is the maximum spatial frequency that can be resolved from the center to the periphery of the screen, as shown in FIG. For this reason, even if the imaging optical system 101 is an optical system with large negative optical distortion, by controlling the Nyquist frequency, image non-uniformity is suppressed from the central part to the peripheral part as an image before distortion correction processing. It is possible to obtain a good correction target image (suppressing deterioration in image quality at the peripheral portion).

  When the imaging optical system 101 has a negative optical distortion, oversampling may be performed so that pixel sampling is performed at a denser pitch toward the periphery of the screen of the image sensor 102 when generating the correction target image. preferable. That is, it is preferable that the sampling pitch in the peripheral part is smaller than the sampling pitch in the central part. Thereby, since the original pixels used for pixel interpolation are densely present in the peripheral portion of the screen with a large stretch ratio in the distortion correction processing, it is expected that the interpolation accuracy is improved.

  In step S109 in FIG. 2, the system controller 110 causes the image processing unit 104 to execute distortion correction processing on the correction target image generated in step S108, as in step S105. In addition, the image processing unit 104 generates a low-resolution image as a final output image by down-sampling the resultant image after the distortion correction processing by a general method. In the present embodiment, a low-resolution image of 12 × 12 × 4 = 576 pixels (final pixel decimation rate ¼) is generated by down-sampling 1/2 in length and width. After generating the output image in this way, this process is terminated.

  On the other hand, if it is determined in step S107 that the distortion correction processing is not to be executed, the system controller 110 reads out pixels at equal pitches every other pixel in the horizontal and vertical directions on the entire screen (equal pitch pixels) in step S110. Sampling). Thus, after generating a low-resolution image as an output image in this way, this processing is terminated.

  The output image generated by the image processing unit 104 is stored in a predetermined format in the image recording medium 109 shown in FIG. The output image is displayed on the display unit 105.

  As described above, in this embodiment, when generating an output image having a predetermined number of pixels smaller than the total number of effective pixels of the image sensor 102, the output of the effective pixels greater than the predetermined number of pixels is sampled, and the center of the screen The correction target image is generated using different sampling pitches in the area and the peripheral area. As a result, it is possible to generate a correction target image in which deterioration in image quality at the periphery of the image due to optical distortion is suppressed. Since distortion correction processing is performed on such a correction target image, it is possible to generate an output image in which distortion components are sufficiently reduced and the image quality is good from the center to the periphery. The same applies to other embodiments described later.

  In addition, the result image after the distortion correction processing obtained in this embodiment can also be obtained by performing distortion correction processing on an image generated by sampling all pixels on the image sensor and then performing reduction processing. It is done. However, in this case, the calculation load in the distortion correction process increases as the number of pixels handled increases. On the other hand, by performing distortion correction processing on the correction target image obtained by optimizing the sampling pitch at the center portion and the peripheral portion of the screen according to the optical distortion as in this embodiment, the calculation load is reduced. Can be reduced. In particular, the larger the ratio (final pixel thinning rate) between the low-resolution image as the output image and the effective pixel number of the image sensor, the higher the calculation load reduction effect. The same applies to other embodiments described later.

  In the present embodiment, the case where the boundary between the center portion and the peripheral portion of the screen is 80% image height has been described. However, this is only an example, and other image heights (for example, 70% image height) are set. It is good also as a boundary of a center part and a peripheral part.

  In the present embodiment, the case where the Nyquist frequency in the peripheral portion with respect to the central portion of the screen is controlled to be doubled to make the Nyquist frequency in the entire screen different in two stages has been described. However, this is only an example, and the Nyquist frequency can be continuously controlled by changing the pixel sampling pitch in multiple stages over the entire screen. For example, when it is desired to set the Nyquist frequency in the peripheral portion to three times the central portion, it can be handled by performing oversampling three times in the vertical and horizontal directions for a predetermined number of pixels. That is, parameters such as the pixel sampling pitch and the final pixel thinning rate can be arbitrarily set. Further, these parameters may be changed according to image processing such as electronic zoom processing and electronic camera shake correction processing.

  Next, an image pickup apparatus that is Embodiment 2 of the present invention will be described. The basic configuration of the imaging apparatus according to the present embodiment is the same as that of the imaging apparatus according to the first embodiment illustrated in FIG. 1, but the imaging process (image generation process) is different from that of the first embodiment. The imaging process in the present embodiment will be described using the flowchart of FIG. This processing is also executed by the system controller 110 in accordance with an imaging program that is a computer program.

  Steps S101 to S107 in FIG. 7 are the same as steps S101 to S107 shown in FIG.

  If it is determined in step S102 that the current imaging mode is the second imaging mode and the distortion correction process is to be executed in step S107, the system controller 110 proceeds to step S201.

  In the present embodiment, unlike the first embodiment, the readout position control unit 120 always selects pixels from which data is to be read out of all effective pixels of the image sensor 102 at an equal pitch. In this embodiment, in step S201, the system controller 110 causes the image processing unit 104 to perform pixel thinning readout at the center of the screen by pixel mixing. Pixel mixing is a process of outputting an average value of outputs (data) of a predetermined number of adjacent pixels as one pixel value.

  FIG. 8 shows an outline of pixel sampling by pixel mixture. Here, the image sensor 102 is assumed to have 2304 effective pixels of 48 pixels in the horizontal direction and 48 pixels in the vertical direction, and will be described in the range of a quarter quadrant. Further, four pixels of RGGB arranged in a Bayer array are treated as one pixel, and the concept of a color filter is omitted.

  Of the effective pixels shown in FIG. 8, the hatched pixels are pixels from which data is read (sampled). Further, four pixels surrounded by a thick line are treated as one pixel by pixel mixture. That is, in this embodiment, after reading out data of all effective pixels of the image sensor 102, pixel thinning processing is performed on the center portion of the screen by pixel mixture. Thereby, a correction target image equivalent to the case where non-equal pitch pixel sampling is performed is generated.

  In this embodiment, as shown in FIG. 5, a sampling result equivalent to non-equal pitch pixel sampling in which the radial Nyquist frequency in the peripheral portion is doubled relative to the central portion of the screen is obtained. The number of pixels to be mixed is made different between the central portion and the peripheral portion of the screen. More specifically, in the central portion from the center of the screen (on the axis) to the 80% image height, the number of pixels to be mixed is set to 2 pixels in the horizontal and vertical directions (sampling pitch is 2 pixels). 4-pixel mixing is performed. On the other hand, in the peripheral portion of 80% image height or higher, the number of pixels to be mixed is set to zero (sampling pitch is 1 pixel), and all pixels are read (pixel mixing is not performed). At this time, the effective pixel number Sl for sampling data when generating the correction target image is 2304 pixels, which corresponds to the region surrounded by the double line in FIG.

  The same value as the average value of the data from these four pixels is used as the pixel values of the four pixels (for example, four pixels in the first row and the second column and the second row and the second column) in which the pixels are mixed. .

  As described above, this embodiment is equivalent to a case where oversampling is performed twice in the vertical and horizontal directions with respect to the number of pixels of the output image (predetermined number of pixels = 576 pixels), and different sampling pitches are used in the central portion and the peripheral portion of the screen. A correct correction target image is generated. Thereby, the Nyquist frequency is controlled. Also in this embodiment, as shown in FIG. 6, it is possible to control the Nyquist frequency, which is the maximum spatial frequency that can be resolved from the center to the periphery of the screen. For this reason, even if the imaging optical system 101 is an optical system with large negative optical distortion, by controlling the Nyquist frequency, image non-uniformity is suppressed from the central part to the peripheral part as an image before distortion correction processing. It is possible to generate a good correction target image (suppressing deterioration of image quality in the peripheral portion).

  In step S109 of FIG. 7, the system controller 110 causes the image processing unit 104 to execute distortion correction processing on the correction target image generated in step S201 in the same manner as in step S109 of FIG. At this time, also in this embodiment, when non-equal pitch pixel sampling is performed, not only information regarding optical distortion is used for coordinate conversion processing as distortion correction processing, but also information of the sampled output from the image sensor 102 is used. Must also be used.

  Also in the present embodiment, the image processing unit 104 performs downsampling on the result image after the distortion correction processing by a general method, so that a low-resolution image (final pixel thinning) as a final output image is obtained. 576-pixel image corresponding to the rate 1/4) is generated. After generating the output image in this way, this process is terminated.

  On the other hand, if it is determined in step S107 that the distortion correction processing is not to be executed, the system controller 110 performs pixel mixing of two pixels in the horizontal and vertical directions on the entire screen in step S202 to reduce the low resolution as the output image. Generate an image. Then, this process ends.

  In the present embodiment, by appropriately setting a region where pixel mixing is performed in the screen, it is possible to expect an effect of averaging noise due to pixel mixing in the central portion of the screen where a main subject often exists.

  Next, an image pickup apparatus that is Embodiment 3 of the present invention will be described. The basic configuration of the image pickup apparatus of the present embodiment is the same as that of the image pickup apparatus of the first embodiment shown in FIG. 1, but doubles the electronic zoom in the image pickup process (image generation process), and the pixel decimation rate is accordingly increased. This is different from the first embodiment. Since the flow of the imaging process itself is the same as the flow shown in the flowchart of FIG. 1, here, a pixel sampling method in the case where the distortion correction process is performed in the second imaging mode will be described.

  Also in this embodiment in which the electronic zoom is performed twice, the Nyquist frequency is controlled so that the Nyquist frequency in the radial direction is twice that in the central portion at the periphery of the screen when the correction target image is generated. Specifically, the oversampling of the output image is performed twice in the vertical and horizontal directions, and the correction target image is generated by changing the pixel thinning rate between the central portion and the peripheral portion of the screen.

  9 and 10 show an outline of non-equal pitch pixel sampling in the second imaging mode. 9 and 10 show only effective pixels in the quadrant of the image sensor 102. FIG. Of the effective pixels, the hatched pixels are pixels from which data is read (sampled). A range surrounded by a thick line is a range in which pixel data is read. The effective number of pixels of the image sensor 102 is 24 × 24 × 4 = 2304 pixels in FIG. 9, and 12 × 12 × 4 = 576 pixels in FIG.

  In FIG. 9, in the central part from the center (on the axis) to 80% image height, a pixel thinning rate (sampling pitch is 4 pixels) for reading one pixel every 4 pixels in the horizontal and vertical directions is set. Yes. On the other hand, in the peripheral portion of 80% image height or higher, a pixel thinning rate (sampling pitch is 4 pixels in the horizontal direction and vertical), in which 1 pixel is read out every 4 pixels in the horizontal direction and 2 pixels are read out in every 4 pixels in the vertical direction. 2 pixels in the direction) is set. Thereby, the Nyquist frequency of the image sensor 102 is controlled.

  At this time, the number of pixels used when generating the correction target image is 12 × 12 × 4 = 1152 pixels, and a region surrounded by a double line in FIG. 9 corresponds to one pixel. Here, the pixel value interpolation method for the non-data pixels is the same as in the first embodiment. At this time, the number of pixels of the low resolution image as the output image is 6 × 6 × 4 = 144 pixels.

  In FIG. 10, in the central portion from the center (on the axis) to 80% image height, a pixel thinning rate (sampling pitch is 2 pixels) is set every other pixel in the horizontal and vertical directions. Further, in the peripheral portion where the image height is 80% or higher, the pixel thinning rate is set to zero (sampling pitch is one pixel), and all pixels are read out. Thereby, the Nyquist frequency of the image sensor 102 is controlled.

  At this time, the number of pixels used when generating the correction target image is 12 × 12 × 4 = 1152 pixels, and a region surrounded by a double line in FIG. 10 corresponds to one pixel. Here, the pixel value interpolation method for the non-data pixels is the same as in the first embodiment. At this time, the number of pixels of the low resolution image as the output image is 6 × 6 × 4 = 144 pixels.

  The pixel data readout area shown in FIG. 10 is ½ the pixel data readout area from the image sensor 102 shown in FIG. 9, but the number of pixels of the output low-resolution image is the same at 144 pixels. It is. For this reason, by switching between the above two processes, a double electronic zoom magnification can be obtained.

  Each embodiment described above is only a representative example, and various modifications and changes can be made to each embodiment in carrying out the present invention.

  An imaging apparatus such as a digital still camera or a video camera that can output a high-quality image that has been satisfactorily corrected for distortion can be provided.

DESCRIPTION OF SYMBOLS 101 Image pick-up optical system 102 Image pick-up element 104 Image processing part 110 System controller 120 Reading position control part

Claims (8)

An image sensor that photoelectrically converts an optical image formed by the optical system;
Image generation means for generating an image by sampling the output of effective pixels of the image sensor and performing distortion correction processing for reducing a component corresponding to distortion aberration of the optical system included in the image; ,
The image generating means includes
When generating an output image having a predetermined number of pixels less than the total number of effective pixels of the image sensor, the output of the effective pixels having a number greater than the predetermined number of pixels is sampled, and the center and the periphery of the image sensor A correction target image is generated so as to be equivalent to a case where a different sampling pitch is used or a different sampling pitch is used.
Performing the distortion correction process on the correction target image;
An imaging apparatus, wherein the output image is generated from a result image after the distortion correction processing. The imaging apparatus according to claim 1, wherein the number S1 of effective pixels whose output is sampled when the image generation unit generates the correction target image satisfies the following condition.
S1 = Se / Lm ²
However, Se is the total number of effective pixels of the image sensor, and Lm is the minimum value of the sampling pitch.   The imaging apparatus according to claim 1, wherein a sampling pitch at the peripheral portion when the image generation unit generates the correction target image is smaller than a sampling pitch at the central portion.   4. The imaging apparatus according to claim 1, wherein the image generation unit generates the correction target image by changing a pixel thinning rate between the central portion and the peripheral portion. 5.   4. The image generation unit according to claim 1, wherein the image generation unit generates the correction target image by changing the number of pixels to be mixed in the central portion and the peripheral portion. 5. Imaging device.   The image generation means, as the distortion correction processing for the correction target image, performs coordinate conversion processing using information related to distortion aberration of the optical system and output information sampled from the effective pixels. Item 6. The imaging device according to any one of Items 1 to 5. An image pickup apparatus having an image pickup device that photoelectrically converts an optical image formed by the optical system generates an image by sampling the output of effective pixels of the image pickup device, and supports distortion aberration of the optical system included in the image An imaging method for performing distortion correction processing for reducing components to be performed,
In the imaging device,
Sampling a correction target image having a predetermined number of pixels less than the total number of effective pixels of the image sensor, sampling outputs of the effective pixels greater than the predetermined number of pixels, and a center portion and a peripheral portion of the image sensor Generating at a different sampling pitch or equivalent to using a different sampling pitch;
Performing the distortion correction process on the correction target image;
And a step of generating the output image from the result image after the distortion correction processing. An image pickup apparatus having an image pickup device that photoelectrically converts an optical image formed by the optical system generates an image by sampling the output of effective pixels of the image pickup device, and supports distortion aberration of the optical system included in the image An imaging program as a computer program executable by a computer of the imaging apparatus that causes distortion correction processing to reduce components to be performed,
In the imaging device,
Sampling a correction target image having a predetermined number of pixels less than the total number of effective pixels of the image sensor, sampling outputs of the effective pixels greater than the predetermined number of pixels, and a center portion and a peripheral portion of the image sensor Generating at a different sampling pitch or equivalent to using a different sampling pitch;
Performing the distortion correction process on the correction target image;
And a step of generating the output image from the result image after the distortion correction processing.