JP5487955B2 - Imaging apparatus and imaging method - Google Patents

Description

  The present invention relates to an imaging apparatus and an imaging method for taking an image.

  In fields such as surveillance cameras, measurement, and medical use, there are many requests for focusing on every corner of the object scene. In order to extend the depth of field (EDOF: Extension of Depth of Field), the F value (focal length / lens aperture) may be increased, and the lens aperture may be simply reduced. However, when the lens diameter is reduced, the spatial resolution is lowered, the amount of light passing through the lens is reduced, and the image becomes dark.

  On the other hand, in recent years, a technique called WFC (Wavefront Coding) has attracted attention. A specific example of WFC is a combination of a phase modulation element that modulates the phase of transmitted light and an image processing technique for digitized image data. WFC is a technology that expands the depth of field without sacrificing spatial resolution or the amount of transmitted light, and can expand the depth of field without reducing the lens aperture. In addition, an imaging optical system that is not easily affected by chromatic aberration and the like can be realized, and a so-called optical zoom lens mechanism can be eliminated, so that the number of lenses can be reduced, and the mechanism and assembly adjustment can be simplified.

  FIG. 12 is a diagram for explaining the outline of the WFC. The optical system 10 includes lenses 11 to 13, and a phase modulation element 10 a to which WFC is applied is inserted between the lens 12 and the lens 13.

  By inserting the phase modulation element 10a, the imaging characteristics of the original optical system having only the lenses 11 to 13 are changed, and an analog image (intermediate image) formed in the state of the optical system 10 including the phase modulation element 10a. Image).

  Since a normal optical system is optically designed so that aberrations are sufficiently corrected, if an extra phase modulation element is inserted, the imaging performance is lowered, and the intermediate image becomes a deteriorated image. At this time, if a phase modulation element whose degree of deterioration does not change with the amount of defocus can be selected, an intermediate image in which the same deteriorated images are arranged along the optical axis can be obtained.

  The phase modulation element 10a is a third-order phase modulation element having a phase distribution in which the degree of deterioration is less changed depending on the defocus amount, and the phase distribution is expressed by the following expression (1). α is a constant, and x and y are position coordinates in the optical axis direction. It is known that the case of having a third order coefficient increases the effect of EDOF.

φ (x, y) = α (x ³ + y ³ ) (1)
The intermediate image deteriorated by the phase modulation element 10a is picked up by the subsequent image pickup element 20 and converted into a digital image. The image processing unit 30 performs digital image processing using a deconvolution filter (inverse filter), removes the deteriorated component from the deteriorated intermediate image, and generates a high-quality final image.

  In WFC, the degree of deterioration of the intermediate image varies little depending on the amount of defocus, and the degree of deterioration becomes known. For this reason, the inverse filter processing can be easily performed, and a clear image restored to the diffraction limit can be obtained regardless of the defocus amount.

  As a conventional technique, a technique for taking an image by expanding the depth of field by WFC has been proposed (for example, Patent Document 1 and Non-Patent Document 1). Further, a technique for generating a restored image even when the center of gravity of an image restoration filter is shifted has been proposed (for example, Patent Document 2). Furthermore, a technique for suppressing image quality deterioration by performing data correction based on an impulse response of an inverse filter has been proposed (for example, Patent Document 3).

JP 2003-235794 A JP 2007-267279 A JP 2004-24659 A

E. R. Dowski and W.T. Cathey, "Extended Depth of Field Through Wavefront Coding", Applied Optics, vol. 34, no 11, pp. 1859-1866, April, 1995

  Image degradation such as image blur is often expressed by a PSF (Point Spread Function). The PSF represents the response of the optical system to the point light source, and is a function representing how much the point image spreads when an ideal point image passes through the optical system.

  For example, when one point of the sample is projected on the screen through the optical system, the spot does not become the same size as the original point due to the diffraction of the lens, but a blurred and spread spot is projected. This spot is an image generated by the PSF, and the blurred image displayed at this time appears as an image obtained by multiplying the original image information by the PSF (convolution).

In order to return the image spread according to the PSF to the original image, an inverse operation, that is, division by PSF (inverse matrix calculation of PSF) may be performed on the obtained image.
Specifically, as inverse filtering (deconvolution filtering), division is performed by an OTF (Optical Transfer Function) representing the spatial frequency transfer characteristics of the imaging optical system obtained by Fourier transforming PSF.

  However, when the OTF has a very small spatial frequency or the like, noise is amplified, and therefore, an appropriate filter condition or inverse function is selected to perform inverse filtering.

  On the other hand, after passing through the phase modulation element, a phenomenon occurs in which the position of an image whose shape is expanded by PSF (hereinafter also simply referred to as PSF) is shifted from the center of the imaging surface to the peripheral side. That is, the PSF imaging that is projected through the optical system including the phase modulation element having the WFC characteristic and the position on the imaging surface of the PSF that is projected through the normal optical system that does not include the phase modulation element. The position on the surface means that a displacement occurs.

  Therefore, when a subject is photographed with an optical system including a phase modulation element, if the subject image is located at the end of the frame (effective imaging area), the frame is out and the image data is lost. There was a problem.

  13 and 14 are diagrams for explaining the frame-out phenomenon. FIG. 13 shows a state in which a subject image 6a of a certain subject is located at the upper left corner of the frame 20a and a subject image 6b of a certain subject is located at the lower right corner. A dotted line frame in the drawing indicates a region where the subject images 6a and 6b shift when the image is taken with an optical system including a phase modulation element.

  FIG. 14 shows the positions of the subject images 6a and 6b when the image is taken by the optical system 10 including the phase modulation element 10a. The subject images 6a and 6b are shifted to the shift area (for example, shifted in the arrow direction).

  In this case, since the subject image 6a is in the frame 20a, the subject image 6a is captured and the data of the subject image 6a can be acquired. However, since a part of the subject image 6b goes out of the frame 20a, the data of the subject image 6b is lost.

  That is, since there is no image pickup device of the frame 20a at the shifted position, the corresponding data is lost. If image processing is performed using image data lacking such data, image degradation will be noticeable.

  To deal with the above problems, it may be possible to take countermeasures with the existing camera shake correction function such as shifting the image pickup element of the frame 20a immediately before the shutter is released, but it is completely possible that data will be lost even with the camera shake correction function. It is impossible to prevent.

  FIG. 15 is a diagram illustrating a state in which data loss occurs even when camera shake correction is performed. The image pickup device itself of the frame 20a shifts in the direction in which the subject images 6a and 6b shift by the camera shake correction function at the moment of shooting.

  In this case, since the subject image 6b fits in the frame 20a-1 after being shifted by camera shake correction, the data of the subject image 6b remains, but this time, the subject image 6a that was visible in the viewfinder is out of the frame 20a-1. This results in data loss of the subject image 6a. Thus, even with the camera shake correction function, data loss cannot be prevented.

The present invention has been made in view of these points, and an object of the present invention is to provide an imaging apparatus that performs high-quality imaging while preventing data loss caused by frame-out.
Another object of the present invention is to provide an imaging method for performing high quality imaging by preventing data loss caused by frame-out.

In order to solve the above problems, an imaging apparatus is provided. The imaging device includes an optical system including a phase modulation element, an imaging element that captures a subject image that has passed through the optical system, and an image processing unit that performs image processing on the subject image. A first imaging area of the area and a second imaging area of the non-imaging display area, and the image processing unit captures the subject image captured in the first imaging area and the second imaging area. The second imaging area of the imaging element is an area for imaging the portion of the subject image that protrudes from the first imaging area, the imaging position of which is shifted by the phase modulation element. Further, this is an area used for image reproduction correction of the subject image that protrudes from the first imaging region, and corresponds to the vertical and horizontal shift amounts of the subject image determined by the phase distribution function of the phase modulation element. Shi Area is provided.

  Data loss is prevented and image deterioration is suppressed, and high-quality shooting becomes possible.

It is a figure which shows the structural example of an imaging device. It is a figure which shows a phase distribution function. It is a figure which shows the optical path by the normal optical system which does not contain a phase modulation element. It is a figure which shows the optical path by the optical system containing a phase modulation element. It is a figure which shows the relationship between a view angle and the shift amount of a Y-axis direction. It is a figure which shows the relationship between a view angle and the shift amount of a Y-axis direction. It is a figure which shows an imaging area. It is a figure which shows an imaging area. It is a figure which shows the structural example of an imaging device. It is a figure for demonstrating trimming control. It is a flowchart which shows operation | movement of an imaging device. It is a figure for demonstrating the outline | summary of WFC. It is a figure for demonstrating a flame-out phenomenon. It is a figure for demonstrating a flame-out phenomenon. It is a figure which shows the state which data loss produces when camera shake correction is performed.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a diagram illustrating a configuration example of an imaging apparatus. The imaging device 1 includes an optical system 10, an imaging element 2, and an image processing unit 3.
The optical system 10 includes a plurality of lenses (not shown) and forms a subject image. A phase modulation element 10a is inserted between the lenses. The phase modulation element 10a is an element that reduces the change in the optical transfer function according to the focal length.

  The image sensor 2 can use a CCD (Charged Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor) sensor, or the like, and captures the subject image that has passed through the optical system 10 and converts the analog image into a digital image. The image processing unit 3 performs image processing on the subject image.

  Here, the imaging element 2 shifts the imaging position by passing through the phase modulation element 10a and the first imaging area (hereinafter referred to as an effective imaging area 2a) that is captured and displayed and is not captured and displayed. And a second imaging area (hereinafter referred to as a correction imaging area 2b) for imaging a portion of the subject image that protrudes from the effective imaging area 2a. The image processing unit 3 performs image processing of subject images captured in the effective imaging area 2a and the correction imaging area 2b.

  Next, before describing the operation of the imaging apparatus 1, the reason why the PSF shifts by passing through the phase modulation element 10a and the relationship between the angle of view and the shift amount will be described. The PSF shift after passing through the phase modulation element 10a is due to the shape of the phase modulation element 10a.

  FIG. 2 is a diagram showing a phase distribution function. Since the expression (1) of the phase distribution function described above can be considered as the thickness of the shape of the phase modulation element 10a, if the thickness is Z, the following expression (1a) is obtained. Therefore, FIG. 2 can also be seen as a schematic view of the surface shape of the phase modulation element 10a.

Z = α (X ³ + Y ³ ) (1a)
The phase modulation element 10a is an element that emits a phase-modulated incident light by having a thickness having a shape change with respect to the X-axis and the Y-axis as in the formula (1a). For this reason, a difference occurs between the optical path length that passes through the portion where the thickness of the element is large and the optical path length that passes through the portion where the thickness is small. This will cause a phenomenon of misalignment.

  Next, how much misalignment occurs will be described with a specific example. FIG. 3 is a diagram showing an optical path by a normal optical system that does not include the phase modulation element 10a. The normal optical system 10-1 includes lenses 11 to 13, and light that has passed through these lenses is imaged on the image sensor 2. As photographing conditions by the optical system 10-1, the focal length is 6 mm, the F value = 3.5, and the maximum angle of view (half value) = 26.5 degrees.

  The F value is a value obtained by dividing the focal length of the lens by the effective aperture, and is an index indicating the brightness of the lens. Further, the angle of view is a range displayed on the image sensor 2. The smaller the focal length, the wider the angle of view, and the larger the focal length, the narrower the angle of view. Therefore, if the focal length is changed, the desired range can be selected.

  FIG. 4 is a diagram showing an optical path by the optical system 10 including the phase modulation element 10a. The optical system 10 includes lenses 11 to 13 and a phase modulation element 10a. A phase modulation element 10 a is inserted between the lens 12 and the lens 13, and light that has passed through the optical system 10 is imaged on the imaging element 2 under the same imaging conditions as those in FIG. 3.

  Comparing FIG. 3 and FIG. 4, it can be seen that in the optical system 10 including the phase modulation element 10a, the imaging position is shifted on the Y axis. Although FIG. 4 shows only the shift of the Y axis, the imaging position is similarly shifted with respect to the X axis.

  5 and 6 are diagrams showing the relationship between the angle of view and the shift amount in the Y-axis direction. The horizontal axis is the angle of view (degree), and the vertical axis is the shift amount (mm). A solid line indicates the shift amount of the normal optical system 10-1, and a dotted line indicates the shift amount of the optical system 10 including the phase modulation element 10a. FIG. 6 is an enlarged view of the state in the circle of FIG.

  Angle of view 26.5 deg. Then, a difference of 90 μm occurs. Thus, the larger the angle of view, the greater the shift amount in the Y-axis direction. That is, it can be seen that the positional shift phenomenon appears more significantly as the angle of view increases.

  Next, the imaging area and operation of the imaging apparatus 1 will be described below. FIG. 7 is a diagram showing an imaging region. The imaging element 2 is further provided with a correction imaging area 2b with respect to the effective imaging area 2a. The effective imaging area 2a is a normal imaging area that is used for shooting a subject and is displayed as a finder for the user.

  The correction imaging area 2b is not used for actual photographing of a subject, and is an area for capturing a subject image that protrudes from the effective imaging area 2a without using a finder display and using it for calculation of image reproduction correction. is there.

  For this reason, an area corresponding to the shift amount of the subject image in the vertical and horizontal directions is provided. The shift amount, that is, the area can be determined by a function (the above-described equation (1a)) of the phase modulation element 10a, and a necessary amount can be calculated from this function.

  Here, when the subject is photographed with the optical system 10 including the phase modulation element 10a, it is assumed that the subject images 5a and 5b are respectively shifted in the arrow directions. In this case, since the subject image 5a is captured in the effective imaging region 2a even after the shift, the image processing unit 3 can acquire the data of the subject image 5a.

  Further, although the subject image 5b protrudes outside the effective imaging region 2a, since the correction imaging region 2b is provided so that the subject image 5b is captured, the correction is performed on the correction imaging region 2b. The subject image 5b exists.

  Therefore, the data of the subject image 5b is not lost, and the image processing unit 3 can acquire the data of the subject image 5b. In this way, the image processing unit 3 acquires the image data of the subject image captured in the effective imaging area 2a and the correction imaging area 2b and performs image processing.

  FIG. 8 is a diagram showing an imaging region. In FIG. 7, the correction imaging region 2 b is provided at a location where the portion of the subject image that protrudes from the effective imaging region 2 a can be captured in the direction in which the imaging position shifts by passing through the phase modulation element 10 a.

  On the other hand, in the case of FIG. 8, a correction imaging region is also provided in a region for capturing a portion of the subject image that protrudes outside the effective imaging region 2 a by performing camera shake correction. That is, as shown in the figure, correction image pickup areas 2b-1 are provided in four directions of the effective image pickup area 2a. As a result, it is also possible to acquire data of a subject image that is out of frame by camera shake correction.

  As described above, the imaging apparatus 1 is used for correction to capture the portion of the subject image that protrudes from the effective imaging region 2a by shifting the imaging position by passing through the effective imaging region 2a and the phase modulation element 10a. The imaging area 2b is configured to perform image processing on the subject image captured in the effective imaging area 2a and the correction imaging area 2b. With such a configuration, data loss caused by frame-out can be prevented, and high-quality imaging with suppressed image deterioration is possible.

  The correction imaging area 2b is different from the blanking area that the imaging element 2 originally has. The blanking area has only a few columns (a width of several μm to several tens of μm depending on the pitch) at the edge of the image sensor, so the area is not sufficient, and basically only one of the X direction and the Y direction is provided. . Further, it is a spare area for performing dark current correction and the like, not used for imaging. Therefore, it is unsuitable to use the blanking area for the correction imaging region 2b having the characteristics as described above.

  Next, the test imaging unit included in the imaging apparatus 1 will be described. FIG. 9 is a diagram illustrating a configuration example of the imaging apparatus. The imaging device 1a includes an optical system 10, an imaging device 2, an image processing unit 3, and a test imaging unit 4. Since the optical system 10, the image pickup device 2, and the image processing unit 3 have been described above, the test photographing unit 4 will be described.

  The test photographing unit 4 performs a test photographing of a pseudo point light source (PSF) via the optical system 10, measures the amount of positional deviation of the point light source photographed by the test photographing, and outputs a measurement value. Here, the image sensor expansion amount for the correction imaging region 2b is almost determined from the above-described function, but actually, an element change, manufacturing / mounting error, and the like are also taken into consideration.

  Therefore, in order to suppress image degradation due to data loss with higher accuracy, the test photographing unit 4 performs pseudo-point light source photographing under conditions such that the aperture is almost closed and the gain is maximum and the exposure is maximum. To measure the amount of positional deviation of the point light source and output the measured value. And this measured value is used as a correction coefficient at the time of an inverse conversion process. Thus, by providing the test photographing unit 4 and actually measuring the positional deviation, it is possible to efficiently suppress image deterioration due to data loss for each individual product.

  The test photographing unit 4 measures not only the amount of positional deviation of the point light source in the horizontal direction (X-axis direction) and the vertical direction (Y-axis direction) but also the rotation amount in the height direction (Z-axis direction). Therefore, it becomes possible to measure the positional deviation with higher accuracy.

  Next, trimming control will be described. FIG. 10 is a diagram for explaining the trimming control. In the imaging apparatus 1, in order to prevent data loss caused by a point image position shift phenomenon caused by passing through the phase modulation element 10a, the effective imaging area 2a is not shifted, but the effective imaging area 2a is corrected. The image pickup area 2b is newly added as an image pickup area.

  In this case, in the area diagonal to the direction in which the subject image shifts, the data on the effective image pickup area 2a shifts, so that an area (data non-area) r1 in which no data exists (black in the figure) appears. region).

  If such a non-data area r1 exists, the aspect ratio changes after shooting, which may adversely affect printout or external monitor display, resulting in image degradation. For this reason, the image processing unit 3 performs control for automatically trimming (cutting out) the non-data area r1 after image processing of the subject image. As a result, it is possible to suppress image degradation that may occur during printout or external monitor display.

FIG. 11 is a flowchart showing the operation of the imaging apparatus 1.
[S1] The effective imaging area 2a is displayed in a finder.
[S2] The subject is photographed.

  [S3] In addition to the subject image being captured in the effective imaging region 2a, when the subject image is shifted by the phase modulation element 10a and protrudes from the effective imaging region 2a, the protruding portion is the correction imaging region 2b. Is imaged.

  [S4] The image processing unit 3 determines how far the correction imaging area 2b protrudes (how far the data is dispersed) by taking into account the phase distribution function of the phase modulation element 10a and the shift amount for camera shake correction. recognize.

[S5] The image processing unit 3 acquires and buffers data captured in all areas of the effective imaging area 2a and data captured in the correction imaging area 2b.
[S6] The image processing unit 3 performs an inverse filter process (inverse transform process) on the buffered image data.

[S7] The image processing unit 3 performs trimming control.
[S8] The image processing unit 3 displays the image data after the inverse filter processing and the trimming as a frame.

(Supplementary note 1) an optical system including a phase modulation element;
An image sensor that images a subject image that has passed through the optical system;
An image processing unit for image processing the subject image;
With
The imaging element has a first imaging area of a shooting display area and a second imaging area of a non-shooting display area,
The image processing unit performs image processing of the subject image captured in the first imaging region and the second imaging region;
An imaging apparatus characterized by that.

  (Supplementary Note 2) The second imaging area of the imaging element is an area that captures a portion of the subject image that protrudes from the first imaging area and whose imaging position is shifted by the phase modulation element. The imaging apparatus according to appendix 1, which is characterized.

  (Supplementary Note 3) The second imaging area of the imaging element is an area used for image reproduction correction of the subject image that protrudes from the first imaging area, and the phase distribution function of the phase modulation element The imaging apparatus according to claim 2, wherein an area corresponding to a shift amount in the vertical and horizontal directions of the subject image determined by is provided.

  (Supplementary Note 4) The second imaging area of the imaging element is provided in an area for imaging a portion of the subject image that protrudes outward from the first imaging area due to camera shake correction. The imaging apparatus according to appendix 1.

(Additional remark 5) It further has a test imaging | photography part,
The test photographing unit performs a test photographing of a pseudo point light source through the optical system, measures a positional deviation amount of the point light source photographed by the test photographing, and outputs a measured value. The imaging apparatus according to 1.

  (Additional remark 6) The said test imaging | photography part measures the shift amount of the vertical and horizontal direction of the said point light source, and the rotation inclination of a height direction, and measures the positional offset amount of the said point light source, The additional characteristic 5 characterized by the above-mentioned. Imaging device.

  (Additional remark 7) The said image process part trims the area | region where the data which arise in the said 1st imaging area do not exist, when the said to-be-photographed image is image | photographed via the said optical system. The imaging device described.

(Supplementary Note 8) In the imaging method,
For the imaging device, a second imaging area of the non-imaging display area is separately provided in the first imaging area of the imaging display area,
The subject image that has passed through the optical system including the phase modulation element is captured by the imaging element,
Image processing of the subject image imaged in the first imaging area and the second imaging area;
An imaging method characterized by the above.

DESCRIPTION OF SYMBOLS 1 Imaging device 2 Imaging element 2a 1st imaging area 2b 2nd imaging area 3 Image processing part 10 Optical system 10a Phase modulation element

Claims (4)

An optical system including a phase modulation element;
An image sensor that images a subject image that has passed through the optical system;
An image processing unit for image processing the subject image;
With
The imaging element has a first imaging area of a shooting display area and a second imaging area of a non-shooting display area,
The image processing unit may have a row image processing of the subject image captured by the first imaging region and the second imaging region,
The second imaging area of the imaging element is an area for imaging a portion of the subject image that protrudes from the first imaging area, the imaging position of which is shifted by the phase modulation element. An area for use in image reproduction correction of the subject image that protrudes from the imaging region, and is provided with an area corresponding to a vertical and horizontal shift amount of the subject image determined by a phase distribution function of the phase modulation element.
An imaging apparatus characterized by that.   A test shooting unit;
  The test photographing unit performs a test photographing of a pseudo point light source through the optical system, measures a positional deviation amount of the point light source taken by the test photographing, and outputs a measurement value. Item 2. The imaging device according to Item 1.   The imaging according to claim 1, wherein the image processing unit trims an area where no data is generated in the first imaging area when the subject image is captured through the optical system. apparatus.   In the imaging method,
  For the imaging device, a second imaging area of the non-imaging display area is separately provided in the first imaging area of the imaging display area,
  The subject image that has passed through the optical system including the phase modulation element is captured by the imaging element,
  Performing image processing of the subject image imaged in the first imaging area and the second imaging area;
  The second imaging area of the imaging element is an area for imaging a portion of the subject image that protrudes from the first imaging area, the imaging position of which is shifted by the phase modulation element. An area for use in image reproduction correction of the subject image that protrudes from the imaging region, and is provided with an area corresponding to a vertical and horizontal shift amount of the subject image determined by a phase distribution function of the phase modulation element.
  An imaging method characterized by the above.

Images