JP5633218B2 - Image processing apparatus and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus that executes processing on an image captured by an optical system.

  Conventionally, various image processing methods have been proposed as a method for removing blur from an image captured by an optical system (see, for example, Patent Documents 1 to 3). Japanese Patent Application Laid-Open No. 2004-228561 discloses that, in the process of reducing color blur on a color image, the color blur to be reduced and the method of estimating the amount of color blur are changed according to the image characteristics and imaging conditions. Has been. Patent Document 2 discloses a method for deconvolution of an image by a Bayer array image sensor. Patent Document 3 discloses a three-dimensional image acquisition apparatus that uses a biological material that is nearly transparent as a sample and obtains a three-dimensional image from the amount of attenuation of sample transmitted light.

JP 2009-124598 A JP 2009-89082 A Japanese Patent No. 4236123

  In the above prior art, processing based on image restoration filtering according to the PSF (Point Spread Function) of the optical system is executed. However, the PSF of an optical system often varies depending on the angle (direction) of incident light. In the above prior art, it is difficult to perform image restoration filtering that can cope with a change in PSF due to the angle of incident light.

  Therefore, an object of the present invention is to enable image processing that can cope with a change in PSF due to an angle of incident light.

An image processing apparatus disclosed in the present application is an image processing apparatus that processes an image captured through an optical system, and includes a filter processing unit that performs filtering on the image according to a transfer function of the optical system. The filter processing unit corresponds to a transfer function of light incident in the direction of the optical axis of the optical system, with a coefficient corresponding to the distance from the position corresponding to the optical axis center in the image to the position to be processed. By linearly combining the first filter and the second filter corresponding to the transfer function of light incident at a certain angle with respect to the optical axis, different filter data depending on the position of the processing target in the image is obtained. And performing filtering using the generated filter data .

  According to the disclosure of the present specification, it is possible to perform image processing that can cope with a change in PSF depending on an angle of view.

FIG. 1 is a functional block diagram illustrating a schematic configuration of an imaging apparatus including an image processing apparatus according to the first embodiment. FIG. 2 is a diagram illustrating an example of a path of light incident on the optical system. (A), (b), and (c) of FIG. 3 show calculation examples of the PSF of the light beam shown in (a), (b), and (c) of FIG. 2, respectively. FIGS. 4A, 4B, and 4C show calculation examples of the MTF (Modulation Transfer Function) of the light beam shown in FIGS. 2A, 2B, and 2C, respectively. FIG. 5 is a functional block diagram illustrating an example of a detailed configuration of the image processing apparatus. FIG. 6 is a flowchart illustrating an operation example of the filter processing unit. FIG. 7 is a diagram illustrating an example of the deconvolution distribution in the image area. FIG. 8 is a diagram for explaining the deconvolution distribution. FIG. 9 is also a diagram for explaining the deconvolution distribution. FIG. 10 is a flowchart illustrating an example of reference distribution calculation. It is a figure which shows the example of the optical axis center light beam in an optical system, and an outermost field angle light beam. FIG. 12 is a flowchart illustrating an operation example of the filter processing unit according to the second embodiment. FIG. 13 is a diagram illustrating an example of a plurality of zones preset in an image. FIG. 14 is a diagram illustrating another example of a plurality of zones preset in an image.

(First embodiment)
[Configuration example of imaging device]
FIG. 1 is a functional block diagram illustrating a schematic configuration of an imaging apparatus including an image processing apparatus according to the first embodiment. An imaging apparatus shown in FIG. 1 includes an optical system 1 that condenses light from a subject K on an image plane, an imaging element 2 that converts light from the subject collected by the optical system 1 into an electronic signal (analog signal), An AFE (Analog front end) 3 that converts an analog signal of a captured image into a digital signal, an image processing device 4 that includes a function of executing a filtering process on an image of the digital signal, and an image that has undergone a filtering process are further necessary A post-processing unit 5 that performs processing to generate a display image and a display unit 9 that displays an image generated by the post-processing unit 5 are provided. The imaging apparatus further includes a drive control device 6 that controls the optical system 1 and a control device 7 that controls the AFE 3, the image processing device 4, and the post-processing device 5.

  The optical system 1 includes lenses 11 a, 11 b, 11 c and a diaphragm 12. The lenses 11a, 11b, 11c, and the diaphragm 12 focus the light from the subject K on the image pickup surface of the image pickup device 1 to form an image of the subject. The drive control device 6 can control the positions of the lenses 11 a, 11 b, 11 c, the diaphragm degree of the diaphragm 12, and the like. The configuration of the optical system 1 is not limited to a specific one. The image pickup device 2 includes, for example, a two-dimensional image pickup device such as a CCD / CMOS. The two-dimensional image pickup device converts an image of a subject into an electronic signal (image signal) and outputs the electronic signal to the AFE 3.

  The AFE 3 includes an A / D (analog / digital) converter 31 and a timing generator 32. The timing generator 32 generates a timing pulse used for driving the image sensor 2 based on a control signal from the control device 7 and outputs the timing pulse to the image sensor 2 and the A / D converter 31.

  The image processing apparatus 4 includes a RAW memory 41 that records an image (RAW image) converted into a digital signal by the A / D converter 31, and a transfer function (PSF or OTF (Optical Transfer) of the optical system 1 for the image. Filter processing unit 42 that performs filtering using filter data according to Function)). The image filtered by the filter processing unit 42 is recorded in the image memory 8.

  The filter processing unit 42 performs filtering at each position using different filter data depending on the position to be processed in the image. Thus, by changing the filter according to the position in the image, it is possible to perform filtering corresponding to the difference in OTF due to the difference in the incident angle of light in the optical system.

  For example, the filter processing unit 42 corrects the filter data indicating the filter corresponding to the transfer function of the optical system 1 recorded in advance according to the position of the processing target in the image, and uses the corrected filter data. It is possible to perform filtering at a position to be processed in the image. This makes it possible to change the filter flexibly according to the position in the image while suppressing the amount of filter data recorded in advance. A detailed example of the filtering process will be described later.

  The filter data can be a set of parameters necessary for filtering, such as a deconvolution kernel. Specifically, the deconvolution kernel is represented by a region in which an image of a circular subject corresponding to the PSF is distributed and data representing the weight of each pixel in the region (such data is referred to as a deconvolution distribution). it can. The reference distribution can be calculated based on the PSF of the optical system 1, for example. The file data is preferably recorded in advance in a memory inside or outside the filter processing unit.

  The post-processing unit 5 reads out the filtered image from the image memory 8, performs a process other than filtering, generates an image for display, and outputs the image to the display unit 9. The display unit 9 includes, for example, a VRAM that records an image and a display that outputs an image of the VRAM. Note that the imaging apparatus does not necessarily have a display function, and instead of the display unit 9, a recording unit (for example, a VRAM) that records an image for display may be provided.

  The image processing device 4 can be constituted by, for example, a DSP (Digital Signal Processor). In this case, the RAW memory 41 may be a DSP built-in memory or an external memory. Further, the post-processing unit 5, the image memory 8, the VRAM for display, and the like may be configured by an integrated DSP together with the image processing device 4, or only the filter processing unit of the image processing device 4 is configured by one DSP. May be. Alternatively, the function of the image processing apparatus 4 can be realized by a predetermined processor executing a predetermined program instead of a processor dedicated to specific processing such as a DSP. Similarly, the drive control device 6, the control device 7, and the post-processing unit 5 can be configured by at least one specific processing dedicated processor or general-purpose processor.

  Note that a program that causes a processor to function as the image processing apparatus 4 and a recording medium that records the program are also included in the embodiments of the present invention. This recording medium is non-transitory and does not include a transient medium such as the signal itself.

[Change in transfer function depending on incident angle of light in optical system 1]
FIG. 2 is a diagram illustrating an example of a path of light incident on the optical system 1. In FIG. 2, (a) is a light beam that is incident in substantially the same direction as the optical axis of the optical system 1, that is, a light beam that is incident in a direction perpendicular to the image plane, and (b) is a fixed angle with respect to the optical axis. (C) indicates the light flux when the angle with respect to the optical axis is maximized.

  (A), (b), and (c) in FIG. 3 show calculation examples of the PSF of the light beam shown in (a), (b), and (c) in FIG. 2, respectively. In the example shown in FIG. 3, the PSF distribution range increases as the angle of the light beam with respect to the optical axis increases. Thus, when the angle of light incident on the optical system 1 changes, the PSF also changes.

  FIGS. 4A, 4B, and 4C show calculation examples of the MTF (Modulation Transfer Function) of the light beam shown in FIGS. 2A, 2B, and 2C, respectively. In the example shown in FIG. 4, the response deteriorates as the angle of the light beam with respect to the optical axis increases.

  Thus, when the angle of light incident on the optical system 1 changes, the light transfer function changes. In the present embodiment, the filter processing unit 42 of the image processing apparatus 4 changes the filter to be applied according to the position in the image. Therefore, filtering that can cope with a change in transfer function due to a change in the angle of light is possible.

[Specific examples of filtering processing]
Here, a specific example of the filtering process by the filter processing unit 42 will be described. The following processing is an example, and the operation of the filter processing unit 42 is not limited to the following example. FIG. 5 is a functional block diagram illustrating an example of a detailed configuration of the image processing apparatus 4. In the image processing device 4 illustrated in FIG. 5, the filter processing unit 42 includes a convolution operation unit 44 and a convolution control unit 45. The convolution operation unit 44 uses the filter data stored in the memory 43 included in the image processing device 4 to execute a deconvolution process on the image stored in the RAW memory 41. The convolution control unit 45 controls the operation of the convolution calculation unit 44. The memory 43 is a register that records, for example, filter data including coefficients corresponding to the reference distribution and the position in the image.

  FIG. 6 is a flowchart illustrating an operation example of the filter processing unit 42. In the example illustrated in FIG. 6, when the convolution operation unit 44 acquires an image from the RAW memory 41 (Op1), the convolution control unit 45 determines a processing target position in the acquired image, that is, a position to be filtered. (Op2). For example, the processing target position can be sequentially determined by selecting pixels or local regions in the order of raster scanning. The processing target position can be represented by, for example, coordinates of a pixel or coordinates indicating a local area.

  When the processing target position is determined, the convolution control unit 45 calculates the distance from the optical axis center of the processing target position (Op3). The optical axis center can be a point on the image corresponding to the point where the optical axis of the optical system 1 intersects the imaging surface. The optical axis of the optical system 1 is often a rotationally symmetric axis of an optical element such as the lens 11 in many cases. In this case, the center of the optical axis can be the center pixel of the image.

  The convolution control unit 45 sets filter data for specifying a filter at the processing target position based on the distance from the optical axis center calculated in Op3 (Op4). Here, as an example, an example will be described in which the map 43 indicating the correspondence between the reference distribution A, the reference distribution B, and the processing target position and the coefficient is stored in the memory 43 in advance. The reference distribution A is an example of data indicating a first filter corresponding to a transfer function of light incident in the direction of the optical axis of the optical system 1, and the reference distribution B is incident at a certain angle with respect to the optical axis. It is an example of the data which shows the 2nd filter corresponding to the transfer function of the light which was performed. The map data of the processing target position and the coefficient is data indicating an example of the coefficient corresponding to the distance from the center of the optical axis to the processing target position.

  The deconvolution distribution is, for example, data representing a distribution that is restored to the original circle by performing a convolution operation on a captured image if the subject is a circle. In the present embodiment, deconvolution distributions in a plurality of reference regions (here, the optical axis center and the outermost position) in the image are recorded in advance in the memory 43 as reference distributions A and B. The sum of the reference distributions A and B multiplied by the coefficients a and b depending on the distance l is calculated as the deconvolution distribution. This deconvolution distribution is used for the deconvolution process as a deconvolution kernel (inverse filter) at a position 1 from the optical axis center. Thus, in this embodiment, the coefficients a and b of the reference distribution are determined by the distance l from the optical axis center to the processing target position.

  FIG. 7 is a diagram illustrating an example of the deconvolution distribution in the image region R. In the example shown in FIG. 7, the deconvolution distribution D2 at a position of the distance l from the optical axis center P and the rotation angle θ around the optical axis center is shown. The deconvolution distribution D1 shows a case where the distance is 1 and the rotation angle is 90 degrees, and the deconvolution distribution D3 shows a case where the distance is 1 and the rotation angle is 0 degrees. In this way, the deconvolution distribution can be treated as changing depending on the distance from the optical axis center P and the rotation angle around the optical axis center. As a result, the deconvolution distribution corresponding to the PSF change due to the change in the incident angle of light can be represented by data.

As shown in FIGS. 8 and 9, the deconvolution distribution includes a reference distribution A represented by a double ellipse and a reference distribution B represented by a larger ellipse, and a coefficient a = f depending on the distance l. ₁ (l) and b = f ₂ (l) can be expressed in a linearly coupled form. Also, as shown in FIG. 9, the deconvolution distribution can be a distribution that further considers the processing target position by being rotated according to the rotation angle θ of the processing target position.

The convolution control unit 45 refers to the map data of the processing target position and the coefficient in the memory 43, and acquires the values of the coefficients a and b corresponding to the distance l calculated in Op3. In this case, as map data, data indicating the correspondence between the distance l and the coefficients a and b is recorded in advance. For example, position 1, position 2, position 3, etc. associated with the coefficients (a1, b1), (a2, b2), (a3, b3),... Of the map data in the memory 43 shown in FIG. , Distances l ₁ , l ₂ , l ₃ ,...

  Then, by adding a product of the coefficient a and the reference distribution A to the product of the coefficient b and the reference distribution B, and applying a rotation process according to the rotation angle θ, a deconvolution distribution ( Deconvolution kernel) can be calculated. Thereby, a filter at the processing target position is determined.

  In the above example, the deconvolution distribution at the processing target position is calculated by correcting the reference distribution based on both the distance l and the rotation angle θ. However, at least one of the distance l and the rotation angle θ is calculated. It is also possible to set a filter according to the processing target position by correcting the reference distribution only.

  The deconvolution operation unit 44 executes the deconvolution process at the processing target position using the filter set in Op4 (Op5). The deconvolution process is a filtering process using, for example, a deconvolution distribution as a filter.

  The processes of Op2 to Op5 are repeated until all the regions R of the acquired image are completed (until determined to be Yes in Op6). Thereby, filtering processing using an appropriate filter is executed for the region R of the acquired image according to the position in the image.

  The filtering process is not limited to the above example. For example, in the filter setting process, deconvolution kernel data is recorded in advance for each of a plurality of positions so as to cover the entire image, and the deconvolution is performed according to the distance from the optical axis center P of the processing target position. Deconvolution processing may be performed by selecting kernel data. Alternatively, the value of the coefficient k is selected according to the distance from the optical axis center of the processing target position, and the result obtained by multiplying the previously recorded deconvolution kernel degradation data by the coefficient is obtained as deconvolution kernel reference data such as a reference distribution. You may subtract from This also makes it possible to generate filter data according to the processing target position.

[Reference distribution calculation example]
Here, an example of calculation of the reference distribution by a computer will be described. FIG. 10 is a flowchart illustrating an example of reference distribution calculation. In the example shown in FIG. 10, the PSF (A) of the luminous flux along the optical axis in the optical system 1 (optical axis central luminous flux) and the luminous flux along the outermost angle of view (outermost angular luminous flux) calculated in advance. ) PSF (B) is input data.

  FIG. 11 is a diagram illustrating an example of the optical axis central light beam F1 and the outermost field angle light beam F2 in the optical system 1. In the example shown in FIG. 11, the optical axis center light flux F1 is light incident along the optical axis that coincides with the rotational symmetry axis of the lenses 11a, 11b, and 11c. The outermost field angle light beam is light having the maximum incident angle with respect to the imaging surface. That is, the image of the light beam at the center of the optical axis corresponds to the center of the image, and the image of the light beam at the outermost angle of view corresponds to the outside of the image. The PSF (A) of the light beam at the center of the optical axis and the PSF (B) of the light beam at the outermost angle of view can be calculated, for example, by simulation using design support software for a commercially available optical system.

  Assuming that an ideal image of the subject is f and an image captured through the optical system 1 is g, PSF (A) and PSF (B) have a relationship represented by the following equation. In the following formula, * represents convolution.

g = PSF (A) * f
g = PSF (B) * f
In Op11 of FIG. 10, the computer calculates the Fourier transform values of the input PSF (A) and PSF (B). If the Fourier transform values of f and g are F and G, respectively, PSF (A) and PSF (B ) Fourier transform values H (A) and H (B) have the following relationship.

G = H (A) × F
G = H (B) × F
In Op12, the computer performs reciprocal processing for H (A) and H (B), and calculates H ⁻¹ (A) and H ⁻¹ (B) that satisfy the relationship of the following formula, for example.

F = H ⁻¹ (A) × G
F = H ⁻¹ (B) × G
In Op 13, ^{computer, H -1 (A), H} -1 inverse Fourier transform value h ^-1 (A) of (B), calculates h ^-1 to (B), respectively. These h ⁻¹ (A) and h ⁻¹ (B) can be used as the reference distribution (A) and the reference distribution (B), respectively. Thus, in this embodiment, the deconvolution kernel (here, the reference distribution) can be calculated and recorded for each of a plurality of representative light beams in the optical system 1. Thereby, the filter processing unit 42 generates a deconvolution kernel by calculating the interpolation or extrapolation of a representative deconvolution kernel so as to match the processing target position in the image, and uses this to generate the deconvolution processing. Can be executed.

  The data used as the reference distributions (A) and (B) is not limited to the above example. For example, in addition to the above two light fluxes, a reference distribution can be calculated from the PSF for other light fluxes, and three or more reference distributions can be recorded in the memory 43 in advance. Further, the combination of the PSF light fluxes on which the reference distribution is based is not limited to the pair of the optical axis center and the outermost angle of view as described above, but based on the PSF of the light flux intermediate between the optical axis center and the outermost angle of view. A reference distribution may be created.

(Second Embodiment)
FIG. 12 is a flowchart illustrating an operation example of the filter processing unit 42 according to the second embodiment. In FIG. 12, the same steps as those in FIG. The configurations of the image processing device 4 and the imaging device including the filter processing unit 42 can be the same as those in the first embodiment.

  In the example shown in FIG. 12, when the convolution calculation unit 44 acquires an image (Op1), the convolution control unit 45 sets a region (zone) to be processed in the image (Op2a). The subsequent processes of Op3 to Op5 are executed for each zone set in Op2a. In Op2a, the convolution control unit 45 sequentially selects zones to be processed from a plurality of zones in a preset image, and the filtering process is repeated for the selected zones.

  In Op3, the distance l between the representative point of the zone to be processed and the optical axis center is calculated. The representative point of each zone is not particularly limited, and may be, for example, a position coordinate at the center of the zone or a point on the boundary of the zone.

  In Op4, as in the first embodiment, the reference distribution (based on the distance l from the optical axis center P of the representative point of the zone and the rotation angle θ around the axis perpendicular to the optical axis center of the representative point) A) A process of generating a deconvolution kernel by correcting (B) is executed. In Op5, filtering is performed using the deconvolution kernel generated in Op4.

  As described above, in this embodiment, the image is divided into a plurality of regions, and the filtering process is executed by changing the filter data of the filter applied to each of the plurality of regions (zones). This makes it possible to execute appropriate filtering corresponding to the change in PSF for each zone. In the above example, the filter data in each zone is generated by correcting the reference distributions (A) and (B) using the distance l and the rotation angle θ in Op4. The setting method is not limited to this. For example, filter data such as a deconvolution kernel is recorded in advance in the memory 43 for each zone, and when a zone to be processed is determined, the deconvolution kernel corresponding to that zone can be read from the memory 43 and used for filtering. .

  FIG. 13 is a diagram illustrating an example of a plurality of zones preset in an image. In the example shown in FIG. 13, the acquired image region R is divided into zones by a plurality of concentric circles centered on the optical axis center point P and straight lines extending radially in the radial direction. That is, the image area to be corrected is divided radially. As described above, by arranging the plurality of zones radially, it is possible to set a filter according to the PSF that changes radially from the center of the optical axis.

  FIG. 14 is a diagram illustrating another example of a plurality of zones preset in an image. In the example shown in FIG. 14, the image is divided so that each zone is rectangular. Further, each zone is set so that the area of the zone becomes smaller as the position is farther from the position corresponding to the center of the optical axis in the image. That is, the area of the image area to be corrected can be reduced according to the distance from the center of the optical axis. Thereby, the filter setting can be changed more finely in the peripheral part of the image. The degree of change in PSF tends to increase as the distance from the optical axis center increases. Therefore, by setting the area smaller in the zone farther from the center of the optical axis, filtering that is more adapted to the degree of change in PSF can be performed. In FIGS. 13 and 14, only a quarter of the acquired image region R is shown as a zone, and the other zones are omitted.

  The embodiment of the present invention has been described above. According to the above-described embodiment, since it is possible to perform appropriate filtering according to the position in the image, it is possible to restore the blur corresponding to the change in the PSF. Therefore, for example, an image restoration effect by filtering can be obtained without inserting a phase modulation element or the like for suppressing a change in PSF into the optical system 1. As a result, the optical system can be simplified. That is, the resolution of the acquired image can be improved using a simple optical system.

  In addition, this invention is not limited to the said 1st-2nd specific embodiment.

  As described above, an imaging apparatus in which an optical system and an imaging element are added to the image processing apparatus is also included in the embodiment of the present invention. The use of this imaging device is not particularly limited. The imaging device can be used for, for example, a digital still camera, a mobile phone camera, a mobile information terminal camera, an image inspection device, an industrial camera for automatic control, an information code reader, and the like.

Claims (8)

An image processing apparatus that processes an image captured through an optical system,
A filtering unit that performs filtering on the image according to a transfer function of the optical system;
The filter processing unit uses a coefficient corresponding to a distance from a position corresponding to the optical axis center in the image to a processing target position as a first transfer function corresponding to the transfer function of light incident in the direction of the optical axis of the optical system. Filter data and a second filter corresponding to the transfer function of light incident at a certain angle with respect to the optical axis, thereby generating different filter data according to the position of the processing target in the image. , Filtering using the generated filter data ,
Image processing device.
The filter processing unit corrects filter data indicating a filter corresponding to a transfer function of the optical system recorded in advance according to a position of a processing target in the image, and uses the corrected filter data to perform the filtering The image processing apparatus according to claim 1, wherein:
The filter processing unit
The filter is corrected according to a direction from a point corresponding to the optical axis center in the image to a position to be processed, and the filtering is executed using filter data of the corrected filter. An image processing apparatus according to 1.
The filter processing unit corresponds to a position of a processing target in the image among filter data of a plurality of filters corresponding to a plurality of transfer functions corresponding to light having different incident angles in the optical system, which is recorded in advance. The image processing apparatus according to claim 1, wherein the filtering is performed using filter data of the selected filter.
The filter processing unit divides the image into a plurality of regions, and generates or selects filter data of the filter to be applied to each of the plurality of regions from filter data of a filter corresponding to the transfer function recorded in advance. The image processing apparatus according to claim 1, wherein filtering processing is executed for each region using the generated or selected filter data.
The image processing apparatus according to claim 5, wherein the plurality of regions are arranged radially with respect to a position corresponding to an optical axis center in the image.
The image processing apparatus according to claim 5, wherein the plurality of regions are set such that the area of the region decreases as the distance from the position corresponding to the optical axis center in the image increases.
An image processing program for causing a computer to process an image captured through an optical system,
Causing the computer to perform a filtering process for performing filtering according to the transfer function of the optical system on the image;
In the filtering process, a coefficient corresponding to the distance from the position corresponding to the optical axis center in the image to the position to be processed is a first function corresponding to the transfer function of light incident in the direction of the optical axis of the optical system. The filter and the second filter corresponding to the transfer function of the light incident at a certain angle with respect to the optical axis are linearly combined to generate different filter data according to the position of the processing target in the image, Perform filtering using the generated filter data
An image processing program.

Images