WO2016098282A1 - Image processing apparatus, image pickup apparatus, image processing method, image processing program, and storage medium - Google Patents

Image processing apparatus, image pickup apparatus, image processing method, image processing program, and storage medium Download PDF

Info

Publication number
WO2016098282A1
WO2016098282A1 PCT/JP2015/005773 JP2015005773W WO2016098282A1 WO 2016098282 A1 WO2016098282 A1 WO 2016098282A1 JP 2015005773 W JP2015005773 W JP 2015005773W WO 2016098282 A1 WO2016098282 A1 WO 2016098282A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
point spread
spread functions
regions
acquisition unit
Prior art date
Application number
PCT/JP2015/005773
Other languages
English (en)
French (fr)
Inventor
Takeshi Watanabe
Original Assignee
Canon Kabushiki Kaisha
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Canon Kabushiki Kaisha filed Critical Canon Kabushiki Kaisha
Publication of WO2016098282A1 publication Critical patent/WO2016098282A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/60—Noise processing, e.g. detecting, correcting, reducing or removing noise
    • H04N25/61—Noise processing, e.g. detecting, correcting, reducing or removing noise the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
    • H04N25/615—Noise processing, e.g. detecting, correcting, reducing or removing noise the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4" involving a transfer function modelling the optical system, e.g. optical transfer function [OTF], phase transfer function [PhTF] or modulation transfer function [MTF]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20021—Dividing image into blocks, subimages or windows

Definitions

  • the present invention relates to an image processing apparatus which corrects a deterioration of an image caused by an optical system.
  • the captured image is typically stored as electronic data, and a method (image restoration processing) of correcting the deterioration of the image caused by the optical system by using image processing has been proposed.
  • image restoration processing image restoration processing
  • the deterioration state of the image varies depending on a position in the image.
  • PTL 1 discloses an image processing method of performing optimum restoration processing depending on a deterioration state at each position in an image while reducing a calculation amount even when an image deterioration varies depending on the position according to a lens characteristic.
  • NPL 1 discloses an image restoration method by a repetition calculation using so-called Lucy-Richardson method.
  • PTL 1 describes neither setting the number of repetition times of calculation which is appropriate depending on each position when performing the image restoration processing, nor a specific method of the image restoration processing.
  • the number of repetition times of calculation is a dominant factor with respect to the intensity of the image restoration. For example, if the number of repetition times of calculation is insufficient, the degree of the restoration is insufficient. On the contrary, if the number of repetition times of calculation is too large, the degree of the restoration is excessive, which results in occurrence of a harmful effect such as ringing. Accordingly, it is necessary to set the number of repetition times appropriately depending on characteristics of the image deterioration at a position or a region in the image.
  • the repetition processing cannot be finished appropriately by comparing the RMS of an image which is not updated with the RMS of an updated image.
  • the (shift-variant) deterioration characteristic which varies depending on a position in a captured image cannot be satisfactorily restored.
  • the present invention provides an image processing apparatus, an image pickup apparatus, an image processing method, and an image processing program, and a storage medium which are capable of satisfactorily restoring a deterioration of an image depending on a position in the image.
  • An image processing apparatus includes an image acquisition unit configured to acquire an image, a condition acquisition unit configured to acquire image capturing condition information of the image, a PSF acquisition unit configured to acquire point spread functions relating to an optical system, a number setting unit configured to set the number of repetition times for each of a plurality of regions in the image based on each of the point spread functions corresponding to one of the plurality of regions, and an image restoration unit configured to perform repetition processing by the number of repetition times by using each of the point spread functions respectively on one of the plurality of regions to perform restoration processing on the image, and the PSF acquisition unit is configured to acquire the point spread functions based on the image capturing condition information.
  • An image pickup apparatus as another aspect of the present invention includes an image pickup element configured to photoelectrically convert an optical image formed via an optical system to output an image signal, an image acquisition unit configured to acquire an image generated based on the image signal, a condition acquisition unit configured to acquire image capturing condition information of the image, a PSF acquisition unit configured to acquire point spread functions relating to the optical system, a number setting unit configured to set the number of repetition times for each of a plurality of regions in the image based on each of the point spread functions corresponding to one of the plurality of regions, and an image restoration unit configured to perform repetition processing by the number of repetition times by using each of the point spread functions respectively on one of the plurality of regions to perform restoration processing on the image, and the PSF acquisition unit is configured to acquire the point spread functions based on the image capturing condition information.
  • An image processing method as another aspect of the present invention includes the steps of acquiring an image, acquiring image capturing condition information of the image, acquiring point spread functions relating to an optical system, setting the number of repetition times for each of a plurality of regions in the image based on each of the point spread functions corresponding to one of the plurality of regions, and performing repetition processing by the number of repetition times by using each of the point spread functions respectively on one of the plurality of regions to perform restoration processing on the image, and in the step of acquiring the point spread functions, the point spread functions are acquired based on the image capturing condition information.
  • An image processing program as another aspect of the present invention causes a computer to execute a process including the steps of acquiring an image, acquiring image capturing condition information of the image, acquiring point spread functions relating to an optical system, setting the number of repetition times for each of a plurality of regions in the image based on each of the point spread functions corresponding to one of the plurality of regions, and performing repetition processing by the number of repetition times by using each of the point spread functions respectively on one of the plurality of regions to perform restoration processing on the image, and in the step of acquiring the point spread functions, the point spread functions are acquired based on the image capturing condition information.
  • a storage medium as another aspect of the present invention stores the image processing program.
  • an image processing apparatus an image pickup apparatus, an image processing method, and an image processing program, and a storage medium which are capable of satisfactorily restoring a deterioration of an image depending on a position in the image can be provided.
  • FIG. 1 is a flowchart of an image processing method in Embodiment 1.
  • FIG. 2A is an explanatory diagrams of an image processing method which corrects a shift-variant deterioration characteristic in Embodiment 2.
  • FIG. 2B is an explanatory diagrams of the image processing method which corrects the shift-variant deterioration characteristic in Embodiment 2.
  • FIG. 3A is an explanatory diagram of the number of repetition times of calculation in Embodiment 2.
  • FIG. 3B is an explanatory diagram of the number of repetition times of calculation in Embodiment 2.
  • FIG. 4 is a configuration diagram of an image processing system in Embodiment 3.
  • FIG. 5 is a block diagram of an image processing apparatus in Embodiment 4.
  • FIG. 6 is a block diagram of an image pickup apparatus in Embodiment 5.
  • the image restoration processing by using the Lucy-Richardson method is represented by the following expression (2) in a real space.
  • symbol f n is an image which is updated (updated image) and symbol f n-1 is an image before updating.
  • Symbol h is a deterioration characteristic of the optical system, and for example it is a PSF.
  • Symbol g is a deteriorated image.
  • the image f n-1 is sequentially updated until the processing is discontinued according to, for example, a predetermined evaluation criterion, and then the image f n is output.
  • the evaluation criterion for the processing includes, for example, a value of root mean square (RMS) of the images f n and f n-1 .
  • an image captured by using a typical digital camera has a deterioration characteristic which varies depending on a position in the image. Furthermore, in many cases, the image may be asymmetrically deteriorated in the periphery of the image because an aberration of an image pickup optical system is large. This means that an amount of the image deterioration which is to be corrected varies depending on a position or a region in the image, and accordingly it is necessary to perform appropriate processing on each region.
  • expression (2) assuming an environment in which there are a number of combinations of the image pickup optical system and the image pickup element as in the case of the lens-interchangeable camera, a satisfactory result cannot be obtained only by applying expression (2).
  • FIG. 1 is a flowchart of the image processing method in this embodiment. Each step in FIG. 1 is performed by each unit of an image processing apparatus.
  • the image processing apparatus of this embodiment acquires an image (captured image) on which image restoration processing is to be performed. Subsequently, at step S102, the image processing apparatus acquires image capturing condition (image capturing condition information) to start the image restoration processing on the image acquired at step S101.
  • the image processing apparatus acquires the image capturing condition from information such as an Exif (Exchangeable image file format) added to the image acquired at step S101, or it directly acquires the image capturing condition.
  • the image capturing condition contains, as the image capturing condition of the optical system (image pickup optical system), a lens identification number (lens ID) to specify a lens used for capturing the image, a focal length at the time of capturing the image, an F number, and an object distance.
  • the image processing apparatus can specify the optical transfer function of the optical system based on the image capturing condition which is a combination of the lens ID, the focal length, the F number, and the object distance.
  • the image capturing condition information indicating a situation at the time of capturing the image, such as information relating to an array of color filters and information relating to an optical low-pass filter, may be contained.
  • the optical transfer function of the optical system is data discretized with a sampling pitch where a folding signal does not occur. It is preferred that the optical transfer function is for example calculated by simulation on a computer based on design data of a lens (optical system), or alternatively it may be measured by using an image pickup element having a small pixel pitch. It is preferred that the pixel pitch in this case is a pixel pitch where the folding signal does not occur as described above.
  • the optical transfer function calculated, for each wavelength, by using a spectral characteristic of an assumed light source and a spectral characteristic of the color filter of the image pickup element is generated according to each of colors R, G, and B.
  • the number of the optical transfer functions of the optical system corresponds to the number of the combinations of the position on the image pickup element and mainly the spectral characteristic of the color filters of R, G, and B when the focal length, the F number, and the object distance are determined as described above.
  • the plurality of optical transfer functions are necessary for each image capturing condition.
  • the number of the combinations of one-hundred (100) representative points on the image pickup element and the color filters of the three colors R, G, and B is 300. Accordingly, specifying the optical transfer functions of the optical system described above means specifying the 300 points.
  • the image processing apparatus acquires the optical transfer function of the optical system based on the image capturing condition acquired at step S102.
  • an acquisition range of the optical transfer function varies depending on a size of the image pickup element which is to be used.
  • the acquisition range of the optical transfer function is for example determined based on a ratio of a range determined depending on the optical system and a range in which the image pickup element can acquire an optical image.
  • the range determined depending on the optical system is for example a range in which an aberration correction is performed at the time of designing the optical system.
  • the image processing apparatus revises (i.e., corrects) the optical transfer function acquired at step S103.
  • the image processing apparatus of this embodiment revises the optical transfer function of the optical system, with respect to various image pickup elements, depending on characteristics (frequency characteristics) of a band of each image pickup element and a shape of the pixel. In order to revise the optical transfer function of the optical system depending on the characteristics of the image pickup element, the following characteristics are considered.
  • the characteristics of the image pickup element there are information relating to a size of the image pickup element, a pixel pitch, an array of color filters, and the like.
  • the size of the image pickup element is information which specifies an image circle to cut out an optical aerial image.
  • the pixel pitch is information relating to a folding signal of the optical transfer function.
  • a characteristic of an optical low-pass filter may be considered.
  • the characteristic of the optical low-pass filter is information relating to the presence or absence of the optical low-pass filter and information relating to a characteristic affected by the optical transfer function according to the optical low-pass filter.
  • the information contains information relating to a separation width, a separation direction, a separation method, and the like of the point spread function by the optical low-pass filter.
  • array information of the R, G, and B pixels can be used as information on the folding signal for a sampling interval of the Bayer array.
  • the sampling pitch of each of the R, G, and B pixels may be different from each other and the image pickup element may have a special arrangement. Accordingly, the information can be considered as well.
  • the information can be acquired as the image capturing condition (image capturing condition information) at step S102.
  • the optical transfer function revised depending on the characteristics of the image pickup element at step S104 is represented by a model as the following expression (3).
  • symbol H S (u,v) is an optical transfer function revised depending on the characteristics of the image pickup element.
  • symbol H(u,v) is an optical transfer function of the optical system
  • symbol OLPF(u,v) is a frequency characteristic of the optical low-pass filter
  • symbol PIX(u,v) is a frequency characteristic depending on a pixel shape of the image pickup element.
  • symbol [] S denotes acquisition of the frequency characteristic within a band of the image pickup element.
  • the image processing apparatus performs a frequency conversion on each of the plurality of optical transfer functions revised at step S104 to convert them into data on a real space, and thus it generates point spread functions (PSFs) containing the characteristic of the image pickup element.
  • PSFs point spread functions
  • the image processing apparatus determines the number of repetition times (the number of repetition times of calculation) in the image restoration processing based on the PSF generated at step S105. The number of repetition times of calculation is determined for each position or each region in the image. In this case, the image processing apparatus divides the image into a plurality of regions for example based on a pixel position of the image pickup element.
  • the image processing apparatus performs the image restoration processing by using the PSF generated at step S105 and expression (2), and then it performs development processing on the restored image.
  • expression (2) a deformation including various regularizations can be also adopted. It is preferred that the image processing method of this embodiment is provided as a program which operates by software or on a hard ware.
  • FIGs. 2A and 2B are explanatory diagrams of an image processing method which corrects a shift-variant deterioration characteristic, and they illustrate an example of image restoration processing using a PSF which varies depending on the characteristic of the image pickup element.
  • This embodiment will describe an image processing method in which a satisfactory result can be obtained in a case where a characteristic of the PSF significantly varies depending on a position in an image.
  • the image processing is performed by using an appropriate (i.e., precise) PSF for each position in the image.
  • an optical transfer function of the optical system corresponding to each position is revised (i.e., corrected) depending on the characteristic of the image pickup element. Then, the revised optical transfer function is converted into a PSF by frequency conversion to be used.
  • the numbers of vertical and horizontal data of the R, G, and B of the image are denoted by V and H, respectively.
  • the numbers V and H may be different from each other.
  • the rectangular region to which the PSF (target pixel) belongs among the plurality of rectangular regions can be calculated immediately without using processing such as conditional branching.
  • a rectangle region (vertical length A and horizontal length B) having grid point positions of the selected four PSFs as vertexes is represented by a region ⁇ m,n>.
  • FIG. 2B illustrates the region ⁇ m,n> in FIG. 2A.
  • the number of repetition times of calculation (the number of repetition times) for the region ⁇ m,n> is calculated based on four PSF kernels of the PSF[m,n], PSF[m,n+1], PSF[m+1,n], and PSF[m+1,n+1] corresponding to the grid point positions.
  • X and Y coordinates in the PSF kernels are as illustrated in FIG. 2B.
  • symbol round[] denotes a round-off function to an integer, that is, a function of rounding off a value in [] to the nearest integer.
  • Symbol V denotes a parameter to determine the reference number of repetition times, and for example it is input by a user. As described above, in this embodiment, the user determines a single parameter V (the reference number of repetition times) and accordingly the number of repetition times of calculation for each region (the number of repetition times IterNum ⁇ m,n>) is automatically set.
  • FIGs. 3A and 3B are explanatory diagrams of the number of repetition times of calculation.
  • the number of repetition times IterNum ⁇ m,n> for d ⁇ m,n> is gently changed, and a difference of the number of repetition times does not substantially occur for adjacent regions with respect to d ⁇ m,n>. Accordingly, each of the insufficient restoration, the excessive restoration, and the generation of unnatural harmful effects on the restored image is avoided.
  • PSFs ij corresponding to the target pixel (i,j) are four PSFs of PSF[m,n], PSF[m,n+1], PSF[m+1,n], and PSF[m+1,n+1], and they can be calculated as follows.
  • a weight W (weight coefficient) for each PSF is determined as follows.
  • W(m,n) (A-a)*(B-b)/(A*B)
  • W(m,n+1) (A-a)*b/(A*B)
  • W(m+1,n) a*(B-b)/(A*B)
  • W(m+1,n+1) a*b/(A*B)
  • expression (2) is used as represented by the following expression (7) based on expression (6).
  • the processing by expression (7) is performed for each of R, G, and B of the image and the repeating calculation (repetition processing) is performed by the number of repetition times IterNum ⁇ m,n> for each region ⁇ m,n> to obtain a final image.
  • a dispersion value of the PSF may be calculated and considered.
  • processing to match centroid positions of respective PSFs each other is performed in advance in order to perform the calculation of expression (6) with high accuracy by using small M and N.
  • Maximum values can be matched each other, in addition to the centroid positions, and the PSFs can be corrected so that an evaluation value for evaluating both of them is satisfied. It is preferred that the image processing apparatus decreases the number of repetition times with increasing the maximum value of the point spread function (PSF).
  • PSF point spread function
  • FIG. 4 is a configuration diagram of an image processing system 400 in this embodiment.
  • the image processing system 400 includes an image processing apparatus 401 (hardware as an information processing apparatus), a display device 402 (monitor), and an input device 403 (input unit such as a keyboard).
  • This embodiment will describe a case in which the image processing method of Embodiment 1 or 2 operates on the image processing system 400 (software in a personal computer).
  • the software (image processing program) which executes the image processing method is installed in the image processing apparatus 401 (personal computer).
  • the software can be installed from a medium 404 (storage medium) such as a CD-ROM or a DVD-ROM. Alternatively, the software may be installed through download via a network (internet).
  • the image processing apparatus 401 stores a plurality of optical transfer functions (optical transfer function data) of the optical system (image pickup optical system).
  • the plurality of optical transfer functions stored in the image processing apparatus 401 are for example all the optical transfer functions for each of various image capturing conditions (image capturing condition information) of the optical system.
  • the optical transfer functions are also downloaded via the medium 404 or the network to be stored in the image processing apparatus 401.
  • software image processing program
  • data such as optical transfer function information and image capturing condition information are stored in a storage such as a hard disk and a memory in the image processing apparatus 401.
  • the image processing apparatus 401 can be configured to acquire at least part of data from an external server via the network.
  • the image processing apparatus 401 runs the installed software to perform the image restoration processing on the captured image.
  • various settings such as the number of repetition times for determining the intensity of the image restoration processing and the application of the image restoration processing only in a predetermined region (region where the image restoration processing is to be performed) of an image, are possible. It is preferred that a user can change these parameters while confirming the restored image on the display device 402.
  • FIG. 5 is a block diagram of an image processing apparatus 500 in this embodiment.
  • the image processing apparatus 500 includes a captured image acquirer 501, an image capturing condition acquirer 502, an optical transfer function memory 503, an optical transfer function acquirer 504, an optical transfer function reviser 505, a frequency converter 506, a number of repetition times setter 507, and an image restorer 508.
  • the captured image acquirer 501 acquires an image (captured image) to which information relating to an image capturing condition (image capturing condition information) is added (step S101 in FIG. 1).
  • the image capturing condition acquirer 502 acquires, as the image capturing condition, information relating an optical system (image pickup optical system) and information relating to an image pickup element (step S102 in FIG. 1).
  • the information relating to the optical system contains information relating to a focal length, an F number, an object distance, or the like, but it is not limited thereto, and it may contain information relating to a diameter of an image circle of the optical system.
  • the diameter of the image circle of the optical system is a radius or a diameter of a unit circle which is predetermined for correcting aberrations at the time of designing the optical system.
  • the information relating to the image pickup element is information relating to a size of the image pickup element (image height), a pixel pitch of the image pickup element, a shape or a size of a pixel, or the like.
  • various elements such as an optical low-pass filter may be provided in the image pickup element.
  • information relating to characteristics of the various elements can be contained in the information relating to the image pickup element.
  • the optical transfer function memory 503 (memory) previously stores an optical transfer function (optical transfer function data) of the optical system.
  • the optical transfer function acquirer 504 (function acquisition unit) acquires a plurality of optical transfer functions from the optical transfer function data stored in the optical transfer function memory 503 based on the information relating to the optical system acquired by the image capturing condition acquirer 502 (step S103 in FIG. 1).
  • the optical transfer function reviser 505 (function revision unit) revises (i.e., corrects), based on the information relating to the image pickup element, the plurality of optical transfer functions acquired based on the information relating to the optical system (step S104 in FIG. 1). This revision contains providing a characteristic of the optical low-pass filter or the pixel, or processing of conversion into an optical transfer function up to a frequency band according to the pixel pitch of the image pickup element.
  • the frequency converter 506 (PSF acquisition unit) performs the inverse Fourier transform of all the optical transfer functions revised by the optical transfer function reviser 505, and it generates a point spread function (PSF) depending on characteristics of the optical system and the image pickup element (step S105 in FIG. 1).
  • the number of repetition times setter 507 (number setting unit) sets the number of repetition times for each region corresponding to the point spread function based on the point spread function generated by the frequency converter 506 (step S106 in FIG. 1). In other words, the number of repetition times setter 507 sets the number of repetition times for each of a plurality of regions in an image based on the point spread function (degree of blur) corresponding to each of the regions.
  • the image restorer 508 (image restoration unit) performs image restoration processing by using the image acquired by the captured image acquirer 501 and the point spread function generated by the frequency converter 506, based on the number of repetition times for each region corresponding to the point spread function (step S107 in FIG. 1). In other words, the image restorer 508 performs repetition processing by the number of repetition times on each of the plurality of regions by using the point spread function to perform the restoration processing on the image.
  • the captured image acquirer 501 acquires the image (captured image), and the image capturing condition information may not be added to the captured image in some cases. Accordingly, in this embodiment, the image processing apparatus 500 can be configured such that the captured image and the image capturing condition are acquired separately. In this case, the image capturing condition acquirer 502 can acquire the image capturing condition information separately from the captured image (for example, via an image pickup optical system controller 606 or a state detector 607 of an image pickup apparatus 600 illustrated in FIG. 6).
  • FIG. 6 is a block diagram of an image pickup apparatus 600 in this embodiment.
  • the image pickup apparatus 600 includes an image processor 604 (image processing apparatus) capable of performing the image processing method of each embodiment described above.
  • an object (not illustrated) is imaged on an image pickup element 602 via an image pickup optical system 601 (optical system) including an aperture stop 601a (or light shielding member) and a focus lens 601b.
  • the image pickup optical system 601 is an interchangeable lens (lens apparatus) removably mounted on an image pickup apparatus body including the image pickup element 602. This embodiment, however, is not limited to this, and can be applied also to an image pickup apparatus including an image pickup apparatus body and an image pickup optical system 601 integrated with each other.
  • An aperture value (F number) is determined by the aperture stop 601a or the light shielding member.
  • the image pickup element 602 photoelectrically converts an object image (optical image) formed via the image pickup optical system 601 to output an image signal (captured image data).
  • An electric signal output from the image pickup element 602 is output to an A/D converter 603.
  • the A/D converter 603 converts the electric signal (analog signal) input from the image pickup element 602 to a digital signal (captured image) and then outputs the digital signal to the image processor 604.
  • the image pickup element 602 and the A/D converter 603 constitute an image pickup unit configured to photoelectrically convert the optical image (object image) formed via the image pickup optical system 601 to output the captured image.
  • the image processor 604 performs predetermined image processing on the digital signal output from the A/D converter 603, i.e., image generated based on the image signal output from the image pickup element 602, by using information of a state detector 607 and a memory 608. Especially, the image processor 604 of this embodiment performs image restoration processing on the captured image to output a corrected image (restored image).
  • the image processor 604 includes units having the same functions as the captured image acquirer 501, the image capturing condition acquirer 502, the optical transfer function memory 503, the optical transfer function acquirer 504, the optical transfer function reviser 505, the frequency converter 506, the number of repetition times setter 507, and the image restorer 508.
  • the memory 608 stores an optical transfer function (optical transfer function data) of the image pickup optical system 601 for each image capturing condition (image capturing condition information), i.e., for each of combinations of a focal length at the time of capturing an image, an F number, a photographing distance, and an image height.
  • the memory 608 has the same function as the optical transfer function memory 503 illustrated in FIG. 5.
  • the image capturing condition is for example acquired by the image pickup optical controller 606 or the state detector 607.
  • the image processor 604 and the memory 608 constitute the image processing apparatus.
  • the optical transfer function memory can be included in the image processor 604, instead of the memory 608 provided outside the image processor 604.
  • the image processor 604 acquires the optical transfer function of the optical system corresponding to the acquired image capturing condition, and it revises the optical transfer function of the optical system by using the information relating to the image pickup element. Then, the image processor 604 (frequency converter) performs the Fourier transform on the revised optical transfer function, and it generates the point spread function depending on characteristics of the image pickup optical system and the image pickup element. The image processor 604 (number of repetition times setter) sets the number of repetition times for each region corresponding to the point spread function based on the generated point spread function.
  • the image processor 604 (image restorer) reads image data from the memory 608, and it performs the repetition processing by the number of repetition times set for each region by using the point spread function depending on the characteristics of the optical system and the image pickup element to perform the image restoration processing.
  • various settings such as application of the image restoration processing only in a predetermined region (region where the image restoration processing is to be performed) of an image, are possible. It is preferred that a user can change these parameters while confirming the restored image on the display 605.
  • the output image (restored image) processed by the image processor 604 is recorded in an image recording medium 609 in a predetermined format.
  • a display 605 displays an image obtained by performing predetermined processing for display on the processed image in this embodiment. Alternatively, the display 605 may display a simply-processed image for high-speed display.
  • the display 605 also displays a GUI to select an image restoration mode or a normal image capturing mode by a user. When the image restoration mode is selected by the user via the GUI on the display 605, a system controller 610 controls the image processor 604 to perform the image processing method of Embodiment 1 or 2.
  • the system controller 610 includes a CPU, an MPU, or the like, and it controls a whole of the image pickup apparatus 600. More specifically, the system controller 610 controls each of the image processor 604, the display 605, the image pickup optical system controller 606, the state detector 607, and the image recording medium 609.
  • the image pickup optical system controller 606 controls the motion of the image pickup optical system 601.
  • the state detector 607 detects a state of the image pickup optical system 601 based on information of the image pickup optical system controller 606.
  • software an image processing program that realizes the image processing method described above can be supplied to the system controller 610 via a network or a storage medium, and then the system controller 610 can read out and execute the image processing program.
  • an image processing apparatus includes an image acquisition unit (captured image acquirer 501), a condition acquisition unit (image capturing condition acquirer 502), a PSF acquisition unit (frequency converter 506), and a number setting unit (number of repetition times setter 507), and an image restoration unit (image restorer 508).
  • the image acquisition unit acquires an image (captured image).
  • the condition acquisition unit acquires image capturing condition information of the image.
  • the PSF acquisition unit acquires point spread functions relating to an optical system.
  • the number setting unit sets the number of repetition times for each of a plurality of regions in the image based on each of the point spread functions corresponding to one of the plurality of regions.
  • the image restoration unit performs repetition processing by the number of repetition times by using each of the point spread functions respectively on one of the plurality of regions to perform restoration processing on the image.
  • the PSF acquisition unit acquires the point spread functions based on the image capturing condition information.
  • the PSF acquisition unit acquires the point spread functions which vary depending on a pixel position of an image pickup element 602.
  • the PSF acquisition unit divides the image into the plurality of regions based on the pixel position of the image pickup element.
  • the image restoration unit specifies a plurality of point spread functions corresponding to a target pixel of the image pickup element from among the point spread functions acquired by the PSF acquisition unit, and performs the restoration processing on the image for each target pixel. More preferably, the image restoration unit determines weight information (weight coefficient) to be applied to the plurality of point spread functions corresponding to the target pixel, and performs the restoration processing on the image for each target pixel based on the weight information.
  • weight information weight coefficient
  • the image processing apparatus includes a function acquisition unit (optical transfer function acquirer 504) which acquires optical transfer functions relating to the optical system and a function revision unit (optical transfer function reviser 505) which revises the optical transfer functions based on information relating to the image pickup element.
  • the PSF acquisition unit generates the point spread functions from the optical transfer functions revised by the function revision unit.
  • the number setting unit sets the number of repetition times based on a maximum value of the point spread function. More preferably, the number setting unit decreases the number of repetition times with increasing the maximum value of the point spread function. Preferably, the number setting unit correlates the plurality of point spread functions with each of the regions to set the number of repetition times for each of the regions based on the plurality of point spread functions.
  • the image restoration unit generates a first updated image by using the image and the point spread functions. Then, the image restoration unit repeatedly performs processing of generating an (N+1)th updated image by using the image, an Nth updated image, and the point spread functions (N is a natural number). For example, the restoration processing on the image is performed by the LR method.
  • an image processing apparatus an image pickup apparatus, an image processing method, and an image processing program, and a storage medium which are capable of satisfactorily restoring a deterioration of an image depending on a position in the image can be provided.
  • Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a 'non-transitory computer-readable storage medium') to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s).
  • computer executable instructions e.g., one or more programs
  • a storage medium which may also be referred to more fully as
  • the computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions.
  • the computer executable instructions may be provided to the computer, for example, from a network or the storage medium.
  • the storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD) TM ), a flash memory device, a memory card, and the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)
  • Studio Devices (AREA)
PCT/JP2015/005773 2014-12-19 2015-11-18 Image processing apparatus, image pickup apparatus, image processing method, image processing program, and storage medium WO2016098282A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2014257080A JP6555881B2 (ja) 2014-12-19 2014-12-19 画像処理装置、撮像装置、画像処理方法、画像処理プログラム、および、記憶媒体
JP2014-257080 2014-12-19

Publications (1)

Publication Number Publication Date
WO2016098282A1 true WO2016098282A1 (en) 2016-06-23

Family

ID=56126196

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2015/005773 WO2016098282A1 (en) 2014-12-19 2015-11-18 Image processing apparatus, image pickup apparatus, image processing method, image processing program, and storage medium

Country Status (2)

Country Link
JP (1) JP6555881B2 (ja)
WO (1) WO2016098282A1 (ja)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190328208A1 (en) * 2016-03-09 2019-10-31 Sony Corporation Image processing device, image processing method and recording medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020164082A1 (en) * 2001-03-23 2002-11-07 Minolta Co., Ltd. Image processing apparatus
JP2009267523A (ja) * 2008-04-22 2009-11-12 Nikon Corp 画像復元装置及び撮像装置
US20140211038A1 (en) * 2013-01-28 2014-07-31 Canon Kabushiki Kaisha Image processing apparatus, image pickup apparatus, image processing method, and non-transitory computer-readable storage medium

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070009169A1 (en) * 2005-07-08 2007-01-11 Bhattacharjya Anoop K Constrained image deblurring for imaging devices with motion sensing
JP2007193508A (ja) * 2006-01-18 2007-08-02 Sanyo Electric Co Ltd 画像処理装置及び画像処理プログラム
JP4927005B2 (ja) * 2008-03-04 2012-05-09 日東光学株式会社 変化要因情報のデータの生成法および信号処理装置
JP2010087857A (ja) * 2008-09-30 2010-04-15 Fujifilm Corp 撮像装置、撮像方法、およびプログラム

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020164082A1 (en) * 2001-03-23 2002-11-07 Minolta Co., Ltd. Image processing apparatus
JP2009267523A (ja) * 2008-04-22 2009-11-12 Nikon Corp 画像復元装置及び撮像装置
US20140211038A1 (en) * 2013-01-28 2014-07-31 Canon Kabushiki Kaisha Image processing apparatus, image pickup apparatus, image processing method, and non-transitory computer-readable storage medium

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190328208A1 (en) * 2016-03-09 2019-10-31 Sony Corporation Image processing device, image processing method and recording medium
US10799088B2 (en) * 2016-03-09 2020-10-13 Sony Corporation Image processing device, image processing method and recording medium
US11642004B2 (en) 2016-03-09 2023-05-09 Sony Corporation Image processing device, image processing method and recording medium

Also Published As

Publication number Publication date
JP2016118856A (ja) 2016-06-30
JP6555881B2 (ja) 2019-08-07

Similar Documents

Publication Publication Date Title
US9100559B2 (en) Image processing apparatus, image pickup apparatus, image processing method, and image processing program using compound kernel
US8849055B2 (en) Image processing apparatus and method
US9225898B2 (en) Image pickup apparatus, image processing system, image pickup system, image processing method, and non-transitory computer-readable storage medium
JP5615393B2 (ja) 画像処理装置、撮像装置、画像処理方法、画像処理プログラム、および、記憶媒体
US11228701B2 (en) Image capturing apparatus
JP2015156615A (ja) 画像処理装置、画像処理方法、制御プログラム、および記録媒体
US20170155881A1 (en) Image processing apparatus, image pickup apparatus, image processing method, and non-transitory computer-readable storage medium for generating restored image
JP6415063B2 (ja) 画像処理装置、画像処理方法、制御プログラム、および記録媒体
US9727956B2 (en) Image processing apparatus, image pickup apparatus, image processing method, and non-transitory computer-readable storage medium
US10339637B2 (en) Image processing apparatus, image pickup apparatus, image processing method, and non-transitory computer-readable storage medium for correcting deterioration of image
EP3474532B1 (en) Image processing apparatus, image processing method, and program
WO2016098282A1 (en) Image processing apparatus, image pickup apparatus, image processing method, image processing program, and storage medium
JP6682184B2 (ja) 画像処理方法、画像処理プログラム、画像処理装置および撮像装置
US10304168B2 (en) Image processing apparatus, image pickup apparatus, image processing method, and non-transitory computer-readable storage medium for correcting deterioration of image
US10354367B2 (en) Image processing apparatus, image pickup apparatus, image processing method, image processing program, and storage medium
US9652832B2 (en) Image processing apparatus, image pickup apparatus, image processing method, and non-transitory computer-readable storage medium which perform image restoration processing
US9692970B2 (en) Image pickup apparatus, method of controlling image pickup apparatus, and non-transitory computer-readable storage medium
JP6672043B2 (ja) 画像処理装置、撮像装置、画像処理方法、画像処理システムおよび画像処理プログラム
JP6468751B2 (ja) 画像処理装置、画像処理方法及びプログラム
JP6486076B2 (ja) 画像処理装置および画像処理方法
RU2631521C2 (ru) Устройство обработки изображения, устройство для съемки изображения и способ обработки изображения
JP2017123532A (ja) 画像処理装置、撮像装置、画像処理方法、画像処理プログラムおよび記録媒体
RU2656009C1 (ru) Устройство обработки изображения, устройство для съемки изображения, способ обработки изображения и постоянный машиночитаемый носитель
JP6452389B2 (ja) 画像処理装置および画像処理方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15869496

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15869496

Country of ref document: EP

Kind code of ref document: A1