WO2017017909A1 - Image processing apparatus, image pickup apparatus, image processing method, image processing program, and storage medium - Google Patents

Image processing apparatus, image pickup apparatus, image processing method, image processing program, and storage medium Download PDF

Info

Publication number
WO2017017909A1
WO2017017909A1 PCT/JP2016/003228 JP2016003228W WO2017017909A1 WO 2017017909 A1 WO2017017909 A1 WO 2017017909A1 JP 2016003228 W JP2016003228 W JP 2016003228W WO 2017017909 A1 WO2017017909 A1 WO 2017017909A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
image processing
processing apparatus
characteristic information
moire
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2016/003228
Other languages
English (en)
French (fr)
Inventor
Takeshi Watanabe
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Canon Inc
Original Assignee
Canon Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Canon Inc filed Critical Canon Inc
Priority to US15/540,452 priority Critical patent/US10354367B2/en
Publication of WO2017017909A1 publication Critical patent/WO2017017909A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/40—Picture signal circuits
    • H04N1/409—Edge or detail enhancement; Noise or error suppression
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/20—Image enhancement or restoration using local operators
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
    • H04N23/12—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths with one sensor only
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H04N23/81—Camera processing pipelines; Components thereof for suppressing or minimising disturbance in the image signal generation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/64—Circuits for processing colour signals
    • H04N9/646—Circuits for processing colour signals for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20092—Interactive image processing based on input by user
    • G06T2207/20104—Interactive definition of region of interest [ROI]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/60—Noise processing, e.g. detecting, correcting, reducing or removing noise
    • H04N25/61—Noise processing, e.g. detecting, correcting, reducing or removing noise the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
    • H04N25/615—Noise processing, e.g. detecting, correcting, reducing or removing noise the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4" involving a transfer function modelling the optical system, e.g. optical transfer function [OTF], phase transfer function [PhTF] or modulation transfer function [MTF]

Definitions

  • the present invention relates to an image processing apparatus that corrects a moire component included in an image.
  • an object image (optical image) is acquired as a digital image
  • the optical image is discretized with pixel pitch intervals by an image pickup element.
  • aliasing occurs. Due to the generation of the aliasing, a moire (periodic pattern) occurs in the digital image.
  • the generation of the moire is not preferable, and in particular a color moire is also called a false color.
  • a method of reducing the generation of the moire a method of inserting an optical low-pass filter to attenuate the frequency response near the Nyquist frequency of the image pickup element is known.
  • PTL 1 discloses an image pickup apparatus that captures a plurality of images while driving optical lenses (image pickup optical system) minutely to acquire an image in which any moire is not generated.
  • PTL 2 discloses an image processing method that stores a frequency at which the moire occurs based on a relationship between a dot density at the time of printing and a surface density of an image itself and that specifies a region of the image where the moire occurs based on frequency characteristics of the image.
  • NPL1 discloses linearity between color components of a locus region in an image, and it discloses a method of reducing a color noise by improving the linearity between the color components when the color noise occurs.
  • NPL2 discloses a method of reducing a phenomenon where the linearity between the color components is collapsed like the color noise.
  • NPL 1 Color Lines: Image Specific Color Representation.
  • NPL 2 A Convex Regularizer for Reducing Color Artifact in Color Image Recovery
  • the image processing method disclosed in PTL 2 is limited to image processing for printing an image. Since NPL 1 and NPL 2 do not describe a correction amount to improve the linearity between the color components, the methods disclosed in NPL 1 and NPL 2 cannot correct a moire component included in an image effectively.
  • the present invention provides an image processing apparatus, an image pickup apparatus, an image processing method, an image processing program, and a storage medium which are capable of effectively correcting a moire component included in an image.
  • An image processing apparatus includes an estimator configured to estimate a moire component included in an image based on optical characteristic information, a determiner configured to determine a correction amount based on the estimated moire component, and a corrector configured to correct the image so as to reduce the moire component included in the image based on the correction amount.
  • An image pickup apparatus as another aspect of the present invention includes an image pickup element configured to photoelectrically convert an optical image formed via an image pickup optical system to output an image signal, an estimator configured to estimate, based on optical characteristic information, a moire component included in an image generated from the image signal, a determiner configured to determine a correction amount based on the estimated moire component, and a corrector configured to correct the image so as to reduce the moire component included in the image based on the correction amount.
  • An image processing method as another aspect of the present invention includes the steps of estimating a moire component included in an image based on optical characteristic information, determining a correction amount based on the estimated moire component, and correcting the image so as to reduce the moire component included in the image based on the correction amount.
  • An image processing program as another aspect of the present invention causes a computer to execute a process including the steps of estimating a moire component included in an image based on optical characteristic information, determining a correction amount based on the estimated moire component, and correcting the image so as to reduce the moire component included in the image based on the correction amount.
  • a storage medium as another aspect of the present invention stores the image processing program.
  • an image processing apparatus an image pickup apparatus, an image processing method, an image processing program, and a storage medium which are capable of effectively correcting a moire component included in an image can be provided.
  • FIG. 1 is a flowchart of an image processing method in Embodiment 1.
  • FIG. 2 is a diagram of illustrating an example of a color line matrix in Embodiment 1.
  • FIG. 3A is a diagram of illustrating linearity of each column in the color line matrix before moire correction processing is performed in Embodiment 1.
  • FIG. 3B is a diagram of illustrating linearity of each column in the color line matrix after the moire correction processing is performed in Embodiment 1.
  • FIG. 4A is a diagram of illustrating an estimation region of a moire generation amount in Embodiment 1.
  • FIG. 4B is a diagram of illustrating the estimation region of the moire generation amount in Embodiment 1.
  • FIG. 5A is an example of an image before the moire correction processing is performed in Embodiment 1.
  • FIG. 1 is a flowchart of an image processing method in Embodiment 1.
  • FIG. 2 is a diagram of illustrating an example of a color line matrix in Embodiment 1.
  • FIG. 5B is an example of an image after the moire correction processing is performed in Embodiment 1.
  • FIG. 6 is a flowchart of an image processing method in Embodiment 2.
  • FIG. 7 is a configuration diagram of an image processing system in Embodiment 3.
  • FIG. 8 is a block diagram of an image pickup apparatus in Embodiment 4.
  • FIG. 1 is a flowchart of the image processing method in this embodiment. Each step in FIG. 1 is performed by each unit of an image processing apparatus.
  • the image processing apparatus (acquirer) of this embodiment acquires an image (captured image) to be subject to image restoration processing. Subsequently, at step S102, the image processing apparatus (acquirer) acquires an image capturing condition (image capturing condition information) to start moire correction processing on the image acquired at step S101.
  • the image processing apparatus may acquire the image capturing condition from for example Exif information associated with the image acquired at step S101 or acquire the image capturing condition directly.
  • the image capturing condition includes information relating to a lens identification number (lens ID) to specify a lens used for photography, a focal length for the photography, and an F number, an object distance, presence or absence of an optical low-pass filter or its characteristic, and a Nyquist frequency of an image pickup element.
  • lens ID lens identification number
  • the image processing apparatus acquires optical characteristic information based on the image capturing condition acquired at step S102.
  • the image processing apparatus acquires (specifies) the optical characteristic information corresponding to the image capturing condition that is a combination of the parameters described above.
  • the optical characteristic information is an optical transfer function (OTF) or a modulation transfer function (MTF) as an absolute value of the OTF.
  • the image processing apparatus of this embodiment performs image processing while dividing a whole of an image into S regions (S division) in a vertical direction and into T regions (T division) in a horizontal direction.
  • S division S regions
  • T division T regions
  • a block (region in the image) whose number is s in the vertical direction and t in the horizontal direction in the image is represented as (s,t), and the (s,t)-th block will be described.
  • expression (1) When the Fourier transform is performed for expression (1) to convert the real space (x,y) to a frequency space (u,v), expression (1) can be represented as expression (2) below.
  • G(u,v) F(u,v)*H(u,v) ...
  • F(u,v), G(u,v), and H(u,v) are results of the Fourier transform of f(x,y), g(x,y), and h(x,y), respectively.
  • the OTF corresponds to H(u,v) in expression (2).
  • the MTF corresponds to an absolute value
  • the OTF may be determined considering an optical low-pass filter, frequency characteristics of a pixel shape, or the like.
  • the OTF greatly varies depending on an image height and a color (wavelength). Accordingly, in this embodiment, the OTF is represented as H(u,v,s,t) considering the block (s,t), that is, image height, in the image in addition to the frequency space (u,v). In this embodiment, it is preferred that a variable relating to the color (wavelength) is added, but this description is omitted.
  • the image processing apparatus estimates a moire generation amount based on the optical characteristic information acquired at step S103, i.e., the OTF represented as H(u,v,s,t). Then, at step S105, the image processing apparatus (determiner) determines a correction amount based on the moire generation amount estimated at step S104.
  • H(u,v,s,t) is a characteristic which is independent of an object. While a real image (optical image) corresponds to G(u,v,s,t) as represented by expression (2), the moire generation amount can be estimated based on H(u,v,s,t).
  • the image processing apparatus can estimate the moire generation amount based on the Nyquist frequency (for example a maximum value, a sum total, an average of the Nyquist frequency) of
  • FIG. 2 is a diagram of illustrating an example of the color line matrix. Symbols M1, M2, and M3 denote row vectors with m rows, and m is equal to the number of pixels in the arbitrary region in the image. An i-th row corresponds to an i-th pixel. The row vectors M1, M2, and M3 correspond to the RGB components in the image, respectively.
  • the image processing apparatus performs a singular value decomposition for the matrix M.
  • V * is an adjoint matrix of V.
  • ⁇ 1 max ⁇ ( ⁇ 1- ⁇ 1),0 ⁇
  • ⁇ 2 max ⁇ ( ⁇ 2- ⁇ 2),0 ⁇
  • Symbols ⁇ 1, ⁇ 2, and ⁇ 3 are correction values, and details thereof will be described below.
  • Symbol max ⁇ A,B ⁇ means a larger value of A and B.
  • the linearity of vectors of M1, M2, and M3 is improved with increasing a ratio of the maximum singular value ⁇ 1 and each of the singular values ⁇ 2 and ⁇ 3. In particular, when a color moire occurs, this ratio is decreased. Accordingly, by replacing ⁇ 1, ⁇ 2, and ⁇ 3 with ⁇ 1, ⁇ 2, and ⁇ 3 described above, this ratio can be improved to improve the linearity of M1, M2, and M3.
  • Symbols ⁇ 1, ⁇ 2, and ⁇ 3 on the right side are estimated values of the moire generation amount (i.e., estimated moire component), and they are calculated based on H(u,v,s,t).
  • Symbols w1, w2, and w3 on the right side are reference weights that are used as weights (weight coefficients) relating to the respective singular values.
  • the Nyquist frequency is a parameter that is determined depending on the image pickup element being used.
  • ⁇ 1 max ⁇
  • ⁇ 2 max ⁇
  • ⁇ 3 max ⁇
  • ⁇ 4 max ⁇
  • W is a maximum correction amount of a system. Accordingly, W can be previously calculated by measuring a CZP chart or the like in reality for each image pickup apparatus (camera) to be used. In this embodiment, it is preferred that W is around 10 to 100. While the method of calculating the weight w1 is described as a representative in this embodiment, the weights w2 and w3 can also be calculated similarly. When a user wishes to change a strength of the moire correction processing, each value can be changed with reference to the determined weights w1, w2, and w3.
  • the image processing apparatus corrects the image (i.e., performs the moire correction processing) so as to reduce the moire component included in the captured image (image) acquired at step S101 based on the correction amount determined at step S105.
  • the image processing apparatus develops the image on which the moire correction processing has been performed.
  • FIGs. 3A and 3B are diagrams of illustrating linearity of each column in a color line matrix before the moire correction processing is performed and after the moire correction processing is performed, respectively.
  • FIG. 3A illustrates a three-dimensional diagram with the color line matrix as axes with respect to an image where a color moire occurs before the moire correction processing is performed.
  • FIG. 3B illustrates a three-dimensional diagram with the color line matrix as axes after the moire correction processing is performed. Comparing FIG. 3A with FIG. 3B, it is understood that the ratio of the maximum singular value is corrected to be increased and the linearity for each color line matrix is improved. Thus, the image where the moire component (especially, color moire) has been reduced can be acquired.
  • the moire generation amount is estimated based on the optical characteristic information of the Nyquist frequency, but this embodiment is not limited thereto.
  • a predetermined width with reference to the Nyquist frequency may be provided such that an estimation region of the moire generation amount is within a range of uN/2 ⁇ u ⁇ uN and vN/2 ⁇ v ⁇ vN.
  • FIGs. 4A and 4B are diagrams of illustrating the estimation region of the moire generation amount for each image height.
  • FIG. 4A illustrates an image height dividing point (m,n) for each region (image height) in an image
  • FIG. 4B illustrates a corresponding optical characteristic with respect to a region of the second quadrant in the image of FIG. 4A.
  • a region between two black frames at an outside and an inside is the estimation region of the moire generation amount.
  • the image processing apparatus estimates the moire generation amount for each image height dividing point (m,n). Then, the image processing apparatus performs the moire correction processing for each region corresponding to the image height dividing point (m,n) by using the estimated moire generation amount.
  • FIGs. 5A and 5B are an example of images before and after the moire correction processing is performed in this embodiment.
  • FIG. 5A illustrates the image before the moire correction processing is performed
  • FIG. 5B illustrates the image after the moire correction processing is performed for each of RGB components.
  • the moire component included in the image can be effectively corrected by the moire correction processing in this embodiment.
  • FIG. 6 is a flowchart of the image processing method in this embodiment. Each step in FIG. 6 is performed by each unit of the image processing apparatus. This embodiment describes a method of performing moire correction processing on a region specified by a user.
  • the image processing method of this embodiment is different from the image processing method of Embodiment 1 described referring to FIG. 1 in that the image processing method of this embodiment includes a step (step S604 in FIG. 6) of acquiring the region (processing region) specified by the user.
  • Other steps (steps S601 to S603, and S605 to S608) in this embodiment are the same as steps S101 to S107 in FIG. 1, and accordingly descriptions thereof are omitted.
  • the image processing apparatus acquires a captured image, an image capturing condition, and optical characteristic information, respectively. Subsequently, at step S604, the image processing apparatus acquires a region (processing region) of the image (captured image) acquired at step S601 that is subject to perform moire correction processing specified by a user. Then, at step S605, the image processing apparatus estimates a moire generation amount relating to the processing region acquired at step S604 based on the optical characteristic information acquired at step S603.
  • the image processing apparatus when the image processing apparatus cannot acquire the optical characteristic information relating to the processing region (i.e., optical characteristic information corresponding to the processing region specified by the user) at step S603, it acquires optical characteristic information relating to a region near the processing region.
  • the image processing apparatus acquires optical characteristic information relating to a region corresponding to an image height which is closest to the processing region.
  • the image processing apparatus may interpolate and generate the optical characteristic information relating to the processing region by using optical characteristic information corresponding to the region (image height) near the processing region.
  • the image processing apparatus specifies the block (s,t) of the image corresponding to the region (processing region) specified by the user.
  • the image processing apparatus estimates a moire generation amount for example based on H(u,v,4,10) as optical characteristic information. Then, at step S606, the image processing apparatus determines a correction amount based on the moire generation amount estimated at step S605. In this embodiment, for example, the image processing apparatus determines the moire generation amount based on a maximum value, a sum total, an average of the Nyquist frequency of
  • ⁇ 1 max ⁇
  • ⁇ 2 max ⁇
  • ⁇ 3 max ⁇
  • ⁇ 4 max ⁇
  • the image processing apparatus corrects (i.e., performs the moire correction) included in the captured image (image) acquired at step S601 based on the correction amount determined at step S606. Then, at step S608, the image processing apparatus develops the image on which the moire correction processing has been performed.
  • FIG. 7 is a configuration diagram of an image processing system 700 in this embodiment.
  • the image processing system 700 includes an image processing apparatus 701 (hardware as an information processing apparatus), a display apparatus 702 (monitor), and an input device 703 (input unit such as a keyboard).
  • This embodiment will describe a case in which the image processing method of Embodiment 1 or 2 operates on the image processing system 700 (software in a personal computer).
  • the software (image processing program) which executes the image processing method is installed in the image processing apparatus 701 (personal computer).
  • the software can be installed from a medium 704 (storage medium) such as a CD-ROM and a DVD-ROM. Alternatively, the software may be installed through download via a network (internet).
  • the image processing apparatus 701 stores a plurality of optical transfer functions (optical transfer function data) of the image pickup optical system.
  • the plurality of optical transfer functions stored in the image processing apparatus 701 are for example all the optical transfer functions for each of various image capturing conditions (image capturing condition information) of the optical system.
  • optical transfer functions are also downloaded via the medium 704 or the network to be stored in the image processing apparatus 701.
  • software image processing program
  • data such as optical transfer function information and image capturing condition information are stored in a storage such as a hard disk and a memory in the image processing apparatus 701.
  • the image processing apparatus 701 can be configured to acquire at least part of data from an external server via the network.
  • the image processing apparatus 701 runs the installed software to perform the moire correction processing on the captured image.
  • the software it is possible to perform various settings (settings of parameters) for example to perform the moire correction processing only in a predetermined region (predetermined target region) in the image. It is preferred that a user can change these parameters while confirming the image on the display apparatus 702 after the restoration processing is performed.
  • FIG. 8 is a block diagram of an image pickup apparatus 800 in this embodiment.
  • the image pickup apparatus 800 includes an image processor 804 (image processing apparatus) that is capable of performing the image processing method in each embodiment described above.
  • an object (not illustrated) is imaged on an image pickup element 802 via an image pickup optical system 801 (optical system) including an aperture stop 801a (or light shielding member) and a focus lens 801b.
  • the image pickup optical system 801 is an interchangeable lens (lens apparatus) removably mounted on an image pickup apparatus body including the image pickup element 802.
  • this embodiment is not limited thereto, and can be applied also to an image pickup apparatus including an image pickup apparatus body and an image pickup optical system 801 integrated with each other.
  • An aperture value (F number) is determined by the aperture stop 801a or the light shielding member.
  • the image pickup element 802 photoelectrically converts an object image (optical image) formed via the image pickup optical system 801 to output an image signal (captured image data).
  • An electric signal output from the image pickup element 802 is output to an A/D converter 803.
  • the A/D converter 803 converts the electric signal (analog signal) input from the image pickup element 802 to a digital signal (captured image) and then outputs the digital signal to the image processor 804.
  • the image pickup element 802 and the A/D converter 803 constitute an image pickup unit that photoelectrically converts the optical image (object image) formed via the image pickup optical system 801 to output the captured image.
  • the image processor 804 performs predetermined image processing on the digital signal output from the A/D converter 803, i.e., image generated based on the image signal output from the image pickup element 802, by using each of pieces of information of a state detector 807 and a memory 808. Especially, the image processor 804 of this embodiment performs the moire correction processing on the captured image to output a corrected image (moire corrected image).
  • the image processor 804 includes an acquisition unit 804a (acquirer or acquisition circuit), an estimation unit 804b (estimator or estimation circuit), a determination unit 804c (determiner or determination circuit), and a correction unit 804d (corrector or correction circuit).
  • the acquisition unit 804a acquires the captured image, the image capturing condition information, and the optical characteristic information.
  • the estimation unit 804b estimates a moire component (moire generation amount) included in the captured image.
  • the determination unit 804c determines the correction amount based on the estimated moire component.
  • the correction unit 804c corrects the captured image so as to reduce the moire component included in the captured image based on the determined correction amount (i.e., performs the moire correction processing).
  • the memory 808 stores an optical transfer function (optical transfer function data) of the image pickup optical system 801 for each image capturing condition (image capturing condition information), i.e., for each of combinations of a focal length at the time of photography, an F number, a photographing distance, and an image height.
  • the memory 808 has the same function as the optical transfer function memory 603 illustrated in FIG. 6.
  • the image capturing condition is for example acquired by the image pickup optical controller 806 or the state detector 807.
  • the image processor 804 and the memory 808 constitute the image processing apparatus.
  • the function of the optical transfer function memory can be included in the image processor 804, instead of the memory 808 provided outside the image processor 804.
  • the image processor 804 estimates the moire generation amount based on the optical transfer function, determines the correction amount based on the estimated moire generation amount, and performs the moire correction processing based on the determined correction amount. Furthermore, the image processor 804 is capable of performing various settings (settings of parameters) for example to perform the moire correction processing only in a predetermined region (predetermined target region) in the image. It is preferred that these parameters can be changed while a user confirms the image on the display apparatus 702 after the restoration processing is performed.
  • the output image (moire corrected image) processed by the image processor 804 is recorded in an image recording medium 809 in a predetermined format.
  • a display 805 displays an image obtained by performing predetermined processing for display on the processed image in this embodiment. Alternatively, the display 805 may display a simply-processed image for high-speed display.
  • the display 805 also displays a GUI to select an image restoration mode or a normal image capturing mode by a user. When the image restoration mode is selected by the user via the GUI on the display 805, a system controller 810 controls the image processor 804 to perform the image processing method of Embodiment 1 or 2.
  • the system controller 810 includes a CPU, an MPU, or the like, and it controls a whole of the image pickup apparatus 800. More specifically, the system controller 810 controls each of the image processor 804, the display 805, an image pickup optical system controller 806, the state detector 807, and the image recording medium 809.
  • the image pickup optical system controller 806 controls the motion of the image pickup optical system 801.
  • the state detector 807 detects a state of the image pickup optical system 801 based on information of the image pickup optical system controller 806.
  • software an image processing program
  • the system controller 810 can read out to execute the image processing program.
  • the image processing apparatus includes the estimator (estimation unit 804b (estimation circuit)), the determiner (determination unit 804c (determination circuit)), and the corrector (correction unit 804d (correction circuit)).
  • the estimator estimates a moire component (moire generation amount) included in an image based on optical characteristic information.
  • the determiner determines a correction amount based on the estimated moire component.
  • the corrector corrects the image so as to reduce the moire component included in the image based on the correction amount.
  • the corrector adjusts linearity of a color component of the image based on the correction amount. More preferably, the determiner determines, as the correction amount, an adjustment amount to adjust the linearity of the color component for each of a plurality of regions in the image. Then, the corrector adjusts the linearity of the color component for each of the plurality of regions based on the adjustment amount.
  • the determiner determines the correction amount based on the optical characteristic information and a Nyquist frequency of an image pickup element.
  • the estimator estimates the moire component based on the optical characteristic information relating to the Nyquist frequency.
  • the estimator estimates the moire component based on the optical characteristic information relating to a predetermined frequency range with reference to the Nyquist frequency.
  • the optical characteristic information is information relating to an optical transfer function (OTF or MTF).
  • the image processing apparatus includes an acquirer (acquisition unit 804a (acquisition circuit)) that acquires the image and image capturing condition information relating to the image. More preferably, the acquirer acquires the optical characteristic information based on the image and the image capturing condition information. More preferably, the acquirer acquires, as the optical characteristic information, a plurality of pieces of optical characteristic information depending on the plurality of regions in the image. Preferably, the determiner determines the correction amount based on each of the plurality of pieces of optical characteristic information depending on the plurality of regions in the image. Preferably, the acquirer acquires optical characteristic information corresponding to a region of the image specified by a user.
  • acquirer acquisition unit 804a (acquisition circuit)
  • the acquirer acquires the optical characteristic information based on the image and the image capturing condition information. More preferably, the acquirer acquires, as the optical characteristic information, a plurality of pieces of optical characteristic information depending on the plurality of regions in the image.
  • the determiner determines the correction amount based on each of the pluralit
  • Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a 'non-transitory computer-readable storage medium') to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s).
  • computer executable instructions e.g., one or more programs
  • a storage medium which may also be referred to more fully as
  • the computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions.
  • the computer executable instructions may be provided to the computer, for example, from a network or the storage medium.
  • the storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD) TM ), a flash memory device, a memory card, and the like.
  • an image processing apparatus an image pickup apparatus, an image processing method, an image processing program, and a storage medium which are capable of effectively correcting a moire component included in an image can be provided.
  • IMAGE PROCESSOR IMAGE PROCESSING APPARATUS
  • ESTIMATION UNIT ESTIMATOR
  • DETERMINATION UNIT DETERMINER
  • CORRECTION UNIT CORRECTOR

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Color Television Image Signal Generators (AREA)
  • Image Processing (AREA)
  • Facsimile Image Signal Circuits (AREA)
  • Color Image Communication Systems (AREA)
PCT/JP2016/003228 2015-07-27 2016-07-07 Image processing apparatus, image pickup apparatus, image processing method, image processing program, and storage medium Ceased WO2017017909A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US15/540,452 US10354367B2 (en) 2015-07-27 2016-07-07 Image processing apparatus, image pickup apparatus, image processing method, image processing program, and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2015147373A JP6708378B2 (ja) 2015-07-27 2015-07-27 画像処理装置、撮像装置、画像処理方法、画像処理プログラム、および、記憶媒体
JP2015-147373 2015-07-27

Publications (1)

Publication Number Publication Date
WO2017017909A1 true WO2017017909A1 (en) 2017-02-02

Family

ID=57884637

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2016/003228 Ceased WO2017017909A1 (en) 2015-07-27 2016-07-07 Image processing apparatus, image pickup apparatus, image processing method, image processing program, and storage medium

Country Status (3)

Country Link
US (1) US10354367B2 (enExample)
JP (1) JP6708378B2 (enExample)
WO (1) WO2017017909A1 (enExample)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6818585B2 (ja) * 2017-02-21 2021-01-20 キヤノン株式会社 画像処理装置、画像処理方法、及び、プログラム
FR3071344B1 (fr) * 2017-09-20 2019-09-13 Idemia Identity And Security Procede de determination d'une fonction d'etalement de point d'un systeme d'imagerie
CN108418997B (zh) * 2018-01-31 2020-12-22 深圳市商巨视觉技术有限公司 去除图像摩尔纹的方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007300203A (ja) * 2006-04-27 2007-11-15 Canon Inc 画像処理装置、画像処理装置の制御方法およびプログラム
US20090263018A1 (en) * 2008-04-16 2009-10-22 Canon Kabushiki Kaisha Image processing apparatus and image processing method for reducing color blur
US20120050583A1 (en) * 2010-08-27 2012-03-01 Canon Kabushiki Kaisha Image processing method, image processing apparatus and image processing program
US20150146046A1 (en) * 2013-11-25 2015-05-28 Kabushiki Kaisha Toshiba Solid-state imaging device, digital camera, and image processing method

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5777441A (en) * 1995-07-10 1998-07-07 Matsushita Electric Industrial Co., Ltd. Moire reducing apparatus
JPH09116911A (ja) * 1995-10-20 1997-05-02 Canon Inc 撮像システム
KR20060065800A (ko) * 2004-12-10 2006-06-14 한국전자통신연구원 헬름홀츠 스테레오를 이용한 깊이 불연속선을 지니는물체의 3차원 형상 획득 장치 및 그 방법
JP2006262081A (ja) 2005-03-17 2006-09-28 Fuji Xerox Co Ltd 画像処理システム
US7379247B2 (en) * 2005-05-23 2008-05-27 Olympus Imaging Corp. Image pickup apparatus
JP4775306B2 (ja) * 2007-04-23 2011-09-21 ソニー株式会社 画像処理装置、撮像装置、および画像表示制御方法、並びにコンピュータ・プログラム
JP6195055B2 (ja) 2012-08-06 2017-09-13 株式会社リコー 撮像装置および撮像方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007300203A (ja) * 2006-04-27 2007-11-15 Canon Inc 画像処理装置、画像処理装置の制御方法およびプログラム
US20090263018A1 (en) * 2008-04-16 2009-10-22 Canon Kabushiki Kaisha Image processing apparatus and image processing method for reducing color blur
US20120050583A1 (en) * 2010-08-27 2012-03-01 Canon Kabushiki Kaisha Image processing method, image processing apparatus and image processing program
US20150146046A1 (en) * 2013-11-25 2015-05-28 Kabushiki Kaisha Toshiba Solid-state imaging device, digital camera, and image processing method

Also Published As

Publication number Publication date
JP6708378B2 (ja) 2020-06-10
JP2017028594A (ja) 2017-02-02
US10354367B2 (en) 2019-07-16
US20170358058A1 (en) 2017-12-14

Similar Documents

Publication Publication Date Title
JP6327922B2 (ja) 画像処理装置、画像処理方法、およびプログラム
JP5968073B2 (ja) 画像処理装置、撮像装置、画像処理方法、および画像処理プログラム
US10958888B2 (en) Image processing apparatus, image processing method and storage medium for storing program
US9992478B2 (en) Image processing apparatus, image pickup apparatus, image processing method, and non-transitory computer-readable storage medium for synthesizing images
JP6312487B2 (ja) 画像処理装置及びその制御方法、並びに、プログラム
US20160080727A1 (en) Depth measurement apparatus, imaging apparatus, and depth measurement method
US20170187938A1 (en) Image pickup apparatus, image processing apparatus, image processing method, and non-transitory computer-readable storage medium for improving quality of captured image
CN104702824A (zh) 摄像装置及摄像装置的控制方法
US20180234616A1 (en) Auto-focusing systems and methods
US10354367B2 (en) Image processing apparatus, image pickup apparatus, image processing method, image processing program, and storage medium
US10097806B2 (en) Image processing apparatus, image pickup apparatus, image processing method, non-transitory computer-readable storage medium for improving quality of image
US9727956B2 (en) Image processing apparatus, image pickup apparatus, image processing method, and non-transitory computer-readable storage medium
US10326951B2 (en) Image processing apparatus, image processing method, image capturing apparatus and image processing program
US10304168B2 (en) Image processing apparatus, image pickup apparatus, image processing method, and non-transitory computer-readable storage medium for correcting deterioration of image
US10339637B2 (en) Image processing apparatus, image pickup apparatus, image processing method, and non-transitory computer-readable storage medium for correcting deterioration of image
EP3474532A1 (en) Image processing apparatus, image processing method, and program
US9652832B2 (en) Image processing apparatus, image pickup apparatus, image processing method, and non-transitory computer-readable storage medium which perform image restoration processing
US10334161B2 (en) Image processing apparatus, image processing method, computer program and imaging apparatus
JP2018088587A5 (enExample)
WO2016098282A1 (en) Image processing apparatus, image pickup apparatus, image processing method, image processing program, and storage medium
JP6570406B2 (ja) 画像処理装置、撮像装置、画像処理方法、プログラム
KR101890188B1 (ko) 필름층을 구비한 피사체에 대한 자동초점 조절장치 및 방법
JP2017126952A (ja) 画像処理装置、撮像装置および画像処理プログラム
JP6566780B2 (ja) 画像処理装置、撮像装置、画像処理方法、画像処理プログラム、および、記憶媒体
JP2018092077A (ja) 焦点検出装置、その制御方法、プログラム及び記憶媒体

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16830017

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 15540452

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16830017

Country of ref document: EP

Kind code of ref document: A1