JP6185819B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus having a function of reconstructing a refocused image.

  2. Description of the Related Art In recent years, an imaging apparatus (light field camera) that can acquire not only the intensity distribution of light but also information on the incident direction of light has been proposed in an imaging apparatus such as an electronic camera.

  For example, according to Non-Patent Document 1, a microlens array is arranged between a photographic lens and an image sensor, and one microlens is associated with a plurality of pixels of the image sensor, thereby allowing light that has passed through the microlens. Are obtained for each incident direction by a plurality of pixels.

  In addition to generating a normal captured image for the pixel signal (light ray information) acquired in this way, a technique called “Light Field Photography” is applied to focus on an arbitrary image plane (refocus plane). Can be reconstructed after shooting. With conventional cameras, it was necessary to re-shoot when the desired subject was not in focus, but it was desirable without re-taking by reconstructing the refocus plane using the image signal taken with the light field camera. Images are obtained.

  In addition, Patent Document 1 proposes an imaging apparatus that can switch between an imaging method based on the “Light Field Photography” technique and a normal high-resolution imaging method by moving a microlens array.

JP 2008-312080 A

Ren. Ng and 7 others, “Light Field Photography with a Hand-Held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02

  However, in the conventional technique disclosed in Non-Patent Document 1 described above, since it is necessary to use a plurality of pieces of light information in order to generate one pixel data, an image with a resolution corresponding to the array density of the image sensor is generated. It is difficult. Further, with the conventional technique disclosed in Patent Document 1 described above, a high-resolution image can be taken, but refocus processing cannot be performed with a high-resolution image.

  SUMMARY OF THE INVENTION An object of the present invention is to provide an image processing apparatus capable of generating an image with a resolution corresponding to the array density of image sensors by refocus processing.

  In order to achieve the above object, according to the present invention, an image processing apparatus that processes image data includes first image data acquisition means for acquiring first image data capable of generating a refocus image, A second image data acquisition unit that acquires second image data representing a normal captured image, a refocus unit that generates a refocus image from the first image data, and a refocus image First converting means for converting the image data into first frequency characteristics that are frequency domain information, and second converting means for converting the second image data into second frequency characteristics that are frequency domain information And an extraction means for extracting the high frequency component of the second frequency characteristic, and a synthesis means for synthesizing the high frequency component and the first frequency characteristic to generate frequency domain information of one image, Zhou The information number region and a third conversion means for generating a third image data by converting the information in the spatial domain.

  According to the present invention, it is possible to provide an image processing apparatus that enables refocus processing with an image having a resolution according to the arrangement density of the imaging elements.

The figure which shows the structure of the image processing apparatus which concerns on 1st Example of this invention. The figure which shows the example of the refocus image and its blur amount The figure which shows the example of the result of coordinate conversion of normal image data Diagram showing an example of coordinate transformation The figure which shows the example of a synthesis | combination in the image processing apparatus which concerns on 1st Example of this invention. FIG. 5 is a diagram illustrating a configuration of an image processing apparatus according to a second embodiment of the invention. The figure which shows the example of the normal image data in the 2nd Example of this invention, and its coordinate transformation result The figure which shows the structure of the image processing apparatus which concerns on 3rd Example of this invention. The figure which shows the flowchart of imaging | photography operation | movement of the image processing apparatus which concerns on 3rd Example of this invention. The figure which shows the relationship between the refocusable range and depth of field in the 3rd Example of this invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  The first embodiment of the present invention will be described below with reference to FIGS.

  FIG. 1 shows a configuration example of an image processing apparatus according to the first embodiment of the present invention. The operation of the image processing apparatus according to the present embodiment is controlled by a CPU of a control unit (not shown) executing a program stored in the memory of the control unit. The control unit may be a unit inherent to the image processing apparatus, or may be a control unit of the imaging apparatus when the image processing apparatus is applied to the imaging apparatus, for example.

  In FIG. 1, 100 is an image processing apparatus that processes image data, 101 is light ray information data (first image data), 102 is normal image data (second image data) representing a normal captured image, and 103 is an output. Image data (third image data). The ray information data 101 is image data that can generate an image focused on a specific image plane (hereinafter referred to as a refocus image). The normal image data 102 is image data captured at the same angle of view as the light ray information data 101. The output image data 103 is image data output from the image processing apparatus 100.

  A first refocus processing unit 110 performs a refocus process on the acquired light ray information data 101 (first image data acquisition), and outputs a refocus image. The focus position (refocus position) at the time of the refocus process may be a position desired by the user through an operation unit constituted by buttons or the like, or may be a predetermined position.

  Reference numeral 111 denotes a first coordinate conversion unit that converts the acquired normal image data 102 (second image data acquisition) into frequency domain data (hereinafter referred to as second frequency characteristics) (second conversion means). .

  Reference numeral 112 denotes a second coordinate conversion unit that converts the refocused image into frequency domain data (hereinafter referred to as first frequency characteristic) (first conversion means). The second coordinate conversion unit 112 uses the same conversion method as the first coordinate conversion unit 111. Therefore, as illustrated, the first coordinate conversion unit 111 and the second coordinate conversion unit 112 may be configured separately, or the first coordinate conversion unit 111 and the second coordinate conversion unit 112 may be configured as one. And processing may be performed in a time-sharing manner.

  A blur amount calculation unit 113 calculates the blur amount of the refocus image. The blur amount calculation is performed by dividing the refocus image into a plurality of blocks and calculating the blur amount for each block. Here, the blur amount is a focus shift amount (defocus amount), and can be calculated using, for example, the light beam information data 101 and the focus position refocused by the first refocus processing unit 110 ( Defocus amount calculation means). However, the present embodiment is not limited to this blur amount calculation method.

  Reference numeral 114 denotes a first high-frequency component extraction unit that extracts a high-frequency component (hereinafter, referred to as a normal high-frequency component) in a specific spatial region of normal image data from the second frequency characteristic and the blur amount of the refocused image.

  Reference numeral 115 denotes a synthesis unit that synthesizes the refocus frequency characteristic and the normal high frequency component. Reference numeral 116 denotes a third coordinate conversion unit that converts the processing result of the synthesis unit 115 into data in the spatial domain (third conversion means). The conversion method of the third coordinate conversion unit 116 is an inverse conversion of the conversion method of the first coordinate conversion unit 111 and the second coordinate conversion unit 112. The output of the third coordinate conversion unit 116 is output image data 103.

  Next, details of the operation of the image processing apparatus according to the present embodiment will be described with reference to FIGS. In the following description, the first coordinate transformation unit 111 and the second coordinate transformation unit 112 use discrete wavelet transformation as a coordinate transformation method, and the third coordinate transformation unit 116 uses inverse discrete wavelet transformation as a coordinate transformation method. It is assumed that

  An example of the output result of the first refocus processing unit 110 is shown in FIG. 2A, and an example of the output result of the blur amount calculation unit 113 in that case is shown in FIG.

  FIG. 2A is an example of a refocus image generated by the first refocus processing unit 110. In the figure, reference numerals 210, 211, and 212 are subjects that are within the angle of view of the refocus image, and the refocus image is an image generated by focusing on the subject 210. In this embodiment, it is assumed that the distance from the imaging device that captured the light ray information data has a relationship of subject 210 <subject 211 <subject 212.

  FIG. 2B is a diagram illustrating an example of a result of calculating the blur amount by the blur amount calculation unit 113 with respect to the refocus image in FIG. The example shown in FIG. 2B is an example in which the refocus image in FIG. 2A is divided into 3 × 4 blocks (hereinafter referred to as B blocks), and the amount of blur is calculated for each block. Note that the number of divisions of the refocus image may be determined as appropriate according to limitations on the calculation capability, frame rate, and the like of the image processing apparatus 100.

  In FIG. 2B, 221 is one divided B block, and 230, 231, and 232 indicate the amount of blur in the B block including the subjects 210, 211, and 212. Since the refocus image in FIG. 2A is an image focused on the subject 210, the amount of blur has a relationship of blur amount 230 <blur amount 231 <blur amount 232. Here, the blur amount of the B block not included in the blur amounts 230, 231, and 232 is larger than the blur amount 232.

  3A shows an example of the output result of the first coordinate conversion unit 111, and FIG. 3B shows an example of the output result of the first high frequency component extraction unit 114.

  In the figure, 300 is an example in which the normal image data 102 is divided into two stages by discrete wavelet transform. The division result of the nth stage (n is a natural number) of the discrete wavelet transform for an arbitrary image is expressed as LLn, LHn, HLn, HHn. LLn is a horizontal low frequency component and a vertical low frequency component, LHn is a horizontal low frequency component and a vertical high frequency component, HLn is a horizontal high frequency component and a vertical low frequency component, and HHn is a horizontal direction. The results of extracting the high-frequency component and the high-frequency component in the vertical direction are shown. The division of the (n + 1) th stage is performed only on the LLn, and LL (n + 1), LH (n + 1), HL (n + 1), and HH (n + 1) are generated as in the nth stage.

  The arrangement of each component in the output result shown in FIG. 3 follows the arrangement of each component shown in FIG. In FIG. 4, 411, 412, and 413 indicate division results in one stage, and 420, 421, 422, and 423 indicate division results in two stages. 310 is a result of extracting only a part of the high frequency components of the output result 300 based on the blur amount shown in FIG. A high frequency component at a location corresponding to a B block having a smaller blur amount is set as an extraction target. In the present embodiment, regions other than the region 420 are set as high frequency components, and some high frequency components are further extracted from regions other than the region 420 based on the blur amounts 230, 231, and 232. Specifically, a process is performed in which an area where the amount of blur is larger than a certain threshold value m (m is a natural number) is deleted from the division results after the mth stage. The extraction result of the high-frequency component shown in FIG. 3B is an example in which extraction processing is performed assuming that blur amount 230 <threshold 1 (first stage) <blur amount 231 <threshold 2 (second stage) <blur amount 232. It is. As shown in FIG. 3B, the area corresponding to the blur amount 231 is deleted from the division result of the first stage, and the area corresponding to the blur amount 232 is determined from the division result of the first stage and the second stage. Information was deleted.

  An example of the output result of the synthesis unit 115 is shown in FIG. In the figure, reference numeral 500 denotes a result obtained by dividing the refocus image into 0 stages by the second coordinate conversion unit 112. The output result 500 may be subjected to processing such as upsampling and downsampling so that the resolution of the refocus image is the same as the resolution of the region 420 of the output result 300. In the synthesizing unit 115, as shown in FIG. 5, the low frequency component of the output result 310 of the first high frequency component extracting unit 114 is replaced with the output result 500 of the second coordinate converting unit 112.

  As described above, in this embodiment, when a refocus image is generated from the ray information data 101, a high frequency component is generated according to the defocus amount of the refocus image from the normal image data 102 captured at the same angle of view. Is extracted and combined with the refocused image. This makes it possible to focus on a specific position and generate a refocus image having the same resolution as that of the normal image data 102.

  In the first embodiment described above, the first coordinate conversion unit 111 is described using the discrete wavelet transform in this embodiment, but the present invention is not limited to this. For example, other coordinate conversion means such as discrete Fourier transform may be used. When performing coordinate transformation that does not leave spatial information after transformation, such as discrete Fourier transformation, the ray information data 101 and the normal image data 102 may be divided into a plurality of blocks and processed in units of blocks. When the light ray information data 101 is divided into a plurality of blocks, the light ray information data 101 may be upsampled to the size of the normal image data 102.

  Next, a second embodiment of the present invention will be described with reference to FIGS. In the first embodiment, the extraction region of the high-frequency component of the normal image data is determined according to the blur amount of the refocus image. However, in this embodiment, the re-image is extracted from the ray information data according to the region where the high-frequency component is extracted. A refocus position for generating a focus image is set. In this case, the extraction area is specified by a user setting or the like.

  FIG. 6 is a diagram showing an example of the configuration of the image processing apparatus according to the second embodiment of the present invention. Reference numeral 600 in FIG. 1 denotes an image processing apparatus according to this embodiment. In the figure, the same parts as those in the first embodiment are denoted by the same reference numerals, and the description thereof is omitted unless particularly necessary.

  In FIG. 6, reference numeral 614 denotes a second high-frequency component extraction unit in the present embodiment, which extracts high-frequency components in a specific spatial region of normal image data and performs second refocus processing described later on information on the extracted region. Part 610. As a region for extracting a high frequency component, a user's desired value may be input through an operation unit including buttons or a predetermined position may be used. The user's desired value can be obtained by, for example, selecting a place where the normal image data 102 is desired to be focused.

  Reference numeral 610 denotes a second refocus processing unit in the present embodiment, which performs a refocus process on the light ray information data 101 and outputs a refocus image. The focal position at the time of the refocus process can be determined from the extraction result of the high frequency component of the second high frequency component extraction unit 614. Specifically, the refocus position is selected so that the focus is on the spatial region from which the high frequency component is extracted. Note that the refocus position determination method is not limited to this. For example, when extracting a high frequency component, if the user inputs a place where the user wants to focus, that place may be used as the refocus position.

  The operation of the image processing apparatus in FIG. 6 will be described in more detail with reference to FIG. In the following description, the first coordinate conversion unit 111 and the second coordinate conversion unit 112 are described as those using discrete wavelet conversion as a coordinate conversion method.

  FIG. 7A shows an example of normal image data, and FIG. 7B shows an example of an output result of the second high-frequency component extraction unit 614 corresponding thereto.

  In the angle of view of the normal image data in FIG. 7A, subjects 210, 211, and 212 are photographed as in FIG. In this embodiment, it is assumed that processing is performed so that the subject 210 is focused. Accordingly, the focus position when the second refocus processing unit 610 performs the refocus processing is also adjusted to the subject 210.

  FIG. 7B shows a result of extracting only high frequency component information about the subject 210 to be focused after dividing the stage of FIG. 7A by the discrete wavelet transform into two stages. As shown in FIG. 7B, all information other than the subject to be focused is deleted. Note that the method of extracting the high frequency component is not limited to this method. For example, the processing may be performed in consideration of the distances of the subjects 210, 211, and 212 from the imaging device that captured the normal image data. Further, for example, the blur intensity may be further input and reflected in the extraction result of the high frequency component in the same manner as in the first embodiment. When the blur intensity is increased, a process of not extracting a high frequency component in a region other than the place where the focus is desired may be performed.

  Specifically, a predetermined threshold value associated with distance information is set for each frequency band, and the difference between the subject distance information and the distance in the depth direction in the coordinate information of the extraction region is calculated as a distance difference, and the distance difference Compare with the threshold. And the information of the high frequency component more than the frequency band which a threshold value shows is deleted with respect to the area | region whose distance difference is more than a threshold value.

  As described above, also in this embodiment, when a refocus image is generated from the light ray information data 101, high frequency components are extracted from the normal image data 102 photographed at the same angle of view according to a desired pin and position. And synthesized with the refocus image. This makes it possible to focus on a specific position and generate a refocus image having the same resolution as that of the normal image data 102.

  The third embodiment of the present invention will be described below. The present embodiment is characterized in that, for example, the configuration of the first embodiment is configured to control shooting of normal image data acquired according to a refocusable range. When extracting a high-frequency component to be combined with the refocus image, it is desirable that the normal image data is in focus in all the refocusable ranges. In order to realize this, this embodiment provides a configuration for controlling the aperture of the photographing lens in accordance with the refocusable range. In the present embodiment, a configuration including a lens unit and a photographing unit is presented as a configuration example of the image processing device, but these may be configured as a device to which the image processing device is applied, for example, an imaging device. The gist of the present embodiment is to generate control information for controlling the photographing condition (the photographing lens aperture) in accordance with the refocusable range.

  FIG. 8 is a diagram illustrating an example of the configuration of the image processing apparatus according to the present embodiment. In the figure, the same parts as those in the first embodiment are denoted by the same reference numerals, and the description thereof is omitted unless necessary.

  In the figure, reference numeral 800 denotes an image processing apparatus in this embodiment. Reference numeral 810 denotes a lens unit that constitutes the photographing optical system, and reference numeral 811 denotes a light beam control unit. The lens unit 810 changes the focusing distance (hereinafter referred to as a focus point) and changes the amount of light input to the lens unit 810. Limit by aperture.

  Reference numeral 820 denotes a light beam information photographing unit, and reference numeral 822 denotes a normal image photographing unit. The light ray information photographing unit 820 receives the light output from the lens unit 810 and generates and outputs light ray information data 101. The normal image capturing unit 822 controls the light beam control unit 811 and receives the light output from the lens unit 810 to generate and output normal image data 102.

  Reference numeral 821 denotes a refocusable range calculation unit that calculates a range in the depth direction that can be refocused with the light ray information data 101 from the light ray information data 101 and outputs it. Reference numeral 823 denotes a depth-of-field calculating unit that calculates and outputs the depth of field of an image captured by the normal image capturing unit 822.

  Reference numeral 830 denotes a first refocus processing unit that performs refocus processing on the light ray information data 101 and outputs a refocused image. A range in which refocusing is possible in the first refocus processing unit 830 (hereinafter, refocus allowable range) is determined by receiving outputs from the refocusable range calculation unit 821 and the depth of field calculation unit 823. As the focus position for the refocus processing, a position desired by the user may be input through an operation unit including buttons or the like, or may be a predetermined position.

  Next, the photographing operation in the image processing apparatus 800 will be described using the flowchart of FIG.

  Step S901 is a step in which the light ray information photographing unit 820 generates and outputs the light ray information data 101. Step S902 is a step in which the refocusable range photographing unit 821 calculates the refocusable range of the light ray information data 101.

  Step S903 is a step in which the normal image photographing unit 822 calculates an aperture value (hereinafter, F value) from the refocusable range and the focus distance obtained from the light beam control unit 811. Step S904 is a step in which the normal image capturing unit 822 captures the normal image data 102 with the F value calculated in step S903.

  Here, a method of setting the refocus allowable range in the first refocus processing unit 830 will be described with reference to FIG.

  FIG. 10 is a diagram showing the relationship between the refocusable range, the depth of field of the normal image data 102, and the refocus allowable range when the distance from the image processing apparatus 800 is taken on the horizontal axis. The zero point on the horizontal axis is the position of the image processing apparatus 800.

  FIG. 10A is a diagram showing a relationship when the depth of field of the normal image data 102 is shallower than the refocusable range. FIG. 8B is a diagram showing a relationship when the depth of field of the normal image data 102 is deeper than the refocusable range. In the figure, reference numerals 1010 and 1030 denote focus points of the lens unit 810. Reference numerals 1001 and 1021 denote refocusable ranges. Reference numerals 1002 and 1022 denote the depth of field of the normal image data 102. Reference numerals 1003 and 1023 denote allowable refocus ranges.

  As shown in the figure, the refocus allowable ranges 1003 and 1023 are “(refocusable range 1001 and 1021) AND (depth of field 1002 and 1022)”.

  Here, the focus point when the light information capturing unit 820 captures the image and the focus point when the normal image capturing unit 822 captures the image are described as being the same, but this is not restrictive. Each focus point may be different. In this case, however, the refocusable ranges 1001 and 1021 and the depths of field 1002 and 1022 need to be controlled so as to overlap each other at least partially.

  Further, although the depth of field after refocusing is not considered in the above description, a refocusable range that can be generated as the output image 103 may be set in consideration of this.

  In the present embodiment described above, the light beam at the time of shooting the normal image data 102 is controlled by the light beam controller 811 in response to the result of the refocusable range of the light beam information data 101, but this is not restrictive. For example, after the depth of field of the normal image data 102 is calculated by the depth of field calculation unit 823, the light information photographing unit 820 controls the light beam at the time of photographing through the light ray control unit 811 in response to this result. good.

  In this embodiment, the method for obtaining the depth of field is calculated based on an image photographed by the normal image photographing unit 822, but the present invention is not limited to this. The calculation may be performed using the state of the light beam control unit 811 when the normal image capturing unit 822 captures an image.

  Although the present embodiment has a refocus image generation configuration based on the configuration of the image processing apparatus of the first embodiment, the refocus image generation configuration of the image processing apparatus of the second embodiment. You may have.

  As described above, also in this embodiment, when a refocus image is generated from the light ray information data 101, a high frequency component is extracted from the normal image data 102 photographed at the same angle of view and synthesized with the refocus image. doing. This makes it possible to focus on a specific position and generate a refocus image having the same resolution as that of the normal image data 102.

  The object of the present invention can also be achieved by supplying a storage medium storing software program codes for realizing the functions of the above-described embodiments to a system or apparatus. That is, it goes without saying that the object of the present invention can also be achieved when the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code itself and the storage medium storing the program code constitute the present invention.

  As a storage medium for supplying the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  In addition, the functions of the above-described embodiment can also be realized when an OS (basic system or operating system) operating on the computer performs part or all of the actual processing based on the instruction of the program code read by the computer. Realized. Needless to say, this case is also included in the present invention.

  Furthermore, after the program code read from the storage medium is written to the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer, the processing based on the instruction of the program code is also performed. Included in the invention. That is, it goes without saying that the present invention also includes the case where the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing based on the instruction of the program code to realize the functions of the above-described embodiment. Yes.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

Claims (20)

In an image processing apparatus that processes image data,
First image data acquisition means for acquiring first image data capable of generating a refocus image;
Second image data acquisition means for acquiring second image data representing a normal captured image corresponding to the first image data;
Refocusing means for generating a refocused image from the first image data;
First conversion means for converting the image data of the refocused image into a first frequency characteristic which is information in a frequency domain;
Second conversion means for converting the second image data into a second frequency characteristic which is information of a frequency domain;
Extraction means for extracting a high frequency component of the second frequency characteristic;
A synthesizing unit that synthesizes the high frequency component and the first frequency characteristic to generate information of a frequency region of one image;
An image processing apparatus comprising: a third conversion unit configured to convert the synthesized frequency domain information into spatial domain information to generate third image data.   The image processing apparatus according to claim 1, wherein the extraction unit extracts, from the second frequency characteristic, a high frequency component that is equal to or higher than a frequency band indicated by the first frequency characteristic.   The defocus amount calculation means for calculating the defocus amount of the refocus image is further provided, and the extraction means extracts a high frequency component from the second frequency characteristic according to the defocus amount. Or the image processing apparatus of 2.   The extraction means compares the defocus amount with a predetermined threshold associated with the defocus amount for each frequency band, and when the defocus amount is equal to or greater than the threshold, a high frequency equal to or greater than the frequency band indicated by the threshold The image processing apparatus according to claim 3, wherein a component is deleted from an area corresponding to the defocus amount.   The defocus amount calculating unit divides the refocus image into a plurality of blocks to calculate a defocus amount for each block, and the extracting unit deletes the high frequency component in units of blocks. The image processing apparatus according to claim 3 or 4.   The extraction unit outputs information on a spatial region from which the high-frequency component has been extracted, and the refocusing unit generates a refocus image from the first image data so that the spatial region is focused according to the information. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   The image processing apparatus according to claim 6, further comprising an input unit that inputs coordinate information of the spatial region, wherein the extraction unit extracts the high-frequency component in accordance with the coordinate information.   8. The extraction means acquires subject distance information, which is subject distance information of the second image data, and extracts a high-frequency component based on the coordinate information and the subject distance information. An image processing apparatus according to 1.   The extraction unit calculates a difference between the subject distance information and a distance in the depth direction in the coordinate information, compares the difference with a threshold value associated with distance information for each frequency band, and the difference is equal to or greater than the threshold value. The image processing apparatus according to claim 8, wherein information on a high frequency component equal to or higher than a frequency band indicated by the threshold is deleted from the region. A depth of field calculating means for calculating a depth of field of the second image data;
Refocusable range calculating means for calculating a refocusable range using the first image data;
2. The apparatus according to claim 1, further comprising: a control unit configured to generate control information on imaging conditions of the second image data based on the refocusable range and the focus distance of the second image data. The image processing device according to any one of claims 1 to 9.   The image processing apparatus according to claim 10, wherein the control unit generates control information for controlling the photographing condition so that the depth of field includes at least a part of the refocusable range.   The image processing apparatus according to claim 10, wherein the control unit generates control information for controlling the photographing condition so that the refocusable range includes at least a part of the depth of field.   The refocusing unit sets an area included in both the refocusable range and the depth of field as a refocus allowable range based on the refocusable range and the depth of field. The image processing apparatus according to claim 10, wherein the image processing apparatus is characterized.   The image processing apparatus according to claim 10, wherein the photographing condition is an aperture value of a photographing optical system.   The image processing apparatus according to claim 1, wherein the first image data and the second image data are images having the same angle of view. In an image processing method for processing image data,
A first image data acquisition step of acquiring first image data capable of generating a refocus image;
A second image data acquisition step corresponding to the first image data and acquiring second image data representing a normal captured image;
A refocusing step for generating a refocused image from the first image data;
A first conversion step of converting the image data of the refocused image into a first frequency characteristic which is information of a frequency domain;
A second conversion step of converting the second image data into a second frequency characteristic which is frequency domain information;
An extraction step of extracting a high frequency component of the second frequency characteristic;
A synthesis step of synthesizing the high frequency component and the first frequency characteristic to generate frequency domain information of one image;
And a third conversion step of converting the synthesized frequency domain information into spatial domain information to generate third image data.   A program causing a computer to execute the image processing method according to claim 16.   A computer-readable storage medium storing a program for causing a computer to execute the image processing method according to claim 17.   A program causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 15.   A computer-readable storage medium storing a program that causes a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 15.

Images