JP2015188251A - Image processing system, imaging apparatus, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to intuitively grasp a focusing state of a subject during imaging or image editing.SOLUTION: The image processing system includes: means for generating a rearrangement image formed by rearranging a subject region of photographed image data in accordance with a distance from an imaging apparatus; and means for generating an image to which information concerning a focusing state when performing image composition processing by using the photographed image data including the rearrangement image is reflected.

Description

  The present invention relates to an image processing device, an imaging device, an image processing method, and a program for obtaining information related to subject focus control.

  In light field photography, the direction and intensity (light field) of light rays passing through each position are calculated from a plurality of viewpoints of image data. Then, using the obtained light field information, an image when passing through a virtual optical system and forming an image on a virtual sensor is calculated. By appropriately setting such virtual optical systems and sensors, the above-described refocusing can be performed. Known imaging devices for acquiring a light field include a Plenoptic Camera (for example, Patent Document 1) in which a microlens array is placed behind a main lens, and a camera array in which small cameras are arranged. In any case, image data of a plurality of viewpoints obtained by imaging the subject from different directions can be obtained by one imaging. In other words, light field photography can be paraphrased as calculating an image acquired by a virtual sensor under virtual optical conditions from image data of a plurality of viewpoints. Hereinafter, the process of calculating an image acquired by the virtual sensor is referred to as “refocus process”. As the refocusing process, there is known a method in which acquired image data of a plurality of viewpoints are subjected to projective transformation on a virtual sensor, added and averaged (for example, Patent Document 2).

  As a method for displaying a refocused image (hereinafter, refocused image) while changing the focus position, for example, there is a method disclosed in Patent Document 3. In the method of Patent Document 3, a user interface (UI) for adjusting a focus position is prepared on a screen on which a refocus image is displayed, and the focus position is changed via the UI. Further, Patent Document 3 discloses a method in which a user indicates a subject to be focused on a screen on which a refocus image is displayed, and the refocus image with the focus position adjusted is displayed on the subject. Is disclosed.

JP 2009-124213 A WO20080504904 US Patent 2008/0131019

Ren Ng, “Fourier Slice Photography”, 2005 ACM Trans. Graph. 24, p735-744

  In other words, in the conventional methods disclosed in the above-mentioned patent documents and the like, it is not specified which range of subjects can be refocused at the time of shooting or image editing, so that the user can perform intended shooting or image editing. There is a problem that it is difficult to do.

  An image processing apparatus according to the present invention includes: a unit that generates a rearranged image in which a region of a subject in captured image data is rearranged according to a distance from the imaging device; and the captured image data including the rearranged image. And a means for generating an image reflecting information relating to an in-focus state when performing image composition processing.

  According to the present invention, the user can intuitively grasp the in-focus state of the subject at the time of shooting or image editing.

1 is a block diagram illustrating an internal configuration of a multi-eye imaging device according to Embodiment 1. FIG. It is an example of the structure which has arrange | positioned the lens array to the image side of an imaging optical system. It is an example of the structure which has arrange | positioned the lens array to the image side of an imaging optical system. It is an example of the structure which has arrange | positioned the lens array to the image side of an imaging optical system. It is an example of the structure (camera array) which arranged the some imaging optical system. It is a figure showing the relationship between a lens array and an image pick-up element. FIG. 3 is a diagram illustrating a relationship between expressions representing a refocus coefficient α _± in the first embodiment. FIG. 3 is a block diagram illustrating an internal configuration of an image processing unit according to the first embodiment. 6 is a flowchart illustrating a flow of processing in the image processing unit according to the first embodiment. (A) is a figure which shows an example of a single viewpoint image, (b) is a figure which shows the distance map as distance information derived | led-out about the single viewpoint image. It is a figure which shows an example of the result of having extracted the to-be-photographed area | region from the distance map. It is a figure which shows an example of the rearrangement image when the scene which concerns on a single viewpoint image is seen from right above. It is the figure which showed the optical arrangement | positioning in the structure of the imaging part shown in FIG. It is a figure which shows an example of a focusing information display image. It is a figure which shows the variation of a focusing information display image. It is a figure which shows an example in the case of displaying a single viewpoint image and / or a synthesized image together on a focusing information display image. FIG. 4 is an enlarged view centering on a lens array and an imaging element portion in the configuration of the imaging unit shown in FIG. 3. FIG. 5 is a diagram illustrating a relationship between expressions representing a refocus coefficient α _± in the second embodiment. It is the figure which looked at the image pick-up part which has the composition of a camera array from the front (object side). FIG. 10 is a diagram (cross-sectional view) of an imaging optical system and an image sensor according to Example 3 as viewed from the side. FIG. 6 is a diagram illustrating a relationship between expressions representing a refocus coefficient α _± in the third embodiment.

<Example 1>
In the present embodiment, an aspect will be described in which a refocusable range can be intuitively grasped using an image in which the coordinates (arrangement) of each subject area is changed according to the distance for each subject area.

  FIG. 1 is a block diagram illustrating an internal configuration of a multi-view imaging apparatus according to the present embodiment. The imaging unit 1500 receives light information of a subject with an imaging element, A / D converts the received signal, and acquires image data (digital data). This image data is stored in a recording medium such as an SD card as imaging data in response to an imaging instruction from the user. The image data acquired by the imaging unit 1500 according to the present embodiment is image data with parallax obtained by capturing the subject space from a plurality of viewpoints (hereinafter referred to as “parallax image data”). The image data acquired by the imaging unit 1500 is also used for a so-called live view function that sequentially displays in real time on a display unit 1506 provided on the back surface of the imaging device. Hereinafter, an image stored in a recording medium in response to an imaging instruction from a user is referred to as a recorded image, and an image displayed in real time in a live view display is referred to as a live view image.

  A central processing unit (CPU) 1501 comprehensively controls each unit described below. A RAM 1502 functions as a main memory, work area, and the like for the CPU 1501. The ROM 1503 stores a control program executed by the CPU 1501 and the like. The bus 1504 is a transfer path for various data. For example, digital data acquired by the imaging unit 1500 is sent to a predetermined processing unit via the bus 1504. The operation unit 1505 that receives a user instruction includes buttons, a mode dial, and the like. For example, a liquid crystal display is used as the display unit 1506 that displays images and characters. The display unit 1506 may have a touch screen function. In that case, a user instruction using the touch screen can be handled as an input of the operation unit 1505. In this embodiment, when a refocusable range is displayed and a focus position at the time of refocusing is specified, it is specified by a user input via such a touch screen.

  A display control unit 1507 performs display control of images and characters displayed on the display unit 1506. The imaging unit control unit 1508 controls the imaging system based on an instruction from the CPU 1501 such as focusing, opening / closing the shutter, and adjusting the aperture stop. A digital signal processing unit 1509 performs various processes such as white balance processing, gamma processing, and noise reduction processing on the digital data received via the bus 1504. The encoder unit 1510 performs processing for converting digital data into a file format such as JPEG or MPEG. The external memory control unit 111 is an interface for connecting to a PC or other media (for example, hard disk, memory card, CF card, SD card, USB memory). The image processing unit 1512 performs image processing such as refocus processing described later from the image data acquired by the imaging unit 1500 or the image data output from the digital signal processing unit 1509. Details of the image processing unit 1512 will be described later. The exposure state prediction unit 1513 predicts the exposure state of the imaging unit 1500 at the time of imaging. Although there are other components of the image pickup apparatus other than those described above, the description thereof is omitted because it is not the main point of the present embodiment.

  First, the configuration of the imaging unit 1500 will be described in detail.

  As described above, the imaging unit 1500 acquires parallax image data, that is, light field data. Configuration examples of the imaging unit 1500 are shown in FIGS. 2 to 4 show a configuration in which a lens array is arranged on the image side of the imaging optical system, and FIG. 5 shows a configuration (camera array) in which a plurality of imaging optical systems are arranged. The configuration of the imaging unit is preferably a multi-view configuration that can simultaneously obtain images from a plurality of viewpoints as shown in FIGS. In the method of imaging multiple times while changing the position using a monocular imaging device, the parallax image data obtained by imaging the subject space at different times is obtained, and if there is a moving object in the subject space, correct parallax information is obtained. It is because it cannot be obtained.

  The parallax image data acquired by the imaging unit 1500 having the configuration illustrated in FIGS. 2 to 5 is subjected to processing such as pixel extraction, rearrangement, and composition, thereby performing refocusing, depth of field control, viewpoint change, and the like. It can be performed. Hereinafter, these processes are referred to as an image composition process, and an image generated by the image composition process is referred to as a composite image. The composite image may be subjected to processing such as noise reduction and processing such as depth of field control. A range in which refocusing is possible in the subject space is referred to as a “focus control range”.

  Here, the internal configuration of the imaging unit 1500 shown in FIG. 2 will be described in detail.

  The lens array 1603 is disposed on the image side conjugate surface with respect to the subject surface 1601 of the imaging optical system 1602. Further, the lens array 1603 is configured such that the exit pupil of the imaging optical system 1602 and the image sensor 1604 are substantially conjugate. Light rays from the subject surface 1601 are incident on different pixels of the image sensor 1604 according to the position and angle of the light rays on the subject surface 1601 via the imaging optical system 1602 and the lens array 1603. Thereby, parallax image data (light field data) is acquired. Here, the lens array 1603 has a role of preventing light rays that have passed through different positions on the subject surface 1601 from entering the same pixel. As a result, the image sensor 1604 acquires an image in which pixel groups in which the same region on the subject surface 1601 is captured from a plurality of viewpoints are arranged.

  Note that an object such as a person or a building is not necessarily present on the subject surface 1601 in FIGS. This is because the focus position can be adjusted after imaging to a person or a building existing behind or in front of the subject surface 1601 by refocusing.

<Refocusing principle>
From here, the refocus processing will be described. Since refocusing is described in detail in Non-Patent Document 1, it will be briefly described here. The basic principle of refocus is the same in any of the configurations shown in FIGS. Here, the configuration of FIG. 2 will be described as an example. In FIG. 2, since the pupil of the imaging optical system is divided into nine two-dimensionally (three one-dimensional), nine viewpoint images are acquired. Here, an image corresponding to a certain divided pupil is referred to as a single viewpoint image. Since the nine single viewpoint images have parallax with each other, the relative positional relationship of the subject on the image changes according to the subject distance. When single-viewpoint images are combined so that a certain subject overlaps, subjects located at different subject distances are shifted and combined. Due to this shift, subjects located at different subject distances are blurred. The blur at this time is determined by the pupil corresponding to the single viewpoint image used for the synthesis. When all nine single viewpoint images are synthesized, the blur of the image acquired by the imaging optical system 1602 can be reproduced. Since the subject to be superimposed by combining the single-viewpoint images is arbitrary, the imaging optical system 1602 can reproduce an image focused on the arbitrary subject. This is the focus control after imaging, that is, the principle of refocusing.

  Here, a method of generating a single viewpoint image in the present embodiment will be described. FIG. 6 is a diagram showing the relationship between the lens array 1603 and the image sensor 1604 in FIG. A broken-line circle 2001 represents a pixel region where a light beam that has passed through one lens (microlens) constituting the lens array 1603 enters. Although FIG. 6 corresponds to the case where a plurality of microlenses are arranged in a lattice shape, the arrangement of the microlenses is not limited to this. For example, it may be arranged with sixfold symmetry (honeycomb structure), or each microlens may be slightly shifted from the regular arrangement. A hatched area 2002 in FIG. 6 represents a pixel on which a light beam that has passed through the same pupil area of the imaging optical system 1602 enters. Therefore, by extracting the pixels in the hatched area 1602, a single viewpoint image in which the subject space is viewed from the lower part of the pupil of the imaging optical system 1602 can be generated. Similarly, another single viewpoint image can be generated by extracting pixels having the same relative position with respect to each circle 2001 of the broken line.

<Focus control range>
Next, a focus control range that is a refocusable range will be described.

  Since refocusing is performed by superimposing single-viewpoint images, it is impossible to refocus the subject that is blurred in each single-viewpoint image. This is because high-frequency components are not obtained even if the blurred images are superimposed, and the blurred images remain.

  Here, the smaller the pupil is divided, the deeper the depth of field in each single-viewpoint image, so the focus control range is expanded. That is, the focus control range depends on the divided pupil of the imaging optical system 1602. However, the depth of field and the focus control range in a single viewpoint image do not necessarily match. This is because the focus control range changes depending on the resolution ratio between the single viewpoint images and the synthesized image obtained by synthesizing them. For example, when the resolution of the composite image is lower than the resolution of the single viewpoint image of each viewpoint, the sampling pitch of the spatial component in the composite image is larger than that of the single viewpoint image. Therefore, the synthesized image has a deeper depth of field than the single-viewpoint image, and the focus control range is expanded accordingly. On the contrary, when the resolution of the composite image is higher than that of the single viewpoint image, the focus control range becomes narrower than the depth of field of the single viewpoint image.

  As a method for making the resolution of the composite image higher than that of the single-viewpoint image, use of pixel-shifted super-resolution is conceivable. As described above, refocusing is performed by shifting single-viewpoint images from each other. If the shift amount at this time is a non-integer multiple of the pixel pitch of the single-viewpoint image, super-resolution with pixel shift can be performed, and the resolution of the composite image can be increased.

  From the discussion so far, it can be seen that in order to acquire an accurate focus control range in the composite image, it is necessary to consider the conditions for compositing the single viewpoint image.

Next, how the focus control range of the composite image is derived will be described. Consider the depth of focus corresponding to the depth of field of the composite image. Let ε be the size of the permissible circle of confusion of the depth of focus, and Δu be the sampling pitch of the angle component of the light ray. At this time, the refocus coefficient α _± is given by the following equation (4).

The image side refocusable range α ₊ s _{2 to} α _- s ₂ expressed by the equation (4) is a range conjugate to the image side optical system 1602 (range conjugate to the imaging optical system 1602). This is the focus control range that is the focusable range. FIG. 7 is a diagram showing the relationship of the expression (4), and the center position of the refocusable range is the focus position of the imaging optical system 1602. (In this case, the position of the lens array 1603 is the center position of the refocusable range). Here, s ₂ is the distance between the image-side main plane of the imaging optical system 1602 and the image-side conjugate surface of the imaging optical system 1602 with respect to the subject surface 1601. In FIG. 7, the image side focus control range refers to a range conjugate with the focus control range via the imaging optical system 1602. Further, [Delta] y is the sampling pitch of the two-dimensional intensity distribution of the light, equal to the pitch delta _LA of the lens array 1603 in the configuration of FIG. In addition, the relationship of Formula (4) is materialized also in any structure of FIGS.

  Expression (4) can be approximated as the following expression (5) because the pixel pitch Δ of the image sensor 1604 is sufficiently small with respect to the exit pupil distance P of the imaging optical system 1602.

  Here, the exit pupil distance P of the imaging optical system 1602 is the distance between the exit pupil plane of the imaging optical system 1602 and the image side conjugate plane of the imaging optical system 1602 with respect to the object plane 1601. N is the number of one-dimensional divisions of the pupil of the imaging optical system 1602, and F is the F value of the imaging optical system 1602. The F value can be derived from the focal length f of the imaging optical system 1602 and the effective lens diameter D by the following equation (6).

That is, the refocusable range (focus control range) is determined according to the angle of view defined by the number of primary divisions of the pupil of the imaging optical system, the focal length of the imaging optical system, and the effective lens diameter. . Furthermore, a refocusable range is determined in accordance with the resolution of composite image data to be described later.

<Image processing unit>
Next, the configuration of the image processing unit will be described in detail.

  FIG. 8 is a block diagram illustrating an internal configuration of the image processing unit 1512 according to the present embodiment.

  The parallax image data acquisition unit 2201 acquires parallax image data supplied from the imaging unit 1500 or the like.

  The optical parameter acquisition unit 2202 acquires the optical parameters of the imaging unit 1500 required by the distance deriving unit 2203 and the focus control range deriving unit 2207. Here, the optical parameters are the configuration of the imaging unit 1500, the exposure state of the diaphragm, the focus position, the focal length of the zoom lens, and the like.

  The distance deriving unit 2203 derives distance information to the subject in the scene. The distance to the subject is derived using the parallax image data supplied from the parallax image data acquisition unit 2201 and the optical parameters supplied from the optical parameter acquisition unit 2202. Details of the distance derivation will be described later.

  The subject area extraction unit 2204 performs processing for extracting a subject area to be processed by the rearranged image generation unit 2205 based on the subject in the scene and the distance information supplied from the distance deriving unit 2203. This subject area extraction process can be rephrased as a process of separating the subject area at regular intervals.

  The rearranged image generation unit 2205 performs processing to change the position (coordinates) of each subject area extracted by the subject area extraction unit 2204 according to the distance information supplied from the distance deriving unit 2203. By this processing, an image (hereinafter referred to as “rearranged image”) in which each subject area is rearranged in accordance with the distance from the imaging device so as to easily grasp the sense of distance for each subject area is generated.

  An image synthesis parameter acquisition unit 2206 acquires parameters necessary for image synthesis (resolution after synthesis, viewpoint, focus position, depth of field, etc.).

  The focus control range derivation unit 2207 derives the focus control range using the optical parameters supplied from the optical parameter acquisition unit 2202 and the image synthesis parameters supplied from the image synthesis parameter acquisition unit 2206.

  A focus position acquisition unit 2208 acquires a focus position at the time of image composition based on a user input via the operation unit 1505. In this embodiment, based on the rearranged image generated by the rearranged image generation unit 2205, the distance specified by the user is specified as the focus position (focus position) via the touch screen as the operation unit 1505. It will be. Information regarding the designated focus position is sent to the display image generation unit 2209.

  The display image generation unit 2209 reflects an image (hereinafter referred to as “focus”, including information related to the focus state (focus position, focus control range, depth of field, etc.) on the rearranged image generated by the rearranged image generation unit 2205. Called a “focus information display image”). At the time of generation, the optical parameter, the image synthesis parameter, the focus control range, and the focus position supplied from each unit described above are used. The generation of the focus information display image may be performed on a live view image before imaging, or may be performed on a recorded image after imaging. Further, the focus information display image may be based on any single viewpoint image included in the parallax image, or may be based on a composite image obtained by combining a plurality of single viewpoint images. Also good. The generated focus information display image data is output to the display unit 1506. Furthermore, the display image generation unit 2209 according to the present embodiment also performs image synthesis processing according to the image synthesis parameter using the parallax image data, but the display image generation unit 2209 performs the image synthesis processing. You may provide separately.

  FIG. 9 is a flowchart illustrating the flow of processing in the image processing unit 1512 according to the present embodiment. This series of processing is performed by reading a computer-executable program describing the following procedure from the ROM 1503 onto the RAM 1502 and then executing the program by the CPU 1501.

  In step 2301, the parallax image data acquisition unit 2201 acquires parallax image data acquired by the imaging unit 1500. As described above, the parallax image data acquired here may be parallax image data as a live view image or parallax image data as a recorded image.

  In step 2302, the optical parameter acquisition unit 2202 and the image synthesis parameter acquisition unit 2206 acquire the optical parameter and the image synthesis parameter, respectively.

  In step 2303, the distance deriving unit 2203 derives distance information in the entire scene. Specifically, the distance of the scene is derived by a method such as performing stereo matching between single viewpoint images included in the parallax image. For example, in the stereo matching method, a reference single-viewpoint image (for example, a single-viewpoint image corresponding to a light beam passing near the center of the pupil) is divided into a plurality of blocks, and the target image is selected from other single-viewpoint images. The block position closest to the next block is detected. Then, using the positional relationship between the two blocks, the positional relationship between the corresponding pupils, and the angle of view of each pupil, the distance of the target block is derived by triangulation. Note that a single viewpoint image whose viewpoint position is other than the center may be selected as the reference single viewpoint image. In addition to the stereo matching method described above, methods for obtaining distance information include a method such as DFD (Depth From Defocus), and a method using a distance measuring unit using infrared rays or the like.

  FIG. 10A shows an example of a single-viewpoint image, and FIG. 10B shows a distance map as distance information derived for the single-viewpoint image of FIG. In the scene obtained from the single viewpoint image shown in FIG. 10A, three types of subjects (persons, buildings, and mountains) exist at different distances based on the imaging device. The distance map shown in FIG. 10B is displayed in shades according to the distance from the imaging device, and the subject “person” in the vicinity of the imaging device is the darkest color and the subject far from the imaging device. The “mountain” is displayed in the brightest color, and the subject “building” between the person and the mountain is displayed in an intermediate brightness color. As described above, in this step, distance information of the subject in the scene is derived.

  Returning to the flowchart of FIG.

In step 2304, the subject area extraction unit 2204 performs processing for extracting the subject area based on the distance map for the entire scene derived in step 2303. This subject area extraction process can be rephrased as a process of separating the subject area at regular intervals. Image areas having substantially the same distance are grouped, and the grouped image areas are extracted as subject areas. As the subject region extraction method, for example, the following method can be used in addition to the subject distance information.
1) A technique such as face recognition is applied to parallax image data to identify an area where a person, animal, plant, or the like exists, and the identified person or the like is extracted as a subject area.
2) Image data of representative objects of interest such as people, animals, and plants are prepared in advance as templates, and subject areas are extracted by pattern matching with these templates.
3) Pre-learning like a neural network, using the learning results to recognize main objects and extracting subject areas.

  Extraction of the subject region in the present embodiment can be realized by applying various known methods such as those described in 1) to 3) other than the method using the distance information of the subject.

  FIGS. 11A and 11B are diagrams showing an example of the result of extracting the subject area from the distance map shown in FIG. 10B, and FIG. 11A shows the subject along the contour of each subject. When the region is extracted, (b) is a case where a rectangular region circumscribing each subject is extracted as the subject region. In (a) and (b) of FIG. 11, each region surrounded by a dotted line is a subject region corresponding to each of “person”, “building”, and “mountain”. As the subject region extraction method, other methods can be applied as long as the region is set so as to include the main subject. For example, the subject area may be defined by an arbitrary shape such as a circle, an ellipse, a triangle, or a polygon.

  Returning to the flowchart of FIG.

  In step 2305, the rearranged image generation unit 2205 generates a rearranged image of the scene using the distance information in the entire scene derived in step 2303 and the subject area in the scene extracted in step 2304.

  FIGS. 12A to 12D are diagrams illustrating examples of rearranged images when the scene related to the single viewpoint image illustrated in FIG. 10A is viewed from directly above.

In any of (a) to (d) of FIG. 12, the position close to the imaging device is the lower side, the position far from the imaging device is the upper side, and “person” is arranged at the closest position to the imaging device. Is disposed at a position farthest from the imaging device. FIG. 12A shows an image region cut out from the single viewpoint image along the subject region extracted in step 2304 (see FIG. 11A above) at a position corresponding to each subject distance. It is the rearranged image rearranged. In the case of this rearranged image, a part of the single viewpoint image is arranged in association with the subject distance, so that the rearranged image closest to the actual image is obtained. (B) in FIG. 12 arranges image areas extracted along the subject area from the distance map derived in step 2303 (see FIG. 10 (b)) at positions corresponding to the subject distances. This is a corrected rearranged image. In the case of this rearranged image, the rearranged image with the subject distance emphasized is obtained. 12C and 12D, the frame of the subject area extracted in step 2304 (see FIGS. 11A and 11B above) is arranged at a position corresponding to each subject distance. This is a corrected rearranged image. In the case of these rearranged images, the rearranged images are simplified.

  Returning to the flowchart of FIG.

  In step 2306, the focus control range deriving unit 2207 derives the focus control range in the case where image synthesis is performed on the parallax image acquired in step 2301 using the optical parameter and image synthesis parameter acquired in step 2302. This will be described in detail below.

As described above, the focus control range changes depending on the resolution ratio between each single-viewpoint image and the synthesized image. Here, for the sake of simplicity, a case where the resolution ratio is 1 is considered. If the image-side focus control range is within the range of the above equation (5), it is possible to acquire a focus-controllable area. Therefore, the image side conjugate surface of the imaging optical system 1602 with respect to the object surface 1601 and the image side focus control range d _refocus only need to satisfy the following expression (7).

FIG. 13 is a diagram illustrating an optical arrangement in the configuration of the imaging unit 1500 illustrated in FIG. 2, and σ is the distance between the image-side main plane of the lens array 1603 and the imaging device 1604. In FIG. 13, two parallel broken lines extending from both ends of the central microlens represent an area of the image sensor 1604 corresponding to the microlens, and hatched pixels in the image sensor 1604 represent a dead zone where no light beam is incident. . In this embodiment, the lens array 1603 is configured so as not to generate a dead zone. In this case, Δ _LA = NΔ. However, the configuration is not limited to this, and a dead zone may exist. From FIG. 13, it can be understood that NF = σ / Δ geometrically. Here, the magnitude of the permissible circle of confusion which determines the depth of focus and characterized by sampling pitch [Delta] y = delta _LA spatial component, equation (7) can be rewritten as the following equation (8).

  Next, consider a general case where the resolution ratio between the single-viewpoint image and the composite image is other than 1. Since the angle of view of the synthesized image and the single-viewpoint image used for the synthesis are the same, when the resolution ratio is other than 1, Δy of the two is different. In general, the allowable circle of confusion is smaller as Δy is smaller and larger as it is larger. Therefore, the above formula (8) can be expanded as the following formula (9) by taking the ratio of the single-viewpoint image and the composite image Δy.

Here, R _mono represents the resolution of a single viewpoint image used for synthesis, and R _comb represents the resolution of the synthesized image. By taking the square root of the ratio of R _mono and R _comb , the ratio Δy is obtained. It can be seen from FIG. 13 that the resolution R _mono of the single viewpoint image is expressed by the following equation (10).

Here, R _total is the number of effective pixels of the image sensor 1604. From the expressions (9) and (10), the following conditional expression (11) that the image-side focus control range should satisfy can be obtained.

In the conditional expression (11), the range is ± 10.0. By setting this to a range of ± 6.0, a sharper composite image can be obtained. More desirably, a sharper composite image can be obtained by setting the range to ± 3.0.

Specific examples of each value are shown below.
The number of effective pixels R _total of the image sensor 1604: 46.7 × 10 ⁶ (pix)
The distance σ between the image side main plane of the lens array 1603 and the image sensor 1604: 0.0374 (mm)
-Pixel pitch Δ of the image sensor 1604: 0.0043 (mm)
Lens array 1603 pitch Δ _LA : 0.0129 (mm)
The focal length f of the imaging optical system 1602: 14.0 (mm)
-F value: 2.9
・ One-dimensional pupil division number N: 3
・ Resolution per single viewpoint image R _mono : 5.2 × 10 ⁶ (pix)
The range of the conditional expression (11), the resolution R _{comb of} the composite image corresponding to each range of the conditional expression (11), and the d _refocus corresponding to each resolution of the composite image are, for example, as shown in the following table (1).

By the user input via the operation unit 1505, the resolution R _{comb of the} composite image is selected from the above three types, for example.

In the above example, for example, in order to generate a composite image of 8.0 × 10 ⁶ pix, since the resolution R _mono per single viewpoint image is 5.2 × 10 ⁶ pix, pixel-shifted super-resolution It can be seen that higher resolution is required. Note that the value of each resolution R _comb of the above-described composite image is an example, and the types are not limited to three. R _comb and d _refocus that satisfy the conditional expression (11) may be determined as appropriate.

  The focus control range can be calculated from the imaging formula using the image side focus control range and the focal length, focus position, and the like of the imaging optical system 1602. The information on the focus control range acquired in this way may be attached to parallax image data or composite image data as tag information and stored in the ROM 1503. Alternatively, a table of focus control ranges corresponding to various conditions is created in advance and stored in the ROM 1503, and instead of performing the above-described calculation for deriving the focus control range, data corresponding to the input conditions is read. It may be.

  As another method of acquiring the focus control range, a method of actually generating a refocused composite image and evaluating the contrast of the subject existing at the focus position is conceivable. However, with this method, since it is necessary to generate a composite image while moving the focus position and sequentially determine whether or not refocusing has been performed, processing takes time. In addition, when the subject does not exist at the refocused focus position, the evaluation cannot be performed, and thus an accurate focus control range cannot be acquired. Therefore, it is desirable to use the method described above for acquiring the focus control range.

  Returning to the flowchart of FIG.

  In step 2307, the display image generation unit 809 reflects the information on the focus state (focus position, focus control range, depth of field, etc.) on the rearranged image generated in step 2305. Is generated. At the time of generation, the optical parameters and image synthesis parameters acquired in step 2302 and the focus control range derived in step 2306 are used.

  FIGS. 14A and 14B are examples of the focus information display image generated based on the rearranged image shown in FIG. 12A, and include a focus position 2801, a focus control range 2802, Each information of the depth of field 2803 is displayed. A focus position 2801 indicated by a black square mark on the slider bar indicates a focus position of the lens at the time of imaging included in the optical parameter or a focus position designated by user input described later. In the focus information display image of (a), the focus position 2801 is at the position of the subject distance d0 (position of the person), and in the focus information display image of (b), the focus position 2801 is at the position of the subject distance d0 ′ (position of the building). Is set. A focus control range 2802 indicated by diagonal lines on the slider bar indicates a range from the subject distance d1 (d1 ′) to the subject distance d2 (d2 ′). In this embodiment, “person” and “building” It can be seen that the distance is within the focus control range. Hereinafter, the term “focus control range” simply refers to this object-side focus control range. A depth of field 2803 indicated by diagonal lines on the rearranged image is a range from the subject distance d3 (d3 ′) to the subject distance d4 (d4 ′). In the focus information display image of FIG. The depth of field at the focus position d0 that is in focus, and the focus information display image of (b) shows the depth of field at the focus position d0 ′ that is in focus on the building.

  By displaying the focus information display image as described above, the user can intuitively grasp both the subject in the scene and the information regarding the focus state. 14A and 14B also show a schematic diagram of the imaging device and a field of view range (view angle) 2804 that can be captured by the imaging device for the sake of convenience of explanation, but are generated in this step. It is not always necessary to include such information in the focus information display image.

  Returning to the flowchart of FIG.

  In step 2308, the display image generation unit 2209 outputs the focus information display image data generated in step 2307 to the display unit 1506.

  In step 2309, the CPU 1501 determines whether or not a user input related to the focus position at the time of the refocus processing has been newly made. In this case, user input is performed via the touch screen, the operation unit 1505 of the imaging apparatus, or the like. For example, the subject to be newly focused can be directly designated on the rearranged image, the subject distance to the new focus position can be directly designated, or the mark representing the focus position 2801 on the slide bar can be designated. Can be considered. If a new focus position has been input, the process returns to step 2307, and a series of processing from step 2307 to step 2309 is repeated. For example, it is assumed that a user input with “building” as a new focus position is made in a state in which an in-focus information display image (FIG. 14A) with “person” as the focus position is displayed. In this case, the focus information display image shown in FIG. 14B is newly displayed. As described above, in FIG. 14B, the depth of field 2803 also changes in accordance with the newly set focus position (subject distance d0 '). That is, when a new focus position at the time of refocus processing is input, there is no change in the focus control range 2802, and the focus position 2801 and the depth of field 2803 at the focus position mainly change.

  It is also conceivable that the newly input focus position is outside the focus control range. In this case, the user may be prompted to input a focus position within the focus control range by notifying a warning or the like. Or you may make it restrict | limit the range which can be input by a user so that the focus position out of a focus control range may not be received.

  On the other hand, if there is no input of a new focus position, this process is terminated.

  In the above-described example, the focus information display image based on the rearranged image overlooking the scene from directly above has been described. However, the focus information display image is not limited to the above-described example, and various modes are conceivable. (A)-(c) of FIG. 15 has shown an example of the variation of a focusing information display image. FIG. 15A is an in-focus information display image based on a rearranged image obtained by looking down the scene from the horizontal direction. FIG. 6B is a focus information display image based on a rearranged image obtained by looking down the scene from an obliquely upward direction. (C) is an in-focus information display image based on a rearranged image viewed from diagonally lateral directions. In this way, the rearranged image that is the base of the focus information display image may be any image as long as the subjects are sequentially arranged in the depth direction based on the distance information.

  In step 2308, the focus information display image is displayed. At this time, the single-viewpoint image acquired by the imaging unit 1500 and / or the composite image synthesized by the display image generation unit 2209 are displayed together. May be. (A)-(c) of FIG. 16 has shown an example in the case of displaying a single viewpoint image and / or a synthesized image together on a focusing information display image.

  FIG. 16A shows a focus information display image (see FIG. 15A described above) based on a rearranged image obtained by looking down at the scene from the horizontal direction, and a single-viewpoint image and two types of synthesized images. A case of displaying an image is shown. The composite image 1 in the figure is in a state in which the focus position is set with “person” as the subject of interest (the “building” and “mountain” subjects other than the subject of interest are out of range of the depth of field) ). The composite image 2 in the figure is in a state in which the focus position is set with “building” as the subject of interest (the subjects other than the subject of interest “person” and “mountain” are out of range of the depth of field) ). When displaying the composite image after such refocusing together, the user can more intuitively confirm the identified target subject by using the composite image in which only the target subject is in focus. . 16A is based on a rearranged image obtained by looking down on the scene from the horizontal direction, it goes without saying that the rearranged image serving as the base may be of any type. In the example of FIG. 16A, two types of composite images are displayed. However, only one type of composite image may be displayed or three or more types may be displayed.

  In FIG. 16B, in addition to the focus information display image (see FIGS. 14A and 14B described above) based on the rearranged image obtained by overlooking the scene from directly above, “person” is added. A case is shown in which a composite image is displayed when a focus position is set as a subject of interest. By displaying in this way, it is possible to efficiently display the rearranged image, information on the focused state, and the composite image within a limited display screen. Note that the in-focus information display image in FIG. 16B omits lateral information with respect to the rearranged image obtained by looking down at the scene shown in FIG. It is based only on what has been made. FIG. 16C shows a case in which a single-viewpoint image is displayed in addition to a focus information display image based on a rearranged image obtained by looking down on the scene from directly above. In this way, by displaying the single viewpoint image together, it is possible to intuitively grasp the in-focus state while confirming the image at the time of imaging. Note that the focus information display image in FIG. 16C is a rearranged image obtained by looking down on the scene shown in FIG. 12D from directly above, and the rearranged image surrounding the subject area with a rectangle. Thus, the information in the horizontal direction is omitted and only the information in the depth direction is used as a base.

  As described above, a single-viewpoint image or a composite image may be displayed together with the focus information display image.

  In the focus information display images shown in FIGS. 14 to 16, information on the focus state, the focus control range, and the depth of field can be simultaneously known as information on the focus state. However, it is not necessary to display all these pieces of information at the same time, and they may be displayed separately. For example, only the focus position and the depth of field may be displayed first, and information on the focus control range may be displayed according to a user instruction.

  As described above, according to the present embodiment, at the time of imaging or editing of the captured image, based on the rearranged image in which subjects are arranged in the depth direction, the focus position, focus control range, Information about the in-focus state such as the depth of field is displayed. Thus, the user can intuitively grasp the in-focus state of the subject at the time of imaging and / or editing of the captured image.

[Example 2]
In the first embodiment, the description has been made on the assumption of an imaging apparatus having an imaging unit having the configuration shown in FIG. Next, a mode based on an imaging apparatus having an imaging unit configured as shown in FIGS. 3 and 4 will be described as a second embodiment. In the following, the description will focus on the points peculiar to the present embodiment.

  In the configuration shown in FIGS. 3 and 4, parallax image data in which a plurality of small images with different imaging viewpoints and imaging ranges are arranged is acquired by the imaging element 1604. In the configuration of FIG. 3, the lens array 1603 is disposed between the imaging optical system and the image side conjugate surface (left side of the image side conjugate surface). On the other hand, in the configuration of FIG. 4, the lens array 1603 is arranged outside (on the right side of the image side conjugate surface) instead of between the imaging optical system and the image side conjugate surface. Therefore, in the configuration of FIG. 4, the lens array 1603 sees the image formed by the imaging optical system 1602 as a real object and re-images it on the image sensor 1604. However, in both cases, the image formed by the imaging optical system 1602 is viewed by the lens array 1603 as an object, and the image is formed on the image sensor 1604. Therefore, both are essentially the same. In the following, the configuration of FIG. 3 will be described as an example, but the contents thereof are similarly established with respect to the configuration of FIG.

  First, an image composition (refocus) process in this embodiment will be described.

  Qualitatively, as in the first embodiment, the divided pupil images of the imaging optical system 1602 may be superimposed with a shift amount corresponding to the subject distance to be focused.

  FIG. 17 is an enlarged view focusing on the lens array 1603 and the image sensor 1604 in the configuration of FIG. In this embodiment, the lens array 1603 is configured by microlenses whose object-side surface is flat and whose image-side surface is convex. Of course, the shape of the lens array 1603 is not limited to this. In FIG. 17, the alternate long and short dash line represents the angle of view of each microlens. By projecting and synthesizing the pixel value obtained by the image sensor 1604 onto the virtual imaging plane via a microlens corresponding to the pixel, a synthesized image focused on the virtual imaging plane can be generated. it can. Here, the virtual imaging plane refers to a plane conjugate with an object side plane to be focused by image synthesis (a plane conjugate via the imaging optical system 1602). For example, in FIG. 3, in order to generate an image focused on the object plane 1601, a virtual imaging plane may be set on the image side conjugate plane 1701. In FIG. 17, the pixels projected at the time of generating the composite image are represented by broken lines by shifting the angle of view of each microlens for easy understanding. The composite image is generated by overlapping pixels similar to the method described above (the method of projecting and synthesizing the pixel value obtained by the image sensor 1604 onto the virtual imaging plane via the microlens corresponding to the pixel). If so, a method may be used in which the respective pixels are translated and combined. At this time, when the regions of the lens array 1603 through which the light beams incident on the pixels have passed are the same, the parallel movement amounts of these pixels are the same. That is, the operation of the pixel at the time of generating the composite image in FIGS. 3 and 4 is determined according to the region of the lens array 1603 through which the light beam incident on the pixel has passed.

  Next, the focus control range in the present embodiment will be described.

The focus control range in the present embodiment is also described by Expression (4) as in the first embodiment. That is, a range conjugated with the refocus range α ₊ s _{2 to} α _- s ₂ on the image side expressed by the above-described formula (4) (a range conjugated with the imaging optical system 1602) is the object side. This is the focus control range that is the refocus range. FIG. 18 is a diagram corresponding to FIG. 7 according to the first embodiment. Δy in FIG. 18 represents the sampling pitch of the two-dimensional intensity distribution of light, and Δy = Δσ ₁ / σ ₂ (σ ₁ : the distance between the image side conjugate plane 1701 and the object side main plane of the lens array 1603, σ _2. : Distance between the image side main plane of the lens array 1603 and the image sensor 1604). This is because the image formed by the imaging optical system 1602 is reduced and formed on the image sensor 1604 at σ ₂ / σ ₁ times when the lens array 1603 sees it as a virtual object. Also in this embodiment, from Δ << P (P: exit pupil distance of the imaging optical system 1602), the equation (4) can be approximated to the equation (5).

  The processing flow in the image processing unit 1512 according to the present embodiment is the same as the flowchart of FIG. 9 according to the first embodiment. However, the focus control range is derived as follows from the difference in the configuration of the imaging unit 1500. (Step 2306).

As is clear from FIG. 18, since NF = σ ₁ / Δ _LA geometrically and Δy = Δσ ₁ / σ ₂ , the following equation (12) is established.

From this expression (12) and the like, the conditional expression (13) to be satisfied by the image-side focus control range d _refocus in this embodiment is obtained.

As in the first embodiment, by setting the range of ± 10.0 in the conditional expression (13) to ± 6.0 or ± 3.0, a sharper composite image can be obtained.

A specific example is shown below.
The number of effective pixels of the image sensor 1604 R _total : 150.0 × 10 ⁶ (pix)
・ Σ ₁ : 0.3712 (mm)
・ Σ ₂ : 0.0740 (mm)
-Pixel pitch Δ of the image sensor 1604: 0.0024 (mm)
The pitch of the lens array 1603 is Δ _LA : 0.0256 (mm)
Focal length f _W at the wide angle end of the imaging optical system 1602: 72.2 (mm)
Focal length f _T at the telephoto end of the imaging optical system 1602: 194.0 (mm)
F value (from wide angle end to telephoto end): 2.9
・ One-dimensional pupil division number N: 5
_-Resolution per single viewpoint image R _mono : 6.0 × 10 ⁶ pix
The range of conditional expression (13), the resolution R _{comb of} the composite image corresponding to each range of conditional expression (13), and the d _refocus corresponding to each resolution of the composite image are as shown in the following table (2), for example. .

As in the first embodiment, the resolution R _{comb of the} composite image is selected from, for example, the above three types by user input via the operation unit 1505.

In the above example, for example, in order to generate a composite image of 10.0 × 10 ⁶ pix, since the resolution per _mono- viewpoint image is R _mono : 6.0 × 10 ⁶ pix, pixel-shifted super-resolution It can be seen that higher resolution is required.

[Example 3]
Next, an embodiment based on an imaging apparatus having an imaging unit having the configuration (camera array) shown in FIG. In the following, the description will focus on the points peculiar to the present embodiment.

  FIG. 19 is a diagram of an imaging unit 1500 having the configuration of the camera array of FIG. 5 according to the present embodiment as viewed from the front (object side), and imaging optical systems 1602a to 1602g and imaging elements in seven individual cameras. The arrangement of 1604a-1604g is shown. In the case of the imaging unit 1500, the camera array has sixfold symmetry with the optical axis of the imaging optical system 1602b as the rotation axis. However, the configuration of the camera array is not limited to this, and the number and arrangement of the imaging optical systems are arbitrary. In addition, each of the imaging elements 1604a to 1604g is arranged corresponding to each of the imaging optical systems 1602a to 1602g. One may be sufficient.

  In the imaging unit 1500 having the configuration shown in FIG. 19, the light beams refracted by the imaging optical systems 1602a to 1602g are received by the imaging elements 1604a to 1604g corresponding thereto. The plurality of images acquired by the image sensors 1604a to 1604g are parallax images obtained by observing the subject space from different viewpoints. By performing image synthesis using these plural parallax images, a light field in the subject space can be obtained.

  FIG. 20 is a view (cross-sectional view) of the imaging optical system 1602a and the image sensor 1604a as viewed from the side. The same applies to the other imaging optical systems 1602b to 1602g and the image sensors 1604b to 1604g, but the configuration of each imaging optical system may be different. The imaging optical system 1602a is a single focus lens, and performs refocusing by changing the interval between the imaging optical system 1602a and the image sensor 1604a.

Similarly to the first embodiment, the image composition processing in the present embodiment may be performed by superimposing the images of the respective viewpoints with a deviation amount corresponding to the subject distance to be focused. Further, the focus control range is as described in the above-described equation (4). FIG. 21 is a diagram corresponding to FIG. 7 of the first embodiment. In the present embodiment, Δy = Δ and Δu = P _mono / F _mono . Here, F _mono represents an F value in each of the imaging optical systems 1602a to 1602g, and P _mono represents an exit pupil distance corresponding to each imaging optical system. In the present embodiment, since Δ << P _mono , the equation (4) can be approximated to the following equation (14).

  The processing flow in the image processing unit 1512 according to the present embodiment is the same as the flowchart of FIG. 9 according to the first embodiment, but the focus control range is derived as follows from the difference in the configuration of the imaging unit 1500. (Step 2302 and Step 2306).

  In the case of the present embodiment, each of the imaging optical systems 1602a to 1602g constituting the imaging unit 1500 has an aperture stop with a variable aperture value. Therefore, in step 2302, the aperture value of each imaging optical system at the time of imaging is acquired as an optical parameter.

In step 2306, based on the same concept as in the first embodiment, the F value of each of the imaging optical systems 1602a to 1602g obtained in step 2302 at the time of imaging is substituted for F _mono to derive the focus control range. . Here, when the resolution of an image formed by the imaging optical system having an F value of F _mono is R _mono , conditional expression (15) that d _refocus should satisfy is obtained.

Similar to the first embodiment, by setting the range of ± 10.0 in the conditional expression (15) to ± 6.0 or ± 3.0, a sharper composite image can be obtained.

A specific example is shown below.
The number of effective pixels R _mono of each of the image sensors 1604a to 1604g: 19.3 × 10 ⁶ (pix)
Pixel pitch Δ: 0.0012 (mm)
-Focal length f of each imaging optical system 1602a to 1602g: 50.0 (mm)
F value (F _mono ): 1.8
The range of conditional expression (15), the resolution R _{comb of} the composite image corresponding to each range of conditional expression (15), and the d _refocus corresponding to each resolution of the composite image are as shown in the following table (3), for example. .

As in the first embodiment, the resolution R _{comb of the} composite image is selected from, for example, the above three types by user input via the operation unit 1505.

When the F value at the time of imaging is a different F value, d _refocus is determined so as to satisfy the conditional expression (15).

[Example 4]
The image composition processing described in the first to third embodiments has been realized using parallax images obtained by imaging from a plurality of different viewpoints acquired by an imaging apparatus having the imaging unit illustrated in FIGS. . The image composition process is not limited to the image composition process using the parallax image. For single-viewpoint imaging data, the focus position and depth of field may be adjusted later by image processing.

  For example, as a method for changing the focus position by image processing, a method is known in which image data is filtered according to the relative distance from the focus position. In this case, first, image data having a deep depth of field and scene distance information are acquired. The distance information is obtained by providing a distance sensor separately from the imaging device, or obtained by analyzing imaging data of the imaging device. Next, the original image data is not used for the subject to be focused (or subject distance), and the original image data is used, and the relative distance from the subject is determined for the area other than the subject to be focused. Find from information. Then, for an area where the relative distance from the subject to be focused is close, a filter process for reducing the blur amount is performed (for example, the filter size is set to be relatively small). On the other hand, for an area where the relative distance from the subject to be focused is long, filter processing that increases the amount of blur is performed (for example, the filter size is set relatively large). The focus position may be changed by such a method.

  As described above, in the method of filtering the imaging data according to the relative distance from the focus position with respect to the single viewpoint image, the focus control range has the depth of field when the single viewpoint image is obtained. This is the focus control range.

  The degree of depth of field can also be adjusted by image processing using a method similar to the above. In other words, in the filtering process for the imaging data, by setting a larger filter size, the amount of blur increases, and image data with a narrow depth of field can be generated. Conversely, in order to generate image data with a deep depth of field, the filter size may be reduced.

[Example 5]
In the first to fourth embodiments, an image synthesis process is performed on a multi-viewpoint image or a single-viewpoint image to generate an image in which the focus position or the like is changed. However, an image whose focus position has been changed can be obtained without performing image synthesis processing.

  For example, there is known a method of acquiring a plurality of imaging data by bracket imaging that acquires a plurality of images while changing imaging parameters little by little. Specifically, imaging data obtained by changing the focus adjustment ring and the aperture adjustment ring provided in the imaging apparatus in stages is acquired. It is assumed that the imaging data by bracket imaging holds the parameters at the time of imaging together with the captured image. That is, in the first to fourth embodiments, a composite image in which the focus position or the like is changed is generated from one imaging data including images of a plurality of viewpoints, but in this embodiment, a plurality of images obtained by bracket imaging are used. A desired captured image is selected from the captured image data and displayed.

  In bracket imaging (focus bracket imaging) in which the focus position is changed in stages, the focus position is changed in stages while using an operation unit that performs focus adjustment provided in the imaging apparatus, and imaging is performed a plurality of times. In FIG. 12A, there are three types of subjects (persons, buildings, and mountains) at different distances from the imaging device, where “person” is the closest position from the imaging device and “building” is medium. Position, “mountain” is located at the most distant position. In the present embodiment, the focus position is continuously changed so that a subject from a nearby subject to a distant subject is included, and imaging data at each focus position is acquired. As a result, it is possible to obtain captured images in which the subjects “person”, “building”, and “mountain” are in focus. Note that in actual focus bracket imaging, imaging is also performed by focusing on a portion where a main subject does not exist (for example, between a person and a building). In the obtained imaging data, information on the focus position at the time of imaging is recorded in association with the captured image as distance information. As in the sixth embodiment, the distance information of the scene may be acquired by providing a distance sensor separately from the imaging device, or may be acquired by analyzing imaging data of the imaging device.

  In this way, image data in which the focus position is changed stepwise is displayed by bracket imaging without performing image synthesis processing. In this case, the focus position farthest from the focus position closest to the imaging device among the focus positions changed during bracket imaging is the focus control range. Further, the depth of field can be handled as substantially the same as the depth of field determined by the aperture adjustment ring of the imaging device at the time of bracket imaging.

  Also, as shown in FIGS. 16A and 16B described above, when an image with the focus position changed is displayed in addition to the focus information display image, the image is supplied from the focus position acquisition unit 2208. A desired captured image is selected and displayed according to the focus position.

  As described above, image data in which the focus position is changed may be obtained using a plurality of pieces of imaging data obtained by bracket imaging.

[Other Embodiments]
The object of the present invention can also be achieved by executing the following processing. That is, a storage medium that records a program code of software that realizes the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU, MPU, etc.) of the system or apparatus is stored in the storage medium. This is the process of reading the code. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code and the storage medium storing the program code constitute the present invention.

Claims (13)

Means for generating a rearranged image in which the area of the subject in the captured image data is rearranged according to the distance from the imaging device;
An image processing apparatus comprising: means for generating an image reflecting information relating to an in-focus state when performing image composition processing using the captured image data including the rearranged image.   The image processing apparatus according to claim 1, wherein the information regarding the in-focus state includes at least one of a focus position, a depth of field, and a focus control range.   The apparatus further comprises distance deriving means for deriving distance information indicating a distance from the imaging device to the subject, and the means for generating the rearranged image is configured such that each subject has a depth based on the distance information derived by the distance deriving means. The image processing apparatus according to claim 1, wherein the image processing apparatuses are rearranged so as to be aligned in a direction. The captured image data is parallax image data obtained by imaging from a plurality of viewpoints, and includes a plurality of single viewpoint images corresponding to the plurality of viewpoints,
The image processing apparatus according to claim 3, wherein the distance deriving unit derives the distance information by performing stereo matching between the plurality of single viewpoint images. The captured image data is parallax image data obtained by imaging from a plurality of viewpoints, and includes a plurality of single viewpoint images corresponding to the plurality of viewpoints,
The means for generating an image reflecting information on the in-focus state generates an image based on one single-viewpoint image of the plurality of single-viewpoint images. The image processing apparatus according to claim 1. The captured image data is parallax image data obtained by imaging from a plurality of viewpoints, and includes a plurality of single viewpoint images corresponding to the plurality of viewpoints,
The means for generating an image reflecting information on the in-focus state generates an image based on a composite image obtained by combining the plurality of single-viewpoint images. The image processing apparatus according to any one of the above. The captured image data is a single viewpoint image,
The image processing apparatus according to claim 1, wherein the image synthesis process is a filtering process for the single viewpoint image.   5. The image processing apparatus according to claim 3, further comprising: a subject area extracting unit that extracts a subject area in the captured image data based on the distance information derived by the distance deriving unit. .   The image processing apparatus according to claim 8, wherein the subject region extraction unit extracts a region along the contour of the subject or a region having an arbitrary shape included in the subject as the subject region. An image pickup apparatus having the image processing apparatus according to any one of claims 1 to 9, wherein the display unit displays an image reflecting information on the in-focus state;
An imaging apparatus comprising: means for adjusting a focus position, a depth of field, and a focus control range in accordance with a user operation based on an image reflecting information on the in-focus state.   The imaging apparatus according to claim 10, wherein the display unit has a live view function, and displays an image reflecting information on the in-focus state in a live view. Generating a rearranged image in which the subject area is rearranged according to the distance from the imaging device from the captured image data;
An image processing method comprising: generating an image reflecting information relating to an in-focus state when performing image composition processing on the rearranged image using the captured image data.   The program for functioning a computer as an image processing apparatus of any one of Claims 1 thru | or 9.