JP2015142366A - Image processing system, control method and program of the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily set the focal distance and the depth of the field of a reconstituted image generated from multi-viewpoint images.SOLUTION: An image processing system detects S602 an area on the image of a matter (target matter) with specific features from an image reconstituted from multi-viewpoint images, and calculates distance distribution S604 and a depth-of-field information S605 of each area. The system receives S607 user's selection of a detected subject, sets S609 a predetermined depth of field from the subject distance of the selected subject, and generates S610 a reconstituted image having the predetermined depth of field set above.

Description

  The present invention relates to an image processing apparatus, a control method thereof, and a program, and more particularly to a technique for generating a reconstructed image focused on a desired subject using a multi-viewpoint image.

  2. Description of the Related Art In recent years, an imaging device (light field camera (LFC)) capable of acquiring light intensity distribution and incident direction information (light space information (also referred to as light field information)) at the time of photographing is known.

  In Non-Patent Document 1, a light beam that has passed through different divided pupil regions of an imaging lens is imaged on each pixel of the imaging element via a microlens array, so that light incident from various directions is separated and recorded. A method is disclosed.

  By applying a method called “Light Field Photography” to the light space information (hereinafter referred to as LF data) acquired in this way, an image focused on an arbitrary focal length (reconstructed image) is captured. It can be generated later.

  When dealing with a conventional digital camera or the like that cannot acquire LF data, if the image after shooting is not focused on the desired subject, it is necessary to take another shot to acquire the image focused on the subject. There is. On the other hand, in the case of shooting using LFC, if a reconstructed image is generated after shooting using the acquired LF data, an image focused on a desired subject can be obtained without being shot again.

  Furthermore, when shooting a scene that includes multiple desired subjects and the distances between the subjects differ greatly in depth direction, if you focus on one of the subjects with a conventional camera, the other subjects will be blurred. It feels so good. On the other hand, in the case of shooting using the LFC, if a reconstructed image focused on each subject is generated and combined after shooting, an image focused on each subject can be obtained.

Ren.Ng, 7 others, “Light Field Photography with a Hand-Held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02

  By using LF data, it becomes possible to change the focus distance (the center of the depth of field) and the depth of field after shooting, which could not be realized in the past. However, a function that allows the user to easily set or change the in-focus distance and the depth of field when generating a reconstructed image using LF data has not been proposed. In particular, when there are multiple subjects, it may be possible to focus on each subject or to change the depth of field depending on the subject. A method of setting is required.

  The present invention has been made in view of the above-described problems of the prior art, and is an image processing apparatus capable of easily setting the in-focus distance and the depth of field of a reconstructed image generated from a multi-viewpoint image. An object of the present invention is to provide a control method and program thereof.

  In order to solve this problem, for example, an image processing apparatus of the present invention has the following configuration. That is, an image processing apparatus capable of generating a reconstructed image focused on an arbitrary subject distance from images of a plurality of viewpoints photographed at the same time, wherein a predetermined subject is obtained from a reconstructed image focused on a predetermined subject distance Detection means for detecting the input, input means for accepting selection by the user of the detected predetermined subject, setting means for setting a predetermined depth of field from the subject distance of the predetermined subject selected by the user, and setting Generating means for generating a reconstructed image having a predetermined depth of field set by the means.

  According to the present invention, it is possible to easily set the in-focus distance and the depth of field of a reconstructed image generated from a multi-viewpoint image.

The block diagram which shows the function structure as an example of LFC which concerns on this embodiment The figure explaining the relationship between the microlens array 105 which concerns on this embodiment, and the imaging part 106. FIG. The figure explaining the relationship between the light beam which passed each area | region of the exit pupil 301 which concerns on this embodiment, and the photoelectric conversion element which photoelectrically converts this light beam. The figure which shows a response | compatibility with each area | region of the exit pupil 301 which concerns on this embodiment, and the photoelectric conversion element matched with each micro lens array. The figure which shows the relationship with the passage position in the imaging surface of the light beam which passes through the specific position of the reconstruction surface which concerns on this embodiment. Flowchart for explaining image reconstruction processing according to the first embodiment FIG. 6 is a diagram for explaining detection of target objects and distance distribution information according to the first embodiment. The figure explaining the depth-of-field information of the target object concerning Embodiment 1 Flowchart for explaining update processing of depth of field information according to the first embodiment The figure explaining the update process of the depth-of-field information according to the distance relationship of the object which concerns on Embodiment 1. FIG. Flowchart for explaining a reconstruction process according to the second embodiment Flowchart for explaining update processing of depth-of-field information of a designated object according to the second embodiment

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following, an example in which the present invention is applied to an LFC capable of recording LF data as an example of an image processing apparatus will be described. However, the present invention can also be applied to an electronic device that does not have a function related to imaging and recording of LF data. These electronic devices include mobile phones, game machines, tablet terminals, personal computers, etc., but these are merely examples.

(Embodiment 1)
<1 LFC configuration>
FIG. 1 is a block diagram illustrating a functional configuration example of an LFC 100 according to the embodiment of the present invention. The control unit 101 is, for example, a CPU, and controls the operation of each block included in the LFC 100 to realize the operation of the LFC 100 including shooting processing or reconstructed image generation processing described later. Specifically, the control unit 101 reads out a program stored in the ROM 102, develops it in the RAM 103, and executes it, thereby controlling the operation of each block.

  The ROM 102 is a rewritable nonvolatile memory, for example, and stores parameters necessary for the operation of each block in addition to the control program of the LFC 100.

  The RAM 103 is a volatile memory. The RAM 103 is used not only as a program development area read from the ROM 102 but also as a storage area for storing intermediate data output in the operation of each block.

  The imaging unit 106 is an imaging element such as a CCD or a CMOS sensor. The imaging unit 106 receives a timing signal output from a timing generator (TG) (not shown) according to an instruction from the control unit 101, and photoelectrically converts an optical image formed on the photoelectric conversion element surface of the imaging element by the imaging optical system 104. Convert and output analog image signal. Note that the imaging optical system 104 includes, for example, an objective lens, a focus lens, a diaphragm, and the like. In addition, the LFC 100 according to the present embodiment includes a microlens array 105 provided between the imaging optical system 104 and the imaging unit 106 in addition to the microlens provided in each photoelectric conversion element of the imaging unit 106.

  Here, the microlens array 105 will be described with reference to FIGS.

  As shown in FIG. 2, the microlens array 105 of the present embodiment includes a plurality of microlenses 201. In FIG. 2, the optical axis of the imaging optical system 104 is the z-axis, the horizontal direction at the lateral position of the LFC 100 is the x-axis, and the vertical direction is the y-axis. In the example of FIG. 2, for the sake of simplicity, the microlens array 105 will be described as being configured by microlenses 201 arranged in 5 rows and 5 columns, but the configuration of the microlens array 105 is not limited thereto. .

  In FIG. 2, the photoelectric conversion element 202 of the image pickup element that forms the image pickup unit 106 is shown by a lattice. A predetermined number of photoelectric conversion elements 202 are associated with each microlens 201, and in the example of FIG. 2, a photoelectric conversion element 202 of 5 × 5 = 25 pixels is associated with one microlens 201. Yes. The light beam incident on one microlens 201 is separated according to the incident direction, and is imaged on the photoelectric conversion element 202 corresponding to the incident direction among the plurality of associated photoelectric conversion elements.

  FIG. 3 illustrates light beams incident on the photoelectric conversion elements 202p1 to p5 associated with one microlens 201. In FIG. 3, the upward direction corresponds to the vertically upward direction. FIG. 3 illustrates an optical path of a light beam incident on each of the photoelectric conversion elements 202p1 to p5 viewed from a horizontal direction orthogonal to the optical axis in a state where the LFC 100 is in the horizontal position. As shown in the figure, the photoelectric conversion elements 202p1 to p5 arranged in the horizontal direction pass through regions a1 to a5 obtained by dividing the exit pupil 301 of the imaging optical system 104 into five in the vertical direction via one microlens 201. Each incident light beam. In addition, the number attached | subjected to each area | region has shown the correspondence with the photoelectric conversion element 202 in which the light beam which passed through enters.

  In the example of FIG. 3, the optical paths of the light beams incident on the photoelectric conversion elements 202 are separated in the vertical direction as viewed from the horizontal direction. However, the separation of the light beams is not limited to the vertical direction, and the same applies in the horizontal direction. To be done. That is, when the exit pupil 301 of the image pickup optical system 104 is divided into regions as shown in FIG. 4A when viewed from the image pickup device side, the light beam that has passed through each region has a photoelectric as shown in FIG. Of the conversion elements 202, the light enters the photoelectric conversion elements with the same identification numbers. Here, it is assumed that the F numbers of the microlenses of the imaging optical system 104 and the microlens array 105 are substantially the same.

  Returning to FIG. 1, an AFE (Analog Front End) 107 and a DFE (Digital Front End) 108 perform a correction process on the image signal generated by the imaging unit 106. Specifically, the AFE 107 performs reference level adjustment (clamp processing) and A / D conversion processing on the analog image signal output from the imaging unit 106, and outputs LF data to the DFE 108. The DFE 108 corrects a slight reference level shift or the like with respect to the input LF data.

  The image processing unit 109 applies various image processing such as color conversion processing to the LF data to which the correction processing by the DFE 108 is applied. In the present embodiment, the image processing unit 109 also performs processing for generating an image (reconstructed image) focused on an arbitrary subject distance from the LF data. For example, the method of “Light Field Photography” as shown in Non-Patent Document 1 described above may be used to generate the reconstructed image.

  The image processing unit 109 performs processing for generating a reconstructed image generated from the LF data and generating a reconstructed image in which the depth of field is changed based on an instruction from the control unit 101. Processing for generating a reconstructed image and processing for changing the depth of field will be described in detail later.

  The display unit 110 is a display device included in the LFC 100, and is a small LCD or the like, for example. The display unit 110 displays the image generated by the image processing unit 109 and focused at an arbitrary distance. As described above, the LF data obtained by A / D converting the analog image signal output from the imaging unit 106 of this embodiment cannot be used for display as it is. Therefore, the display unit 110 displays image data generated by the image processing unit 109 by adding data of a plurality of pixels corresponding to each microlens, for example. The display unit 110 may be an external display device.

  The recording medium 111 is a recording device that is detachably connected to the LFC 100 such as a built-in memory included in the LFC 100, a memory card, an HDD, or the like. The recording medium 111 records LF data and a reconstructed image focused on an arbitrary focal length generated from these LF data.

  The operation input unit 112 is a user interface of the LFC 100 having, for example, a power button, a touch panel, and the like. When operated by the user, the operation input unit 112 outputs a control signal and a command corresponding to the operation to the control unit 101.

  In the present embodiment, the LF data is acquired by the imaging unit 106 having the configuration shown in FIG. 4, but a plurality of cameras with different viewpoints are collected as in, for example, Japanese Patent Application Laid-Open No. 2011-22796. The configuration may be regarded as the imaging unit 106. In addition, unlike the optical system of FIG. 4, a configuration in which a light beam from the imaging optical system is imaged on the microlens array and an imaging device is provided on the imaging surface so that the object plane and the imaging device are in a conjugate relationship. But you can. Furthermore, the image from the imaging optical system is re-imaged on the microlens array (this is called re-imaging because the image once formed is diffused), and the image plane A configuration in which an image pickup element is provided in the apparatus may be used. A configuration in which a mask (gain modulation element) with an appropriate pattern is inserted into the optical path of the photographing optical system can also be used.

<2 Reconstructed image generation processing>
An outline of a process for generating a reconstructed image focused on a specific subject will be described with reference to FIGS. 4 and 5.

  First, the distance (subject distance) of a specific subject included in the shooting range can be obtained by the following method. The image processing unit 109 generates a pair of images (referred to as an A image and a B image) from light beams that have passed through different divided pupil regions from the LF data, and calculates a defocus amount for a specific subject from the phase difference between the images. To detect. Based on the current in-focus distance and the detected defocus amount, the control unit 101 can calculate the distance of a specific subject.

In the example of FIG. 4B, among the pixels associated with each microlens, the pixel values of the first column and the second column from the right end are added to the divided pupil region on the left half of the exit pupil 301. A corresponding A image can be generated. Further, by adding the pixel values of the fourth column and the fifth column, a B image corresponding to the divided pupil region on the right half of the exit pupil 301 can be generated. That is, when this is expressed by an expression


become that way. The pair of images obtained in this way are images having the center of gravity of the corresponding divided pupil region as the optical axis.

  In other words, since the pair of images has an image shift due to the shift of the optical axis, the image shift amount (pupil division phase difference) can be detected for the subject by performing a correlation operation on the two images. . From the image shift amount obtained in this way, the focal length for focusing on each subject included in the shooting range of the LF data can be analyzed. For example, a reconstructed image that focuses on a specific subject is generated. Can do.

  Next, generation of a reconstructed image that focuses on a subject at a specific distance will be described. In the LFC 100 of the present embodiment, as described above, each of the plurality of pixels assigned to one microlens receives the light flux that has passed through the divided pupil regions having different exit pupils of the imaging lens. This is the same for all the microlenses of the microlens array 105. In addition, since light beams that have passed through the imaging lens from different directions are incident on each microlens, all the pixels of the imaging element receive light beams that are incident from different directions.

  For this reason, in the following, the optical path of the light beam incident on each pixel of the LF data obtained by photographing is represented by the coordinates (u, v) of the pupil region that has passed through the exit pupil and the position coordinates (x ′) of the corresponding microlens. , Y ′), each light flux is defined and described. In the generation of the reconstructed image, the pixel (x, y) on the reconstructed surface corresponding to an arbitrary focal distance for generating the reconstructed image is integrated by integrating a light beam having an optical path passing through the point. A value can be obtained.

  In FIG. 5, the optical path of the light beam in the horizontal plane (xz plane) seen from the vertical direction at the lateral position of the LFC 100 is shown. Hereinafter, the optical path of the light beam passing through each pixel on the reconstruction plane in the xz plane will be described, but the same applies to the yz plane.

If the coordinates (u, v) of the pupil region and the pixel coordinates on the reconstruction plane are (x, y), the micro beam on the micro lens array 105 on which the light beam passing through the pupil region and the pixels on the reconstruction plane is incident. The position coordinates (x ′, y ′) of the lens are


It is represented by F is the distance from the imaging lens to the microlens array, and αF is the distance from the exit pupil of the photographing lens to the reconstruction surface (α is the refocus coefficient: a variable coefficient for determining the distance to the reconstruction surface). is there.

If the output of the photoelectric conversion element that receives the luminous flux is L (x ′, y ′, u, v), the pixel output E (x, y) of the coordinates (x, y) of the image formed on the reconstruction surface y) is obtained by integrating L (x ′, y ′, u, v) with respect to the pupil region of the photographing lens,


It is represented by By using (u, v) as the representative coordinates of the pupil region, the equation can be calculated by simple addition. By performing such calculation for each pixel coordinate on the reconstruction surface, a reconstructed image focused on an arbitrary subject distance can be generated.
<3 Depth of field change processing>
The depth-of-field changing process executed in the LFC 100 of this embodiment having such a configuration will be specifically described with reference to the flowchart of FIG. Each step will be described with reference to FIGS. 7 to 10 which are necessary for the description.

  The processing corresponding to the flowchart can be realized by the control unit 101 reading, for example, a program stored in the ROM 102, developing the program in the RAM 103, and executing the program. Note that this processing will be described as being started when, for example, the LFC 100 is set in the browsing mode and is activated and an instruction to generate a reconstructed image is given to the LF data recorded on the recording medium 111. In addition, it is not limited to the viewing mode after shooting, and processing for changing the depth of field for the purpose of displaying a reconstructed image on the display unit 110 at the time of shooting and setting parameters at the time of playback in the acquired LF data. May be performed.

  In step S <b> 601, the image processing unit 109 acquires LF data that can be reconstructed from the recording medium 111 according to an instruction from the control unit 101. Furthermore, the image processing unit 109 generates a reconstructed image corresponding to a predetermined subject distance in order to generate an image used for the subsequent object detection process. In this embodiment, the predetermined subject distance is set to infinity and a reconstructed image is generated. However, a setting value by a user or a setting value determined experimentally is used. A reconstructed image may be generated.

  In step S <b> 602, the image processing unit 109 detects a region on the image of an object (target object) having a specific feature from the reconstructed image generated from the LF data. In this embodiment, the LFC 100 has a function of detecting a region corresponding to the face and torso of a person included in the image. In this step, the LFC 100 corresponds to the face and torso of the person using the reconstructed image generated in S601. It is assumed that the area is detected. The subject to be detected is not limited to a human face area, and may be a specific subject that the LFC 100 has a detection function. FIG. 7A shows an example in which an object A 701, an object B 702, and an object C 703 are detected.

  In step S603, the image processing unit 109 calculates a distance distribution to the subject for each divided region in the reconstructed image with respect to the acquired LF data. Reference numeral 704 in FIG. 7A indicates an example of a divided region of one distance distribution. The size of the divided area is defined according to the minimum resolution of the reconstructed image that can be generated from the LF data. The number of divisions of the divided area 704 may be determined as appropriate according to limitations on the calculation capability, frame rate, etc. of the LFC 100.

  The image processing unit 109 calculates information about the representative distance to the subject included in the area for each of the set divided regions 704. Specifically, the image processing unit 109 generates two types of reconstructed images (detection images) for detecting a defocus amount corresponding to light beams that have passed through two different divided pupil regions. As described above, the detection image generated in this step may be one that divides the area of the exit pupil 301 into divided pupil areas of the left half and the right half, and corresponds to the light flux that has passed through each divided pupil area. However, the detection image may be an image corresponding to the light beam that has passed through the exit pupil 301 and has passed through two types of divided pupil regions having different optical axes, and the method for selecting the divided pupil region is not limited thereto.

  The image processing unit 109 calculates the defocus amount for the subject in each divided region from the two obtained detection images, and calculates the representative distance according to the defocus amount. The representative distance may be, for example, a distance for a subject located in the center of the divided region, or may be an average value of distances obtained for subjects in the divided region.

  In S604, the image processing unit 109 generates distance distribution information of each target object from the representative distance for each divided region 704 obtained in S603 and the positional relationship on the image of the target object detected in S602 (FIG. 7). (B)). The distance distribution with respect to each target object has a spread according to the shape and size of the object, but in this embodiment, for the sake of simplification, a distance that represents the distance of the target object (representative distance of the target object) is given. And FIG. 7B is a diagram showing representative distances of the target objects A701, B702, and C703, and shows that the distances to the target objects are different.

  The image processing unit 109 acquires the representative distance of each divided region 704 corresponding to the positional relationship of the target object on the image, and calculates the representative distance of the target object. In the present embodiment in which the target object is a person, the representative distance corresponding to the face area of the person is calculated (selected) as the representative distance of the target object. However, the representative distances of all the divided areas including the body area are related to the same person. May be calculated as the representative distance of the target object.

  In step S605, the image processing unit 109 generates depth-of-field information of the target object. FIG. 8 is a diagram schematically showing the subject distance and the depth of field of the target object shown in FIG. 7A, where the horizontal axis is the subject distance. The depth of field information of the object A 701 is the subject distance 401, the closest-side depth of field 402, and the infinite-side depth of field 403. The depth-of-field information of the object B 702 includes a subject distance 404, a closest-side depth of field 405, and an infinite-side depth of field 406. The depth of field information of the object C703 includes the subject distance 407, the closest-side depth of field 408, and the infinite-side depth of field 409.

  First, the image processing unit 109 calculates the subject distance of the target object using the representative distance of the target object generated in S604. In addition, the image processing unit 109 acquires the closest depth of field and the infinite depth of field at the subject distance of each target object. In general, the depth of field can be obtained from the physical value of the imaging apparatus such as the focal length of the lens, the aperture value (F number), and the size of the imaging device, and the subject distance. However, in this embodiment, it is assumed that the near side and infinite side depths of field are calculated in advance for a certain subject distance and F number, and are stored as a depth of field table. In order to obtain depth-of-field information on the subject distance of the target object, the image processing unit 109 uses the values of the near-side depth of field and the infinite-side depth of field for a predetermined subject distance and F-number as the depth of field. Get from the depth table.

  In step S606, the control unit 101 causes the display unit 110 to display the reconstructed image focused on the object at infinity, generated by the image processing unit 109 in step S601, for the target object specifying operation by the user.

  In step S <b> 607, the control unit 101 determines whether an operation for specifying a target object has been performed via the operation input unit 112. When determining that the target object designation operation has been performed through the operation input unit 112, the control unit 101 advances the process to step S608, and when determining that the target object designation operation has not been performed, moves the process to step S612. Wait for input from the operation input unit 112. Hereinafter, the designated target object is referred to as a designated object.

  For example, when the coordinates on the screen are designated through the operation input unit 112 and the designated coordinates are included in any of the target object areas detected by the image processing unit 109 in S602, the control unit 101 designates the designated coordinates. It is determined that the target object in the region including is designated.

  In step S608, the image processing unit 109 specifies the relationship between the subject distance between the designated object and another target object based on the subject distance of each target object generated in step S605. For example, in S607, it is assumed that the object A701 is designated among the target objects in FIG. Then, based on the subject distance of each target object generated in S605, the subject distance relationship that the object A701 exists on the closest side among the object A701, the object B702, and the object C703 is specified.

  As a result of comparing the subject distances between the designated object and all other target objects, the image processing unit 109 determines the subject distance relationship between the designated object and the other target objects when the designated object is closest. As the information to be shown, for example, an identifier indicating that it is the closest side is set. If the designated object is on the most infinite side, an identifier indicating that the designated object is on the infinite side is set, and if it is neither (in the middle), no identifier is set.

  In S609, the image processing unit 109 specifies based on the depth-of-field information of each target object generated in S605 and information indicating the relationship of the subject distance between the specified object and other target objects set in S608. A process for changing the depth of field of the object is performed.

  A process of changing the depth of field of the designated object will be described with reference to the flowchart of FIG. 9 and FIG. In step S501, the image processing unit 109 refers to information indicating the relationship between the subject distance between the designated object and another target object. If an identifier indicating that the object is close is set, the process proceeds to step S502. If an identifier to that effect is set, the process proceeds to S503. If the identifier is not set in the information indicating the subject distance relationship between the designated object and another target object, the image processing unit 109 advances the process to step S504.

  In step S502, the image processing unit 109 performs processing for moving the subject distance in the depth-of-field information of the designated object to the infinite side. On the other hand, in step S503, the image processing unit 109 performs a process of moving the subject distance in the depth-of-field information of the designated object to the closest side. That is, the subject distance of the designated object is moved in a direction in which the difference between the subject distance of the designated object and the subject distance of another target object is reduced. When the processes of S502 and S503 are completed, the image processing unit 109 advances the process to S504.

  Note that the processing in S502 and S503 is not necessarily required if the predetermined depth of field can be expanded without moving the subject distance. In other words, the depth of field occurs with the same width on the near side and the infinity side with the focus distance as the center, but if the specified object exists on the near side or the infinity side, the subject distance in one direction must be changed without changing the subject distance. Expansion of depth of field reaches the limit. For this reason, in the present embodiment, when the designated object exists on the close side and the infinite side, the subject distance is moved by a predetermined amount so as to move away from the close end or the infinite side.

  In step S504, the image processing unit 109 expands the range of the closest depth of field and the infinite depth of field of the specified object. The image processing unit 109 acquires the near side depth of field and the infinite side depth of field from the depth of field information of the designated object. Next, with the newly set subject distance as the center, the subject distance A is set at the position corresponding to the closest depth of field acquired from the subject distance, and the position corresponding to the infinite side depth of field acquired from the subject distance. The subject distance B is set to. If the subject distance of the designated object has changed in S502 or S503, the subject distance after the change is the newly set subject distance. For example, the image processing unit 109 determines the near-side depth of field of the specified object, for example, the near-side depth of field obtained from the subject distance A on the near side and the near-side depth of field acquired from the depth-of-field information. Are combined to expand the near field depth of the specified object. Similarly, for the infinite depth of field, the image processing unit 109 calculates the infinite depth of field obtained from the subject distance B on the infinite side and the infinite depth of field acquired from the depth of field information. Add together.

  In addition, in order to adjust the amount of movement of the subject distance and the amount of change of the depth of field that are updated by one change process of the depth of field information, what is previously incorporated as a fixed value may be further added. Good. These may be values that can be set by the user from the menu screen or the like. Setting the movement amount and the change amount is synonymous with setting the change speed of the depth of field. Further, in the present embodiment, an example has been described in which the close-side depth of field and the infinite depth-of-field are each changed in expanding the depth of field of the designated object. Only the side depth of field may be performed. Further, in the case of a designated object on the infinite side, processing such as changing only the closest depth of field may be performed.

  In step S505, the image processing unit 109 updates the depth of field information. First, the image processing unit 109 updates the subject distance in the depth of field information with the subject distance at the center. In addition, the image processing unit 109 updates the depth-of-field information using the depth of field expanded in S504 with respect to the closest-side depth of field in the depth-of-field information of the designated object.

  FIG. 9 shows how the depth of field is updated (enlarged) in the process of FIG. 9 in each of the case where the designated object exists on the near side, the infinite distance, and the intermediate position. 10 will be further described.

  FIG. 10A schematically shows a depth-of-field changing process performed when the designated object is present on the near side (for example, when the designated object is A701 in FIG. 7). Since the object A 701 exists on the closest side, the subject distance of the depth of field information of the object A 701 is moved to a predetermined amount infinite side as indicated by 401 to 1001a in FIG. In step S504, the closest depth of field of the depth of field information of the object A701 is expanded from 402 to 1002a in FIG. 10A, and the infinite depth of field is expanded from 403 to 1003a in FIG. 10A. Is done. When the subject distance corresponding to the closest depth of field 402 from the subject distance 1001a is set as the subject distance A, the depth of field 1002a is calculated from the closest depth of field 402 calculated at the subject distance A. It is obtained by adding the depth. Similarly, assuming that the subject distance on the infinite side corresponding to the depth of field 402 from the subject distance 1001a is the subject distance B, the infinite side depth of field calculated at the subject distance B is added to the infinite side depth of field 403. Thus, the infinite depth of field 1003a is obtained.

  FIG. 10B shows a change in the depth of field that is performed when the designated object is positioned between the target object on the near side and the target object on the infinite side (for example, when the designated object is the object B702 in FIG. 7). The process is shown schematically. Since the object B702 is located between the closest subject and the infinite subject, the subject distance 404 in the depth-of-field information does not move. In step S504, the closest depth of field of the depth of field information of the object B 702 is changed from 405 to 1002b in FIG. 10B, and the infinite depth of field of the depth of field information is changed to FIG. ) From 406 to 1003b, the range of the depth of field is expanded. Assuming that the subject distance corresponding to the near field depth 405 from the subject distance 404 is the subject distance A, the depth of field 1002b is obtained from the near field depth 405 determined by the subject distance A. It is obtained by adding the depth. Similarly, assuming that the subject distance on the infinite side corresponding to the depth of field 406 from the subject distance 404 is the subject distance B, the infinite side depth of field calculated at the subject distance B is added to the infinite side depth of field 406. Infinite side depth of field 1003b is obtained.

  FIG. 10C schematically illustrates the depth-of-field changing process performed when the designated object is on the infinity side (for example, when the designated object is the object C703 in FIG. 7). Since the subject distance of the object C703 is on the infinite side, the subject distance of the depth-of-field information of the object C703 is moved closer to a predetermined amount as indicated by 407 to 1001c in FIG. 10C in S503. In S504, the near-side depth of field of the depth-of-field information of the object C703 is changed from 408 to 1002c in FIG. 10C, and the infinite-side depth of field of the depth-of-field information is shown in FIG. ) From 409 to 1003c, the range of the depth of field is expanded. Depth of field 1002c is defined as the near field depth obtained at the subject distance A when the subject distance corresponding to the near field depth 408 from the subject distance 1001c is the subject distance A. It is obtained by adding the depth. Similarly, when the subject distance on the infinite side corresponding to the depth of field 409 from the subject distance 1001c is the subject distance B, the infinite side depth of field calculated at the subject distance B is added to the infinite side depth of field 409. Infinite side depth of field 1003c is obtained.

  In this way, the user changes the depth of field by updating the depth of field after moving the subject distance to be focused according to the relationship of the subject distance between the specified object and another target object. The depth of field can be easily changed without intentionally specifying.

  In step S610, the image processing unit 109 generates a reconstructed image corresponding to the depth of field information changed in step S609. For example, the image processing unit 109 generates a reconstructed image with the three subject distances calculated in step S504 of FIG. 9 as focal planes, and performs another reconstruction on the reconstructed image focused on the central subject distance. A composite reconstructed image is generated by combining the in-focus portions of the constituent images. Thereby, a reconstructed image with an expanded depth of field can be generated. Note that the composition process may be, for example, addition of corresponding pixels of the three reconstructed images. At this time, the edge strength for each region may be calculated, and weighted addition corresponding to the edge strength may be performed for each region. The image processing unit 109 outputs the combined reconstructed image.

  More specifically, for the composite reconstructed image, first, the image processing unit 109 reconstructs the subject distance (focusing distance) that is the center calculated in S504, the subject distance A and the subject distance B on the closest side, respectively. Each reconstructed image is generated by setting it as the focal plane of the image. Thereby, a reconstructed image at the subject distance of the designated object and a reconstructed image at the subject distances A and B can be obtained. Then, by generating a new reconstructed image obtained by combining these reconstructed images, a reconstructed image with an expanded depth of field can be obtained. Note that the method for expanding the depth of field is not limited to the above, and the subject distance A and the subject distance B may be set at positions displaced from the central subject distance by a predetermined step respectively on the close side and the infinity side. Good. Further, the subject distance is not limited to three, and more subject distances may be set.

  In step S <b> 611, the display unit 110 displays the reconstructed image output from the image processing unit 109 according to an instruction from the control unit 101.

  In step S <b> 612, the control unit 101 determines the presence / absence of further processing based on the input from the operation input unit 112. For example, when the selected state is selected in a certain image region by an operation (touch operation) such as a touch on the touch panel, the control unit 101 advances the process to S607 in order to determine the designation of the target object. In addition, when the selection state continues in a certain image area, such as a long press operation on the touch panel, for example, the control unit 101 returns the process to S609 in order to continue changing the depth of field information. Regarding the continuation of the change of the depth of field information, when the continuation of the selected state is stopped, it is determined as a continuation release instruction, the movement to S609 is stopped, and the change of the depth of field is stopped. Further, when there is no input from the user or the like, the control unit 101 enters an input waiting state and returns the process to S612. Furthermore, when the user performs an end operation on the display of the reconstructed image, the control unit 101 ends the series of processes.

  According to the present embodiment, by controlling the depth-of-field changing method based on the relationship between the subject distance of the subject specified by the user and the subject distance of other subjects other than the subject, the user can The focusing distance and the depth of field can be easily set.

  In this embodiment, the example in which the depth of field is continuously changed when the subject is continuously selected has been described. However, when the subject is selected, the depth of field is automatically changed at a predetermined time period. You may make it do. In this case, for example, when the user performs a touch operation on a predetermined area of the touch panel, the change of the depth of field is stopped. Further, the time period may be set by the user from a menu screen or the like. In the present embodiment, the operation of the touch panel has been described as an operation by contact, but an input to a display device such as a touch panel may not necessarily be in contact depending on the detection method. Although not shown, the present invention includes a method for reducing the depth of field in addition to increasing the depth of field of the designated object. For example, the user may select whether to enlarge or reduce in advance before specifying a subject, or when a user continuously selects a subject and changes the depth of field, The processing may be reduced by pressing the button. When the user selects to reduce the depth of field, the image processing unit 109 may determine the subject distance so that the depth of field is reduced in S609.

(Embodiment 2)
With reference to the flowchart of FIG. 11, the reconstructed image generation processing in Embodiment 2 of the present invention will be described. In the present embodiment, in addition to the user specifying the subject to be the designated object, the desired depth of field is reflected by instructing the update direction of the depth of field or the update direction and the update width. A reconstructed image is generated. In addition, a different part from the flowchart in Embodiment 1 of this invention shown in FIG. 6 is demonstrated in detail, and description is abbreviate | omitted about the overlapping part.

If the control unit 101 determines in step S <b> 607 that the target object designation operation has been performed in the same manner as the process described with reference to FIG. 6, the process proceeds to step S <b> 1101.
In step S <b> 1101, the control unit 101 acquires the update direction and the update width of the depth of field for the specified object based on the information input through the operation input unit 112. The control unit 101 advances the process to step S1102 when an operation for instructing the update direction and the update width of the depth of field with respect to the designated object is obtained as an input. If the instruction operation for the specified object cannot be obtained, the process returns to S1101 to wait for the instruction operation from the user. If the instruction operation cannot be obtained within the predetermined time, the control unit 101 times out the process, moves the process to S612, and waits for an input from a new user.

  For example, the control unit 101 detects an operation involving a movement of the position of a touch operation such as a flick operation or a swipe operation on the touch panel through the operation input unit 112. When it is determined that the detected operation is a swipe operation, the control unit 101 further confirms the movement locus and acquires the size of the locus. For example, when the swipe operation is an operation in the vertical direction, each is assigned to the closest or infinite direction of the direction of the depth of field, and the size of the movement locus is assigned to the width of change of the depth of field.

  In step S1102, the image processing unit 109 changes the depth of field of the designated object based on the update direction and the updated width of the depth of field for the designated object acquired in step S1101. The process for updating the depth of field of the designated object in S1102 will be described with reference to FIG. Only parts different from the flowchart of FIG. 9 in the first embodiment of the present invention will be described.

  In step S1201, the image processing unit 109 determines the direction in which the depth of field is updated based on the operation instruction for the specified object acquired in step S1101. When the update direction of the depth of field in the operation instruction indicates the infinite side, the image processing unit 109 proceeds to S502 described in the first embodiment and changes the subject distance to the infinite side. Further, when the update direction of the depth of field in the operation instruction is the close side, the image processing unit 109 proceeds to S503 and changes the subject distance to the close side. When completing the processes of S502 and S503, the image processing unit 109 advances the process to S1202.

  In S1202, the image processing unit 109 changes the depth of field based on the subject distance of the designated object and the width for updating the depth of field acquired in S1101. The image processing unit 109 acquires the near side depth of field and the infinite side depth of field from the depth of field information of the designated object. Next, with the newly set subject distance as the center, the subject distance A is obtained at the position obtained by adding the update width obtained from the closest depth of field obtained from the subject distance, and the infinite side subject obtained from the subject distance. The subject distance B is set at a position where the update width acquired in the depth of field is added. If the subject distance of the designated object has changed in S502 or S503, the subject distance after the change is the newly set subject distance. The image processing unit 109 updates the depth-of-field information using the set subject distance in the same manner as in S505. In this way, by obtaining three subject distances based on the updating direction and the updating width of the depth of field obtained from the operation instruction, three reconstructed images at the respective subject distances are obtained in S612. A combined reconstructed image is obtained by combining.

  According to the present embodiment, the reconstructed image according to the user's intention is obtained by acquiring the renewal direction of the depth of field and the renewal width by using the touch panel or the like and generating the reconstructed image with the depth of field changed The depth of field can be easily set.

  As described above, according to the present invention, an image processing apparatus capable of easily setting the in-focus distance and the depth of field of a reconstructed image generated from a multi-viewpoint image, and a control method and program thereof are provided. It becomes possible to provide.

[Other Embodiments]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (15)

An image processing apparatus capable of generating a reconstructed image focused on an arbitrary subject distance from images of a plurality of viewpoints photographed at the same time,
Detecting means for detecting a predetermined subject from a reconstructed image focused on a predetermined subject distance;
Input means for accepting selection by the user of the detected predetermined subject;
Setting means for setting a predetermined depth of field from the subject distance of the predetermined subject selected by the user;
An image processing apparatus comprising: generating means for generating a reconstructed image having the predetermined depth of field set by the setting means. The setting unit changes the predetermined depth of field when an instruction to select the same predetermined subject is continuously received from the input unit,
The image processing apparatus according to claim 1, wherein the generation unit generates a reconstructed image having the changed predetermined depth of field.   The image processing apparatus according to claim 2, wherein the instruction to select is performed by a touch operation on an area where the predetermined subject is displayed. The setting means changes the predetermined depth of field until an instruction to cancel the selection is received by the input means,
4. The image processing apparatus according to claim 1, wherein the generation unit generates a reconstructed image having the changed predetermined depth of field. 5.   The image processing apparatus according to claim 4, wherein the release instruction is performed by a touch operation on an area where the reconstructed image is displayed.   The said setting means is comprised so that the said predetermined depth of field may be changed for every predetermined width, The magnitude | size of the said predetermined width is determined according to a user's instruction | indication. The image processing apparatus according to any one of 2 to 5.   The said setting means is comprised so that the said predetermined depth of field may be changed by a predetermined time period, The said time period is determined according to a user's instruction | indication. The image processing apparatus according to any one of the above.   The setting means is configured such that, among the predetermined subjects detected from the reconstructed image focused on the predetermined subject distance, the selected predetermined subject is closer to the infinite side or the infinite side than the other predetermined subjects. When changing the subject distance of the selected predetermined subject, a predetermined amount is changed in a direction in which the difference from the subject distance of the other predetermined subject is reduced, and the subject distance after the change is set as a center. The image processing apparatus according to claim 1, wherein the predetermined depth of field is set.   The said setting means sets the said depth of field after changing the object distance of the said predetermined | prescribed subject to the near side or the infinite side according to a user's instruction | indication. An image processing apparatus according to 1.   The image processing apparatus according to claim 9, wherein the user instruction is performed by an operation accompanied by a movement of a position of a touch operation on an area where the reconstructed image is displayed.   The generating means combines the reconstructed image at the subject distance of the predetermined subject selected by the user with the reconstructed image at the subject distance set to the near side and the infinite side from the subject distance, respectively. The image processing apparatus according to claim 1, wherein a reconstructed image having a depth of field set by the setting unit is generated. Imaging means for outputting images of the plurality of viewpoints;
An image processing apparatus comprising: the image processing apparatus according to claim 1.   The imaging apparatus according to claim 12, wherein the imaging unit outputs the plurality of images by photoelectrically converting a light beam that has passed through a pupil-divided imaging optical system. A control method of an image processing apparatus capable of generating a reconstructed image focused on an arbitrary subject distance from images of a plurality of viewpoints photographed at the same time,
A detecting step for detecting a predetermined subject from the reconstructed image focused on a predetermined subject distance;
An input step in which an input means receives a selection by the user of the detected predetermined subject;
A setting step in which the setting means sets a predetermined depth of field from the subject distance of the predetermined subject selected by the user;
And a generation step of generating a reconstructed image having the predetermined depth of field set in the setting step.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 11.