JP2015084518A - Image processing apparatus, image processing method, program and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To set a suitable reconfiguration image generation condition on an image signal capable of generating a reconfiguration image focused on a plurality of object distances.SOLUTION: An image processing apparatus acquires light field data and selects an object. The image processing apparatus then outputs restriction information, that is, information to disable focusing on the selected object, and information related to a reconfiguration image generated by reconfiguring the light field data.

Description

  The present invention relates to an image processing apparatus, an image processing method, a program, and a recording medium, and more particularly to a technique for generating a reconstructed image from output data after shooting.

  In recent years, in an imaging device such as a digital camera, reconstruction is performed by recording light intensity distribution and light incident direction information as output data at the time of shooting, for example, focusing on an arbitrary subject distance from the output data after recording. Techniques for generating images have been proposed.

  In Non-Patent Document 1, a microlens array is arranged between a photographic lens and an image sensor, and the light flux that has passed through different divided pupil regions of the image sensor to each pixel (photoelectric conversion element) of the image sensor via the microlens array. A method is disclosed in which light incident from various directions is separated and recorded by forming an image. The output data (Light Field Data, hereinafter referred to as LF data) obtained in this way records the light flux that is incident on the adjacent pixels from different directions.

  From the LF data, by extracting the light flux in the same direction from the pixels associated with each microlens, it is possible to generate an image taken from that direction. In addition, by applying a technique called “Light Field Photography”, an arbitrary subject distance is set, and the output of the pixel that records the light flux that has passed through each point on the focal plane where the light flux from the subject distance converges Is added, it is possible to artificially generate (reconstruct) a pixel of an image focused on a specific subject distance after shooting.

Ren.Ng, 7 others, “Light Field Photography with a Hand-Held Plenoptic Camera”, Stanford University Computer Science Tech Report CTSR 2005-02

  However, since the reconstructed image focused on various subject distances and the reconstructed images of various viewpoints with deep depth can be generated from the LF data as described above, the privacy information included in the shooting range is appropriate. May not be protected.

  The present invention has been made in view of the above-described problems, and is an image processing apparatus that sets a suitable reconstructed image generation condition for an image signal that can generate a reconstructed image focused on a plurality of subject distances. An object is to provide an image processing method, a program, and a recording medium.

  In order to achieve the above-described object, the image processing apparatus of the present invention includes a data acquisition unit that acquires light field data, a selection unit that selects a subject, and information that does not focus on the subject selected by the selection unit. And output means for outputting restriction information that is information relating to a reconstructed image generated by reconstructing light field data.

  With such a configuration, according to the present invention, it is possible to set a suitable reconstructed image generation condition for an image signal that can generate a reconstructed image focused on a plurality of subject distances.

1 is a block diagram showing a functional configuration of a digital camera 100 according to an embodiment of the present invention. The figure for demonstrating the relationship between the microlens array 105 which concerns on embodiment of this invention, and the imaging part 106. FIG. The figure for demonstrating the relationship between the light beam which passed each area | region of the exit pupil 301 which concerns on embodiment of this invention, and the photoelectric conversion element which photoelectrically converts this light beam. The figure which showed the response | compatibility with each area | region of the exit pupil 301 which concerns on embodiment of this invention, and the photoelectric conversion element matched with each micro lens The figure for demonstrating the relationship with the passage position in the imaging surface of the light beam which passes the specific position in the reconstruction surface which concerns on embodiment of this invention The flowchart which illustrated the restriction information addition processing performed with digital camera 100 concerning the embodiment of the present invention. The figure for demonstrating the distance information map which concerns on embodiment of this invention. The figure which showed the face area detected in the restriction | limiting information addition process which concerns on embodiment of this invention The figure which illustrated the relative distance table which concerns on embodiment of this invention The figure which illustrated restriction information concerning an embodiment of the present invention The flowchart which illustrated the reproduction | regeneration processing performed with the digital camera 100 which concerns on embodiment of this invention. The figure which showed an example of the reconstruction image produced | generated in the digital camera 100 which concerns on embodiment of this invention. The figure for demonstrating the relationship between the light beam which passed through each area | region of the exit pupil 301 which concerns on the modification of this invention, and the photoelectric conversion element which photoelectrically converts this light beam. The flowchart which showed the block diagram which showed the function structure of the digital camera 100 concerning the modification of this invention

[Embodiment]
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following embodiment, an example in which the present invention is applied to a digital camera that can generate an image focused on an arbitrary subject distance from LF data after shooting as an example of an image processing apparatus will be described. . However, the present invention can be applied to any device capable of generating an image focused on an arbitrary subject distance from LF data.

  In this specification, the following terms are defined and described.

・ "Light Field (LF) data"
This indicates light space information describing a three-dimensional subject space like an image signal output from the imaging unit 106 included in the digital camera 100 of the present embodiment. In the present embodiment, each pixel of the image signal (pixel signal) indicates a signal intensity corresponding to a light beam having a different combination of the pupil region and the incident direction of the imaging optical system 104 that has passed.

・ "Reconstructed image"
This is an image that is newly generated by, for example, synthesizing the pixels constituting the LF data in a desired combination. In this embodiment, an image that is generated from the LF data and focused on an arbitrary subject distance is included. Specifically, the LF data pixels are rearranged according to the pixel arrangement on the focal j plane (reconstruction plane, refocus plane) corresponding to the object distance to be generated, and LF corresponding to one pixel (unit pixel) of the reconstructed image. The pixel value of the pixel is obtained by adding the pixel values of a plurality of pixels (sub-pixels) of the data. The pixel arrangement on the reconstruction plane is determined based on the incident direction (incident angle) of the light beam incident when the image sensor is present on the reconstruction plane. One pixel of a reconstructed image can be generated by adding pixel values of a plurality of pixels corresponding to one microlens in the pixel arrangement. In addition, the reconstructed image includes an image of each viewpoint generated from subpixels (subpixels of the same viewpoint) having the same incident direction from the microlens, which exist for each microlens. When generating this viewpoint image, the images from other viewpoints (incident direction from the microlens) may be aligned and added based on the amount of movement of the parallax.

<< Configuration of Digital Camera 100 >>
FIG. 1 is a block diagram showing a functional configuration of a digital camera 100 according to an embodiment of the present invention.

  The control unit 101 is a CPU, for example, and controls the operation of each block included in the digital camera 100. Specifically, the control unit 101 controls an operation of each block by reading an operation program for a shooting process or a refocus moving image generation process, which will be described later, stored in the ROM 102, and developing and executing the program on the RAM 103.

  The ROM 102 is, for example, a rewritable nonvolatile memory, and stores parameters necessary for the operation of each block in addition to the operation program for each block of the digital camera 100.

  The RAM 103 is a volatile memory. The RAM 103 is used not only as a development area for the operation program of each block included in the digital camera 100 but also as a storage area for storing intermediate data output in the operation of each block.

  The imaging unit 106 is an imaging element such as a CCD or a CMOS sensor. The imaging unit 106 receives a timing signal output from a timing generator (TG) (not shown) according to an instruction from the control unit 101, and photoelectrically converts an optical image formed on the photoelectric conversion element surface of the imaging element by the imaging optical system 104. Convert and output analog image signal. Note that the imaging optical system 104 includes, for example, an objective lens, a focus lens, a diaphragm, and the like. In addition, the digital camera 100 according to the present embodiment includes a microlens array 105 between the imaging optical system 104 on the optical axis and the imaging element separately from the microlens provided in each photoelectric conversion element of the imaging element. .

<Relationship between microlens and photoelectric conversion element>
Here, the microlens array 105 provided between the imaging optical system 104 on the optical axis and the imaging element in the digital camera 100 of the present embodiment will be described with reference to the drawings.

  As shown in FIG. 2, the microlens array 105 of the present embodiment includes a plurality of microlenses 201. In FIG. 2, the optical axis of the imaging optical system 104 is the z axis, the horizontal direction of the digital camera 100 is the x axis, and the vertical direction is the y axis. In the example of FIG. 2, for the sake of simplicity, the microlens array 105 will be described as being configured by microlenses 201 arranged in 5 rows and 5 columns, but the configuration of the microlens array 105 is not limited thereto. .

  In FIG. 2, the photoelectric conversion element 202 of the image pickup element that forms the image pickup unit 106 is shown by a lattice. A predetermined number of photoelectric conversion elements 202 are associated with each microlens 201, and in the example of FIG. 2, a photoelectric conversion element 202 of 5 × 5 = 25 pixels is associated with one microlens 201. Yes. The light beam that has passed through one microlens 201 is separated according to the incident direction, and is imaged on the corresponding photoelectric conversion element 202.

FIG. 3 illustrates light beams incident on the photoelectric conversion elements 202 p _{1 to} p ₅ corresponding to one microlens 201. In FIG. 3, the upward direction corresponds to the vertically upward direction. In the figure, an optical path of a light beam incident on each photoelectric conversion element 202 viewed from the horizontal direction in a state where the digital camera 100 is in the horizontal position is illustrated. As shown in the drawing, the photoelectric conversion elements 202p _{1 to} p ₅ arranged in the horizontal direction have regions a ₁ to a which are obtained by dividing the exit pupil 301 of the imaging optical system 104 into five in the vertical direction via one micro lens 201. light beam passing through the a ₅ enters respectively. In addition, the number attached | subjected to each area | region has shown the correspondence with the photoelectric conversion element 202 in which the light beam which passed through enters.

  In the example of FIG. 3, the optical path of the light beam incident on each photoelectric conversion element 202 viewed from the horizontal direction is shown, but the light beam separation is not limited to the vertical direction, and is similarly performed in the horizontal direction. That is, when the exit pupil 301 of the image pickup optical system 104 is classified into the regions as shown in FIG. 4A when viewed from the image pickup device side, the light beam that has passed through each region has a photoelectric as shown in FIG. Of the conversion elements 202, the light enters the photoelectric conversion elements with the same identification numbers. Here, it is assumed that the F numbers (F values) of the microlenses of the imaging optical system 104 and the microlens array 105 are substantially the same.

  An AFE (analog front end) 107 and a DFE (digital front end) 108 perform correction processing and the like on the image signal generated by the imaging unit 106. Specifically, the AFE 107 performs reference level adjustment (clamp processing) and A / D conversion processing on the analog image signal output from the imaging unit 106, and outputs LF data to the DFE 108. The DFE 108 corrects a slight reference level shift or the like with respect to the input LF data.

  The image processing unit 109 performs various types of image processing such as white balance and color conversion processing on the LF data to which correction processing by the DFE 108 is applied, compression processing for compressing a formed image, and synthesis processing for combining a plurality of images. Etc. In the present embodiment, the image processing unit 109 also performs processing for generating an image (reconstructed image) that focuses on an arbitrary subject distance from the LF data. For example, the method of “Light Field Photography” as shown in Non-Patent Document 1 described above may be used to generate the reconstructed image.

<Reconstructed image generation method>
Here, an outline of a method for generating a reconstructed image focused on a specific subject distance will be described with reference to the drawings.

  First, the subject distance focused on a specific subject included in the shooting range can be obtained by the following method. First, the image processing unit 109 generates an image corresponding to two light beams that have passed through different divided pupil regions from the LF data, and detects a difference (defocus amount) of a specific subject image between the images. Based on the defocus amount detected in this way, the control unit 101 can calculate the subject distance to a specific subject.

In the example of FIG. 4B, for each microlens, the divided pupil region in the left half of the exit pupil 301 is generated by adding the pixel values of the pixels in the first column and the second column among the corresponding pixels. A corresponding A image can be generated. Further, a B image corresponding to the divided pupil region on the right half of the exit pupil 301, which is generated by adding the pixel values of the pixels in the fourth column and the fifth column among the corresponding pixels, can be generated. That is, when this is expressed by an expression
become that way. The two types of reconstructed images obtained in this way are images having the centroid position of each corresponding divided pupil region as a pseudo optical axis.

  In other words, since the two types of reconstructed images have image shifts due to optical axis shifts, the amount of image shift (pupil division phase difference) is detected for each subject by performing a correlation operation on the two images. can do. By multiplying the image shift amount obtained in this way by the focal position of the imaging optical system 104 and a conversion coefficient determined by the optical system, the subject distance for each subject included in the photographing range of the LF data is analyzed. Based on the obtained subject distance, for example, a reconstructed image that focuses on a specific subject can be generated as an additional image.

  Next, generation of a reconstructed image that focuses on a specific subject distance will be described. In the digital camera 100 of the present embodiment, as described above, each of the plurality of pixels assigned to one microlens receives the light flux that has passed through the divided pupil regions having different exit pupils of the imaging lens. This is the same for all the microlenses of the microlens array 105. In addition, since light beams that have passed through the imaging lens from different directions are incident on each microlens, all the pixels of the imaging element receive light beams that are incident from different directions.

  For this reason, in the following, the optical path of the light beam incident on each pixel of the LF data obtained by photographing is represented by the coordinates (u, v) of the pupil region that has passed through the exit pupil and the corresponding microlens on the microlens array. Each light flux is defined and explained as position coordinates (x ′, y ′). In the generation of a reconstructed image, by integrating a light beam having an optical path passing through the point for a pixel (x, y) on a reconstruction surface where a light beam incident from a subject distance for generating the reconstructed image converges. Pixel values can be obtained.

  In FIG. 5, the optical path of the light beam on the horizontal plane (xz plane) viewed from the vertical direction at the lateral position of the digital camera 100 is shown. Hereinafter, the optical path of the light beam passing through each pixel on the reconstruction plane in the xz plane will be described, but the same applies to the yz plane.

If the coordinates (u, v) of the pupil area and the pixel coordinates on the reconstruction plane are (x, y), the light beam passing through the pupil division area and the pixels on the reconstruction plane is incident on the microlens array 105. The position coordinates (x ′, y ′) of the microlens are
It is represented by Note that F is a distance from the imaging lens to the microlens array, and αF is a distance from the photographing lens to the reconstruction surface (α is a refocus coefficient: a variable coefficient for determining the distance to the reconstruction surface).

If the output of the photoelectric conversion element that receives the luminous flux is L (x ′, y ′, u, v), the pixel output E (x) of the coordinates (x, y) of the image formed on the reconstruction plane , Y) is obtained by integrating L (x ′, y ′, u, v) with respect to the pupil region of the photographing lens,
It is represented by By using (u, v) as the representative coordinates of the pupil region, the equation can be calculated by simple addition.

  The display unit 110 is a display device included in the digital camera 100 such as a small LCD. The display unit 110 is used to display a user interface screen of the digital camera 100, use as an electronic viewfinder, or display a photographed image. The display unit 110 displays a reconstructed image focused on an arbitrary subject distance that is generated and output by the image processing unit 109. As described above, the LF data obtained by A / D converting the analog image signal output from the imaging unit 106 according to the present embodiment does not connect the images in adjacent pixels. Therefore, not the LF data but the image data generated by the image processing unit 109 is displayed on the display unit 110.

  The recording medium 111 is a recording device that is detachably connected to the digital camera 100 such as a built-in memory of the digital camera 100, a memory card, or an HDD. The recording medium 111 records LF data and a reconstructed image focused on an arbitrary subject distance generated from these LF data. Alternatively, the generated image or the like is transmitted (output) to an external device (not shown) such as a personal computer via the communication unit 116.

  The operation input unit 112 is a user interface that the digital camera 100 has, such as a power button and a shutter button. When the operation input unit 112 detects that the user interface has been operated by the user, the operation input unit 112 outputs a control signal corresponding to the operation to the control unit 101. For example, the operation input unit 112 outputs various information related to shooting, such as setting of the shooting mode, to the control unit 101. A signal output from the release switch is used as an AE or AF operation start trigger or a shooting start trigger. Upon receiving these start triggers, the control unit 101 controls each unit of the imaging apparatus including the imaging unit 106 and the display unit 110.

  In the present embodiment, it is assumed that LF data is acquired by photographing processing and processing for generating a reconstructed image is performed. However, the LF data used for generation is from the recording medium 111 or via the communication unit 116. It may be acquired.

《Restriction information addition processing》
Here, processing relating to refocus restriction, which is characteristic processing of the present embodiment, will be described. Since reconstructed images focused on various subject distances can be generated from the LF data as described above, there is a possibility that privacy information included in the shooting range is not properly protected. That is, for example, when a document that normally does not permit photographing or copying is included in the photographing range, a reconstructed image that can read the document may be generated. Further, in recent years, images obtained by shooting are often released to online image sharing services and the like, and it is conceivable that LF data is released in a state that can be arbitrarily reconstructed. For example, a person who has unintentionally reflected in the shooting range may infringe portrait rights against the intention of the person depending on the conditions for generating a reconstructed image from LF data. Therefore, in this embodiment, it is determined that there is a subject to be protected, a refocus range where the subject is focused, and information for limiting refocus to the refocus range is provided. It is added to LF data. Moreover, not only the reconstructed image for refocusing but also restriction information for making it difficult to focus on the object to be protected or making it difficult to view when a reconstructed image is generated for some purpose such as a viewpoint change or a perspective change. Is added to the LF data.

  A specific process of the restriction information adding process executed by the digital camera 100 of the present embodiment having such a configuration will be described with reference to the flowchart of FIG. The processing corresponding to the flowchart can be realized by the control unit 101 reading out a corresponding processing program stored in, for example, the ROM 102, developing it in the RAM 103, and executing it. Note that the restriction information adding process will be described as being started when shooting is performed using the digital camera 100, for example.

  In step S <b> 601, the control unit 101 sets a distance information acquisition area (hereinafter referred to as “D area”) for acquiring information on the distance to the subject for the LF data generated by imaging. The D area indicates a two-dimensional area in the pixel array constituting the target LF data, which is defined in units of pixel groups (photoelectric conversion elements 202) assigned to the microlens 201. That is, the size of the D area is defined according to the minimum resolution of the reconstructed image that can be generated from the LF data. For example, in the example of FIG. 2, since 5 × 5 pixels assigned to one microlens 201 correspond to the minimum resolution of the reconstructed image, the D area is a multiple of 5 in the horizontal direction and a multiple of 5 in the vertical direction. Is defined as a region having a number of pixels.

  The D area is appropriate for the accuracy required for the distance to the subject, the calculation capability of the device, the amount of calculation, the required frame rate, etc., with the pixel group corresponding to the minimum resolution that can be generated from the target LF data as a unit. May be set to any size.

  In step S602, the control unit 101 calculates, for each of the D areas set in step S601, information on the representative distance to the subject included in the area. Specifically, for each D area of the target LF data, the control unit 101 generates two types of reconstructed images (detection images) for defocus amount detection corresponding to the light beams that have passed through two different divided pupil regions. The image processing unit 109 generates the image. As described above, the detection image generated in this step may be one that divides the area of the exit pupil 301 into divided pupil areas of the left half and the right half, and corresponds to the light flux that has passed through each divided pupil area. However, the embodiment of the present invention is not limited to this, and the detection image may be an image corresponding to the light flux that has passed through the exit pupil 301 and has passed through two types of divided pupil regions having different optical axes. However, the selection method is not limited to this. Here, when one D area is configured including pixel groups corresponding to a plurality of microlenses, the result of only the pixel group corresponding to the central microlens may be substituted, or each microlens is supported. You may use the average value of the distance obtained about the pixel group to perform.

  Then, the control unit 101 calculates the representative distance of each D area according to the obtained defocus amount analysis result. The representative distance may be, for example, a distance for a subject located at the center of each D area, or may be an average value of distances obtained for subjects in the area.

  In the present embodiment, the control unit 101 generates a distance information map corresponding to the shooting range of the LF data using the representative distance information for each D area obtained in this way. For example, a reconstructed image generated by using a pixel corresponding to the central divided pupil region among the pixels associated with each microlens 201 of the LF data so that the depth of field becomes deep is shown in FIG. If so, the distance information map is as shown in FIG. The example of FIG. 7A shows an example in which subjects 701, 702, and 703 exist within the shooting range. In the distance information map of FIG. 7B, the closer the subject is to the digital camera 100 at the time of shooting, the contrast is higher. Shown high (high hatching density).

  In step S <b> 603, the control unit 101 causes the image processing unit 109 to generate a reconstructed image having a deep depth of field as illustrated in FIG. 7A, and detects a human face area in the reconstructed image. Hereinafter, in the present embodiment, a description will be given on the assumption that a human face is detected as a specific subject among photographed subjects, and the restriction on the generation condition of a reconstructed image from LF data is determined based on the face region. However, the implementation of the present invention is not limited to this. For example, the subject to be detected is a region having a specific feature in the image, such as a building, an animal, or a character, and the restriction on the generation condition is determined according to the detection result. It may be done.

  When face areas A, B, and C are respectively detected in the detection image of FIG. 7A as shown in FIG. 8, the control unit 101 assigns a face ID for identification to each. Further, the control unit 101 identifies the subject distance with reference to the distance information map for each face area, and manages the detected face area using a table (relative distance table) as shown in FIG.

  In step S604, the control unit 101 determines whether a face area has been detected in step S603. In the restriction information addition processing according to the present embodiment, it is assumed that LF data is recorded with restriction information that is a restriction on a reconstructed image generation condition only when a human face area is detected. That is, the determination in this step is a determination condition as to whether or not the restriction information is associated. However, as described above, since the present invention is not limited to the detection of the face area, the determination condition is detected. It may be changed as appropriate according to the type of privacy information. If it is determined that the face area has been detected, the control unit 101 moves the process to S605. If it is determined that the face area has not been detected, the control unit 101 moves the process to S610.

  In step S <b> 605, the control unit 101 selects, as a determination target (target face region), one of the detected face regions that has not been determined whether it is a subject subject to privacy protection. That is, in this step, the control unit 101 selects, as the target face area, one face area in which it is not determined whether or not the subject is not allowed to be focused in the reconstructed image.

  In step S606, the control unit 101 determines whether the target face area is a privacy protection target. The determination as to whether or not it is a privacy protection target may be made based on the following criteria, for example.

<Privacy protection criteria>
(1) Size standard of face area For example, the face area having the largest size among the detected face areas may be detected, and some or all of the other face areas may be determined as privacy protection targets. . In other words, it is highly likely that the face area that has been photographed to have the largest face area is assumed to be the main subject by the photographer. May be set as a privacy protection target as a face area of a person having a low relationship.

(2) Position reference of face area For example, among the detected face areas, the face area whose arrangement position in the detection image is closest to the center is detected, and part or all of the other face areas are subject to privacy protection. May be determined. In other words, since it is highly likely that the face area that has been shot so that the face area is captured in the center is assumed to be the main subject by the photographer, May be set as a privacy protection target as a face area of a person having a low relationship.

(3) Face area standard for recognition target For example, even if a face area of a person registered in advance as a recognition target is detected among the detected face areas, other face areas are determined as privacy protection targets. Good. That is, when a person's face is set as a recognition target in advance, it is considered that the relationship between the photographer and the person is guaranteed, so that a face area that could not be recognized is set as a privacy protection target. Good.

(4) Main Subject Category Criteria For example, if the target selected as the main subject by setting the shooting mode or the like is a landscape or the like, the face area of the person may all be determined as a privacy protection target. That is, not only a person's face but also a target category (person, landscape, building, etc.) to be selected as a main subject is predetermined, images belonging to other categories are set as privacy protection targets. Also good.

(5) User Selection As described above, the control unit 101 not only determines a privacy protection target with reference to a predetermined standard, but also is selected by an operation button (not shown) provided in the digital camera 100, for example. The subject may be determined as a privacy protection target. Alternatively, a subject other than the selected subject may be determined as a privacy protection target.

(6) Character string reference When a reconstructed image is generated, such as a car license plate or an address, if the character string that can be viewed is included in the image, all such character string areas are protected for privacy. It may be determined as a target. For the determination of the character string, a known OCR (optical character recognition) technique or the like can be used.

  Based on such a determination criterion, the control unit 101 determines the target face area in this step. If the control unit 101 determines that the target face area is a privacy protection target, it moves the process to S607, and if it determines that the target face area is not a protection target, it moves the process to S608.

  In step S <b> 607, the control unit 101 associates information indicating that the target face area is a privacy protection target, for example, in the relative distance table, and moves the process to step S <b> 608.

  In step S608, the control unit 101 determines whether there is a face area that has not yet been selected as the target face area. If the control unit 101 determines that there is a face area that has not yet been selected as the target face area, the control unit 101 returns the process to step S605. If the control unit 101 determines that there is no face area yet, the process proceeds to step S609.

  In step S <b> 609, the control unit 101 refers to the subject distance of the face area (protected face area) associated with the information indicating that it is a privacy protection target, and sets the generation condition of the reconstructed image focused on the protected face area. It associates with LF data as restriction information. For example, when the face areas 802 and 803 are determined as the protected face areas in the example of FIG. 8, the control unit 101 sets the conditions for generating a reconstructed image including 10 m that is the subject distance of the face area within the depth of field. Include in the restriction information as a generation condition to suppress the setting. The depth of field varies depending on the pseudo aperture value in the reconstructed image, that is, the degree of limitation of the divided pupil region corresponding to the pixel used to generate the reconstructed image. For this reason, the range of generation conditions in which the protected face area can be included within the depth of field is generated as a table as shown in FIG. 10 in association with a pseudo aperture value that can be set in the generation of the reconstructed image. The table is associated with LF data as restriction information.

  As shown in FIG. 10, when a pixel corresponding to a light beam having a large aperture (small aperture value), that is, passing through a wide range of pupil regions, is used to generate a reconstructed image, the depth of field is Since it is shallow, the range of the subject distance that limits the setting is narrowed. Conversely, when using a pixel corresponding to a light beam that has passed through a narrower pupil area with a small aperture opening (a large aperture value) to generate a reconstructed image, the depth of field is deep. The range of the subject distance that restricts is widened.

  In this embodiment, it is assumed that the depth of field is grasped in advance for each of the generation conditions of the reconstructed image that can be generated from the LF data, and the restriction information is generated according to the subject distance of the protected face area. The implementation of the present invention is not limited to this. For example, the range of subject distance, which is restriction information, is not based on whether or not it is included in the focusable range in the depth of field, but whether the protected face area is detected as a face area within the depth of field It may be set based on whether or not. In other words, even when the reconstructed image is out of focus and is in a so-called “blurred” state, the setting of the generation condition may be limited so that the protected face area can be recognized.

  In step S610, the control unit 101 records the LF data on the recording medium 111 and completes the restriction information adding process.

  By doing in this way, in the digital camera 100 of the present embodiment, a reconstructed image generation condition that can identify a subject that is a privacy protection target can be associated with LF data as restriction information.

《Reproduction processing》
Next, a detailed description will be given, with reference to the flowchart of FIG. 11, of a reproduction process for reproducing the LF data with the restriction information added and recorded while generating a reconstructed image. The processing corresponding to the flowchart can be realized by the control unit 101 reading out a corresponding processing program stored in, for example, the ROM 102, developing it in the RAM 103, and executing it. In addition, this reproduction | regeneration process demonstrates as what is started when the browsing instruction | indication of LF data (target LF data) is made by the user, for example.

  In step S <b> 1101, the control unit 101 reads out information on initial values (a focus position or a subject distance to be focused and a pseudo aperture value) that are set in advance for generating target LF data. Set the generation conditions. The initial value of the generation condition may be, for example, a fixed value that is determined regardless of the LF data stored in the ROM 102, or is determined by a shooting condition that is recorded in association with each LF data. It may be a value unique to the LF data. In this embodiment, a reconstructed image refocused to a specific subject distance (reconstruction plane) is illustrated as a reconstructed image. However, the present invention is not limited to this, and an image from a specific viewpoint as described above is generated. There can be a plurality of reconstruction methods such as a reconstructed image.

  In step S1102, the control unit 101 determines whether the generation condition set in step S1101 is a generation condition restricted for the target LF data. Specifically, the control unit 101 determines whether or not the restriction information is associated with the target LF data, and if the restriction information is associated, the set generation condition is included in the generation condition restricted by the restriction information. To determine whether or not If the control unit 101 determines that the set generation condition is a generation condition that is restricted with respect to the target LF data, the control unit 101 moves the process to S1103.

  In step S1103, the control unit 101 sets a generation condition that is not restricted by the restriction information as a new generation condition of the reconstructed image. Specifically, the control unit 101 refers to information on the range of the subject distance that limits the reconstruction associated with the aperture value of the currently set generation condition. Then, the subject distance that is the boundary value of the range and is closest to the subject distance of the currently set generation condition is set as the subject distance of the new generation condition. For example, when the restriction information is as shown in FIG. 10 and the currently set generation condition is a pseudo aperture value of 5.6 and the subject distance is 8 m, 3.5 m is set as the subject distance of the nearest boundary value. Is done.

  If the subject distance range that limits reconstruction is the entire subject distance range that can be set as a generation condition, the next smallest aperture value is selected for the pseudo aperture value. It's okay.

  In step S1104, the image processing unit 109 generates a reconstructed image from the target LF data according to the set generation condition under the control of the control unit 101.

  In step S <b> 1105, the control unit 101 causes the display unit 110 to display the reconstructed image generated by the image processing unit 109.

  In step S1106, the control unit 101 determines whether or not the reproduction state of the target LF data, that is, the state in which the reconstructed image is generated and reproduced from the target LF data is continued. That is, the control unit 101 determines whether or not the user has given an instruction to leave from the state related to viewing the target LF data. If it is determined that the reproduction state of the target LF data is continued, the control unit 101 moves the process to S1107, and if it is determined that the reproduction is not continued, this reproduction process is completed.

  In step S <b> 1107, when the reconstructed image generation condition is changed, the control unit 101 changes the generation condition and returns the process to step S <b> 1102.

  By doing so, it is possible to avoid that privacy information is generated as a reconstructed image in an identifiable manner by referring to the restriction information in the reproduction of LF data.

  In the present embodiment, the generation condition is changed to the boundary value of the limited subject distance when the generation condition of the reconstructed image matches the generation condition that is limited. However, the embodiment of the present invention is not limited to this. It is not limited to. For example, the generation condition after the change is not a boundary value, but may be a value that is outside the range and is as close as possible to the set value before the change. If the subject subject to privacy protection is within a predetermined subject distance range from the subject desired to be set as the main subject in shooting, the subject subject to privacy protection is excluded as shown in FIG. The region thus obtained may be generated as a reconstructed image. In other words, when the focus state of the privacy information cannot be made different from that of the subject that is preferably focused on the reconstructed image, the shooting range (image area) in which reconfiguration is performed so that the privacy information is not included in the reconstructed image. ) May be limited, and such shooting range information may also be established as the limit information. In addition, if the generation condition of the reconstructed image matches a generation condition that is restricted, the fact that the reconstructed image is prohibited may be notified and the reconstruction process may not be performed.

  Further, in the present embodiment, the restriction information addition process and the reproduction process are exemplified, and the addition of the restriction information to the LF data and the reproduction of the LF data to which the restriction information is added have been described. In addition, this can be realized even when the restriction information is not added to the LF data at the time of shooting. That is, for example, even if the restriction information is not added to the LF data, the generation condition may be restricted by determining the presence or absence of the privacy information when the LF data is reproduced. The determination of the generation conditions to be limited is not limited to recording LF data generated by shooting, but when registering LF data shot by the digital camera 100 in an information processing apparatus such as a PC, hardware or application Or the like. The determination may be performed by an upload application or an arbitrary configuration on the server when uploading to a photo sharing server such as SNS.

  As described above, the image processing apparatus according to the present embodiment can set a suitably reconfigurable range for an image signal that can generate a reconstructed image focused on a specific subject distance.

[Modification]
In the above-described embodiment, the LF data acquired by the configuration of the imaging optical system 104, the microlens array 105, and the imaging unit 106 illustrated in FIG. 3 is targeted, but the present invention has the configuration illustrated in FIG. The obtained LF data may be used. The configuration of FIG. 13 corresponds to a plenoptic camera described in the literature “Todor Georgiev and Andrew Lumsdaine,“ Superresolution with Plenoptic 2.0 Camera ”, Signal Recovery and Synthesis, Optical Society of America, 2009”. The configuration of FIG. 13 is different in that the focal point of the imaging optical system 104 is provided not at the surface of the microlens array 105 but at a position closer to the exit pupil of the imaging optical system 104.

  In the above-described embodiment, the light beam that has passed through one imaging optical system 104 is divided by each microlens 201 to obtain pixel outputs corresponding to each divided pupil region. However, the embodiment of the present invention may be realized by handling a plurality of imaging data obtained by a so-called multi-lens camera having a plurality of optical systems as shown in FIG. 14 in the same manner as the LF data in this embodiment. Good. In other words, the output of the plurality of imaging elements that are optically imaged by each of the plurality of optical systems is an image corresponding to the light flux that has passed through different divided pupil regions, that is, an image of the subject taken from different directions. The output of the eye camera is equivalent to the LF data in this embodiment. As a method for generating a reconstructed image from such a multi-lens camera, a method described in JP 2011-022796 A may be used.

[Other Embodiments]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

  100: Digital camera, 101: Control unit, 102: ROM, 103: RAM, 104: Imaging optical system, 105: Micro lens array, 106: Imaging unit, 107: AFE, 108: DFE, 109: Image processing unit, 110 : Display unit, 111: recording medium, 112: operation input unit, 116: communication unit, 118: GPS, 119: electronic compass, 201: microlens, 202: photoelectric conversion element, 301: exit pupil

Claims (21)

Data acquisition means for acquiring light field data;
A selection means for selecting a subject;
An image processing apparatus comprising: output means for outputting restriction information, which is information for not focusing on the subject selected by the selection means, and is information for restricting reconstruction of the light field data. .   The image processing apparatus according to claim 1, further comprising a recording unit that records the restriction information output by the output unit in association with the light field data. The selection means includes detection means for detecting a predetermined subject in a reconstructed image that can be generated from the light field data,
The image processing apparatus according to claim 1, wherein the selection unit selects at least a part of the predetermined subjects detected by the detection unit.   The image processing apparatus according to claim 3, wherein the predetermined subject includes at least one of a human face, a character string, and a building. Distance acquisition means for acquiring distance information corresponding to the subject distance of the subject selected by the selection means;
The said output means produces | generates the said restriction | limiting information based on the distance information obtained by the said distance acquisition means about the to-be-photographed object selected by the said selection means, The any one of Claim 1 thru | or 4 characterized by the above-mentioned. The image processing apparatus described. The restriction information includes a generation condition for generating a reconstructed image from the light field data,
The output unit includes, in the restriction information, a generation condition in which the subject distance obtained for the subject selected by the selection unit is included in the depth of field when the reconstructed image is generated. The image processing apparatus according to claim 5.   The image processing apparatus according to claim 6, wherein the generation condition included in the restriction information is a condition that is excluded when a reconstructed image is generated from the light field data.   The generation condition for generating the reconstructed image includes information on at least one of a pseudo aperture state in the reconstructed image and a subject distance to be focused. Image processing device. Signal acquisition means for acquiring light field data;
Information acquisition means for acquiring restriction information for refocusing the light field data so as not to focus on a specific subject;
Setting means for setting conditions for generating a reconstructed image generated from the light field data;
An image processing apparatus comprising: a determination unit that determines a generation condition of a reconstructed image to be finally generated based on the generation condition set by the setting unit and the restriction information.   The image processing apparatus according to claim 9, further comprising a generation unit that generates a reconstructed image from the light field data based on the generation condition determined by the determination unit. The restriction information includes a generation condition for generating a reconstructed image from the light field data,
The determination means determines a generation condition different from the generation condition set by the setting means when the generation condition set by the setting means is a generation condition included in the restriction information. Item 15. The image processing apparatus according to Item 9 or 10.   12. The image processing apparatus according to claim 9, wherein the generation condition included in the restriction information is a condition excluded when generating a reconstructed image from the light field data. .   13. The determination unit, when a generation condition set by the setting unit is the excluded condition, determines a generation condition not included in the restriction information as a new generation condition. An image processing apparatus according to 1. The excluded condition defines information on a range of subject distance to be focused at least,
The determining means, when the generation condition set by the setting means is the excluded condition, generates a new object distance outside the range of the object distance to be focused determined by the excluded condition. The image processing apparatus according to claim 13, wherein the image processing apparatus is included in a condition. The excluded condition defines information on a subject distance range to be focused on each of a plurality of pseudo aperture states,
The determination unit changes at least one of the aperture state of the generation condition set by the setting unit and the subject distance to be focused when the generation condition set by the setting unit is the excluded condition. The image processing apparatus according to claim 13, wherein the new generation condition that does not correspond to the excluded condition is determined.   2. The light field data according to claim 1, wherein each of the pixel signals has a signal intensity corresponding to a light beam having a different combination of a pupil region and an incident direction of an imaging optical system when the light field data is imaged. 16. The image processing device according to any one of items 15 to 15.   The light field data is ray space information describing a three-dimensional space of a subject, and a reconstructed image that focuses on an arbitrary subject distance can be generated by reconstructing the data. Item 17. The image processing apparatus according to any one of Items 1 to 16. A signal acquisition process for acquiring light field data;
A selection process for selecting a subject;
An output step of outputting output restriction information for outputting restriction information, which is information for not focusing on the subject selected in the selection step, and is information for restricting reconstruction of the light field data. A featured image processing method. A signal acquisition step of acquiring light field data capable of generating a reconstructed image focused on a plurality of subject distances;
An information acquisition step for acquiring restriction information for preventing the subject from being focused on the generation of a reconstructed image that can be generated from the light field data;
A setting step for setting a generation condition of a reconstructed image generated from the light field data;
An image processing method comprising: a determination step of determining a generation condition of a reconstructed image to be finally generated based on the generation condition set in the setting step and the restriction information.   A program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 17.   A computer-readable recording medium on which the program according to claim 20 is recorded.