JP2015084517A - Image processing apparatus, image processing method, program and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To set a suitable reconfiguration image generation condition on an image signal capable of generating a reconfiguration image focused on a plurality of object distances.SOLUTION: An image processing apparatus acquires light field data and selects an object. The image processing apparatus then outputs restriction information, that is, information to make the selected object hard to be viewed irrespective of a generation condition at the time of reconfiguration, and information related to a reconfiguration image generated by reconfiguring the light field data.

Description

  The present invention relates to an image processing apparatus, an image processing method, a program, and a recording medium, and more particularly to a technique for generating a reconstructed image from output data after shooting.

  In recent years, in an imaging apparatus such as a digital camera, by recording information on the light intensity distribution and the light incident direction at the time of shooting as output data, reconstruction is performed after the recording, for example, focusing on an arbitrary subject distance Techniques for generating images have been proposed.

  In Non-Patent Document 1, a microlens array is arranged between a photographic lens and an image sensor, and the light flux that has passed through different divided pupil regions of the image sensor to each pixel (photoelectric conversion element) of the image sensor via the microlens array. A method is disclosed in which light incident from various directions is separated and recorded by forming an image. The output data (Light Field Data, hereinafter referred to as LF data) obtained in this way records the light flux that is incident on the adjacent pixels from different directions.

  From the LF data, by extracting the light flux in the same direction from the pixels associated with each microlens, it is possible to generate an image taken from that direction. In addition, by applying a technique called “Light Field Photography”, an arbitrary subject distance is set, and the output of the pixel that records the light flux that has passed through each point on the focal plane where the light flux from the subject distance converges Is added, it is possible to artificially generate (reconstruct) a pixel of an image focused on a specific subject distance after shooting.

Ren.Ng, 7 others, “Light Field Photography with a Hand-Held Plenoptic Camera”, Stanford University Computer Science Tech Report CTSR 2005-02

  However, since the reconstructed image focused on various subject distances and the reconstructed images of various viewpoints with deep depth can be generated from the LF data as described above, the privacy information included in the shooting range is appropriate. May not be protected.

  The present invention has been made in view of the above-described problems, and is an image processing apparatus that sets a suitable reconstructed image generation condition for an image signal that can generate a reconstructed image focused on a plurality of subject distances. An object is to provide an image processing method, a program, and a recording medium.

  In order to achieve the above object, an image processing apparatus according to the present invention includes a data acquisition unit that acquires light field data, a selection unit that selects a subject, and a subject selected by the selection unit as a generation condition at the time of reconstruction. Regardless of this, it is characterized by comprising output means for outputting restriction information, which is information for making it difficult to visually recognize, and is information relating to a reconstructed image generated by reconstructing light field data.

  With such a configuration, according to the present invention, it is possible to set a suitable reconstructed image generation condition for an image signal that can generate a reconstructed image focused on a plurality of subject distances.

1 is a block diagram showing a functional configuration of a digital camera 100 according to an embodiment of the present invention. The figure for demonstrating the relationship between the microlens array 105 which concerns on embodiment of this invention, and the imaging part 106. FIG. The figure for demonstrating the relationship between the light beam which passed each area | region of the exit pupil 301 which concerns on embodiment of this invention, and the photoelectric conversion element which photoelectrically converts this light beam. The figure which showed the response | compatibility with each area | region of the exit pupil 301 which concerns on embodiment of this invention, and the photoelectric conversion element matched with each micro lens The figure for demonstrating the relationship with the passage position in the imaging surface of the light beam which passes the specific position in the reconstruction surface which concerns on embodiment of this invention The flowchart which illustrated the restriction information addition processing performed with digital camera 100 concerning the embodiment of the present invention. The figure for demonstrating the distance information map which concerns on embodiment of this invention. The figure which showed the face area detected in the restriction | limiting information addition process which concerns on embodiment of this invention The figure which illustrated the relative distance table which concerns on embodiment of this invention The figure explaining the privacy information protection area which concerns on embodiment of this invention The figure explaining the restriction | limiting information table which concerns on embodiment of this invention 6 is a flowchart illustrating a playback process executed by the digital camera 100 according to the first embodiment of the invention. The figure for demonstrating the relationship between the light beam which passed through each area | region of the exit pupil 301 which concerns on the modification of this invention, and the photoelectric conversion element which photoelectrically converts this light beam. The flowchart which showed the block diagram which showed the function structure of the digital camera 100 concerning the modification of this invention The figure explaining the refocus initial setting value which concerns on embodiment of this invention The figure explaining the reconstructed image after the image processing which concerns on embodiment of this invention The flowchart which illustrated the reproduction | regeneration processing performed with the digital camera 100 which concerns on Embodiment 2 of this invention.

[Embodiment 1]
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following embodiment, an example in which the present invention is applied to a digital camera that can generate an image focused on an arbitrary subject distance from LF data after shooting as an example of an image processing apparatus will be described. . However, the present invention can be applied to any device capable of generating an image focused on an arbitrary subject distance from LF data.

  In this specification, the following terms are defined and described.

・ "Light Field (LF) data"
This indicates light space information describing a three-dimensional subject space like an image signal output from the imaging unit 106 included in the digital camera 100 of the present embodiment. In the present embodiment, each pixel of the image signal (pixel signal) indicates a signal intensity corresponding to a light beam having a different combination of the pupil region and the incident direction of the imaging optical system 104 that has passed.

・ "Reconstructed image"
This is an image that is newly generated by, for example, synthesizing the pixels constituting the LF data in a desired combination. In this embodiment, an image that is generated from the LF data and focused on an arbitrary subject distance is included. Specifically, the LF data pixels are rearranged according to the pixel arrangement on the focal j plane (reconstruction plane, refocus plane) corresponding to the object distance to be generated, and LF corresponding to one pixel (unit pixel) of the reconstructed image. The pixel value of the pixel is obtained by adding the pixel values of a plurality of pixels (sub-pixels) of the data. The pixel arrangement on the reconstruction plane is determined based on the incident direction (incident angle) of the light beam incident when the image sensor is present on the reconstruction plane. One pixel of a reconstructed image can be generated by adding pixel values of a plurality of pixels corresponding to one microlens in the pixel arrangement. In addition, the reconstructed image includes an image of each viewpoint generated from subpixels (subpixels of the same viewpoint) having the same incident direction from the microlens, which exist for each microlens. When generating this viewpoint image, the images from other viewpoints (incident direction from the microlens) may be aligned and added based on the amount of movement of the parallax.

<< Configuration of Digital Camera 100 >>
FIG. 1 is a block diagram showing a functional configuration of a digital camera 100 according to an embodiment of the present invention.

  The control unit 101 is a CPU, for example, and controls the operation of each block included in the digital camera 100. Specifically, the control unit 101 controls an operation of each block by reading an operation program for a shooting process or a refocus moving image generation process, which will be described later, stored in the ROM 102, and developing and executing the program on the RAM 103.

  The ROM 102 is, for example, a rewritable nonvolatile memory, and stores parameters necessary for the operation of each block in addition to the operation program for each block of the digital camera 100.

  The RAM 103 is a volatile memory. The RAM 103 is used not only as a development area for the operation program of each block included in the digital camera 100 but also as a storage area for storing intermediate data output in the operation of each block.

  The imaging unit 106 is an imaging element such as a CCD or a CMOS sensor. The imaging unit 106 receives a timing signal output from a timing generator (TG) (not shown) according to an instruction from the control unit 101, and photoelectrically converts an optical image formed on the photoelectric conversion element surface of the imaging element by the imaging optical system 104. Convert and output analog image signal. Note that the imaging optical system 104 includes, for example, an objective lens, a focus lens, a diaphragm, and the like. In addition, the digital camera 100 according to the present embodiment includes a microlens array 105 between the imaging optical system 104 on the optical axis and the imaging element separately from the microlens provided in each photoelectric conversion element of the imaging element. .

<Relationship between microlens and photoelectric conversion element>
Here, the microlens array 105 provided between the imaging optical system 104 on the optical axis and the imaging element in the digital camera 100 of the present embodiment will be described with reference to the drawings.

  As shown in FIG. 2, the microlens array 105 of the present embodiment includes a plurality of microlenses 201. In FIG. 2, the optical axis of the imaging optical system 104 is the z axis, the horizontal direction of the digital camera 100 is the x axis, and the vertical direction is the y axis. In the example of FIG. 2, for the sake of simplicity, the microlens array 105 will be described as being configured by microlenses 201 arranged in 5 rows and 5 columns, but the configuration of the microlens array 105 is not limited thereto. .

  In FIG. 2, the photoelectric conversion element 202 of the image pickup element that forms the image pickup unit 106 is shown by a lattice. A predetermined number of photoelectric conversion elements 202 are associated with each microlens 201, and in the example of FIG. 2, a photoelectric conversion element 202 of 5 × 5 = 25 pixels is associated with one microlens 201. Yes. The light beam that has passed through one microlens 201 is separated according to the incident direction, and is imaged on the corresponding photoelectric conversion element 202.

FIG. 3 illustrates light beams incident on the photoelectric conversion elements 202 p _{1 to} p ₅ corresponding to one microlens 201. In FIG. 3, the upward direction corresponds to the vertically upward direction. In the figure, an optical path of a light beam incident on each photoelectric conversion element 202 viewed from the horizontal direction in a state where the digital camera 100 is in the horizontal position is illustrated. As shown in the drawing, the photoelectric conversion elements 202p _{1 to} p ₅ arranged in the horizontal direction have regions a ₁ to a which are obtained by dividing the exit pupil 301 of the imaging optical system 104 into five in the vertical direction via one micro lens 201. light beam passing through the a ₅ enters respectively. In addition, the number attached | subjected to each area | region has shown the correspondence with the photoelectric conversion element 202 in which the light beam which passed through enters.

  In the example of FIG. 3, the optical path of the light beam incident on each photoelectric conversion element 202 viewed from the horizontal direction is shown, but the light beam separation is not limited to the vertical direction, and is similarly performed in the horizontal direction. That is, when the exit pupil 301 of the image pickup optical system 104 is classified into the regions as shown in FIG. 4A when viewed from the image pickup device side, the light beam that has passed through each region has a photoelectric as shown in FIG. Of the conversion elements 202, the light enters the photoelectric conversion elements with the same identification numbers. Here, it is assumed that the F numbers (F values) of the microlenses of the imaging optical system 104 and the microlens array 105 are substantially the same.

  An AFE (analog front end) 107 and a DFE (digital front end) 108 perform correction processing and the like on the image signal generated by the imaging unit 106. Specifically, the AFE 107 performs reference level adjustment (clamp processing) and A / D conversion processing on the analog image signal output from the imaging unit 106, and outputs LF data to the DFE 108. The DFE 108 corrects a slight reference level shift or the like with respect to the input LF data.

  The image processing unit 109 performs various types of image processing such as white balance and color conversion processing on the LF data to which correction processing by the DFE 108 is applied, compression processing for compressing a formed image, and synthesis processing for combining a plurality of images. Etc. In the present embodiment, the image processing unit 109 also performs processing for generating an image (reconstructed image) that focuses on an arbitrary subject distance from the LF data. For example, the method of “Light Field Photography” as shown in Non-Patent Document 1 described above may be used to generate the reconstructed image.

<Reconstructed image generation method>
Here, an outline of a method for generating a reconstructed image focused on a specific subject distance will be described with reference to the drawings.

  First, the subject distance focused on a specific subject included in the shooting range can be obtained by the following method. First, the image processing unit 109 generates an image corresponding to two light beams that have passed through different divided pupil regions from the LF data, and detects a difference (defocus amount) of a specific subject image between the images. Based on the defocus amount detected in this way, the control unit 101 can calculate the subject distance to a specific subject.

In the example of FIG. 4B, for each microlens, the divided pupil region in the left half of the exit pupil 301 is generated by adding the pixel values of the pixels in the first column and the second column among the corresponding pixels. A corresponding A image can be generated. Further, a B image corresponding to the divided pupil region on the right half of the exit pupil 301, which is generated by adding the pixel values of the pixels in the fourth column and the fifth column among the corresponding pixels, can be generated. That is, when this is expressed by an expression
become that way. The two types of reconstructed images obtained in this way are images having the centroid position of each corresponding divided pupil region as a pseudo optical axis.

  In other words, since the two types of reconstructed images have image shifts due to optical axis shifts, the amount of image shift (pupil division phase difference) is detected for each subject by performing a correlation operation on the two images. can do. By multiplying the image shift amount obtained in this way by the focal position of the imaging optical system 104 and a conversion coefficient determined by the optical system, the subject distance for each subject included in the photographing range of the LF data is analyzed. Based on the obtained subject distance, for example, a reconstructed image that focuses on a specific subject can be generated as an additional image.

  Next, generation of a reconstructed image that focuses on a specific subject distance will be described. In the digital camera 100 of the present embodiment, as described above, each of the plurality of pixels assigned to one microlens receives the light flux that has passed through the divided pupil regions having different exit pupils of the imaging lens. This is the same for all the microlenses of the microlens array 105. In addition, since light beams that have passed through the imaging lens from different directions are incident on each microlens, all the pixels of the imaging element receive light beams that are incident from different directions.

  For this reason, in the following, the optical path of the light beam incident on each pixel of the LF data obtained by photographing is represented by the coordinates (u, v) of the pupil region that has passed through the exit pupil and the corresponding microlens on the microlens array. Each light flux is defined and explained as position coordinates (x ′, y ′). In the generation of a reconstructed image, by integrating a light beam having an optical path passing through the point for a pixel (x, y) on a reconstruction surface where a light beam incident from a subject distance for generating the reconstructed image converges. Pixel values can be obtained.

  In FIG. 5, the optical path of the light beam on the horizontal plane (xz plane) viewed from the vertical direction at the lateral position of the digital camera 100 is shown. Hereinafter, the optical path of the light beam passing through each pixel on the reconstruction plane in the xz plane will be described, but the same applies to the yz plane.

If the coordinates (u, v) of the pupil area and the pixel coordinates on the reconstruction plane are (x, y), the light beam passing through the pupil division area and the pixels on the reconstruction plane is incident on the microlens array 105. The position coordinates (x ′, y ′) of the microlens are
It is represented by Note that F is a distance from the imaging lens to the microlens array, and αF is a distance from the photographing lens to the reconstruction surface (α is a refocus coefficient: a variable coefficient for determining the distance to the reconstruction surface).

If the output of the photoelectric conversion element that receives the luminous flux is L (x ′, y ′, u, v), the pixel output E (x) of the coordinates (x, y) of the image formed on the reconstruction plane , Y) is obtained by integrating L (x ′, y ′, u, v) with respect to the pupil region of the photographing lens,
It is represented by By using (u, v) as the representative coordinates of the pupil region, the equation can be calculated by simple addition.

  The display unit 110 is a display device included in the digital camera 100 such as a small LCD. The display unit 110 is used to display a user interface screen of the digital camera 100, use as an electronic viewfinder, or display a photographed image. The display unit 110 displays a reconstructed image focused on an arbitrary subject distance that is generated and output by the image processing unit 109. As described above, the LF data obtained by A / D converting the analog image signal output from the imaging unit 106 according to the present embodiment does not connect the images in adjacent pixels. Therefore, not the LF data but the image data generated by the image processing unit 109 is displayed on the display unit 110.

  The recording medium 111 is a recording device that is detachably connected to the digital camera 100 such as a built-in memory of the digital camera 100, a memory card, or an HDD. The recording medium 111 records LF data and a reconstructed image focused on an arbitrary subject distance generated from these LF data. Alternatively, the generated image or the like is transmitted (output) to an external device (not shown) such as a personal computer via the communication unit 116.

  The operation input unit 112 is a user interface that the digital camera 100 has, such as a power button and a shutter button. When the operation input unit 112 detects that the user interface has been operated by the user, the operation input unit 112 outputs a control signal corresponding to the operation to the control unit 101. For example, the operation input unit 112 outputs various information related to shooting, such as setting of the shooting mode, to the control unit 101. A signal output from the release switch is used as an AE or AF operation start trigger or a shooting start trigger. Upon receiving these start triggers, the control unit 101 controls each unit of the imaging apparatus including the imaging unit 106 and the display unit 110.

  In the present embodiment, it is assumed that LF data is acquired by photographing processing and processing for generating a reconstructed image is performed. However, the LF data used for generation is from the recording medium 111 or via the communication unit 116. It may be acquired.

《Restriction information addition processing》
Here, processing relating to refocus restriction, which is characteristic processing of the present embodiment, will be described. Since reconstructed images focused on various subject distances can be generated from the LF data as described above, there is a possibility that privacy information included in the shooting range is not properly protected. That is, for example, when a document that normally does not permit photographing or copying is included in the photographing range, a reconstructed image that can read the document may be generated. Further, in recent years, images obtained by shooting are often released to online image sharing services and the like, and it is conceivable that LF data is released in a state that can be arbitrarily reconstructed. For example, a person who has unintentionally reflected in the shooting range may infringe portrait rights against the intention of the person depending on the conditions for generating a reconstructed image from LF data. Therefore, in the present embodiment, the presence of such a subject to be protected, a refocus range in which the subject is in focus or visible, and the refocus range (a plurality of generation conditions) are determined. In the reconstructed image, restriction information for making it difficult to visually recognize the subject to be protected is generated and output. The output restriction information is added to LF data or the like, or used when generating a reconstructed image in the digital camera 100. In addition to reconstructed images for refocusing, when reconstructed images are generated for some purpose, such as viewpoint changes or perspective changes, restriction information is generated to make it difficult to visually recognize the subject to be protected. And added to the LF data.

  A specific process of the restriction information adding process executed by the digital camera 100 of the present embodiment having such a configuration will be described with reference to the flowchart of FIG. Processing of each flow is performed in each unit in FIG. 1 according to the control unit 101 or an instruction from the control unit 101. Further, this process may be performed at the time of performing the normal shooting process, or may be performed, for example, when transmitting LF data captured to an external device. When adding restriction information when transmitting captured LF data to an external device, the LF data already captured and stored in the memory is read after a request to transmit the LF data is made to the external device. You may perform the process after step S1502.

  First, in step S1501, when an imaging instruction is issued by operating the operation input unit 112, imaging processing is performed and imaging data (LF data) is acquired. At this time, imaging may be performed in a state where a desired subject is focused by focusing by driving a focusing lens included in the imaging optical system 104 in advance. Further, the imaging unit 106 may acquire an imaging signal from before the imaging instruction, and may perform exposure control, subject (face) detection, white balance control, and the like by analyzing the image.

  In step S1502, the control unit 101 sets a distance information acquisition area (hereinafter referred to as a D area) for acquiring information on the distance to the subject for the LF data generated by imaging. The D area indicates a two-dimensional area in the pixel array constituting the target LF data, which is defined in units of pixel groups (photoelectric conversion elements 202) assigned to the microlens 201. That is, the size of the D area is defined according to the minimum resolution of the reconstructed image that can be generated from the LF data. For example, in the example of FIG. 2, since 5 × 5 pixels assigned to one microlens 201 correspond to the minimum resolution of the reconstructed image, the D area is a multiple of 5 in the horizontal direction and a multiple of 5 in the vertical direction. Is defined as a region having a number of pixels.

  The D area is appropriate for the accuracy required for the distance to the subject, the calculation capability of the device, the amount of calculation, the required frame rate, etc., with the pixel group corresponding to the minimum resolution that can be generated from the target LF data as a unit. May be set to any size.

  In step S1503, the control unit 101 calculates information on the representative distance to the subject included in the area for each of the D areas set in step S1502. Specifically, for each D area of the target LF data, the control unit 101 generates two types of reconstructed images (detection images) for defocus amount detection corresponding to the light beams that have passed through two different divided pupil regions. The image processing unit 109 generates the image. As described above, the detection image generated in this step may be one that divides the area of the exit pupil 301 into divided pupil areas of the left half and the right half, and corresponds to the light flux that has passed through each divided pupil area. However, the embodiment of the present invention is not limited to this, and the detection image may be an image corresponding to the light flux that has passed through the exit pupil 301 and has passed through two types of divided pupil regions having different optical axes. However, the selection method is not limited to this. Here, when one D area is configured including pixel groups corresponding to a plurality of microlenses, the result of only the pixel group corresponding to the central microlens may be substituted, or each microlens is supported. You may use the average value of the distance obtained about the pixel group to perform.

  Then, the control unit 101 calculates the representative distance of each D area according to the obtained defocus amount analysis result. The representative distance may be, for example, a distance for a subject located at the center of each D area, or may be an average value of distances obtained for subjects in the area.

  In the present embodiment, the control unit 101 generates a distance information map corresponding to the shooting range of the LF data using the representative distance information for each D area obtained in this way. For example, a reconstructed image generated by using a pixel corresponding to the central divided pupil region among the pixels associated with each microlens 201 of the LF data so that the depth of field becomes deep is shown in FIG. If so, the distance information map is as shown in FIG. The example of FIG. 7A shows an example in which subjects 701, 702, and 703 exist within the shooting range. In the distance information map of FIG. 7B, the closer the subject is to the digital camera 100 at the time of shooting, the contrast is higher. Shown high (high hatching density).

  In step S <b> 1504, the control unit 101 causes the image processing unit 109 to generate a reconstructed image having a deep depth of field as illustrated in FIG. 7A, and detects and recognizes a human face area from the reconstructed image. . Hereinafter, in the present embodiment, a description will be given on the assumption that a human face is detected as a specific subject among photographed subjects, and the restriction on the generation condition of a reconstructed image from LF data is determined based on the face region. However, the implementation of the present invention is not limited to this. For example, the subject to be detected is a region having a specific feature in the image, such as a building, an animal, or a character, and the restriction on the generation condition is determined according to the detection result. It may be done.

  When three faces can be detected as shown in FIG. 8 in the detection image of FIG. 7A, face recognition is performed for each, and the face ID and the distance (subject distance) from the digital camera 100 are acquired. The subject distance is acquired using the distance map of FIG. The face ID and subject distance acquired as described above are numbered and recorded as a detected face relative distance table as shown in FIG.

  Next, in step S1505, it is determined whether one or more faces have been detected in step S1504. If one or more faces can be detected, the process advances to step S1506. If no face has been detected, the process advances to step S1509 to save the image and end the process.

  In step S1506, it is determined whether at least one of the faces detected in step S1504 is a privacy protection target. If it is determined that it is a privacy protection target, the process proceeds to S1507, and if it is not determined, the process proceeds to S1508. The determination as to whether or not it is a privacy protection target may be made based on the following criteria, for example.

<Privacy protection criteria>
(1) Size standard of face area For example, the face area having the largest size among the detected face areas may be detected, and some or all of the other face areas may be determined as privacy protection targets. . In other words, it is highly likely that the face area that has been photographed to have the largest face area is assumed to be the main subject by the photographer. May be set as a privacy protection target as a face area of a person having a low relationship.

(2) Position reference of face area For example, among the detected face areas, the face area whose arrangement position in the detection image is closest to the center is detected, and part or all of the other face areas are subject to privacy protection. May be determined. In other words, since it is highly likely that the face area that has been shot so that the face area is captured in the center is assumed to be the main subject by the photographer, May be set as a privacy protection target as a face area of a person having a low relationship.

(3) Face area standard for recognition target For example, even if a face area of a person registered in advance as a recognition target is detected among the detected face areas, other face areas are determined as privacy protection targets. Good. That is, when a person's face is set as a recognition target in advance, it is considered that the relationship between the photographer and the person is guaranteed, so that a face area that could not be recognized is set as a privacy protection target. Good.

(4) Main Subject Category Criteria For example, if the target selected as the main subject by setting the shooting mode or the like is a landscape or the like, the face area of the person may all be determined as a privacy protection target. That is, not only a person's face but also a target category (person, landscape, building, etc.) to be selected as a main subject is predetermined, images belonging to other categories are set as privacy protection targets. Also good.

(5) User Selection As described above, the control unit 101 not only determines a privacy protection target with reference to a predetermined standard, but also is selected by an operation button (not shown) provided in the digital camera 100, for example. The subject may be determined as a privacy protection target. Alternatively, a subject other than the selected subject may be determined as a privacy protection target.

(6) Character string reference When a reconstructed image is generated, such as a car license plate or an address, if the character string that can be viewed is included in the image, all such character string areas are protected for privacy. It may be determined as a target. For the determination of the character string, a known OCR (optical character recognition) technique or the like can be used.

  In step S1507, the image processing unit 109 adds restriction information to the imaging data. Specifically, when it is determined that B and C in FIG. 8 are privacy protection targets, a blurring amount at which faces in B and C in FIG. 8 are not detected is calculated and added. Although a detailed calculation method will be described later, the location information of the privacy protection area shown in FIG. 10 and the restriction information table shown in FIG. 11 are added to the imaging data. For example, as the position information of the subject corresponding to the privacy protection area, the start point (x1, y1) (x2, y2) and end point (x1 ′, y1 ′) (x2 ′, y2 ′) of the coordinates of the face frames B and C in FIG. ) (FIG. 11). In the restriction information table of FIG. 11, the amount of blur applied to the privacy protection area is held for each aperture value (depth of field) and focal position (subject distance). For example, B and C in FIG. 8 are determined to have a subject distance (focusing position) of 10 m in the distance map of FIG. 7B, and in the case of an aperture value of 2, B and C have a subject distance of 6 m to 14 m. Assume that the face can be identified. Considering that the range is blurred by image processing for privacy protection, the amount of blur varies until the object cannot be identified, and the size varies depending on the subject distance. For example, at the subject distance 10 m where the privacy protection target exists as shown in the upper side of FIG. 11, the blurring amount is increased because the object can be identified unless it is greatly blurred. On the other hand, the subject distances of 6 m and 14 m are out of the subject distance of the target, and the target cannot be recognized even if the blur is reduced, so the blur amount is reduced. In this way, the blur amount is obtained for the subject distance range where it is assumed that the privacy protection target can be recognized well, and is stored in the restriction information table. On the other hand, when the aperture value is 5.6, the depth of field becomes deep and the faces B and C can be recognized at a wider subject distance of 3.5 m to 16.5 m, so the range is blurred. In this case, as shown in the lower side of FIG. 11, the blur amount is large at the subject distance of 10 m, the blur amount is small at the subject distances of 3.5 m and 16.5 m, and the blur amount is medium at the subject distances of 6 m and 14 m. become.

  In the above-described example, the refocus process is described as the reconstruction process for the reconstructed image to which the restriction information is added. In a reconstructed image generated mainly using subpixels corresponding to some incident directions (pupil regions), such as changing the viewpoint, the depth of field becomes deep, so blurring processing with a large blurring amount is applied. It is good to set information. Note that the method of changing the blur amount can be realized by changing the characteristics of the Gaussian filter, for example. Specifically, three types of cut-off frequencies of the spatial frequency of the image may be set and the filter coefficients corresponding to each may be stored in the memory. When the amount of blurring is small, the blurring process is performed using the filter coefficient having the highest cutoff frequency. When the amount of blurring is large, the blurring process is performed using the filter coefficient having the lowest cutoff frequency. Similarly, when the amount of blurring is medium, blurring processing is performed using a filter coefficient having a characteristic in which the cutoff frequency takes an intermediate band. In this way, the blur amount can be adjusted. In the present embodiment, the blurring process is described as the process applied to the privacy protection target area. However, the present invention is not limited to this, and various methods are conceivable as image processing for privacy protection, such as mask processing for replacing with a predetermined image, and changing the value of a luminance signal or color signal to make the target unrecognizable. It is done.

  Next, in step S1508, it is determined whether or not all the faces managed in the face information table of FIG. If there is a face that has not been determined, the process returns to step S1506 to repeat the process. If all faces have been determined, the process proceeds to step S1509.

  Finally, in step S1509, the imaging data (LF data), the privacy protection information in FIG. 10, and the restriction information table in FIG. At this time, flag information or the like that prescribes protection processing for the privacy protection area of the imaged data using privacy protection protection information is added as necessary, and the flag is set when reproducing the imaged data. You may make it read.

<Reproduction processing>
Next, a procedure for performing processing for protecting privacy information during image reproduction using restriction information will be described with reference to the flowchart of FIG. Here, a description will be given of an example of a procedure for reproducing the imaging data to which the privacy protection area in FIG. 10 and the restriction information table in FIG.

  First, in step S 1801, when an image reproduction instruction is issued by operating the operation input unit 112, imaging data (LF data) is read from the recording medium 111 and temporarily stored in the RAM 103.

  In step S1802, the subject distance and the initial aperture value of the image data are read from the RAM 103. The LF data can reconstruct an image at an arbitrary subject distance within a refocusable range. Therefore, when displaying an image, it is necessary to set the initial values of the subject distance and the aperture value. In this embodiment, information on whether or not a subject satisfying a specific condition is detected and information on which region in the imaging data the detected subject exists is read from the header information associated with the imaging data. . It is also conceivable to detect a representative subject at the time of LF data acquisition and set the subject distance at a position where the subject exists. Alternatively, since the position of the focus lens at the time of shooting is known in advance, the position may be set as the subject distance, but the present idea is not limited.

  Next, in step S1803, whether the subject distance within the aperture value of the initial setting value read in step S1802 is within the subject distance range to which privacy protection processing should be applied to a predetermined range in the image described in the restriction information table. Determine if. If it is not within the range of the restriction information table, the process advances to step S704 to generate a reconstructed image with the set value. If it is within the range of the restriction information table, the process proceeds to step S1805. In the present embodiment, the subject distance 6 m, which is the initial setting value in FIG. 15, is a range of “aperture value 5.6, subject distance 3.5 m to 16.5 m” that is a blurring target range defined by the restriction information table in FIG. Since it is within the range, the process advances to step S1805.

  In step S1805, the image processing unit 109 (protected image generating unit) blurs the privacy protection target face corresponding to the subject distance range included in the restriction information table with the blur amount described in the restriction information table. Generated image (protected image). In this embodiment, the faces B and C are within the limit information table shown in FIG. Therefore, a blurred image is generated by setting the privacy protection areas of the faces B and C shown in FIG. 10 to the blurring amount recorded in the restriction information shown in FIG. In the case of the initial setting value, since the aperture value is 5.6 and the subject distance is 6 m, a blurred image with the blurring amount “medium” is generated from FIG.

  Assume that the reconstructed image shown in FIG. 16 is generated by the above processing. In FIG. 16, the area surrounded by the dotted line is replaced with the blurred image.

  By performing the above processing, as shown in FIG. 16, it is possible to blur only the privacy protection area with an appropriate blur amount and create an image in which privacy is protected.

  In step S 1806, the image created in step S 1804 or S 1805 is displayed on the display unit 110.

  Finally, in step S1807, it is determined whether or not the reproduction mode is continued. If the reproduction mode is continued, a refocus instruction is waited in step S1808.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

  In this embodiment, based on the database illustrated in FIG. 11, in step S1803, it is determined whether the subject distance is within the range of the subject distance to which the blurring process should be applied to the predetermined range in the image. Protection processing is applied to the privacy area by blurring processing or the like. However, the present invention is not limited to this. For areas in the image determined to be subject to privacy protection (areas B and C in FIG. 10), blur processing and mask processing are always performed as privacy protection processing regardless of the subject distance. You can go.

  For example, in the embodiment, the face is mainly described as the subject. However, the subject is not limited to the face, and an implementation system in which a database of in-focus objects is created on the Internet, and the object registered therein is subjected to blurring processing. Various modifications are possible. In this case, the image processing unit 109 has a database for detecting registered in-focus prohibited objects, and it is necessary to detect in-focus prohibited objects registered in advance using LF data.

  Also, it can be easily imagined that by adding restriction information (blurring process) to the entire range other than the main subject, only the main subject can be conspicuous, and various modifications and changes can be made within the scope of the gist. Is possible.

[Embodiment 2]
In the second embodiment of the present invention, privacy protection processing is applied to image data (LF data) to which no restriction information is added to the privacy protection area or blur processing is not performed, and image display is performed. I do. FIG. 17 is a flowchart showing a procedure of privacy protection processing during image reproduction in the present embodiment.

  First, in step S2101, the imaging data recorded on the recording medium 111 is read. Here, the light field image may be acquired from the external device using the communication unit 116, and the method of reading the imaging data is not limited. It is also conceivable that imaging is performed by the imaging unit 106 and LF data is output in step S2101.

  Next, since step S2102 is the same process as step S1802 of the first embodiment, a description thereof will be omitted.

  In step S2103, a reconstructed image is created based on the subject distance and aperture value read in step S2102.

  Next, since the process of creating a distance information map and performing face detection from step S2104 to step S2106 is the same as step S1502 to step S1504 of the first embodiment, description thereof will be omitted.

  Next, in step S2107, it is determined whether one or more faces have been detected in step S2106. If one or more faces can be detected, the process advances to step S2108. If no face is detected, the process advances to step S2113 to display the reconstructed image on the display unit 110.

  Next, since step S2108 is the same process as step S1506 of the first embodiment, a description thereof will be omitted.

  Next, in step S2109, image processing (blurring processing) for privacy protection is performed on the privacy protection area to create an image in which the privacy protection target cannot be recognized well, and the corresponding area of the reconstructed image is overwritten. To do.

  Next, since step S2110 is the same process as step S1508 of the first embodiment, a description thereof will be omitted.

  In step S2111, the reconstructed image is displayed on the display unit 110. Alternatively, in step S2111, the reconstructed image is transmitted via the communication unit 116 in response to a user operation or a request from an external device. For example, when a user uploads an image to SNS or the like using LF data, a reconstructed image subjected to privacy protection processing may be generated and transmitted to a server that is an external device. It is also conceivable that the digital camera 100 starts the processing of this flowchart in response to a transmission request for reconstructed image data obtained from LF data in the digital camera 100 from an external device.

  Finally, in step S2112, it is determined whether or not the reproduction mode is continued. If the reproduction mode is continued, the refocus instruction is waited in step S2113. If a refocus instruction is given in step S2113, a reconstructed image is created with the instructed set value in step S2114, and the process proceeds to step S2106 to repeat the process.

  In this embodiment, privacy protection information is associated with image data and stored by the digital camera 100, and protection processing is performed by the digital camera 100 based on the privacy protection information during playback. However, the present invention is not limited to this, and an image processing system may be constructed in which recorded image data and protection information are read by another image processing apparatus, and the image data is protected and reproduced.

  Further, in the present embodiment, an example has been described in which so-called LF data is acquired as imaging data, and protection processing is performed when the reconstructed image is generated. However, it is obtained from an image pickup apparatus that picks up image data obtained by a normal image pickup device that is not LF data (for example, the pixels under the same microlens as referred to in the image pickup device of the present embodiment are added to form unit pixels). The present invention can also be applied to image data.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

  For example, privacy protection information that is already registered on the playback device side (for example, the playback device owner's face, address, etc.) is excluded from privacy protection and displayed without blurring. May be.

[Modification]
In the above-described embodiment, the LF data acquired by the configuration of the imaging optical system 104, the microlens array 105, and the imaging unit 106 illustrated in FIG. 3 is targeted, but the present invention has the configuration illustrated in FIG. The obtained LF data may be used. The configuration of FIG. 13 corresponds to a plenoptic camera described in the literature “Todor Georgiev and Andrew Lumsdaine,“ Superresolution with Plenoptic 2.0 Camera ”, Signal Recovery and Synthesis, Optical Society of America, 2009”. The configuration of FIG. 13 is different in that the focal point of the imaging optical system 104 is provided not at the surface of the microlens array 105 but at a position closer to the exit pupil of the imaging optical system 104.

  In the above-described embodiment, the light beam that has passed through one imaging optical system 104 is divided by each microlens 201 to obtain pixel outputs corresponding to each divided pupil region. However, the embodiment of the present invention may be realized by handling a plurality of imaging data obtained by a so-called multi-lens camera having a plurality of optical systems as shown in FIG. 14 in the same manner as the LF data in this embodiment. Good. In other words, the output of the plurality of imaging elements that are optically imaged by each of the plurality of optical systems is an image corresponding to the light flux that has passed through different divided pupil regions, that is, an image of the subject taken from different directions. The output of the eye camera is equivalent to the LF data in this embodiment. As a method for generating a reconstructed image from such a multi-lens camera, a method described in JP 2011-022796 A may be used.

[Other Embodiments]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

  100: Digital camera, 101: Control unit, 102: ROM, 103: RAM, 104: Imaging optical system, 105: Micro lens array, 106: Imaging unit, 107: AFE, 108: DFE, 109: Image processing unit, 110 : Display unit, 111: recording medium, 112: operation input unit, 116: communication unit, 118: GPS, 119: electronic compass, 201: microlens, 202: photoelectric conversion element, 301: exit pupil

Claims (25)

Data acquisition means for acquiring light field data;
A selection means for selecting a subject;
Information relating to a reconstructed image generated by reconstructing the light field data, for making the subject selected by the selection means difficult to visually recognize over the reconstructed image reconstructed under a plurality of generation conditions And an output means for outputting restriction information, which is information.   The image processing apparatus according to claim 1, further comprising a recording unit that records the restriction information output by the output unit in association with the light field data. The selection means includes detection means for detecting a predetermined subject in the reconstructed image generated from the light field data,
The image processing apparatus according to claim 1, wherein the selection unit selects at least a part of the predetermined subjects detected by the detection unit.   The restriction information is information specifying that a process for making it difficult to visually recognize the subject selected by the selection unit is applied to a reconstructed image generated by reconstructing the light field data. The image processing apparatus according to claim 1, wherein the image processing apparatus is characterized in that:   The image processing apparatus according to claim 4, wherein the information defining that the process for making it difficult to visually recognize is determined for each reconstructed image generation condition.   The information defining that the process for making it difficult to view is applied defines a range of a subject distance to which the process is applied according to a depth of field of a generated reconstructed image. Item 6. The image processing apparatus according to Item 4 or 5. The process for making it difficult to visually recognize is a blurring process for an area in the image,
The restriction information is defined such that the reconstructed image focused on a subject distance closer to the subject distance focused on the subject selected by the selection unit increases the intensity of blurring processing. The image processing apparatus according to claim 4.   The image processing apparatus according to claim 4, wherein the process for making the image difficult to visually recognize is a mask process for a region in the image.   The process for making it difficult to visually recognize is a process for converting a luminance signal or a color signal of an area corresponding to the subject selected by the selection unit so that the subject is less likely to be visually recognized. The image processing apparatus according to any one of 4 to 7. Data acquisition means for acquiring light field data;
Information acquisition means for acquiring restriction information for making it difficult to visually recognize a subject over a plurality of generation conditions of a reconstructed image for a reconstructed image that can be generated from the light field data;
Setting means for setting conditions for generating a reconstructed image generated from the light field data;
An image processing apparatus comprising: a determination unit that determines a generation condition of a reconstructed image to be finally generated based on the generation condition set by the setting unit and the restriction information.   The image processing apparatus according to claim 10, further comprising a generation unit that generates a reconstructed image from the light field data based on the generation condition determined by the determination unit.   The said generating means applies the process for making it difficult to visually recognize the said subject to the area | region corresponding to the designated subject in a reconstruction image based on the said restriction | limiting information. Image processing apparatus.   The restriction information is information defining that a process for making it difficult to visually recognize the designated subject is applied to a reconstructed image generated by reconstructing the light field data. The image processing apparatus according to claim 12.   The image processing apparatus according to claim 12 or 13, wherein the information defining that the processing for making it difficult to visually recognize is determined for each generation condition of a reconstructed image.   The information defining that the process for making it difficult to view is applied defines a range of a subject distance to which the process is applied according to a depth of field of a generated reconstructed image. Item 15. The image processing device according to any one of Items 12 to 14. The process for making it difficult to visually recognize is a blurring process for an area in the image,
13. The restriction information is defined such that a reconstructed image focused on a subject distance closer to a subject distance focused on the designated subject increases the intensity of blurring processing. 16. The image processing device according to any one of items 15 to 15.   The image processing apparatus according to any one of claims 12 to 16, wherein the process for making it difficult to visually recognize is a mask process for a region in an image.   The process for making it difficult to visually recognize is a process of converting a luminance signal or a color signal of an area corresponding to the designated subject so that the subject is difficult to visually recognize. The image processing apparatus according to any one of the above.   19. The restriction information is information for making it difficult to visually recognize a subject other than a main subject in a reconstructed image generated by reconstructing the light field data. The image processing apparatus according to claim 1.   2. The light field data according to claim 1, wherein each of the pixel signals has a signal intensity corresponding to a light beam having a different combination of a pupil region and an incident direction of an imaging optical system when the light field data is imaged. 20. The image processing device according to any one of items 1 to 19.   The light field data is ray space information describing a three-dimensional space of a subject, and a reconstructed image that focuses on an arbitrary subject distance can be generated by reconstructing the data. Item 21. The image processing apparatus according to any one of Items 1 to 20. A data acquisition process for acquiring light field data;
A selection process for selecting a subject;
Information relating to a reconstructed image generated by reconstructing the light field data, for making it difficult to visually recognize the subject selected in the selection step over the reconstructed image reconstructed under a plurality of generation conditions. And an output step of outputting restriction information that is information. A data acquisition process for acquiring light field data;
An information acquisition step for acquiring restriction information for making it difficult to visually recognize a subject over a plurality of reconstruction image generation conditions for a reconstructed image that can be generated from the light field data;
A setting step for setting a generation condition of a reconstructed image generated from the light field data;
An image processing method comprising: a determination step of determining a generation condition of a reconstructed image to be finally generated based on the generation condition set in the setting step and the restriction information.   A program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 21.   A computer-readable recording medium on which the program according to claim 24 is recorded.