JP7373297B2 - Image processing device, image processing method and program - Google Patents

Description

The present invention relates to an image processing device, an image processing method, and a program, and particularly relates to a technique for suitably extracting a desired subject area from an image.

By classifying a group of pixels included in a captured image by the amount of defocus of the image of the pixel and applying image processing to pixels belonging to a specific defocus range, different depths of field can be expressed. There is a technique for generating images (Patent Document 1).

Japanese Patent Application Publication No. 2008-015754

By the way, one of the impressive image expressions is a so-called shallow depth of field expression in which only the distance range of the object of interest is focused, and other areas are blurred. The technique disclosed in Patent Document 1 also makes it possible to generate an image with a similar expression by performing blurring processing on pixels outside the defocus range of a region including a desired subject.

However, in the method disclosed in Patent Document 1, in which pixels are specified and processed based on the amount of defocus, an image with a desired expression may not be generated depending on the distribution of the subject. For example, in a distribution in which a subject different from the desired subject is arranged in a distance range that extends in the depth direction and includes the subject distance of the desired subject, when the above-mentioned image processing is performed, Patent Document 1 With this method, it is not possible to separate and identify these objects. That is, for example, when trying to generate an image with a shallow depth of field, not only the desired subject but also the distance range including the different subjects will appear in the image in a focused state, which is a preferable method. The expression may not be realized.

The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an image processing device, an image processing method, and a program that suitably extract a region of a subject of interest based on the distance distribution of the subject. do.

In order to achieve the above-mentioned object, the image processing device of the present invention shows a distance distribution in the depth direction of the subject in the imaging range for a captured image of a subject in the imaging range and a region corresponding to the captured image. an acquisition means for acquiring distance information storing a value; a determining means for determining a subject of interest in the captured image; and a setting means for setting an analysis region for analyzing distance distribution based on the position of the subject of interest in the captured image. and identifying means for specifying a value range in which objects included in the analysis area are distributed based on the frequency distribution of values included in the analysis area of the distance information, and distance information indicating information included in the value range. The present invention is characterized by comprising an extraction means for extracting a subject area including at least a subject of interest by extracting a region.

With such a configuration, according to the present invention, it is possible to suitably extract the region of the subject of interest based on the distance distribution of the subject.

A block diagram showing the functional configuration of a digital camera 100 according to an embodiment and a modification of the present invention A diagram showing the configuration of an image sensor included in the image capture unit 105 according to an embodiment and a modification of the present invention. A block diagram for explaining the configuration of the image processing unit 107 according to the embodiment and modification of the present invention Flowchart illustrating image processing processing executed by the digital camera 100 according to the embodiment and modification of the present invention Diagram for explaining subject distribution in an imaging range illustrated in an embodiment of the present invention Diagram for explaining derivation of defocus amount according to embodiments and modified examples of the present invention Another diagram for explaining derivation of the defocus amount according to the embodiment and modification of the present invention A diagram illustrating a defocus map according to an embodiment of the present invention A diagram illustrating a histogram constructed for the entire area of a defocus map according to conventional technology A diagram for explaining extraction of a region within a predetermined defocus range according to conventional technology A diagram for explaining an analysis region forming a histogram for a subject of interest according to an embodiment of the present invention A diagram illustrating a histogram constructed for a region corresponding to a subject of interest according to an embodiment of the present invention Diagram for explaining regions extracted based on a histogram according to an embodiment of the present invention Another diagram for explaining regions extracted based on a histogram according to an embodiment of the present invention Diagram for explaining regions forming a histogram for a subject of interest according to a modification of the present invention Diagram for explaining the configuration of a histogram with low processing load according to an embodiment of the present invention Another diagram for explaining the configuration of a histogram with a low processing load according to an embodiment of the present invention Yet another diagram for explaining the configuration of a histogram with low processing load according to the embodiment of the present invention

[Embodiment]
Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the claimed invention. Although a plurality of features are described in the embodiments, not all of these features are essential to the invention, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar components are designated by the same reference numerals, and redundant description will be omitted.

An embodiment described below applies the present invention to a digital camera, which is an example of an image processing device, and is capable of acquiring information (distance information) corresponding to a distance distribution in the depth direction (depth direction) of a subject in an imaging range. An example of application will be explained. However, the present invention is applicable to any device capable of acquiring a captured image and distance information related to the imaging range of the captured image.

Furthermore, in this specification, "distance information" is two-dimensional information indicating the defocus amount distribution of each pixel of the captured image, and hereinafter referred to as a "defocus map." However, as described above, the distance information only needs to be information that corresponds to the distance distribution in the depth direction of the subject in the imaging range, for example, it may be a depth map that indicates the subject distance of each pixel, or it may be a depth map that indicates the subject distance of each pixel, or it may be a depth map that indicates the subject distance of each pixel. It may be two-dimensional information indicating the amount of image shift used for derivation. That is, the distance information is not limited to the above information as long as it indicates a change according to the distance distribution in the depth direction.

《Digital camera configuration》
FIG. 1 is a block diagram showing the functional configuration of a digital camera 100 according to an embodiment of the present invention.

The system control unit 101 is a control device, such as a CPU, that controls each block included in the digital camera 100. The system control unit 101 controls the operation of each block included in the digital camera 100 by reading out an operation program for each block included in the digital camera 100 from the ROM 102, loading it into the RAM 103, and executing the program.

The ROM 102 is, for example, a rewritable nonvolatile memory such as a flash ROM. The ROM 102 stores operation programs for each block included in the digital camera 100, as well as parameters necessary for the operation of each block. On the other hand, the RAM 103 is a rewritable volatile memory. The RAM 103 is used not only as a storage area for the operation programs of each block included in the digital camera 100, but also as a temporary storage area for intermediate data output by the operation of each block. In this embodiment, it is assumed that the system control unit 101 and the image processing unit 107 (described later) use the RAM 103 as a work memory.

The optical system 104 forms a subject image on the imaging unit 105 . The optical system 104 includes, for example, a fixed lens, a variable magnification lens that changes the focal length, a focus lens that adjusts the focus, and the like. The optical system 104 also includes a diaphragm, and by adjusting the aperture diameter of the optical system using the diaphragm, the amount of light during photographing is adjusted.

The imaging unit 105 is, for example, an imaging device such as a CCD image sensor or a CMOS image sensor. The imaging unit 105 photoelectrically converts the optical image formed on the imaging surface of the image sensor by the optical system 104 to obtain an analog image signal. The imaging unit 105 outputs the obtained analog image signal to the A/D conversion unit 106.

The A/D conversion unit 106 obtains digital image data (hereinafter simply referred to as image data) by applying A/D conversion processing to the input analog image signal. The A/D converter 106 outputs the obtained image data to the RAM 103 and stores it therein.

The image processing unit 107 performs various image processing on the image data stored in the RAM 103. Specifically, the image processing unit 107 performs image processing such as white balance adjustment, color interpolation, and reduction/enlargement processing. The image processing unit 107 records the image data after the image processing onto the recording medium 108 .

Further, the image processing unit 107 includes blocks that perform processing as shown in FIG. 3 in order to realize the functions according to the present invention. Although the details will be described later, the image processing unit 107 includes a generation unit 300 that generates a defocus map (distance information) indicating the distribution of the amount of defocus for the imaging range in which the image has been captured, and a generation unit 300 that It includes a detection unit 301 that detects. The image processing unit 107 also includes a region setting unit 302 that sets an area for analysis regarding the target subject set based on the detection result by the detection unit 301, and a frequency map (of defocus amount) for the set area. a histogram). Furthermore, the image processing unit 107 includes an area extraction unit 304 that extracts the finally determined area of the subject of interest, and an effect processing unit 305 that performs image processing to transform the captured image into a form that shows a desired image expression. .

The recording medium 108 is, for example, a recording device such as a memory card that may be configured to be removably attached to the digital camera 100. In the recording medium 108, image data (captured image) to which image processing is applied by the image processing unit 107, image signals (RAW images) converted from A/D by the A/D conversion unit 106, etc. are recorded as recorded images. be done.

The display unit 110 is, for example, a display device such as an LCD, and presents various information on the digital camera 100. The display unit 110 functions as a digital viewfinder by displaying (through-display) A/D-converted image data when the image capturing unit 105 is capturing an image.

The operation input unit 109 includes a user input interface such as a release switch, a setting button, a mode setting dial, etc., and upon detecting an operation input made by the user, outputs a control signal corresponding to the operation input to the system control unit 101. . Further, in an embodiment in which the display unit 110 includes a touch panel sensor, the operation input unit 109 also functions as an interface that detects a touch operation performed on the display unit 110.

These functional blocks included in the digital camera 100 are basically connected by a bus 111, and are configured such that signals can be exchanged between the blocks via the bus 111.

In this embodiment, processing will be described as hardware implemented by circuits and processors corresponding to each block included in the digital camera 100. However, the implementation of the present invention is not limited to this, and the processing of each block may be realized by a program that performs the same processing as that of each block.

《Configuration of imaging section》
Next, the detailed configuration of the image sensor included in the image capture unit 105 of this embodiment will be described with reference to FIG. 2.

As shown in FIG. 2(a), the image sensor has a plurality of pixels 200 regularly arranged in two dimensions. Specifically, the plurality of pixels 200 are arranged, for example, in a two-dimensional grid. Note that the arrangement configuration of the pixels 200 is not limited to a grid-like arrangement, and other arrangement configurations may be adopted.

In order to realize the distance measurement function of the imaging plane phase difference distance measurement method, each pixel 200 includes a microlens 201 and a pair of photoelectric conversion units 202a and 202b (hereinafter referred to as pupil division) as shown in FIG. 2(b). pixels 202a and b). The pupil division pixels 202a and 202b are photoelectric conversion units that are configured to have the same shape and size and have a rectangular imaging surface whose longitudinal direction is the y-axis direction. In each pixel 200, the pupil division pixels 202a and 202b are arranged symmetrically with respect to the perpendicular bisector of the microlens 201 along the y-axis direction as an axis of symmetry. Note that the shape of the imaging surface of the pupil division pixels 202a and 202b is not limited to this, and may be any other shape. Furthermore, the arrangement of the pupil division pixels 202a and 202b is not limited to being arranged in parallel in the x direction, and other arrangements may be adopted.

With such a configuration, the imaging unit 105 of this embodiment can capture the A image related to the image signal output from the pupil division pixel 202a that all pixels of the image sensor have, and the image similarly output from the pupil division pixel 202b. A B image related to the signal is output. The A image and the B image have a parallax that corresponds to the distance from the in-focus position.

More specifically, in each pixel 200, the pupil division pixels 202a and 202b are configured to receive different light fluxes among the light fluxes incident through the microlens 201. The optical image related to the light beam passing through the area is photoelectrically converted. In other words, since images A and B are generated for the light beams that have passed through different areas of the exit pupil (pupil division areas), the subject is photographed at a photographing position that is shifted by the difference in the center of gravity of the pupil division areas. The relationship between the images is taken, and parallax will occur. In other words, the A image and the B image correspond to a group of images obtained by imaging the imaging range from different viewpoints.

In this embodiment, the image sensor of the imaging unit 105 is configured as shown in FIG. 2 to obtain the A image and the B image used for deriving the distance distribution of the object in the imaging range. , the method of acquiring the A image and the B image is not limited to this. For example, images A and B may be a group of images captured by multiple imaging devices installed with a baseline length apart, or a single imaging device having multiple optical systems and imaging units (so-called A group of images acquired by a binocular camera, etc.) may be used as images A and B, respectively.

《Derivation of defocus amount》
The generation unit 300 generates a defocus map having a pixel structure corresponding to the captured image, which includes the amount of defocus of each subject image included in the captured image, as information indicating the distance distribution of the subject in the depth direction of the captured image. generate. When the defocus amount is derived based on the acquired image group (image A and image B) having parallax as in this embodiment, it may be performed as follows, for example.

Derivation of the defocus amount includes, for example, a process of dividing an image (image A and image B) 700 into minute blocks 701 shown by broken lines, as shown in FIG. For example, when each pixel of the target image A is a pixel of interest, the micro block 701 may be set for a region of a predetermined size centered on the pixel. In the following description, the microblock 701 will be described as being set as a square area of m×m pixels centered on the pixel of interest, but the microblock 701 may have any shape or size. Further, the minute block 701 is set for each pixel of interest, and the minute blocks 701 may overlap between different pixels of interest. Further, the generation unit 300 may derive the defocus amount after applying a process of applying a bandpass filter to the A image and the B image whose phase difference is to be detected in advance.

For example, when a microblock 701 is set for each pixel of image A and image B, correlation calculation processing is performed for each pixel (pixel of interest) between both images, and the deviation of the image included in the microblock 701 corresponding to the pixel is calculated. The amount (image shift amount) is derived. When the number of data (number of pixels) of a (pair of) micro blocks 701 determined for the pixel of interest at the same position in the A image and the B image is m, the pixel data of the pair of micro blocks 701 are each expressed as E(1 ) to E(m), and F(1) to F(m). In this case, in the correlation calculation, if the shift amount (of data) is k (integer) [pixel], the correlation amount C(k) is C(k)=Σ|E(n)−F(n+k)|
It can be derived as Here, it is assumed that the Σ operation is performed on n, and n and n+k are limited to the range of 1 to m. Further, the shift amount k is a relative shift amount in units of detection pitches of a pair of image data. In this way, when the correlation amount is derived for a pair of pupil division images (a pair of microblocks 701) related to one pixel of interest, the shift amount k and the correlation amount C(k) are discretely shown in the graph of FIG. 6, for example. The relationship is as shown. At this time, since the correlation amount C(k) is the minimum at the image shift amount where the correlation is highest, the following three-point interpolation method is used to shift the amount to give the minimum value C(x) for the continuous correlation amount. Derive the quantity x.
x=kj+D/SLOP
C(x)=C(kj)−|D|
D={C(kj-1)-C(kj+1)}/2
SLOP=MAX{C(kj+1)-C(kj), C(kj-1)-C(kj)}
Here, kj is the shift amount k that minimizes the discrete correlation amount C(k). The shift amount x obtained in this way is included in the distance information as the image shift amount for one pixel of interest. Note that the unit of the image shift amount x is also [pixel].

Therefore, the defocus amount DEF for each pixel of interest is calculated using the image shift amount x.
DEF=KX・PY・x
It can be derived as Here, PY is the pixel pitch of the image sensor (distance between pixels constituting the image sensor, unit [mm/pixel]), and KX is the size of the aperture angle of the center of gravity of the light flux passing through a pair of ranging pupils. It is a conversion coefficient determined by the Note that the size of the aperture angle of the center of gravity of the light flux passing through a pair of distance measuring pupils changes depending on the size of the aperture aperture (F number) of the lens, so it is determined according to the information set at the time of imaging. shall be

In this way, the generation unit 300 can derive the defocus amount of the subject at each pixel of the captured image by repeatedly calculating while shifting the pixel position of interest one pixel at a time. When the generation unit 300 derives the defocus amount of each pixel, it generates a defocus map, which is two-dimensional information having the same structure as the captured image, and uses this as the pixel value as distance information. In other words, the amount of defocus is a value that changes depending on the amount of deviation in the position in the depth direction from the distance of the subject that is in focus in the captured image, so the defocus map is a value that changes depending on the distance in the depth direction of the subject at the time of imaging. This is information equivalent to distribution.

《Image processing》
Hereinafter, in the digital camera 100 of this embodiment, based on the captured image and the defocus map, an image that is focused only on the distance range of the subject of interest and has a narrower depth of field than when the image was captured is generated. The image processing process will be explained using the flowchart of FIG. The processing corresponding to the flowchart can be realized by the system control unit 101 reading out a corresponding processing program stored in, for example, the ROM 102, loading it into the RAM 103, and executing it. This image processing process will be described assuming that, for example, the digital camera 100 is set to a shooting mode that generates an image whose expression has been changed after shooting, and is started when an operation input related to a shooting instruction is detected.

In S401, the imaging unit 105 images a subject under the control of the system control unit 101. When an analog image signal is obtained by imaging, the A/D converter 106 A/D converts it and stores it in the RAM 103 . In this image processing process, the image data stored in the RAM 103 is obtained by adding the A image captured by the pupil division pixel 202a, the B image captured by the pupil division pixel 202b, and the A image and the B image. This is a captured image (an image corresponding to a state in which the pupils are not divided).

In the following description, it is assumed that objects are distributed as shown in FIG. 5 in the imaging range. As shown in the figure, the imaging range includes a human subject 500 with his hand raised, a hedge 501 extending from a position closer to the digital camera 100 than the human subject 500 to a position farther than the human subject 500, and a hedge 501 extending further than the hedge 501. A tree 502 that exists far away is included. Further, it is assumed that the captured image is captured with an imaging setting that focuses on the human subject 500. In the image processing process of this embodiment, the human subject 500 is selected as a subject of interest in a step described later, and the subject of interest is focused only in a distance range, and objects in other distance ranges are blurred. , the following description assumes that an image with a shallow depth of field is generated. Further, for the sake of simplicity, the description will be made assuming that the distance distribution in the depth direction is derived only for the human subject 500, the hedge 501, and the tree 502, excluding the ground distributed in the imaging range.

In S402, under the control of the system control unit 101, the generation unit 300 generates a defocus map corresponding to the captured image based on the A image and the B image stored in the RAM 103 in S401. FIG. 8 shows an example of a defocus map visualized by converting the defocus amount, which is a pixel value, into a gray scale value. To make the explanation easier to understand, in the example in Figure 8, the closer the object distance is, the closer the pixel is to white (higher pixel value), and the further away the object is, the closer to black the value (lower pixel value). Assume that it has been converted.

By the way, when the human subject 500, the hedge 501, and the tree 502 are arranged in the distribution as described above, a histogram (frequency distribution) based on the defocus amount is constructed for the defocus map as in the technique of Patent Document 1. In this case, the distribution will be as shown in FIG. In FIG. 9, a mountain in the histogram indicated by 901 indicates the distribution of defocus amount in a pixel group related to a tree 502 located far away. On the other hand, the peak of the histogram indicated by 900 indicates the distribution of defocus amounts in the pixel group related to the human subject 500 and the hedge 501.

Here, even if pixels in the defocus range indicated by d1 to d2, which indicate the frequency of exceeding a predetermined threshold value 902, are extracted as described in Cited Document 1, as shown in FIG. Only the area related to the hedge 501 (white area in the figure) is extracted. Therefore, even if processing is performed based on such extraction results, it is not possible to obtain an image with a shallow depth of field that focuses only on the distance range of the human subject 500 that is desired to be the subject of interest, as in this embodiment. I can't. In other words, in an embodiment in which the hedge 501 has a shape that extends widely in the depth direction and is arranged to include the subject distance range of the human subject 500, only the distance range of the human subject 500 is Separation and extraction cannot be performed using the described technique. Therefore, even if blurring processing is applied to areas outside the extracted area, the distance range in the depth direction of the hedge 501 will be in focus, making it impossible to suitably narrow the depth of field, resulting in an image with the desired expression. Not.

Therefore, in the image processing according to the present invention, when specifying the value range of the defocus amount to be extracted based on the histogram of the defocus amount, instead of using the histogram for the entire area of the captured image, Use a histogram for regions. The details of the process will be further explained below.

In S403, the detection unit 301 performs processing to detect a predetermined subject included in the captured image under the control of the system control unit 101. As described above, in this embodiment, it is necessary to specify the area of the human subject 500, so the detection unit 301 detects the face area of the person included in the captured image, and determines the position and size (FACE_SIZE_X, FACE_SIZE_Y) information. In this embodiment, the detection unit 301 will be described as detecting a person's face area, but the implementation of the present invention is not limited to this. For example, the detection target may be a person's torso, a passenger car, a pet, a plant, etc. It may be set as .

In this embodiment, the detection unit 301 detects whether or not a predetermined type of subject is included in the captured image, and sets the subject of interest based on the detection result. It will be readily understood that the implementation of the invention is not limited to this. That is, the method of setting the subject of interest from the captured image does not need to be based on image analysis, and may be set based on the user's selection operation of the subject of interest, for example. may be set for a specific area.

In S404, under the control of the system control unit 101, the area setting unit 302 sets an analysis area constituting a histogram in the defocus map corresponding to the captured image, based on the detection result of the detection unit 301 in S403. In this embodiment, since the face area of a person is detected by the detection unit 301, the area setting unit 302 defines an analysis area 1102 that is defined relative to the size of the face area 1100, as shown in FIG. 11, for example. is set as the analysis area that constitutes the histogram. In the example of FIG. 11, the analysis area 1102 has a size obtained by multiplying the size (FACE_SIZE_X, FACE_SIZE_Y) of the face area 1100 of the human subject 500 by a preset magnification rate (MAG_X, MAG_Y). The arrangement position of the analysis region 1102 is determined so that the upper end is a position moved from the upper end of the face region 1100 by FACE_MARGIN (1101) corresponding to the size of the face region 1100 in the Y-axis direction.

Here, the magnification rate for determining the size of the analysis region 1102 is preferably changed depending on the depth of field in the captured image. For example, if you take a portrait shot by focusing on the face of a human subject 500 under imaging conditions that result in a shallow depth of field, the boundary between the image of the subject's torso will be optically blurred, resulting in a shallow depth of field. tends to expand outwards than under deep imaging conditions. Therefore, the magnification ratio may be set such that the shallower the depth of field in the captured image, the larger the area used for extracting the defocus amount range.

Note that in this embodiment, in order to detect a person's face area, the face area 1100 and the analysis area 1102 are set as shown in FIG. 11. It is not limited. For example, in an embodiment in which a person's torso is also detected, an area that more closely matches the human subject 500 can be set based on the human body detection result. Furthermore, the size of the analysis area forming the histogram may be dynamically changed and set depending on the type of subject to be detected or the type of subject selected as the subject of interest. Here, the analysis area 1102 does not need to be set to exactly match the outline of the subject as shown in the figure.

In S405, under the control of the system control unit 101, the configuration unit 303 configures a histogram based on the defocus map for the analysis area that configures the histogram set in S404. As shown in FIG. 11, the histogram generated in this step is composed only of pixels included in the analysis area 1102 set for the human subject 500, and therefore appears as shown in FIG. 12, for example. In other words, in this step, a histogram can be constructed in an area more suited to the human subject 500 than in the case where a histogram is simply generated for the entire area of the defocus map. Therefore, even if there is another subject at a common subject distance with the human subject 500, the defocus amount range in which pixels related to the human subject 500 are distributed is determined regardless of the distance range of the other subject. can be obtained.

In S406, the system control unit 101 determines, based on the histogram of defocus amounts for the configured analysis region 1102, the range of defocus amounts (minimum defocus amount d1 and maximum defocus amount) that indicates the frequency of the number exceeding the threshold value 1201. Specify the quantity d2). The system control unit 101 stores information on the specified defocus amount range in the RAM 103.

In S407, under the control of the system control unit 101, the area extraction unit 304 extracts pixels (areas) indicating the defocus amount included in the range specified in S406 from the entire area of the defocus map. When pixels included in the range of the defocus amount specified based on the analysis region 1102 set according to the face region 1100 of the human subject 500 are extracted in this way, the result is as shown in FIG. 13. As shown in the figure, when only pixels included in the defocus amount range specified based on the analysis region 1102 are extracted, only images of the human subject 500 and the hedge 501 at positions included in the range are extracted. Ru.

In the digital camera 100 of this embodiment, a subject area is extracted based on a histogram related to defocus amount, unlike a histogram of brightness information or color information, so one subject is distributed over a certain distance range. It is possible to perform extraction that takes advantage of the Therefore, even if the defocus amount range is set based on the analysis region 1102, the hands and other parts of the body raised by the human subject 500 can also be extracted, as shown in FIG.

Note that the peaks in the histogram of the defocus map tend to have wider bases and larger fluctuations in the frequency of the defocus amount when images are acquired under imaging conditions with a shallower depth of field. be. Therefore, in the case of imaging conditions with a shallow depth of field, the system control unit 101 reduces the threshold value used for range extraction of the defocus amount, and sets the range so that a wider area can be extracted. It is preferable to do so. With this configuration, when an image is captured under an imaging condition where the face of the human subject 500 is focused and the depth of field is shallow, such as in portrait photography, the area is smaller than the face area. It is also possible to extract the body part that is in focus.

In S408, the region extraction unit 304 separates the image of the region extracted in S407 from the captured image under the control of the system control unit 101. That is, the region extraction unit 304 separates a partial image of the region shown in white in FIG. 13 from the captured image corresponding to the defocus map.

Note that in this embodiment, since an image is generated that focuses on the distance range of the human subject 500, the area of the hedge 501 existing in this range will also be separated. In the in-focus expression, the area of the hedge 501 may also be excluded. In this case, the region extracting unit 304 performs processing to exclude, for example, regions corresponding to the distance range of the human subject 500 that are not continuous with the face region 1100 that is the reference of the analysis region 1102, as shown in FIG. Only the area of the human subject 500 can be separated as shown in FIG.

In S409, under the control of the system control unit 101, the effect processing unit 305 performs blurring processing on the region of the captured image that was not separated in S408 by applying, for example, a low-pass filter. In addition, the image processing unit 107 generates a processed image with a shallower depth of field than that at the time of imaging by combining the image to which the blurring process has been applied and the image separated in S408, and stores the processed image in the recording medium 108. Record it and complete the main image processing process.

As described above, according to the image processing apparatus of this embodiment, the region of the subject of interest can be suitably extracted based on the distance distribution of the subject. More specifically, the image processing device determines the distance range of a desired object based on a captured image and distance information indicating a distance distribution of the object obtained corresponding to the captured image. Identify. Then, based on the specified distance range, the area of the subject included in the distance range of the desired subject can be extracted from the captured image, so the image processing device can generate images of various expressions using this. It can be carried out.

In this embodiment, an image with a shallow depth of field is generated in which only the distance range of the subject of interest is in focus and objects in other distance ranges are blurred. However, the present invention The implementation is not limited to this. That is, the image processing applied by the effect processing unit 305 to the separated region or the non-separated region is not limited to blurring processing, but includes various image processing such as brightness correction and contrast correction. It's fine. In other words, the present invention is applicable to any expression that can be realized by dividing the application of image processing depending on whether the object of interest is within the distance range or outside the range.

Furthermore, in the present embodiment, the digital camera 100 has an image sensor having a distance measurement function based on the imaging plane phase difference distance measurement method, acquires a group of images having a parallax relationship together with the captured image, and generates a defocus map. However, the implementation of the present invention is not limited to this. The defocus map related to the imaging range may not be acquired simultaneously with the captured image, but may be recorded in advance on the recording medium 108 or the like.

Furthermore, in this embodiment, the defocus map has been described as being generated based on a group of images having a parallax relationship, but if it corresponds to the captured image and it is possible to obtain the distance distribution of the subject in the imaging range, The method is not limited to this method. The defocus map generation method may be, for example, a DFD (Depth From Defocus) method in which the amount of defocus is derived from the correlation between two images with different focuses or aperture values. Alternatively, the distance distribution of the subject may be derived using information related to the distance distribution obtained from a distance measurement sensor module such as a TOF (Time of Flight) method. Regardless of the distance distribution obtained by any method, the present invention can be similarly implemented by limiting the area from which the histogram related to the distance distribution is obtained to the area related to the subject of interest.

Although the present embodiment has been described as using a defocus map that refers to the amount of defocus as the distance information indicating the distance distribution of the subject in the imaging range, the implementation of the present invention is not limited to this. In other words, the information indicating the distance distribution of the object is not determined by the amount of defocus, but by using, for example, the amount of shift (image shift amount) derived in deriving the amount of defocus, or the object distance derived based on the amount of defocus as a value. It may also be something that is stored. In the former case, there is no need to hold the conversion coefficient KX determined by the pixel pitch PY of the image sensor or the aperture angle of the center of gravity of the light beam passing through the pair of distance measuring pupils, and the amount of information can be reduced. In the latter case, since the numerical value is easy for the user to understand, for example, by presenting it during the processing process via the display unit 110, it is possible to suitably show the user what kind of expression the image will be generated. I can tell you.

Furthermore, in the image processing processing of the present embodiment, a mode has been described in which imaging is performed under the condition that the human subject 500 as the subject of interest is in focus, but the implementation of the present invention is not limited to this. That is, according to the present invention, even when an out-of-focus subject is set as the subject of interest, pixels existing within the distance range of the subject are identified by referring to the amount of defocus in the area of the subject. Therefore, the presence or absence of focus during imaging is irrelevant to the implementation of the present invention.

Note that from the viewpoint of reducing the processing load and the amount of data of the histogram itself, the amount of calculations related to the histogram configuration can be reduced by reducing the number of classes represented by the horizontal axis of the histogram.

For example, as shown in FIG. 16, consider a mode in which a plurality of subjects are distributed in the imaging range. FIG. 16 is a bird's-eye view showing the arrangement of subjects in the imaging range. In the figure, an imaging point 1600 indicates the position where the digital camera 100 is installed (the position of the imaging surface), and it is assumed that a lens with a focal length of 200 mm is employed as the optical system 104 to perform imaging. Here, it is assumed that the subject A1601 exists 2 m further back than the imaging point 1600, and the digital camera 100 performs imaging while focusing on the subject A1601. In addition, the object B1602 is 2.1m behind the imaging point 1600, the object C1603 is 1m behind the imaging point 1600, and the object D1604 is 100m behind the imaging point 1600. It is a scene where there is. In this arrangement, it is assumed that only the distance range of the focused subject A1601 is extracted from the defocus map. Note that the number of classes in the histogram is set to 32 classes (0 to 31), and when configuring the histogram, the value of the defocus amount is classified into one of the following categories, from the minimum value of 0 to the maximum value of 31. It is normalized as follows.

The imaging relationship between the subject side (object side) with the lens as the boundary and the image sensor (image plane side) is expressed as shown in FIG. If the focal length of the lens is f, then
1/Zo+1/So=1/f
1/Zdef+1/(So+def)=1/f
becomes. Here, Zo is the object side distance of the focused subject, Zdef is the object side distance of the defocused subject, So is the image side distance of the focused subject, and Sdef is the image side distance of the defocused subject. From these formulas, So=(Zo・f)/(Zo−f)
Sdef=(Zdef・f)/(Zdef−f)
can be derived, so the defocus amount def for the defocused subject is:
def=Sdef-So
It can be derived as

As shown in FIG. 17, when the relationship between the subject distance and the amount of deforce for each subject is derived, if you try to construct a histogram for the entire area of the defocus map corresponding to the captured image, at least from subject D to subject C. It is necessary to include the amount of defocus. More specifically, Chairman's normalization is required so that at least the defocus amount of -21.82 mm of the subject D1604 is classified as 0 in the histogram, and the defocus amount of 27.78 mm of the subject C is classified as 31 in the histogram. Therefore, the class width of the defocus amount in one class is (|−21.82|+|27.78|)/31=1.6 mm. Therefore, the defocus amount of the subject B1602 is smaller than the resolution of the defocus amount in the histogram, and the subject A1601 and the subject B1602 cannot be separated based on the histogram. Here, it is possible to separate subject A1601 and subject B1602 by narrowing the class width of the histogram and increasing the number of classes according to the obtained defocus distribution. A lower number of classes is advantageous in terms of the amount of calculation.

On the other hand, according to the present invention, since the histogram is configured by limiting it to the area related to the subject of interest, not only does the number of samples need to be small in the first place, but also the distances of subjects that are not included in the area can be excluded. . That is, since the histogram acquisition area is limited to the vicinity of the subject A1601, it is not necessary to include the defocus amount of the subject C1603 or the subject D1604 in the normalization, for example, so that the histogram class can be assigned near the defocus amount of 0 mm. . In other words, since a predetermined number of classes can be assigned to the distribution of defocus amounts included in the histogram acquisition area, it is possible to perform a more detailed analysis by narrowing the range of classes. Therefore, the subject A 1601 and the subject B 1602 can be separated with reference to the histogram of the defocus map without increasing the number of histogram classes. As a result, it is possible to suitably extract the distance range of only the subject A1601 from the defocus map.

[Modified example]
In the above-described embodiment, a mode has been described in which a face area of a human subject is detected from a captured image to identify a subject of interest. However, in such a mode, for example, there are cases in which the detection unit 301 detects a plurality of face areas. It is possible. At this time, in order to specify the distance range of one human subject so that only one human subject is in focus, the following processing may be performed.

For example, as shown in FIG. 15, there are a human subject 1500a and a human subject 1500b located behind it in the imaging range, and the center of the face area of each subject detected by the detection unit 301 is the face center. Assume that a face center 1501a and a face center 1501b are arranged. At this time, in order to extract only the area of the subject within the distance range of the human subject 1500a, an analysis area 1502 indicated by a broken line is created by multiplying the face area of the human subject 1500a by a preset magnification rate (MAG_X, MAG_Y). Suppose you set it. However, as shown in the figure, in a mode where the human subject 1500a and the human subject 1500b are close to each other, the image of the human subject 1500b may be included in the analysis area 1502. As a result, even if a histogram of the defocus amount is constructed for the analysis region 1502, there is a possibility that the distance range between the human subject 1500a and the human subject 1500b cannot be appropriately separated.

Therefore, when the detection unit 301 produces a plurality of face areas in this way and you want to extract only the distance range of the subject related to any of the face areas, the magnification rate is such that the analysis area that makes up the histogram is the area of the human subject 1500b. The adjustment may be made such that the adjustment area 1503 excludes the. Specifically, the area setting unit 302 sets the adjustment area 1503 by correcting the magnification rate according to the distance between the coordinates (FACE1_POS_X, FACE1_POS_Y) of the face center 1501a and the coordinates (FACE2_POS_X, FACE2_POS_Y) of the face center 1501b. do it.

The invention is not limited to the embodiments described above, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the following claims are hereby appended to disclose the scope of the invention.

[Other embodiments]
The present invention provides a system or device with a program that implements one or more functions of the embodiments described above via a network or a storage medium, and one or more processors in a computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100: Digital camera, 101: System control unit, 102: ROM, 103: RAM, 104: Optical system, 105: Imaging unit, 106: A/D conversion unit, 107: Image processing unit, 108: Recording medium, 109: Operation input unit, 110: Display unit, 300: Generation unit, 301: Detection unit, 302: Area setting unit, 303: Configuration unit, 304: Area extraction unit, 305: Effect processing unit

Claims (14)

acquisition means for acquiring a captured image of a subject in an imaging range, and distance information storing a value indicating a distance distribution in a depth direction of the subject in the imaging range for an area corresponding to the captured image;
determining means for determining a subject of interest for the captured image;
Setting means for setting an analysis region for analyzing distance distribution based on the position of the subject of interest in the captured image;
Identification means for identifying a value range in which objects included in the analysis area are distributed based on a frequency distribution of values included in the analysis area of the distance information;
Extracting means for extracting a subject region including at least the subject of interest by extracting a region of the distance information indicating information included in the value range;
An image processing device comprising: A generation unit that generates a processed image by applying different image processing to pixels included in the subject area extracted by the extraction unit and pixels not included in the subject area, of the captured image. The image processing apparatus according to claim 1, further comprising: an image processing apparatus; The generating means further extracts a region of the subject of interest from the subject region, and extracts a region of the subject of interest from among pixels included in the subject region of the captured image and a region of the subject of interest. The image processing apparatus according to claim 2, wherein different image processing is applied to pixels that are not included. 4. The specifying means specifies the value range based on a value that indicates a frequency exceeding a threshold value in a frequency distribution of values included in the analysis area of the distance information. The image processing device according to item 1 . The image processing apparatus according to claim 4 , wherein the specifying unit specifies the value range by reducing the frequency threshold as the depth of field of the captured image becomes shallower. 6. The image processing apparatus according to claim 4 , wherein the specifying means changes a class width in the frequency distribution according to a range of values included in the analysis area. The determining means determines the object of interest from among the detected predetermined objects by detecting a predetermined object included in the captured image,
7. The setting means sets an area of a predetermined size including an area where an image of the predetermined subject corresponding to the subject of interest appears as the analysis area. The image processing device described in . 8. The image processing apparatus according to claim 7 , wherein the setting unit sets the analysis area by enlarging the area of the predetermined size as the depth of field of the captured image becomes shallower. The setting means includes an image of the predetermined subject, which is different from the subject of interest, detected by the determining means, in an area of a predetermined size that includes an area where an image of the predetermined subject corresponding to the subject of interest appears. 9. The image processing apparatus according to claim 7 , wherein when the analysis area includes an area where an image of the different predetermined subject appears, the analysis area is transformed so as not to include an area where an image of the different predetermined subject appears. The distance information is two-dimensional information having the same pixel structure as the captured image,
The image according to any one of claims 1 to 9 , wherein the value stored in each pixel of the distance information is a defocus amount of an image of a subject existing at the pixel in the captured image. Processing equipment. The distance information is two-dimensional information having the same pixel structure as the captured image, and is information in which a value is stored based on a group of images captured in the imaging range from a plurality of different viewpoints,
Any one of claims 1 to 9 , wherein the value stored in each pixel of the distance information is a relative image shift amount in the group of images of a subject existing at the pixel in the captured image. The image processing device according to item 1. The distance information is two-dimensional information having the same pixel structure as the captured image,
The image processing according to any one of claims 1 to 9 , wherein the value stored in each pixel of the distance information is a distance in the depth direction of a subject existing at the pixel in the captured image. Device. an acquisition step of acquiring a captured image of a subject in an imaging range, and distance information storing a value indicating a distance distribution in a depth direction of the subject in the imaging range for an area corresponding to the captured image;
a determining step of determining a subject of interest for the captured image;
a setting step of setting an analysis region for analyzing distance distribution based on the position of the subject of interest in the captured image;
a specifying step of specifying a value range in which objects included in the analysis area are distributed based on a frequency distribution of values included in the analysis area of the distance information;
an extraction step of extracting a subject region including at least the subject of interest by extracting a region of the distance information indicating information included in the value range;
An image processing method comprising: A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 12 .

Images