JP2020181401A - Image processing system, image processing method and program - Google Patents

Abstract

To provide an image processing system that suitably extracts an area of a subject of interest based on a distance distribution of the subject.SOLUTION: An image processing system includes: acquisition means that acquires a captured image of a subject in an imaging range and distance information storing a value indicating a distance distribution in a depth direction of the subject in an imaging range for an area corresponding to the captured image; determination means that determines a subject of interest for the captured image; setting means that sets, based on a position of the subject of interest in the captured image, an analysis area for analyzing the distance distribution; identification means that identifies, based on information included in the analysis area of the distance information, a value range in which the subject included in the analysis area is distributed; and extraction means that extracts at least a subject area including the subject of interest by extracting an area of the distance information indicating information included in the value range.SELECTED DRAWING: Figure 4

Description

The present invention relates to an image processing apparatus, an image processing method and a program, and more particularly to a technique for suitably extracting a desired subject region in an image.

Pixels included in the captured image are classified according to the amount of defocus of the image of the pixel, and image processing is applied to the pixels belonging to a specific defocus range to show different expressions of depth of field. There is a technique for generating an image (Patent Document 1).

Japanese Unexamined Patent Publication No. 2008-015754

By the way, as one of the impressive image expressions, there is a so-called shallow depth of field expression in which only the distance range of the subject to be noticed is focused and the other areas are out of focus. Also in the technique of Patent Document 1, it is possible to generate an image having the same expression by performing the blur addition processing on the pixels other than the defocus range of the region including the desired subject.

However, in the method of specifying pixels by the amount of defocus and performing processing as in Patent Document 1, there is a possibility that an image having a desired expression may not be generated depending on the distribution of the subject. For example, in a distribution in which a subject different from the desired subject is arranged in a distance range widened in the depth direction including the subject distance of the desired subject, when the above-mentioned image processing is performed, Patent Document 1 With this method, these subjects cannot be separated and specified. That is, for example, when trying to generate an image with a shallow depth of field, not only a desired subject but also a distance range including the different subject appears in the image in a focused state, which is preferable. The expression may not be realized.

The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an image processing apparatus, an image processing method, and a program for appropriately extracting a region of a subject of interest based on the distance distribution of the subject. To do.

In order to achieve the above object, the image processing apparatus of the present invention shows a distance distribution of a captured image of a subject in the imaging range and a region corresponding to the captured image in the depth direction of the subject in the imaging range. Acquisition means for acquiring distance information storing values, determination means for determining the subject of interest for the captured image, and setting means for setting an analysis area for analyzing the distance distribution based on the position of the subject of interest in the captured image. And, based on the information included in the analysis area of the distance information, the specific means for specifying the value range in which the subject included in the analysis area is distributed and the area of the distance information indicating the information included in the value range are extracted. By doing so, it is characterized by having at least an extraction means for extracting a subject area including a subject of interest.

With such a configuration, according to the present invention, it is possible to suitably extract a region of a subject of interest based on the distance distribution of the subject.

A block diagram showing a functional configuration of a digital camera 100 according to an embodiment and a modification of the present invention. The figure which showed the structure of the image pickup element included in the image pickup unit 105 which concerns on embodiment and modification of this invention. A block diagram for explaining the configuration of the image processing unit 107 according to the embodiment and the modification of the present invention. A flowchart illustrating an image processing process executed by the digital camera 100 according to an embodiment and a modification of the present invention. The figure for demonstrating the subject distribution of the imaging range exemplified in the Embodiment of this invention. The figure for demonstrating the derivation of the defocus amount which concerns on embodiment and modification of this invention. Another diagram for explaining the derivation of the defocus amount according to the embodiment and the modified example of the present invention. The figure which illustrated the defocus map which concerns on embodiment of this invention Diagram illustrating a histogram configured over the entire defocus map of the prior art The figure for demonstrating the extraction of the region of a predetermined defocus range which concerns on a prior art The figure for demonstrating the analysis area which constitutes a histogram about the attention subject which concerns on embodiment of this invention. The figure which illustrated the histogram configured about the region corresponding to the attention subject which concerns on embodiment of this invention. The figure for demonstrating the region extracted based on the histogram which concerns on embodiment of this invention. Another figure for explaining the region extracted based on the histogram according to the embodiment of the present invention. The figure for demonstrating the region which constitutes a histogram about the attention subject which concerns on the modification of this invention. The figure for demonstrating the structure of the histogram with low processing load which concerns on embodiment of this invention. Another diagram for explaining the configuration of the histogram having a low processing load according to the embodiment of the present invention. Yet another diagram for explaining the configuration of a histogram having a low processing load according to the embodiment of the present invention.

[Embodiment]
Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate description is omitted.

In one embodiment described below, the present invention is applied to a digital camera, which is an example of an image processing device, capable of acquiring information (distance information) corresponding to a distance distribution in the depth direction (depth direction) of a subject in an imaging range. An applied example will be described. However, the present invention is applicable to any device capable of acquiring a captured image and distance information relating to the imaging range of the captured image.

Further, in the present specification, the "distance information" is assumed to be two-dimensional information indicating the defocus amount distribution of the image of each pixel of the captured image, and will be referred to as a "defocus map" below. However, the distance information may be any information corresponding to the distance distribution in the depth direction of the subject in the imaging range as described above, and may be, for example, a depth map showing the subject distance of each pixel, or the defocus amount. It may be two-dimensional information indicating the amount of image shift used for derivation. That is, the distance information is not limited to these as long as it is information showing a change according to the distance distribution in the depth direction.

<< Configuration of digital camera >>
FIG. 1 is a block diagram showing a functional configuration of the digital camera 100 according to the embodiment of the present invention.

The system control unit 101 is a control device that controls each block included in the digital camera 100, such as a CPU. The system control unit 101 controls the operation of each block included in the digital camera 100 by reading the operation program of each block included in the digital camera 100 from the ROM 102, expanding the program into the RAM 103, and executing the program.

The ROM 102 is a rewritable non-volatile memory such as a flash ROM. The ROM 102 stores, in addition to the operation program of each block included in the digital camera 100, parameters and the like necessary for the operation of each block. On the other hand, the RAM 103 is a rewritable volatile memory. The RAM 103 is used not only as a development area for an operation program of each block included in the digital camera 100, but also as a temporary storage area for intermediate data output by the operation of each block. In the present embodiment, the system control unit 101 and the image processing unit 107 described later use the RAM 103 as a work memory.

The optical system 104 forms a subject image on the imaging unit 105. The optical system 104 includes, for example, a fixed lens, a variable magnification lens that changes the focal length, a focus lens that adjusts the focus, and the like. The optical system 104 also includes a diaphragm, and the amount of light at the time of photographing is adjusted by adjusting the aperture diameter of the optical system with the diaphragm.

The image pickup unit 105 is, for example, an image pickup element such as a CCD image sensor or a CMOS image sensor. The image pickup unit 105 obtains an analog image signal by photoelectrically converting an optical image formed on the image pickup surface of the image pickup device by the optical system 104. The imaging unit 105 outputs the obtained analog image signal to the A / D conversion unit 106.

The A / D conversion unit 106 obtains digital image data (hereinafter, simply referred to as image data) by applying the A / D conversion process to the input analog image signal. The A / D conversion unit 106 outputs the obtained image data to the RAM 103 and stores it.

The image processing unit 107 performs various image processing on the image data stored in the RAM 103. Specifically, the image processing unit 107 performs image processing such as white balance adjustment, color interpolation, and processing related to reduction / enlargement. The image processing unit 107 records the image data after image processing on the recording medium 108.

Further, the image processing unit 107 includes a block that performs processing as shown in FIG. 3 in order to realize the function according to the present invention. Although the details will be described later, the image processing unit 107 includes a generation unit 300 that generates a defocus map (distance information) showing the distribution of the defocus amount for the imaging range in which the image was taken, and a specific subject in the imaging range. Includes a detection unit 301 that detects Further, the image processing unit 107 has an area setting unit 302 for setting an area for analysis for the subject of interest set based on the detection result by the detection unit 301, and a frequency map of the defocus amount for the set area ( It includes a component 303 that constitutes a histogram). Further, the image processing unit 107 includes an area extraction unit 304 that extracts the finally determined region of the subject of interest, and an effect processing unit 305 that performs image processing to make the captured image show a desired image expression. ..

The recording medium 108 is a recording device such as a memory card that may be detachably configured to be attached to and detached from the digital camera 100. The recording medium 108 records image data (images) to which image processing has been applied by the image processing unit 107, image signals (RAW images) A / D converted by the A / D conversion unit 106, and the like as recorded images. Will be done.

The display unit 110 is, for example, a display device such as an LCD, and presents various types of information in the digital camera 100. The display unit 110 functions as a digital viewfinder by displaying (through display) the A / D converted image data when the image pickup unit 105 is performing imaging.

The operation input unit 109 includes a user input interface such as a release switch, a setting button, and a mode setting dial, and when it detects an operation input made by the user, it outputs a control signal corresponding to the operation input to the system control unit 101. .. Further, in the embodiment in which the display unit 110 is provided with a touch panel sensor, the operation input unit 109 also functions as an interface for detecting a touch operation performed on the display unit 110.

These functional blocks included in the digital camera 100 are basically connected by a bus 111, and signals can be exchanged between the blocks via the bus 111.

In the present embodiment, it is assumed that the processing is realized by a circuit or a processor corresponding to each block included in the digital camera 100 as hardware. However, the implementation of the present invention is not limited to this, and the processing of each block may be realized by a program that performs the same processing as each block.

<< Configuration of imaging unit >>
Next, the detailed configuration of the image pickup device included in the image pickup unit 105 of the present embodiment will be described with reference to FIG.

In the image pickup device, as shown in FIG. 2A, a plurality of pixels 200 are regularly arranged in two dimensions. Specifically, the plurality of pixels 200 are arranged in a two-dimensional lattice pattern, for example. The arrangement of the pixels 200 is not limited to the grid-like arrangement, and other arrangements may be adopted.

In order to realize the distance measuring function of the imaging surface phase difference distance measuring method, each pixel 200 has a microlens 201 and a pair of photoelectric conversion units 202a and b (hereinafter, pupil division, respectively) as shown in FIG. 2 (b). It has pixels 202a and b). The pupil division pixels 202a and b are photoelectric conversion units having the same shape and the same size, in which the imaging surface has a rectangular shape with the y-axis direction as the longitudinal direction. In each pixel 200, the pupil division pixels 202a and b are arranged line-symmetrically with the perpendicular bisector along the y-axis direction of the microlens 201 as the axis of symmetry. The shape of the imaging surface of the pupil division pixels 202a and b is not limited to this, and may be a shape. Further, the arrangement mode of the pupil division pixels 202a and b is not limited to those arranged in parallel in the x direction, and other arrangement modes may be adopted.

With such a configuration, the image pickup unit 105 of the present embodiment has the A image related to the image signal output from the pupil division pixel 202a included in all the pixels of the image pickup element, and the image output from the pupil division pixel 202b in the same manner. The B image related to the signal is output. The A image and the B image have a parallax relationship according to the distance from the in-focus position.

More specifically, in each pixel 200, the pupil division pixels 202a and b are configured to receive different light fluxes among the light fluxes received through the microlens 201, so that the exit pupils of the optical system 104 are different. The optical image of the luminous flux that has passed through the region is photoelectrically converted. That is, since the A image and the B image are generated for the luminous flux passing through the different regions (pupil division region) of the exit pupil, the subject is placed at the shooting position deviated by the difference in the center of gravity position of the pupil division region. The relationship is that the images are taken, and parallax will occur. In other words, the A image and the B image correspond to image groups obtained by imaging the imaging range from different viewpoints.

In this embodiment, the image sensor of the image pickup unit 105 is configured as shown in FIG. 2 to acquire the A image and the B image used for deriving the distance distribution of the subject in the imaging range. , The method of acquiring the A image and the B image is not limited to this. For example, the image group imaged by a plurality of image pickup devices installed at a distance of the baseline length may be an A image or a B image, or one image pickup device having a plurality of optical systems and an image pickup unit (so-called so-called). The image group acquired by the binocular camera or the like may be used as an A image and a B image, respectively.

<< Derivation of defocus amount >>
The generation unit 300 provides a defocus map having a pixel structure corresponding to the captured image, including the defocus amount of the image of each subject included in the captured image, as information indicating the distance distribution of the subject in the depth direction of the imaging range. Generate. When the defocus amount is derived based on the image group (A image and B image) having the parallax acquired as in the present embodiment, it may be performed as follows, for example.

Derivation of the defocus amount includes, for example, a process of dividing the image (A image and B image) 700 into the minute block 701 shown by the broken line as shown in FIG. The minute block 701 may be set for a region of a predetermined size centered on the pixel, for example, when each pixel of the target A image is a pixel of interest. In the following description, the minute block 701 will be described as being set as a square region of m × m pixels centered on the pixel of interest, but the shape and size of the minute block 701 may be any. Further, the minute block 701 is set for each pixel of interest, and the minute blocks 701 may overlap between different pixels of interest. Further, the generation unit 300 may derive the defocus amount after applying a process of applying a bandpass filter to the A image and the B image for detecting the phase difference in advance.

For example, when a minute block 701 is set for each pixel of the A image and the B image, a correlation calculation process is performed for each pixel (pixel of interest) between the two images, and the image shift included in the minute block 701 corresponding to the pixel is shifted. The amount (image shift amount) is derived. When the number of data (number of pixels) of the (pair) microblocks 701 defined for the pixel of interest at the same position in the A image and the B image is m, the pixel data of the pair of microblocks 701 is E (1). ) To E (m) and F (1) to F (m). In this case, in the correlation calculation, assuming that the shift amount (of the data) is k (integer) [pixel], the correlation amount C (k) is C (k) = Σ | E (n) -F (n + k) |
Can be derived with. Here, the Σ operation is performed on n, and n and n + k are limited to the range of 1 to m. The shift amount k is a relative shift amount in units of the detection pitch of the pair of image data. In this way, when the correlation amount is derived for the pair of pupil-divided images (pair of minute blocks 701) related to one pixel of interest, the shift amount k and the correlation amount C (k) are discretely shown in the graph of FIG. The relationship is as shown. At this time, since the correlation amount C (k) is minimized in the image shift amount having the highest correlation, the minimum value C (x) is given to the continuous correlation amount by using the following three-point interpolation method. The quantity x is derived.
x = kj + D / SLOP
C (x) = C (kj)-| D |
D = {C (kj-1) -C (kj + 1)} / 2
SLOP = MAX {C (kj + 1) -C (kj), C (kj-1) -C (kj)}
Here, kj is a shift amount k that minimizes the discrete correlation amount C (k). The amount of shift x obtained in this way is included in the distance information as the amount of image shift in one pixel of interest. The unit of the image shift amount x is also assumed to be [pixel].

Therefore, the defocus amount DEF in each pixel of interest is determined by using the image shift amount x.
DEF = KX ・ PY ・ x
Can be derived with. Here, PY is the pixel pitch of the image sensor (distance between pixels constituting the image sensor. Unit [mm / pixel]), and KX is the size of the opening angle of the center of gravity of the luminous flux passing through the pair of distance measuring pupils. It is a conversion coefficient determined by the value. Since the size of the opening angle of the center of gravity of the luminous flux passing through the pair of ranging pupils changes according to the size (F value) of the aperture opening of the lens, it is determined according to the information of the setting at the time of imaging. Shall be.

In this way, the generation unit 300 can derive the defocus amount of the subject in each pixel of the captured image by repeatedly calculating while shifting the position of the pixel of interest one pixel at a time. When the defocus amount of each pixel is derived, the generation unit 300 generates a defocus map, which is two-dimensional information having the same structure as the captured image, as distance information, using this as the pixel value. That is, since the defocus amount is a value that changes according to the amount of deviation of the position in the depth direction from the in-focus subject distance in the captured image, the defocus map is the distance in the depth direction of the subject at the time of imaging. This is information equivalent to the distribution.

《Image processing》
Hereinafter, in the digital camera 100 of the present embodiment, based on the captured image and the defocus map, an image is generated in which the depth of field is narrower than that at the time of imaging, in which only the distance range of the subject of interest is focused. The image processing process will be described with reference to the flowchart of FIG. The processing corresponding to the flowchart can be realized by the system control unit 101 reading, for example, the corresponding processing program stored in the ROM 102, developing it in the RAM 103, and executing it. This image processing process will be described as being started when, for example, the digital camera 100 is set to a shooting mode for generating an image whose expression is changed after imaging, and an operation input related to an imaging instruction is detected.

In S401, the imaging unit 105 images the subject under the control of the system control unit 101. When an analog image signal is obtained by imaging, the A / D conversion unit 106 performs A / D conversion and stores it in the RAM 103. For the image data stored in the RAM 103 in this image processing process, the A image captured by the pupil division pixel 202a and the B image captured by the pupil division pixel 202b are added together with the A image and the B image. It is a captured image composed of (an image corresponding to a state in which the pupil is not divided).

In the following description, it is assumed that the subjects are distributed in the imaging range as shown in FIG. As shown, the imaging range includes the human subject 500 with the hand raised, the hedge 501 extending from a position closer to the digital camera 100 than the human subject 500 to a position farther than the human subject 500, and further than the hedge 501. A distant tree 502 is included. Further, it is assumed that the captured image is captured with an imaging setting that focuses on the human subject 500. In the image processing process of the present embodiment, the person subject 500 is selected as the subject of interest in the step described later, the subject of interest is focused only in the distance range, and the subject in the other distance range is blurred. , It will be described as generating an image having a shallow depth of field. Further, for the sake of simplicity, the distance distribution in the depth direction will be described assuming that the ground distributed in the imaging range is excluded and only the human subject 500, the hedge 501, and the tree 502 are derived.

In S402, the generation unit 300 generates a defocus map corresponding to the captured image based on the A image and the B image stored in the RAM 103 in S401 under the control of the system control unit 101. FIG. 8 illustrates the defocus map by converting the amount of defocus, which is a pixel value, into a grayscale value and visualizing it. In order to make the explanation easier to understand, in the example of FIG. 8, the closer the subject distance is, the closer the value is to white (higher pixel value), and the farther away the pixel is, the closer the value is to black (low pixel value). It is assumed that it has been converted.

By the way, when the human subject 500, the hedge 501 and the tree 502 are arranged in the distribution as described above, a histogram (frequency distribution) based on the defocus amount is configured for the defocus map as in the technique of Patent Document 1. In this case, the distribution is as shown in FIG. In FIG. 9, the peak of the histogram shown by 901 shows the distribution of the defocus amount in the pixel group related to the tree 502 arranged in the distance. On the other hand, the peak of the histogram shown by 900 shows the distribution of the defocus amount in the pixel group related to the human subject 500 and the hedge 501.

Here, even if the pixels in the defocus range shown by d1 to d2, which indicate the frequency exceeding a predetermined threshold value 902 as described in Cited Document 1, are extracted, as shown in FIG. 10, the human subject 500 And the area related to the hedge 501 (white area in the figure) is only extracted. Therefore, even if processing is performed based on such an extraction result, it is possible to obtain an image with a shallow depth of field that focuses only on the distance range of the person subject 500 to be the subject of interest as in the present embodiment. Can't. In other words, in the embodiment in which the hedge 501 has a shape that extends widely in the depth direction and is arranged including the range of the subject distance of the person subject 500, only the distance range of the person subject 500 is included in Reference 1. Separation and extraction cannot be performed by the described technique. Therefore, even if the blurring process is applied to the outside of the extracted area, the depth of field cannot be narrowed appropriately because the hedge 501 is focused on the distance range in the depth direction, and the image of the desired expression is obtained. It does not become.

Therefore, in the image processing process according to the present invention, when specifying the value range of the defocus amount to be extracted based on the histogram of the defocus amount, the value range of the defocus amount to be extracted is not the histogram for the entire area of the captured image, but the subject to be the focus Use a histogram for the region. The details of the processing will be further described below.

In S403, the detection unit 301 performs a process of detecting a predetermined subject included in the captured image under the control of the system control unit 101. As described above, since it is necessary to specify the region of the human subject 500 in the present embodiment, the detection unit 301 detects the face region of the person included in the captured image, and the position and size of the face region (FACE_SIZE_X,, FACE_SIZE_Y) information is acquired. In the present embodiment, the detection unit 301 will be described as detecting the face region of a person, but the embodiment of the present invention is not limited to this, and for example, the body of a person, a passenger car, a pet, a plant, etc. are the detection targets. It may be set as.

In the present embodiment, the detection unit 301 detects whether or not a predetermined type of subject is included in the captured image, and sets the subject of interest based on the detection result. It will be easily understood that the practice of the invention is not limited to this. That is, the method of setting the subject of interest from the captured image does not have to be based on image analysis, and may be set based on, for example, a user's selection operation of the subject of interest, or in the captured image. It may be defined as being set for a specific area of.

In S404, the area setting unit 302 sets the analysis area constituting the histogram in the defocus map corresponding to the captured image based on the detection result of the detection unit 301 in S403 under the control of the system control unit 101. In the present embodiment, since the face region of the person is detected by the detection unit 301, the region setting unit 302 uses the analysis region 1102, which is determined relative to the size of the face region 1100, for example, as shown in FIG. Is set as the analysis area that constitutes the histogram. In the example of FIG. 11, the analysis area 1102 has a size obtained by multiplying the size (FACE_SIZE_X, FACE_SIZE_Y) of the face area 1100 of the human subject 500 by a preset enlargement ratio (MAG_X, MAG_Y). The arrangement position of the analysis area 1102 is determined so that the position moved from the upper end of the face area 1100 by FACE_MARGIN (1101) according to the size of the face area 1100 in the Y-axis direction is the upper end.

Here, the enlargement ratio for determining the size of the analysis region 1102 is preferably changed according to the depth of field in the captured image. For example, when portrait photography is performed by focusing on the face of a person subject 500 under an imaging condition in which the depth of field becomes shallow, the boundary portion of the image of the body of the subject is optically blurred and the depth of field is deep. Tends to expand outwards than under deep imaging conditions. Therefore, the enlargement ratio may be set so that the shallower the depth of field in the captured image, the larger the region used for range extraction of the defocus amount.

In the present embodiment, in order to detect the face region of a person, the face region 1100 and the analysis region 1102 will be described as being set in the manner shown in FIG. 11, but the analysis region constituting the histogram is set accordingly. It is not limited. For example, in the embodiment of detecting the body of a person, a region more suitable for the person subject 500 can be set based on the detection result of the human body. Further, the analysis area constituting the histogram may be dynamically resized and set according to the type of the subject to be detected or the type of the subject selected as the subject of interest. Here, as shown in the figure, the analysis area 1102 does not need to be set so as to exactly match the contour of the subject.

In S405, under the control of the system control unit 101, the configuration unit 303 configures a histogram based on the defocus map for the analysis region that constitutes the histogram set in S404. As shown in FIG. 11, the histogram generated in this step is composed of only the pixels included in the analysis area 1102 set for the human subject 500, and therefore appears as shown in FIG. 12, for example. In other words, in this step, the histogram can be configured in an area more suitable for the human subject 500 than in the case of simply generating a histogram for the entire area of the defocus map. Therefore, even if there is another subject existing at a subject distance common to the human subject 500, the range of the defocus amount in which the pixels related to the human subject 500 are distributed is set regardless of the distance range of the other subject. Can be obtained.

In S406, the system control unit 101 determines the range of the defocus amount (minimum defocus amount d1 and maximum defocus) indicating the frequency of the number exceeding the threshold value 1201 based on the histogram of the defocus amount for the configured analysis region 1102. The quantity d2) is specified. The system control unit 101 stores the information of the specified defocus amount range in the RAM 103.

In S407, the area extraction unit 304 extracts pixels (areas) indicating the amount of defocus included in the range specified in S406 from the entire area of the defocus map under the control of the system control unit 101. In this way, when the pixels included in the range of the defocus amount specified based on the analysis area 1102 set in accordance with the face area 1100 of the human subject 500 are extracted, the pixels are as shown in FIG. As shown in the figure, when only the pixels included in the defocus amount range specified based on the analysis area 1102 are extracted, only the image of the human subject 500 and the position of the hedge 501 included in the range is extracted. To.

In the digital camera 100 of the present embodiment, unlike the histogram of the luminance information and the color information, the subject area is extracted based on the histogram related to the defocus amount, so that one subject is distributed in a certain distance range. It is possible to perform extraction that makes the best use of. Therefore, even if the range of the defocus amount is set based on the analysis area 1102, as shown in FIG. 13, other parts of the hand and the torso mentioned by the human subject 500 can also be extracted.

As for the peaks of the histogram of the defocus map, the more the captured image is acquired under the imaging conditions with a shallow depth of field, the wider the base of the peaks and the greater the fluctuation in the frequency of the defocus amount. is there. Therefore, in the case of an imaging condition in which the depth of field is shallow, the system control unit 101 reduces the threshold value used for extracting the range of the defocus amount, and sets the range so that a wider area can be extracted. It is preferable to do so. With this configuration, when imaging is performed under imaging conditions where the depth of field is shallow and the face portion of the human subject 500 is in focus, such as portrait photography, the image is deduced compared to the face area. It is possible to extract even the body part that seems to be in focus.

In S408, the region extraction unit 304 separates the image of the region extracted in S407 of the captured images under the control of the system control unit 101. That is, the region extraction unit 304 separates the partial image of the region shown in white in FIG. 13 from the captured image corresponding to the defocus map.

In the present embodiment, since the image of the expression that focuses on the distance range of the human subject 500 is generated, the area of the hedge 501 existing in the range is also described as being separated, but only for the human subject 500. In the in-focus expression, the area of the hedge 501 may also be excluded. In this case, if the area extraction unit 304 processes, for example, from the area corresponding to the distance range of the human subject 500, the area that is not continuous with the face area 1100 that is the reference of the analysis area 1102, FIG. It is possible to separate only the area of the human subject 500 as described above.

In S409, under the control of the system control unit 101, the effect processing unit 305 performs blurring processing by applying, for example, a low-pass filter to the region of the captured image that was not separated in S408. Further, the image processing unit 107 generates a processed image having a shallower depth of field than at the time of imaging by synthesizing the image to which the blurring process is applied and the image separated in S408, and uses the recording medium 108 as the image processing unit 107. Record and complete this image processing process.

As described above, according to the image processing apparatus of the present embodiment, the region of the subject of interest can be suitably extracted based on the distance distribution of the subject. More specifically, the image processing apparatus limits the captured image to a desired subject area based on the distance information indicating the distance distribution of the captured image and the subject obtained corresponding to the captured image, and limits the distance range of the subject. To identify. Then, based on the specified distance range, the area of the subject included in the desired distance range of the subject can be extracted from the captured image, so that the image processing apparatus can generate images of various expressions using this. It can be carried out.

In the present embodiment, it has been described that an image having a shallow depth of field is generated in which the subject is focused only in the distance range of the subject of interest and the subjects in the other distance ranges are blurred. The implementation of is not limited to this. That is, the image processing applied by the effect processing unit 305 to the separated area or the non-separated area is not limited to the blurring process, but includes various image processes such as brightness correction and contrast correction. You can. In other words, the present invention can be applied as long as it is an expression realized by dividing the application of image processing according to whether the subject of interest is within the distance range or outside the range.

Further, in the present embodiment, the digital camera 100 has an image pickup element having an image pickup surface phase difference ranging function, acquires an image group having a parallax with the captured image, and generates a defocus map. However, the implementation of the present invention is not limited to this. The defocus map related to the imaging range is not acquired at the same time as the captured image, but may be previously recorded on a recording medium 108 or the like.

Further, in the present embodiment, the defocus map has been described as being generated based on a group of images having a parallax relationship, but if it corresponds to the captured image and the distance distribution of the subject in the imaging range can be acquired, it is possible. It is not limited to this method. The defocus map generation method may be, for example, a DFD (Depth From Defocus) method in which the amount of defocus is derived from the correlation between two images having different focus and aperture values. Alternatively, the distance distribution of the subject may be derived by using the information related to the distance distribution obtained from the distance measurement sensor module such as the TOF (Time of Flight) method. Regardless of the distance distribution obtained by any of the methods, the present invention can be similarly realized by limiting the region for acquiring the histogram related to the distance distribution to the region related to the subject of interest.

In the present embodiment, the defocus map that refers to the defocus amount is used as the distance information indicating the distance distribution of the subject in the imaging range, but the embodiment of the present invention is not limited to this. That is, the information indicating the distance distribution of the subject is not the defocus amount, but the shift amount (image shift amount) derived in deriving the defocus amount or the subject distance derived based on the defocus amount as a value. It may be something to store. In the former case, it is not necessary to hold the conversion coefficient KX determined by the pixel pitch PY of the image sensor and the opening angle of the center of gravity of the light flux passing through the pair of ranging pupils, and the amount of information can be reduced. Further, in the latter case, since the numerical value is easy for the user to grasp, for example, by presenting the image in the process of processing via the display unit 110, it is suitable for the user to know what kind of expression the image is generated. I can tell.

Further, in the image processing process of the present embodiment, the mode in which the image is taken under the condition that the person subject 500 as the subject of interest is in focus has been described, but the embodiment of the present invention is not limited to this. That is, according to the present invention, even when a subject in an out-of-focus state is set as a subject of interest, pixels existing in the distance range of the subject are specified by referring to the amount of defocus in the area of the subject. Therefore, the presence or absence of focusing at the time of imaging is irrelevant to the practice of the present invention.

From the viewpoint of reducing the processing load and the amount of data in the histogram itself, it is possible to reduce the amount of calculation related to the histogram configuration by reducing the number of classes represented by the horizontal axis of the histogram.

For example, as shown in FIG. 16, a mode in which a plurality of subjects are distributed in the imaging range will be examined. FIG. 16 is a bird's-eye view showing the arrangement of the subject in the imaging range. In the figure, the imaging point 1600 indicates the position where the digital camera 100 is installed (the position of the imaging surface), and it is assumed that a lens having a focal length of 200 mm is adopted as the optical system 104 to perform imaging. Here, it is assumed that the subject A1601 exists 2 m behind the imaging point 1600, and the digital camera 100 performs imaging in a state where the subject A1601 is in focus. Subject B1602 is distributed 2.1 m behind the imaging point 1600, subject C1603 is distributed 1 m behind the imaging point 1600, and subject D1604 is distributed 100 m behind the imaging point 1600, and the subject is widely distributed in the depth direction. It is a scene where. In such an arrangement mode, it is assumed that only the distance range of the focused subject A1601 is extracted from the defocus map. The number of classes of the histogram is set to 32 classes (0 to 31), and when the histogram is configured, the defocus amount value is classified into any of 0 to 31 for the minimum value and 31 for the maximum value. Is normalized as.

The imaging relationship between the subject side (object side) and the image sensor (image plane side) with the lens as the boundary is shown as shown in FIG. Let f be the focal length of the lens.
1 / Zoo + 1 / So = 1 / f
1 / Zdef + 1 / (So + def) = 1 / f
Will be. Here, Z is the object-side distance of the in-focus subject, Zdef is the object-side distance of the defocused subject, So is the image-plane-side distance of the in-focus subject, and Sdef is the image-plane-side distance of the defocused subject. From these equations, So = (Zo · f) / (Zo-f)
Sdef = (Zdef · f) / (Zdef−f)
Since the defocus amount def related to the defocus subject can be derived,
def = Sdef-So
Can be derived as.

As shown in FIG. 17, when the relationship between the subject distance of each subject and the amount of deforce is derived, when trying to construct a histogram for the entire defocus map corresponding to the captured image, at least from subject D to subject C Defocus amount should be included. More specifically, it is necessary to normalize the chairman so that at least the defocus amount of subject D1604 of 21.82 mm is set to 0 in the histogram and the defocus amount of subject C of 27.78 mm is classified into 31 of the histogram. Therefore, the class width of the defocus amount in one class is (| -21.82 | + | 27.78 |) / 31 = 1.6 mm. Therefore, the defocus amount of the subject B1602 is smaller than the resolution of the defocus amount in the histogram, and the subject A1601 and the subject B1602 cannot be separated based on the histogram. Here, for example, if the class width of the histogram is narrowed and the number of classes is increased according to the obtained defocus distribution, it is possible to separate the subject A1601 and the subject B1602, but as described above, the histogram The lower the number of classes, the more advantageous in terms of the amount of calculation.

On the other hand, according to the present invention, since the histogram configuration is limited to the region related to the subject of interest, not only the number of samples can be reduced in the first place, but also the distance of the subject not included in the region can be excluded. .. That is, since the acquisition area of the histogram is limited to the vicinity of the subject A1601, for example, it is not necessary to include the defocus amount of the subject C1603 and the subject D1604 in the normalization, so that the histogram class can be assigned in the vicinity of the defocus amount of 0 mm. .. That is, since a predetermined number of classes can be assigned to the distribution of the defocus amount included in the acquisition area of the histogram, it is possible to set the class width to be narrow and perform more detailed analysis. Therefore, the subject A1601 and the subject B1602 can be separated with reference to the histogram of the defocus map without increasing the class number of the histogram. As a result, it is possible to suitably extract the distance range of only the subject A1601 from the defocus map.

[Modification example]
In the above-described embodiment, the aspect of detecting the face region of the human subject from the captured image to identify the subject of interest has been described, but in such an embodiment, for example, there is a case where the detection unit 301 detects a plurality of face regions. Can be done. At this time, in order to specify the distance range of the subject so that only one person subject is in focus, the following processing may be performed.

For example, as shown in FIG. 15, a human subject 1500a and a human subject 1500b located behind the human subject 1500a exist in the imaging range, and the center of the face region of each subject detected by the detection unit 301 is the face center. It is assumed that the 1501a and the face center 1501b are arranged. At this time, in order to extract only the area of the subject in the distance range of the human subject 1500a, the analysis area 1502 shown by the broken line is obtained by multiplying the face area of the human subject 1500a by a preset enlargement ratio (MAG_X, MAG_Y). Suppose you set it. However, as shown in the figure, in the embodiment in which the human subject 1500a and the human subject 1500b are in close proximity to each other, the analysis area 1502 may include an image of the human subject 1500b. As a result, even if a histogram of the defocus amount is configured for the analysis region 1502, it may not be possible to suitably separate the distance range between the human subject 1500a and the human subject 1500b.

Therefore, when the detection unit 301 produces a plurality of face regions in this way and wants to extract only the distance range of the subject related to any of the face regions, the enlargement ratio is such that the analysis region constituting the histogram is the region of the human subject 1500b. The adjustment area 1503 may be adjusted so as to exclude the above. Specifically, the area setting unit 302 sets the adjustment area 1503 by correcting the enlargement ratio according to the distance between the coordinates of the face center 1501a (FACE1_POS_X, FACE1_POS_Y) and the coordinates of the face center 1501b (FACE2_POS_X, FACE2_POS_Y). do it.

The invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100: Digital camera, 101: System control unit, 102: ROM, 103: RAM, 104: Optical system, 105: Imaging unit, 106: A / D conversion unit, 107: Image processing unit, 108: Recording medium, 109: Operation input unit, 110: Display unit, 300: Generation unit, 301: Detection unit, 302: Area setting unit, 303: Configuration unit, 304: Area extraction unit, 305: Effect processing unit

Claims (15)

An acquisition means for acquiring an image captured by capturing a subject in the imaging range and distance information storing a value indicating a distance distribution in the depth direction of the subject in the imaging range for a region corresponding to the captured image.
A determination means for determining a subject of interest for the captured image, and
A setting means for setting an analysis region for analyzing a distance distribution based on the position of the subject of interest in the captured image, and
A specific means for specifying a value range in which subjects included in the analysis area are distributed based on the information included in the analysis area in the distance information, and
An extraction means for extracting at least a subject area including the attention subject by extracting a region of the distance information indicating information included in the value range.
An image processing device characterized by having. A generation means for generating a processed image by applying different image processing to the pixels included in the subject area extracted by the extraction means and the pixels not included in the subject area in the captured image. The image processing apparatus according to claim 1, further comprising. The generation means further extracts the region of the subject of interest from the subject region, and among the pixels included in the subject region of the captured image, the pixels included in the region of the subject of interest and the region of the subject of interest The image processing apparatus according to claim 2, wherein different image processing is applied to pixels that are not included. The image processing apparatus according to any one of claims 1 to 3, wherein the specifying means specifies the value range based on the frequency distribution of values included in the analysis region of the distance information. .. The image processing according to claim 4, wherein the specific means specifies the value range based on a value indicating a frequency exceeding a threshold value in a frequency distribution of values included in the analysis region of the distance information. apparatus. The image processing apparatus according to claim 5, wherein the specifying means reduces the threshold value of the frequency as the depth of field of the captured image becomes shallower to specify the value range. The image processing apparatus according to claim 5 or 6, wherein the specific means changes the class width in the frequency distribution according to a range of values included in the analysis region. By detecting a predetermined subject included in the captured image, the determination means determines the subject of interest from the detected predetermined subjects.
Any one of claims 1 to 7, wherein the setting means sets a region having a predetermined size including a region in which an image of the predetermined subject corresponding to the subject of interest appears as the analysis region. The image processing apparatus according to the section. The image processing apparatus according to claim 8, wherein the setting means expands a region having a predetermined size to set the analysis region as the depth of field of the captured image becomes shallower. The setting means has an image of the predetermined subject different from the attention subject detected by the determination means in a region of a predetermined size including a region in which an image of the predetermined subject corresponding to the attention subject appears. The image processing apparatus according to claim 8 or 9, wherein when the region in which the image appears is included, the analysis region is deformed so as not to include the region in which the image of a different predetermined subject appears. The distance information is two-dimensional information having the same pixel structure as the captured image.
The image according to any one of claims 1 to 10, wherein the value stored in each pixel of the distance information is the defocus amount of the image of the subject existing in the pixel in the captured image. Processing equipment. The distance information is two-dimensional information having the same pixel structure as the captured image, and is information in which a value is stored based on an image group in which the imaging range is imaged from a plurality of different viewpoints.
Any of claims 1 to 10, wherein the value stored in each pixel of the distance information is the relative image shift amount of the subject existing in the pixel in the captured image in the image group. The image processing apparatus according to item 1. The distance information is two-dimensional information having the same pixel structure as the captured image.
The image processing according to any one of claims 1 to 10, wherein the value stored in each pixel of the distance information is the distance in the depth direction of the subject existing in the pixel in the captured image. apparatus. An acquisition step of acquiring an image captured by capturing a subject in the imaging range and distance information storing a value indicating a distance distribution in the depth direction of the subject in the imaging range for a region corresponding to the captured image.
A determination process for determining a subject of interest for the captured image, and
A setting step of setting an analysis region for analyzing a distance distribution based on the position of the subject of interest in the captured image, and
A specific step of specifying a value range in which subjects included in the analysis area are distributed based on the information included in the analysis area of the distance information, and
An extraction step of extracting at least a subject area including the attention subject by extracting the area of the distance information indicating the information included in the value range.
An image processing method characterized by having. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 13.