JP5673454B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

2. Description of the Related Art Conventionally, an imaging apparatus that determines a blur amount of a plurality of regions obtained by dividing each image based on a plurality of images having substantially the same composition and different focus positions, and generates an image subjected to a blur process according to the blur amount. It is disclosed (for example, see Patent Document 1).
For example, after capturing a subject in-focus image in which the subject is in focus and a background in-focus image in which the focus position is shifted backward, a predetermined edge detection filter is applied to each image, and pixels at corresponding positions are captured. An evaluation value is calculated. Then, based on the calculated evaluation value, a background process is performed to identify a background area and perform a blurring process for enhancing the blur of the background area, or a process for identifying a subject area and extracting the subject area.

JP 2011-10194 A

  However, an optical image of an object in the foreground or background that is not in focus is blurred and the optical image is spread. In particular, for example, when the intensity of light emitted from a high-luminance object such as a fluorescent lamp and incident on the image sensor is strong, depending on the resolving power of the imaging lens due to diffraction limits and aberrations, The contour portion of the spread optical image appears clearly. As a result, there is a problem in that an edge of the optical image of the object is erroneously detected in the focused subject image, and the subject region is specified although it is the background region.

  Accordingly, an object of the present invention is to provide an image processing apparatus, an image processing method, and a program capable of appropriately specifying a subject area in a subject focused image.

In order to solve the above problems, an image processing apparatus according to the present invention provides:
A first acquisition unit that acquires a subject focused image captured by the imaging unit while focusing on a predetermined subject in the background, and a predetermined imaging condition that is different from the imaging condition of the subject focused image by the imaging unit Second acquisition means for acquiring one image that is captured and has a composition substantially equal to the subject focused image; and the subject focused image based on image information of each of the subject focused image and the one image An estimation means for estimating a contour expansion region corresponding to an object outside the depth of field and having a contour portion enlarged from the contour portion of the corresponding region of the object in the one image, and the estimation And specifying means for specifying a subject area including the predetermined subject in the subject in-focus image based on the estimation result of the contour enlargement region by the means.

The image processing method according to the present invention includes:
An image processing method using an image processing device, a process of obtaining a subject in-focus image captured by an imaging device while focusing on a predetermined subject in a background, and a process of acquiring the subject in-focus image by the imaging device Based on a process of obtaining one image that is imaged under a predetermined imaging condition different from the imaging condition and has a composition that is substantially equal to the subject focused image, and image information of each of the subject focused image and the one image In the in-focus image of the subject, an enlarged contour region corresponding to an object that exists outside the depth of field and whose contour portion is larger than the contour portion of the corresponding region of the object in the one image is estimated. And a process of identifying a subject area in which the predetermined subject is included in the subject focused image based on the estimation result of the contour enlargement region.

The program according to the present invention is
The first acquisition means for acquiring a subject in-focus image captured by the image capturing apparatus by focusing a predetermined subject in the background on the computer of the image processing apparatus, and the image capturing condition of the object in-focus image by the image capturing apparatus Based on image information of each of the second acquisition means for acquiring one image captured under different predetermined imaging conditions and having a composition substantially equal to the subject focused image, the subject focused image and the one image, Estimating a contour expansion region corresponding to an object existing outside the depth of field in the in-focus image of the subject and having a contour portion enlarged from the contour portion of the corresponding region of the object in the one image. And an identifying means for identifying a subject area in which the predetermined subject is included in the subject in-focus image based on the estimation result of the contour enlargement area by the estimating means. It is characterized by a door.

  According to the present invention, it is possible to appropriately specify a subject area in a subject focused image.

It is a block diagram which shows schematic structure of the imaging device of one Embodiment to which this invention is applied. 3 is a flowchart illustrating an example of an operation related to an image generation process performed by the imaging apparatus in FIG. 1. 3 is a flowchart illustrating an example of an operation related to a luminance change region estimation process in the image generation process of FIG. 2. It is a figure which shows typically an example of the image which concerns on the image generation process of FIG. It is a figure which shows typically an example of the image which concerns on the image generation process of FIG. It is a figure for demonstrating the image generation process of FIG.

  Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.

  The imaging apparatus 100 according to the present embodiment is imaged under a predetermined imaging condition that is different from the imaging conditions of the subject focused image P1 and the subject focused image P1 captured by focusing on the predetermined subject S in the background. Based on each image information of one image (subject unfocused image P2) having a composition substantially equal to the focused image P1, corresponding to an object outside the depth of field in the focused subject image P1, A contour enlarged region A is estimated in which the contour portion is enlarged from the contour portion of the corresponding region of the object in one image. Then, the imaging apparatus 100 specifies a subject region B that includes the predetermined subject S in the subject focused image P1 with reference to the estimation result of the contour enlarged region A.

FIG. 1 is a block diagram illustrating a schematic configuration of an imaging apparatus 100 according to an embodiment to which the present invention is applied.
As shown in FIG. 1, the imaging device 100 specifically includes an imaging unit 1, an imaging control unit 2, an image data generation unit 3, a memory 4, an image processing unit 5, and a display control unit 6. , A display unit 7, a recording medium control unit 8, an operation input unit 9, and a central control unit 10.
The imaging unit 1, the imaging control unit 2, the image data generation unit 3, the memory 4, the image processing unit 5, the display control unit 6, the recording medium control unit 8, and the central control unit 10 are connected via a bus line 11. ing.

The imaging unit 1 captures a predetermined subject S as an imaging unit and generates a frame image.
Specifically, the imaging unit 1 includes a lens unit 1a, an electronic imaging unit 1b, and a lens driving unit 1c.

The lens unit 1a includes a plurality of lenses such as a zoom lens and a focus lens, for example.
The electronic imaging unit 1b includes, for example, an image sensor (imaging device) such as a charge coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS). Then, the electronic imaging unit 1b converts the optical image that has passed through the various lenses of the lens unit 1a into a two-dimensional image signal.
For example, although not shown, the lens driving unit 1c includes a zoom driving unit that moves the zoom lens in the optical axis direction, a focusing driving unit that moves the focus lens in the optical axis direction, and the like.
The imaging unit 1 may include a diaphragm (not shown) that adjusts the amount of light passing through the lens unit 1a in addition to the lens unit 1a, the electronic imaging unit 1b, and the lens driving unit 1c.

The imaging control unit 2 controls the imaging of the subject by the imaging unit 1. That is, the imaging control unit 2 includes a timing generator, a driver, and the like, although not illustrated. Then, the imaging control unit 2 scans and drives the electronic imaging unit 1b with a timing generator and a driver, and converts the optical image that has passed through the lens into a two-dimensional image signal for each predetermined period by the electronic imaging unit 1b. A frame image is read out for each screen from the imaging area of the electronic imaging unit 1b and output to the image data generation unit 3.
Further, the imaging control unit 2 controls the driving of the focusing driving unit of the lens driving unit 1c when imaging the subject, and moves the focus lens in the optical axis direction to adjust the focusing position of the lens unit 1a. To do. Specifically, the imaging control unit 2 moves the focus lens in the optical axis direction by the focusing drive unit when the subject focused image P1 (see FIG. 4A) is captured, and focuses on the background. Match to a predetermined subject S (for example, a stuffed animal). Further, the imaging control unit 2 moves the focus lens in the front-rear direction with respect to the predetermined subject S by the focus driving unit when imaging the subject non-focus image P2 (see FIG. 4B). Focus so that S is outside the depth of field.
Imaging of the subject focused image P1 and the subject non-focused image P2 is continuously performed with a predetermined time interval. More specifically, the imaging of the subject focused image P1 and the subject non-focused image P2 is performed without changing the distance between the main body of the imaging apparatus 100 and the predetermined subject S and the focal length (angle of view) of the lens unit 1a. It is performed continuously with a predetermined time interval. As a result, the subject focused image P1 and the subject non-focused image P2 have substantially the same composition.
Here, it is preferable to capture the subject focused image P1 and the subject non-focused image P2 while the apparatus main body is fixed to a tripod or the like so as not to cause a compositional shift. When the apparatus main body is imaged by hand, a guide display is performed so that a semi-transparent image of the subject focused image P1 is superimposed on the live view image when the subject non-focused image P2 is captured. May be.

The imaging control unit 2 may adjust the in-focus position of the lens unit 1a by moving the electronic imaging unit 1b in the optical axis direction instead of the focus lens of the lens unit 1a.
Further, in addition to AF (automatic focusing process), the imaging control unit 2 may perform adjustment control of conditions when imaging a subject such as AE (automatic exposure process) and AWB (automatic white balance).

The image data generation unit 3 appropriately adjusts the gain for each RGB color component with respect to the analog value signal of the frame image transferred from the electronic imaging unit 1b, and then performs sample holding by a sample hold circuit (not shown). The digital signal is converted into digital data by a / D converter (not shown), color processing including pixel interpolation processing and γ correction processing is performed by a color process circuit (not shown), and then a digital luminance signal Y and color difference signal Cb , Cr (YUV data).
The luminance signal Y and the color difference signals Cb and Cr output from the color process circuit are DMA-transferred to a memory 4 used as a buffer memory via a DMA controller (not shown).

  The memory 4 is composed of, for example, a DRAM (Dynamic Random Access Memory) or the like, and temporarily stores data processed by the image processing unit 5 or the central control unit 10.

The image processing unit 5 includes a first image acquisition unit 5a, a second image acquisition unit 5b, an enlarged contour region estimation unit 5c, an edge detection unit 5d, a subject region specification unit 5e, and an image generation unit 5f. doing.
Note that each unit of the image processing unit 5 includes, for example, a predetermined logic circuit, but the configuration is an example and the present invention is not limited thereto.

The first image acquisition unit 5a acquires the subject focused image P1.
That is, the first image acquisition unit 5a acquires, as a first acquisition unit, a subject focused image P1 (see FIG. 4A) captured by the imaging unit 1 while focusing on a predetermined subject S in the background. To do. Specifically, the first image acquisition unit 5a captures an image captured by the image capturing unit 1 while being focused on a predetermined subject S (for example, a stuffed animal) in the background, and is generated by the image data generating unit 3. Image data (YUV data) of the focused image P <b> 1 is acquired from the memory 4.

The second image acquisition unit 5b acquires the subject non-focus image P2.
In other words, the second image acquisition unit 5b, as the second acquisition unit, is imaged by the imaging unit 1 under a predetermined imaging condition different from the imaging condition of the subject focused image P1, and has a composition substantially equal to the subject focused image P1. A subject non-focus image P2 (one image; see FIG. 4B) is acquired. Specifically, the second image acquisition unit 5b is picked up by the image pickup unit 1 while being focused so that the predetermined subject S in the background exists outside the depth of field, and is generated by the image data generation unit 3. Image data (YUV data) of the subject non-focused image P <b> 2 is acquired from the memory 4.
Note that the subject non-focused image P2 shown in FIG. 4B is an image (background focused image) picked up by moving the focusing position backward with respect to the predetermined subject S, but is an example. However, the present invention is not limited to this. For example, an image (foreground focused image; not shown) captured by moving the focusing position in the forward direction with respect to the predetermined subject S may be used.

The contour expansion region estimation unit 5c estimates the contour expansion region A in the subject focused image P1.
Here, the contour expansion region A is a region corresponding to an object outside the depth of field when the predetermined subject S is focused in the subject focused image P1. That is, since the contour enlarged region A is a blurred region that is not in focus, the contour portion of the contour enlarged region A in the subject focused image P1 is the subject in the subject non-focused image (one image) P2. It will be in the state expanded rather than the outline part of the corresponding | compatible area | region C (refer FIG.4 (b)) corresponding to the outline expansion area | region A. FIG.
Further, in the contour enlarged region A, for example, at least one of the color and brightness of the contour portion has a steep change with respect to the peripheral region. Here, the steep change refers to a change to such an extent that an edge is detected in the contour portion of the contour enlarged region A when the edge detection process (described later in detail) is performed on the subject focused image P1. say. For example, as the contour expansion region A, a high-luminance object (for example, a light source such as a fluorescent lamp) whose luminance level of light incident on an image sensor such as a CCD or CMOS of the imaging unit 1 is larger than the upper limit of the detectable range. Brightness change area A1 corresponding to. The luminance change area A1 is, for example, an area in which a high brightness object such as a light source such as a fluorescent lamp is imaged in a backlight state and is in an overexposed state in the captured image depending on the exposure condition of the imaging unit 1. Including.
In other words, if edge detection processing is performed on the subject focused image P1, not only the predetermined subject S that is in focus but also the edge of an object that corresponds to the contour expansion area A that is not in focus. Is also detected. Similarly, when the edge detection process is performed on the subject non-focused image P2, the edge of the object in the corresponding region C corresponding to the focused contour enlarged region A is detected. The object is in a state of being reduced more than the edge of the object corresponding to the contour enlargement area A in the subject focused image P1, and the size differs.

  Further, the contour enlarged region estimation unit 5c has a sharp brightness value of the contour portion in the subject focused image P1 with respect to the surrounding region based on the brightness information of the subject focused image P1 and the subject non-focused image P2. The brightness change area A1 that has changed to is estimated as the contour enlarged area A. Specifically, the contour enlargement area estimation unit 5c includes a first determination unit c1 and a second determination unit c2, and based on the determination results of the first determination unit c1 and the second determination unit c2. The brightness change area A1 is estimated.

The first determination unit c1 determines, as a first determination unit, whether or not a difference in luminance value of each corresponding pixel between the subject focused image P1 and the subject non-focused image P2 is larger than the first determination value. To do.
Specifically, the first determination unit c1 acquires the luminance signal (Y1) of the YUV data of the subject focused image P1 acquired by the first image acquisition unit 5a, and the second image acquisition unit 5b. A luminance signal (Y2) in the acquired subject non-focused image P2 is acquired. Then, the first determination unit c1 calculates a luminance value difference for each of a plurality of corresponding pixels (x, y) between the subject focused image P1 and the subject non-focused image P2, and the luminance value difference is calculated. Is greater than the first determination value Thr1 according to the following equation (1).
Y1 (x, y) -Y2 (x, y)> Thr1 Formula (1)
Note that the value of the first determination value Thr1 may be set based on a predetermined operation of the operation input unit 9 by the user, or a predetermined value set as a default or a set white balance mode (for example, , Under a fluorescent lamp, under sunlight, etc.) may be automatically set to a predetermined value.

The 2nd determination part c2 determines whether the luminance value of each pixel of the to-be-photographed object P1 is larger than a 2nd determination value as a 2nd determination means.
Specifically, the second determination unit c2 acquires a luminance signal (Y1) in the YUV data of the subject focused image P1 acquired by the first image acquisition unit 5a. The second determination unit c2 determines each of the plurality of pixels (x, y) in the subject focused image P1 that the difference in luminance value is larger than the first determination value Thr1 by the first determination unit c1. After specifying the pixel, it is determined according to the following formula (2) whether or not the luminance value is larger than the second determination value Thr2 for each of the pixels.
Y1 (x, y)> Thr2 Formula (2)
Note that the value of the second determination value Thr1 may be set based on a predetermined operation of the operation input unit 9 by the user, similarly to the first determination value Thr1, or may be a predetermined value or a predetermined value set as a default. It may be automatically set to a predetermined value based on the set white balance mode (for example, under fluorescent light or sunlight).

Further, the contour enlargement region estimation unit 5c identifies a region including pixels that satisfy both of the above expressions (1) and (2) based on the determination results of both the first determination unit c1 and the second determination unit c2. To do.
Then, the contour enlargement area estimation unit 5c sets a pixel that satisfies both the above formulas (1) and (2) (a white area in FIG. 5A) as the first pixel value “1”, and Luminance change region information M (see FIG. 5A) such as a binary image representing a pixel other than the pixel (region represented by black in FIG. 5A) as the second pixel value “0” is generated.
That is, in the luminance change area information M, a pixel having the first pixel value “1” corresponds to a luminance change area corresponding to a high-luminance object such as a light source such as a fluorescent lamp in the subject focused image P1. It corresponds to the edge of A1.

Note that, as the contour expansion region A, for example, the luminance change region A1 corresponding to a high-luminance object such as a light source such as a fluorescent lamp has been described as an example. It may be a region having a change.
As described above, the contour enlargement area estimation unit 5c determines the depth of field in the subject focused image P1 based on the image information of the subject focused image P1 and the subject non-focused image (one image) P2. Corresponding to an object existing outside, an estimation means for estimating an enlarged contour area A in which the contour portion is enlarged from the contour portion of the region corresponding to the object in one image is configured.

The edge detection unit 5d detects an edge in the subject focused image P1.
That is, the edge detection unit 5d detects an edge related to an object in the subject focused image P1 as detection means. Specifically, the edge detection unit 5d performs a differentiation operation on the YUV data of the subject focused image P1 acquired by the first image acquisition unit 5a using a predetermined differential filter (for example, a high-pass filter). Thus, an edge detection process is performed to detect a sharp change in luminance value, color, or density as an edge.
Thereby, for example, the edge detection unit 5d corresponds to the edge of the predetermined subject S that is in focus in the subject focused image P1 and the contour expansion region A (luminance change region A1) that is not in focus. It also detects the edges of objects.

The subject area specifying unit 5e specifies a subject area B in which a predetermined subject S is included in the subject focused image P1.
That is, the subject region specifying unit 5e specifies, as a specifying unit, a subject region B that includes the predetermined subject S in the subject focused image P1 based on the estimation result of the contour enlarged region A by the contour enlarged region estimation unit 5c. To do. Specifically, the subject region specifying unit 5e acquires the luminance change region information M generated by the contour enlargement region estimation unit 5c, and all the edges detected by the edge detection unit 5d (that is, the predetermined subject S). And the edge corresponding to the luminance change area A1), the subject area B is specified by reducing the evaluation value of each edge corresponding to the luminance change area A1.
For example, the subject area specifying unit 5e calculates an evaluation value according to the strength of each edge, the number of pixels existing in a candidate area surrounded by a predetermined number of edges, and the candidate area having the highest evaluation value is determined as a predetermined value. A subject area B including the subject S is specified.
At this time, the subject area specifying unit 5e reduces, for example, the evaluation value (reliability) of each edge corresponding to the luminance change area A1 and the candidate area surrounded by the edges among the plurality of candidate areas. It is difficult to be specified as B. Further, the subject area specifying unit 5e is calculated according to the number of pixels constituting each candidate area, for example, by reducing (shrinking) the size of the candidate area surrounded by the edge corresponding to the luminance change area A1. The evaluation value is relatively lowered to make it difficult to identify the subject area B.

  Note that the subject area specifying unit 5e performs the specification of the subject area B in the subject focused image P1 on the basis of the estimation result of the luminance change area A1 by the contour enlargement area estimation unit 5c. An estimation result of an area where the color, density, or the like as the area A has a steep change relative to the surrounding area may be used.

The image generation unit 5f generates a blurred image P3 in which the background region other than the subject region B in the subject focused image P1 is blurred.
That is, the image generation unit 5f performs blur processing on the subject focused image P1 as an image generation unit, and blurs a background region other than the subject region B in the subject focused image P1 (see FIG. 5 (b)). Specifically, the image generating unit 5f sets binary information in which the alpha value of the subject region B specified by the subject region specifying unit 5e is “1” and the alpha value of the other background region is “0”. Generate. Then, the image generation unit 5f applies a low-pass filter to the binary information to generate an intermediate value (0 ≦ α ≦ 1) in the boundary region, and indicates the position of the subject region B in the subject focused image P1. An alpha map (not shown) as position information is generated. Then, the image generation unit 5f performs blurring processing on the subject focused image P1 using the generated alpha map, and each of the background region and the boundary region in the subject focused image P1 has a predetermined intensity. A blurred image P3 blurred by is generated.
As the other processing, for example, there is a method performed by applying a Gaussian blur filter to the image data of the subject in-focus image P1, but this is an example, and the method is not limited to this. If so, it can be arbitrarily changed as appropriate.

The display control unit 6 performs control to read out the display image data temporarily stored in the memory 4 and display it on the display unit 7.
Specifically, the display control unit 6 includes a VRAM (Video Random Access Memory), a VRAM controller, a digital video encoder, and the like. The digital video encoder reads the luminance signal Y and the color difference signals Cb and Cr read from the memory 4 and stored in the VRAM (not shown) under the control of the central control unit 10 through the VRAM controller. Are periodically read out, and a video signal is generated based on these data and output to the display unit 7.

  The display unit 7 is, for example, a liquid crystal display panel, and displays an image captured by the electronic imaging unit 1b based on a video signal from the display control unit 6 on a display screen. Specifically, the display unit 7 sequentially updates a plurality of frame images generated by imaging the subject by the imaging unit 1 and the imaging control unit 2 at a predetermined frame rate in the still image capturing mode and the moving image capturing mode. While displaying the live view image. The display unit 7 displays an image (rec view image) recorded as a still image or an image being recorded as a moving image.

The recording medium control unit 8 is configured such that the recording medium R is detachable, and controls reading of data from the loaded recording medium R and writing of data to the recording medium R.
That is, the recording medium control unit 8 records the recording image data encoded in a predetermined compression format (for example, JPEG format) by the encoding unit (not shown) of the image processing unit 5 on the recording medium R. .
The recording medium R is composed of, for example, a non-volatile memory (flash memory) or the like.

  The operation input unit 9 is for performing a predetermined operation of the imaging apparatus 100. Specifically, the operation input unit 9 includes a shutter button related to an imaging instruction of a subject, a selection determination button related to an instruction to select an imaging mode and a function, a zoom button related to an instruction to adjust the zoom amount, and the like (all not shown). And outputs a predetermined operation signal to the central control unit 10 in accordance with the operation of each button of the operation unit.

  The central control unit 10 controls each unit of the imaging device 100. Specifically, although not shown, the central control unit 10 includes a CPU (Central Processing Unit) and the like, and performs various control operations according to various processing programs (not shown) for the imaging apparatus 100.

Next, image generation processing by the imaging apparatus 100 will be described with reference to FIGS.
FIG. 2 is a flowchart illustrating an example of an operation related to the image generation process. FIG. 3 is a flowchart illustrating an example of an operation related to the luminance change region estimation process in the image generation process.

The image generation processing is performed when the blur control image generation mode is selected from a plurality of operation modes displayed on the menu screen based on a predetermined operation of the selection determination button of the operation input unit 9 by the user. 10 is a process executed by each unit of the imaging apparatus 100 under the control of 10.
In the image generation process shown in FIG. 2, the luminance change area A1 is estimated as the contour expansion area A, and the subject area B is specified based on the estimation result of the luminance change area A1.

As shown in FIG. 2, first, the display control unit 6 displays a live view image on the display screen of the display unit 7 based on a plurality of frame images generated by imaging the subject by the imaging unit 1 (step S1). .
Specifically, the imaging control unit 2 causes the electronic imaging unit 1b to convert the optical image that has passed through the lens unit 1a into a two-dimensional image signal (RGB image data) at predetermined intervals, and the image data generation unit 3 The two-dimensional image signal output and input from the electronic imaging unit 1b is converted into a digital image signal at predetermined intervals to generate YUV data for each frame image. The YUV data of each frame image generated by the image data generation unit 3 is sequentially output to the memory 4 and stored in the memory 4. The display control unit 6 performs control to read the display image data temporarily stored in the memory 4 and display the live view image on the display unit 7.

Next, the CPU of the central control unit 10 determines whether or not an imaging instruction output from the operation input unit 9 is input based on a predetermined operation (for example, full pressing operation) of the shutter button of the operation input unit 9 by the user. Is determined (step S2).
Here, if it is determined that the imaging instruction is not input (step S2; NO), the CPU of the central control unit 10 returns the process to step S1, and determines that the imaging instruction is input in step S2. (Step S2; YES), the display control unit 6 performs control to display the live view image on the display screen of the display unit 8.

Then, after the adjustment of the composition of the predetermined subject S by the user is completed, the CPU of the central control unit 10 captures an image when the shutter button of the operation input unit 9 is predetermined (for example, fully pressed) by the user. If it is determined that an instruction has been input (step S2; YES), the imaging control unit 2 moves the focus lens in the optical axis direction by the focus driving unit of the lens driving unit 1c to focus on a predetermined subject in the background. A subject focused image P1 (see FIG. 4A) is imaged in accordance with S (for example, a stuffed animal) (step S3). The image data generation unit 3 converts the image signal of the subject focused image P1 into a digital image signal and generates YUV data of the subject focused image P1.
The YUV data of the subject focused image P1 generated by the image data generation unit 3 is output to the memory 4 and temporarily stored in the memory 4.

Subsequently, the imaging control unit 2 moves the focus lens in the front-rear direction with respect to the predetermined subject S by the focus driving unit to focus the predetermined subject S so that it is outside the depth of field. A subject non-focus image P2 (see FIG. 4B) having a composition substantially equal to P1 is captured (step S4). The image data generation unit 3 converts the image signal of the subject non-focus image P2 into a digital image signal, and generates YUV data of the subject non-focus image P2.
The YUV data of the subject non-focused image P <b> 2 generated by the image data generation unit 3 is output to the memory 4 and temporarily stored in the memory 4.

Next, the image processing unit 5 performs a luminance change region estimation process (see FIG. 3) for estimating a luminance change region A1 in which the luminance value of the contour portion changes sharply with respect to the peripheral region in the subject focused image P1. This is performed (step S5).
Here, with reference to FIG. 3, the brightness change region estimation processing will be described in detail.

As shown in FIG. 3, the first image acquisition unit 5a of the image processing unit 5 acquires the image data (YUV data) of the subject focused image P1 stored in the memory 4 (step S51). Subsequently, the second image acquisition unit 5b acquires image data (YUV data) of the subject non-focus image P2 stored in the memory 4 (step S52).
Note that the image data of the subject non-focused image (one image) P2 is acquired after the acquisition of the image data of the subject focused image P1, but the order of the acquisition processing of these images is only an example. However, the order may be reversed, that is, the image data of the subject focused image P1 may be acquired after the image data of the subject non-focused image P2 is acquired.

Next, the first determination unit c1 of the contour enlargement area estimation unit 5c determines whether the difference in luminance value of each corresponding pixel between the subject focused image P1 and the subject non-focused image P2 is larger than the first determination value. It is determined whether or not (step S53).
Specifically, the first determination unit c1 acquires the luminance signal (Y1) in the YUV data of the subject focused image P1 and the luminance signal (Y2) in the subject non-focused image P2, and then adjusts the subject alignment. Whether each of the plurality of pixels (x, y) corresponding to the focus image P1 and the subject non-focus image P2 has a luminance value difference larger than the first determination value Thr1 is determined according to the following formula (1). judge.
Y1 (x, y) -Y2 (x, y)> Thr1 Formula (1)

If it is determined in step S53 that the difference in luminance value is greater than the first determination value Thr1 (step S53; YES), the second determination unit c2 of the contour enlargement area estimation unit 5c determines the subject in-focus image P1. Among the plurality of pixels, it is determined whether or not the luminance value of the pixel determined by the first determination unit c1 that the difference in luminance value is greater than the first determination value Thr1 is greater than the second determination value (step S54). ).
Specifically, the second determination unit c2 acquires the luminance signal (Y1) in the YUV data of the subject focused image P1, and then, among the plurality of pixels (x, y) of the subject focused image P1, For each pixel that is determined to have a luminance value difference greater than the first determination value Thr1, it is determined whether the luminance value is greater than the second determination value Thr2 according to the following equation (2).
Y1 (x, y)> Thr2 Formula (2)

If it is determined in step S54 that the luminance value is greater than the second determination value Thr2 (step S54; YES), the contour enlargement area estimation unit 5c determines that the luminance value is the second determination in the subject focused image P1. A region including a pixel determined to be larger than the value Thr2 is specified (step S55). That is, the contour enlargement area estimation unit 5c specifies an area including pixels that satisfy both the above expressions (1) and (2) in the subject focused image P1.
Then, the contour enlargement area estimation unit 5c sets a pixel that satisfies both the above formulas (1) and (2) (a white area in FIG. 5A) as the first pixel value “1”, and Generate luminance change area information M (see FIG. 5A) such as a binary image representing pixels other than the pixels (area represented by black in FIG. 5A) as the second pixel value “0”. (Step S56), the luminance change region estimation process is terminated.
The generated luminance change area information M is temporarily stored in a predetermined storage unit (for example, the memory 4).

  On the other hand, when it is determined in step S53 that the difference in luminance value is not greater than the first determination value Thr1 (step S53; NO), or in step S54, the luminance value is greater than the second determination value Thr2. If it is determined that it is not large (step S54; NO), the image processing unit 5 skips steps S55 and S56 and ends the luminance change region estimation process.

Returning to FIG. 2, the edge detection unit 5d of the image processing unit 5 performs an edge detection process for detecting an edge in the subject focused image P1 (step S6).
Specifically, the edge detection unit 5d applies, for example, a high-pass filter to the YUV data of the subject focused image P1, and detects a sharp change in luminance value, color, density, or the like as an edge.

Subsequently, the image processing unit 5 determines whether the luminance change area information M is obtained in the luminance change area estimation process according to whether or not the luminance change area information M is stored in a predetermined storage unit (for example, the memory 4). It is determined whether or not it has been generated (step S7).
Here, if it is determined that the brightness change area information M has been generated (step S7; YES), the subject area specifying unit 5e uses the estimation result of the brightness change area A1 by the contour enlargement area estimation unit 5c as a reference. A subject area B including a predetermined subject S is specified in the focused image P1 (step S8). Specifically, the subject region specifying unit 5e acquires the luminance change region information M generated by the contour enlargement region estimation unit 5c, and all the edges detected by the edge detection unit 5d (that is, the predetermined subject S). In addition, the evaluation value is calculated according to the intensity of the edge of the luminance change area A1, the number of pixels existing in the candidate area surrounded by the predetermined number of edges, and the like. At this time, the subject area specifying unit 5e reduces the evaluation value of each edge corresponding to the brightness change area A1, thereby making it difficult to specify the brightness change area A1 as the subject area B. Then, the subject area specifying unit 5e specifies the candidate area having the highest evaluation value as the subject area B including the predetermined subject S.

Next, the image generation unit 5f performs a blurring process on the subject focused image P1, and blurs a background region other than the subject region B in the subject focused image P1 (FIG. 5B). Reference) is generated (step S9).
Specifically, the image generation unit 5f performs a blurring process on the subject focused image P1 using an alpha map as position information indicating the position of the subject region B in the subject focused image P1. A blurred image P3 is generated by blurring the background area and the boundary area in the subject focused image P1 with a predetermined intensity.

On the other hand, if it is determined in step S7 that the brightness change area information M has not been generated (step S7; NO), the subject area specifying unit 5e does not consider the brightness change area A1 and the subject focused image. A subject area B including a predetermined subject S in P1 is specified (step S10). Specifically, the subject area specifying unit 5e is a candidate surrounded by the strength of all edges detected by the edge detection unit 5d (that is, the edges related to the predetermined subject S and the luminance change area A1) or a predetermined number of edges. An evaluation value is calculated according to the number of pixels existing in the region. Then, the subject area specifying unit 5e specifies the candidate area having the highest evaluation value as the subject area B including the predetermined subject S (step S10).
Thereafter, the image processing unit 5 shifts the processing to step S9, and the image generation unit 5f performs a process other than the subject region B specified in step S10 in the subject focused image P1 in substantially the same manner as described above. A blurred image (not shown) with a blurred background region is generated (step S9).

  Then, the recording medium control unit 8 obtains the YUV data of the blurred image P3 generated by the image generation unit 5f and encoded in a predetermined compression format (for example, JPEG format) by the encoding unit of the image processing unit 5. Then, the image is recorded on the recording medium R (step S11), and the image generation process is terminated.

  As described above, according to the imaging apparatus 100 of the present embodiment, in the subject focused image P1, based on the respective image information of the subject focused image P1 and the subject non-focused image (one image) P2, Estimating a contour expansion region A corresponding to an object existing outside the depth of field and having a contour portion enlarged from the contour portion of the region corresponding to the object in one image. Since the subject region B including the predetermined subject S is specified in the subject focused image P1 on the basis of the result, the contour portion of the optical image in which an object existing outside the depth of field is expanded in the subject focused image P1. Even when the contour enlargement area A in which is clearly shown is included, the subject area B including the predetermined subject S in the subject focused image P1 is determined based on the estimation result of the contour enlargement area A. By specifying, outside the depth of field To hardly cause erroneous detection of an edge of the optical image of the object a particular subject region B can be properly carried out.

Specifically, for example, the luminance level of light emitted from a high-luminance object such as a fluorescent lamp and incident on the electronic imaging unit 1b is larger than the upper limit of the detectable range (the light intensity is high), a lens When the resolution of the portion 1a is low, the contour portion of the optical image in which the object is spread appears clearly on the subject focused image P1. That is, the contour portion of the contour enlarged region A (luminance change region A1) in the subject focused image P1 is more than the contour portion of the corresponding region C corresponding to the contour magnified region A in the subject non-focused image (one image) P2. May also be magnified, which may cause false detection of the edge of an optical image of a high-brightness object.
Even in such a case, it is possible to estimate all the brightness change areas A1 in which the brightness value of the contour portion has sharply changed in the subject focused image P1 in advance, thereby detecting all of the detected in the subject focused image P1. Of the edge of the optical image of the high-luminance object is caused by lowering the evaluation value of the edge corresponding to the luminance change area A1 among the edges (the edges related to the predetermined subject S and the luminance change area A1) Therefore, it is possible to appropriately specify the subject area B in the subject focused image P1.

In addition, each of the subject focused image P1 captured by focusing on a predetermined subject S in the background and the subject non-focused image P2 captured by moving the focusing position in the front-rear direction with respect to the predetermined subject S. Since the brightness change area A1 is estimated in the subject focused image P1 based on the brightness information of the subject, the brightness change area A1 in which the brightness value of the contour portion as the contour enlargement area A has sharply changed is appropriately estimated. Can do.
Specifically, in the subject focused image P1, the difference between the luminance values of corresponding pixels between the subject focused image P1 and the subject non-focused image P2 is larger than the first determination value, and the luminance Since the region including pixels whose values are larger than the second determination value is estimated as the luminance change region A1, not only the difference in luminance value of each corresponding pixel between the subject focused image P1 and the subject non-focused image P2 Considering the luminance value of each pixel of the subject focused image P1, the luminance change area A1 can be appropriately estimated in the subject focused image P1. That is, even if the difference in luminance value is larger than the first determination value, pixels whose luminance value is not larger than the second determination value can be excluded. For example, high luminance such as a fluorescent lamp An area corresponding to the object can be appropriately estimated as the luminance change area A1.

In addition, when the blurred image P3 in which the background region other than the subject region B in the subject focused image P1 is blurred is generated, the subject region B and the background region cannot be properly determined, so that the background region is the background region. Nevertheless, the subject region B is identified and an unnatural blurred image P4 (see FIG. 6) is not generated.
That is, the blurred image P4 shown in FIG. 6 is determined to be a subject region even though the contour enlargement region A (luminance change region A1) is the background region, and the degree of blur of the contour enlargement region A is other than It is smaller overall than the background area. In particular, the central portion of the contour enlarged region A is in a state that is not blurred at all. On the other hand, according to the imaging apparatus 100 of the present embodiment, the contour expansion region A can be given a degree of blur substantially the same as that of other background regions, and the entire background region other than the subject region B can be applied. A blurred image P3 (see FIG. 5B) to which natural blur is added can be generated.

The present invention is not limited to the above-described embodiment, and various improvements and design changes may be made without departing from the spirit of the present invention.
For example, although the contour enlargement area A (luminance change area A1) is estimated using the subject non-focus image P2 as one image, this is an example and the present invention is not limited to this. One image used for the estimation can be arbitrarily changed as appropriate.
That is, one image may be an image that is captured under a predetermined imaging condition different from the imaging condition of the subject focused image P1 and has a composition that is substantially equal to the subject focused image P1, for example, as one image, The exposure time of the electronic imaging unit 1b of the imaging unit 1 may be shorter than the subject focused image P1, and a short-time exposure image captured by the imaging unit 1 may be used. Then, the contour enlargement area estimation unit 5c, based on the brightness information of the subject focused image P1 and the short-time exposure image, changes the brightness change area A1 in which the brightness value of the contour portion in the subject focused image P1 has changed sharply. May be estimated as the contour enlarged region A. That is, for example, a high-brightness object such as a light source such as a fluorescent lamp has a high intensity of light incident on the electronic imaging unit 1b. Therefore, charges can be accumulated even if the exposure time of the electronic imaging unit 1b is short. it can. As a result, in the short-time exposure image captured with the exposure time of the electronic imaging unit 1b shorter than that of the subject focused image P1, the region corresponding to the pixel in which the charge of the electronic imaging unit 1b is accumulated has high brightness. It is considered that there is a high possibility of dealing with
Therefore, the contour enlargement area estimation unit 5c acquires a luminance signal in the short-time exposure image and specifies a high-luminance area corresponding to a high-luminance object in the short-time exposure image. Then, the contour enlargement area estimation unit 5c acquires a luminance signal in the YUV data of the subject focused image P1, and determines an area corresponding to the high luminance area of the short-time exposure image in the subject focused image P1. It is estimated as the luminance change area A1 (contour enlarged area A).

  Even in such a configuration, by specifying the subject region B including the predetermined subject S in the subject focused image P1 on the basis of the estimation result of the contour enlarged region A, an object outside the depth of field can be identified. The subject area B can be specified appropriately by making it difficult to cause erroneous detection of the edge of the optical image.

Moreover, in the said embodiment, although the determination process by the 2nd determination part c2 was performed after the determination process by the 1st determination part c1, the order of these determination processes is an example, and is restricted to this. Instead, the order may be reversed, that is, the determination process by the first determination unit c1 may be performed after the determination process by the second determination unit c2.
Furthermore, the contour enlarged region estimation unit 5c is based on the determination results of both the first determination unit c1 and the second determination unit c2, that is, a region including pixels that satisfy both the above expressions (1) and (2). Is estimated as the luminance change region A1, but it is an example and not limited to this, and includes either one of the first determination unit c1 and the second determination unit c2, and the above formula (1 ) And formula (2), a region including pixels that satisfy only one of the two may be estimated as the luminance change region A1.
Specifically, the contour enlargement area estimation unit 5c performs only the determination processing by the first determination unit c1, and the pixel (the above-described difference in luminance value is larger than the first determination value Thr1 in the subject focused image P1). When a pixel that satisfies only equation (1) is specified, an area including the pixel is estimated as the luminance change area A1. In addition, the contour enlargement area estimation unit 5c performs only the determination process by the second determination unit c2, and the pixel whose luminance value is larger than the second determination value Thr2 in the subject focused image P1 (only the above equation (2)). When a pixel satisfying the above is specified, an area including the pixel is estimated as the luminance change area A1.

  In the above embodiment, the subject region B is specified using the detection result of the edge in the subject focused image P1, but whether or not the edge in the subject focused image P1 is used. These can be arbitrarily changed as appropriate. That is, the imaging apparatus 100 does not necessarily include the edge detection unit 5d, and can specify a subject region B that includes the predetermined subject S in the subject focused image P1 based on the estimation result of the contour enlargement region A. Any configuration may be used.

Furthermore, in the above embodiment, the image generation unit 5f generates the blurred image P3 in which the background region other than the subject region B in the subject focused image P1 is blurred. However, the image generation unit 5f is not necessarily provided. It is not necessary to do so, and whether or not to generate the blurred image P3 can be arbitrarily changed as appropriate.
Further, the post-processing after the subject area B is specified by the imaging apparatus 100 is not limited to the generation of the blurred image P3, but is an example and can be arbitrarily changed as appropriate. That is, the imaging apparatus 100 may perform a so-called focus bracket imaging by using the specified subject region B to capture a plurality of images in which the focus position is changed in the front-rear direction with respect to the predetermined subject S. Alternatively, a process of cutting out an image including the subject region B from the subject focused image P1 may be performed.

  Furthermore, the configuration of the imaging apparatus 100 is merely an example illustrated in the above embodiment, and is not limited thereto. Further, although the imaging apparatus 100 has been exemplified as the image processing apparatus, the present invention is not limited to this, and any configuration may be used as long as the image processing according to the present invention can be executed.

In addition, in the above embodiment, the functions of the first acquisition unit, the second acquisition unit, the estimation unit, and the identification unit are controlled by the central control unit 10 in the first image acquisition unit 5a, the second image acquisition unit 5a, and the second image acquisition unit 5a. The image acquisition unit 5b, the contour enlarged region estimation unit 5c, and the subject region specification unit 5e are configured to be driven. However, the configuration is not limited to this, and a predetermined program or the like is executed by the central control unit 10. It is good also as a structure implement | achieved by doing.
In other words, a program including a first acquisition processing routine, a second acquisition processing routine, an estimation processing routine, and a specific processing routine is stored in a program memory (not shown) that stores the program. Then, the first acquisition processing routine causes the CPU of the central control unit 10 to function as a first acquisition unit that acquires the focused image P1 captured by the imaging device while focusing on the predetermined subject S in the background. Anyway. In addition, the CPU of the central control unit 10 is imaged under a predetermined imaging condition different from the imaging condition of the subject focused image P1 by the imaging device by the second acquisition processing routine, and has a composition substantially equal to the subject focused image P1. You may make it function as the 2nd acquisition means which acquires the image of this. Further, the CPU of the central control unit 10 corresponds to an object outside the depth of field in the subject focused image P1 based on the image information of the subject focused image P1 and one image by the estimation processing routine. Then, the contour portion may be made to function as an estimation means for estimating the contour enlargement region A formed by enlarging the contour portion of the region corresponding to the object in one image. In addition, the CPU of the central control unit 10 by the specifying process routine specifies the subject region B including the predetermined subject S in the subject focused image P1 based on the estimation result of the contour enlarged region A by the estimating unit. You may make it function as.

  Similarly, the first determination unit, the second determination unit, the detection unit, and the image generation unit may be realized by executing a predetermined program or the like by the CPU of the central control unit 10.

  Furthermore, as a computer-readable medium storing a program for executing each of the above processes, a non-volatile memory such as a flash memory or a portable recording medium such as a CD-ROM is applied in addition to a ROM or a hard disk. Is also possible. A carrier wave is also used as a medium for providing program data via a predetermined communication line.

Although several embodiments of the present invention have been described, the scope of the present invention is not limited to the above-described embodiments, but includes the scope of the invention described in the claims and equivalents thereof.
The invention described in the scope of claims attached to the application of this application will be added below. The item numbers of the claims described in the appendix are as set forth in the claims attached to the application of this application.
[Appendix]
<Claim 1>
First acquisition means for acquiring a focused subject image captured by the imaging means while focusing on a predetermined subject in the background;
Second acquisition means for acquiring one image captured by the imaging means under a predetermined imaging condition different from the imaging condition of the subject focused image and having a composition substantially equal to the subject focused image;
Based on the image information of each of the subject in-focus image and the one image, the object in the subject in-focus image corresponds to an object existing outside the depth of field, and the contour portion corresponds to the object in the one image. An estimation means for estimating a contour expansion region that is enlarged from the contour portion of the corresponding region;
Specifying means for specifying a subject area in which the predetermined subject is included in the subject focused image with reference to the estimation result of the contour enlarged region by the estimating means;
An image processing apparatus comprising:
<Claim 2>
The one image includes a subject non-focused image captured by the imaging means by moving a focusing position in the front-rear direction with respect to the predetermined subject,
The estimating means further includes:
Based on the luminance information of each of the subject focused image and the subject non-focused image, a luminance change region in which the luminance value of the contour portion has sharply changed in the subject focused image is estimated as the contour enlarged region. The image processing apparatus according to claim 1.
<Claim 3>
The estimation means includes
First determination means is further provided for determining whether or not a difference in luminance value of each corresponding pixel between the subject focused image and the subject non-focused image is greater than a first determination value. The image processing apparatus according to claim 2, wherein an area including a pixel having a luminance value difference larger than the first determination value is estimated as the luminance change area based on a determination result by the determination unit.
<Claim 4>
The estimation means includes
Second determination means for determining whether or not the luminance value of each pixel of the subject focused image is larger than a second determination value, and based on the determination result by the second determination means, the luminance value is The image processing apparatus according to claim 2, wherein an area including pixels larger than a second determination value is estimated as the luminance change area.
<Claim 5>
The one image includes a short-time exposure image captured by the imaging unit with an exposure time of the imaging unit shorter than that of the subject focused image,
The estimating means further includes:
Estimating, as the enlarged contour region, a luminance change region in which a luminance value of a contour portion has sharply changed in the subject focused image based on luminance information of each of the subject focused image and the short-time exposure image. The image processing apparatus according to claim 1, wherein:
<Claim 6>
6. The brightness change area is an area corresponding to a high brightness object in which a brightness level of light incident on an image sensor of the imaging means is larger than an upper limit of a detectable range. An image processing apparatus according to claim 1.
<Claim 7>
Detecting means for detecting an edge in the subject focused image;
The specifying means is:
7. The subject area is specified by reducing an evaluation value of an edge corresponding to the contour enlargement area estimated by the estimation means among edges detected by the detection means. The image processing apparatus according to any one of the above.
<Claim 8>
2. The image processing apparatus according to claim 1, further comprising: an image generation unit configured to perform a blurring process on the subject focused image to generate a blurred image in which a background region other than the subject region in the subject focused image is blurred. The image processing device according to any one of?
<Claim 9>
An image processing method using an image processing apparatus,
Processing to obtain a subject in-focus image captured by the imaging device while focusing on a predetermined subject in the background;
Processing for obtaining one image having a composition that is imaged by the imaging device under a predetermined imaging condition different from the imaging condition of the subject focused image and having substantially the same composition as the subject focused image;
Based on the image information of each of the subject in-focus image and the one image, the object in the subject in-focus image corresponds to an object existing outside the depth of field, and the contour portion corresponds to the object in the one image. A process for estimating a contour enlargement region that is enlarged from the contour portion of the corresponding region;
A process of specifying a subject area that includes the predetermined subject in the subject focused image with reference to the estimation result of the contour enlargement region;
An image processing method comprising:
<Claim 10>
The computer of the image processing device
First acquisition means for acquiring a focused subject image captured by the imaging apparatus while focusing on a predetermined subject in the background;
A second acquisition unit configured to acquire an image captured by the imaging apparatus under a predetermined imaging condition different from an imaging condition of the subject focused image and having a composition substantially equal to the subject focused image;
Based on the image information of each of the subject in-focus image and the one image, the object in the subject in-focus image corresponds to an object existing outside the depth of field, and the contour portion corresponds to the object in the one image. An estimation means for estimating a contour expansion region that is enlarged from the contour portion of the corresponding region;
Specifying means for specifying a subject area including the predetermined subject in the subject focused image with reference to the estimation result of the contour enlargement region by the estimating means;
A program characterized by functioning as

DESCRIPTION OF SYMBOLS 100 Imaging device 1 Imaging part 2 Imaging control part 5 Image processing part 5a 1st image acquisition part 5b 2nd image acquisition part 5c Outline expansion area estimation part c1 1st determination part c2 2nd determination part 5d Edge detection part 5e Subject area specification Part 5f image generation part 10 central control part

Claims (10)

First acquisition means for acquiring a focused subject image captured by the imaging means while focusing on a predetermined subject in the background;
Second acquisition means for acquiring one image captured by the imaging means under a predetermined imaging condition different from the imaging condition of the subject focused image and having a composition substantially equal to the subject focused image;
Based on the image information of each of the subject in-focus image and the one image, the object in the subject in-focus image corresponds to an object existing outside the depth of field, and the contour portion corresponds to the object in the one image. An estimation means for estimating a contour expansion region that is enlarged from the contour portion of the corresponding region;
Specifying means for specifying a subject area in which the predetermined subject is included in the subject focused image with reference to the estimation result of the contour enlarged region by the estimating means;
An image processing apparatus comprising: The one image includes a subject non-focused image captured by the imaging means by moving a focusing position in the front-rear direction with respect to the predetermined subject,
The estimating means further includes:
Based on the luminance information of each of the subject focused image and the subject non-focused image, a luminance change region in which the luminance value of the contour portion has sharply changed in the subject focused image is estimated as the contour enlarged region. The image processing apparatus according to claim 1. The estimation means includes
First determination means is further provided for determining whether or not a difference in luminance value of each corresponding pixel between the subject focused image and the subject non-focused image is greater than a first determination value. The image processing apparatus according to claim 2, wherein an area including a pixel having a luminance value difference larger than the first determination value is estimated as the luminance change area based on a determination result by the determination unit. The estimation means includes
Second determination means for determining whether or not the luminance value of each pixel of the subject focused image is larger than a second determination value, and based on the determination result by the second determination means, the luminance value is The image processing apparatus according to claim 2, wherein an area including pixels larger than a second determination value is estimated as the luminance change area. The one image includes a short-time exposure image captured by the imaging unit with an exposure time of the imaging unit shorter than that of the subject focused image,
The estimating means further includes:
Estimating, as the enlarged contour region, a luminance change region in which a luminance value of a contour portion has sharply changed in the subject focused image based on luminance information of each of the subject focused image and the short-time exposure image. The image processing apparatus according to claim 1, wherein:   6. The brightness change area is an area corresponding to a high brightness object in which a brightness level of light incident on an image sensor of the imaging means is larger than an upper limit of a detectable range. An image processing apparatus according to claim 1. Detecting means for detecting an edge in the subject focused image;
The specifying means is:
7. The subject area is specified by reducing an evaluation value of an edge corresponding to the contour enlargement area estimated by the estimation means among edges detected by the detection means. The image processing apparatus according to any one of the above.   2. The image processing apparatus according to claim 1, further comprising: an image generation unit configured to perform a blurring process on the subject focused image to generate a blurred image in which a background region other than the subject region in the subject focused image is blurred. The image processing device according to any one of? An image processing method using an image processing apparatus,
Processing to obtain a subject in-focus image captured by the imaging device while focusing on a predetermined subject in the background;
Processing for obtaining one image having a composition that is imaged by the imaging device under a predetermined imaging condition different from the imaging condition of the subject focused image and having substantially the same composition as the subject focused image;
Based on the image information of each of the subject in-focus image and the one image, the object in the subject in-focus image corresponds to an object existing outside the depth of field, and the contour portion corresponds to the object in the one image. A process for estimating a contour enlargement region that is enlarged from the contour portion of the corresponding region;
A process of specifying a subject area that includes the predetermined subject in the subject focused image with reference to the estimation result of the contour enlargement region;
An image processing method comprising: The computer of the image processing device
First acquisition means for acquiring a focused subject image captured by the imaging apparatus while focusing on a predetermined subject in the background;
A second acquisition unit configured to acquire an image captured by the imaging apparatus under a predetermined imaging condition different from an imaging condition of the subject focused image and having a composition substantially equal to the subject focused image;
Based on the image information of each of the subject in-focus image and the one image, the object in the subject in-focus image corresponds to an object existing outside the depth of field, and the contour portion corresponds to the object in the one image. An estimation means for estimating a contour expansion region that is enlarged from the contour portion of the corresponding region;
Specifying means for specifying a subject area including the predetermined subject in the subject focused image with reference to the estimation result of the contour enlargement region by the estimating means;
A program characterized by functioning as