JP2011035650A - Image pickup apparatus, image processing method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image pickup apparatus for simply extracting a subject region at one shot by discriminating an image pickup environment suitable for extraction of the subject region in one shot. <P>SOLUTION: The image pickup apparatus 100 includes: a degree-of-non-flatness computing section 7 for computing a degree of non-flatness of image frames of an image for extracting a first subject; a degree-of-non-flatness determining section 8a for determining whether or not the degree of non-flatness of the image frames of the image for extracting the first subject is a predetermined value or lower; a background generating section 8d for generating a background image for extraction on the basis of the image frames of the image for extracting the first subject when it is determined that the degree of non-flatness is the predetermined value or lower; and a cutout image generating section 8f for extracting a subject region from an image including the subject on the basis of difference information of each corresponding pixel between an image including the subject picked up by an electronic image pickup section 2 in accordance with an image pickup instruction from an operation input section 12 and the background image for extraction. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an imaging apparatus, an image processing method, and a program for extracting a subject area from a captured image.

  Conventionally, there is an application that uses an imaging device to capture an image with a subject in the background and a background image without the subject, generate difference information from the background image and the image with the subject, and extract only the subject. (For example, refer to Patent Document 1).

Japanese Patent Laid-Open No. 10-21408

However, as in the above-mentioned Patent Document 1, when an image with a subject in the background and a background image without the subject are shot in two shots, the angle of view easily moves between the two shots. In the process of extracting the subject, there is a problem that a difference occurs in the pixel value of the background itself, and the background portion is erroneously recognized as the subject and the subject cannot be properly extracted.
On the other hand, a technique for clipping a subject from a single photographed image generated by one photographing is also known, but it is difficult to apply depending on the photographing environment.

  Accordingly, an object of the present invention is to provide an imaging apparatus and an image that can determine a shooting environment suitable for extraction of a subject area in one shooting and can easily perform extraction of a subject area in one shooting. A processing method and a program are provided.

In order to solve the above problem, an imaging apparatus according to claim 1 is provided.
Imaging means, generating means for generating an image frame of a subject extraction image for extracting a subject area imaged by the imaging means, and non-flatness of the image frame of the subject extraction image generated by the generating means A first determination unit that determines whether or not the non-flatness of the image frame of the subject extraction image calculated by the calculation unit is equal to or less than a predetermined value; An image frame of the subject extraction image when the non-flatness is determined to be less than or equal to a predetermined value by the first imaging instruction means for instructing imaging of a subject-existing image in which a subject exists. A background generation unit that generates a background image for extraction based on the image, the subject presence image captured by the imaging unit in accordance with an imaging instruction by the first imaging instruction unit, and the Based on the difference information of each pixel corresponding to the output for the background image, it is characterized by comprising, a subject extracting unit for extracting the subject region from the subject-including image.

The invention according to claim 2 is the imaging apparatus according to claim 1,
The generating means images each of a first subject extraction image in which no subject exists in the background and a second subject extraction image in which the subject exists in the background by the imaging means, and generates each image frame. And a second determination means for determining whether or not a difference between corresponding pixels of the image frame of the first subject extraction image and the image frame of the second subject extraction image is equal to or less than a predetermined value. The background generation means generates the background image for extraction based on a pixel whose degree of difference is determined to be equal to or less than a predetermined value by the second determination means.

The invention according to claim 3 is the imaging apparatus according to claim 2,
The background generation unit generates the extraction background image having the same color as a pixel whose degree of difference is determined to be less than or equal to a predetermined value by the second determination unit.

The invention according to claim 4 is the imaging apparatus according to claim 2 or 3,
Imaging conditions for setting the imaging conditions of the first subject extraction image first captured by the imaging unit and the second subject extraction image captured after the first subject extraction image is captured to the same condition It is characterized by further comprising setting means.

The invention according to claim 5 is the imaging apparatus according to claim 4,
The imaging condition setting means is further characterized in that after the second subject extraction image is captured, an imaging condition for capturing the subject existing image by the imaging means is set.

Invention of Claim 6 is an imaging device as described in any one of Claims 1-5,
First extraction information generating means for generating extraction information for extracting the subject region from the subject existing image based on difference information of corresponding pixels between the subject existing image and the extraction background image; Further, the subject extraction means extracts the subject region from the subject existing image using the extraction information generated by the first extraction information generation means.

The invention according to claim 7 is the imaging apparatus according to any one of claims 1 to 6,
The calculation means extracts a plurality of feature points of the subject extraction image as the non-flatness, and the first determination means further includes the plurality of feature points extracted by the calculation means being a predetermined value or less. The background generation means further generates the background image for extraction when the first determination means determines that the feature point is equal to or less than a predetermined value. Yes.

The invention according to claim 8 is the imaging apparatus according to any one of claims 1 to 7,
The calculation means calculates, as the non-flatness, a variation amount of pixel values in a plurality of image regions of the subject extraction image, and the first determination means further includes the plurality of the calculation values calculated by the calculation means. It is determined whether or not the variation amount of the pixel value in the image area is equal to or less than a predetermined value, and the background generation unit further determines that the variation amount of the pixel value is equal to or less than the predetermined value by the first determination unit. In such a case, the extraction background image is generated.

The invention according to claim 9 is the imaging apparatus according to any one of claims 1 to 8,
The image pickup means picks up the subject existing image when the first determination means determines that the non-flatness of the subject extraction image is a predetermined value or less.

The invention according to claim 10 is the imaging apparatus according to any one of claims 1 to 9,
When the first determination unit determines that the non-flatness is not equal to or less than a predetermined value, the imaging unit instructs the imaging unit to capture a subject nonexistent image in the same background as the background of the subject existing image. A second imaging instruction means, wherein the subject extraction means is a difference between corresponding pixels of the subject non-existence image captured by the imaging means and the subject presence image in accordance with an imaging instruction by the second imaging instruction means; The subject area is extracted from the subject existing image based on the information.

The invention according to claim 11 is the imaging apparatus according to claim 10,
Second extraction information generating means for generating extraction information for extracting the subject region from the subject presence image based on difference information of corresponding pixels between the subject non-existence image and the subject presence image. Further, the subject extraction means is characterized in that the subject area is extracted from the subject existing image using the extraction information generated by the second extraction information generation means.

An image processing method according to a twelfth aspect of the present invention includes:
A subject extraction image for extracting a subject area imaged by the imaging means in an imaging apparatus comprising an imaging means and an imaging instruction means for instructing imaging of a subject existing image in which a subject is present in the background by the imaging means A generation step for generating the image frame, a calculation step for calculating the non-flatness of the image frame of the subject extraction image generated by the generation step, and an image of the subject extraction image calculated by the calculation step A determination step for determining whether or not the non-flatness of the frame is equal to or less than a predetermined value, and when it is determined by this determination step that the non-flatness is equal to or less than a predetermined value, the image frame of the subject extraction image A background generation step of generating a background image for extraction based on the image, and imaging by the imaging unit according to an imaging instruction by the imaging instruction unit Based on the difference information of each pixel corresponding to the subject-image and the extracted background image, it is characterized in that to execute the steps of extracting the subject region from the subject-including image.

The program of the invention described in claim 13 is:
A computer of an imaging apparatus comprising imaging means and imaging instruction means for instructing imaging of a subject existing image in which a subject is present in the background by the imaging means, subject extraction for extracting a subject area imaged by the imaging means Generating means for generating an image frame of the image for calculation, calculating means for calculating the non-flatness of the image frame of the image for subject extraction generated by the generating means, and the image of the image for subject extraction calculated by the calculating means A determination unit that determines whether or not the non-flatness of the frame is equal to or less than a predetermined value, and when the non-flatness is determined to be equal to or less than the predetermined value by the determination unit, based on the image frame of the subject extraction image Background generation means for generating a background image for extraction, and the subject existing image imaged by the imaging means in accordance with an imaging instruction by the imaging instruction means The extraction based on the corresponding differential information of each pixel of the background image, is characterized in that to function as the subject extracting unit for extracting the subject region from the subject-including image and.

  According to the present invention, it is possible to determine a shooting environment suitable for extracting a subject area in one shooting, and it is possible to easily extract a subject area in one shooting in this shooting environment.

It is a block diagram which shows schematic structure of the imaging device of Embodiment 1 to which this invention is applied. 3 is a flowchart illustrating an example of an operation related to subject clipping processing by the imaging apparatus of FIG. 1. FIG. 3 is a flowchart showing a continuation of the subject clipping process in FIG. 2. FIG. 3 is a flowchart illustrating an example of an operation related to a background generation process in the subject clipping process of FIG. 2. 3 is a flowchart illustrating an example of an operation related to a first area detection process in the subject clipping process of FIG. 2. It is a figure which shows typically an example of the image for demonstrating the object clipping process of FIG. It is a figure which shows typically an example of the image for demonstrating the object clipping process of FIG. It is a block diagram which shows schematic structure of the imaging device of Embodiment 2 to which this invention is applied. 9 is a flowchart illustrating an example of an operation related to a subject clipping process performed by the imaging apparatus in FIG. 8. 10 is a flowchart showing a continuation of the subject clipping process in FIG. 9. 10 is a flowchart illustrating an example of an operation related to a second area detection process in the subject clipping process of FIG. 9. It is a figure which shows typically an example of the image for demonstrating the object clipping process of FIG. 10 is a flowchart illustrating an example of an operation related to a first area detection process according to Modification 1;

  Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.

[Embodiment 1]
When it is determined that the non-flatness of the image frame of the first subject extraction image P2a is equal to or less than a predetermined value, the imaging apparatus 100 according to the first embodiment is based on the image frame of the first subject extraction image P2a. An extraction background image P3 is generated, and a subject region is extracted from the subject presence image P1a based on difference information between corresponding pixels of the extraction background image P3 and the subject presence image P1a.

FIG. 1 is a block diagram illustrating a schematic configuration of an imaging apparatus 100 according to the first embodiment to which the present invention is applied.
As illustrated in FIG. 1, the imaging apparatus 100 includes a lens unit 1, an electronic imaging unit 2, an imaging control unit 3, an image data generation unit 4, an image memory 5, a resolution conversion unit 6, and non-flatness. A calculation unit 7, an image processing unit 8, a recording medium 9, a display control unit 10, a display unit 11, an operation input unit 12, and a CPU 13 are provided.
Further, the imaging control unit 3, the non-flatness calculation unit 7, the image processing unit 8, and the CPU 13 are designed as, for example, a custom LSI 1A.

The lens unit 1 includes a plurality of lenses and includes a zoom lens, a focus lens, and the like.
Although not shown, the lens unit 1 includes a zoom drive unit that moves the zoom lens in the optical axis direction when the subject S is imaged, a focus drive unit that moves the focus lens in the optical axis direction, and the like. May be.

  The electronic imaging unit 2 is composed of an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-oxide Semiconductor), for example, and converts an optical image that has passed through various lenses of the lens unit 1 into a two-dimensional image signal. To do.

  The lens unit 1 and the electronic imaging unit 2 configured in this way constitute imaging means.

Although not shown, the imaging control unit 3 includes a timing generator, a driver, and the like. Then, the imaging control unit 3 scans and drives the electronic imaging unit 2 with a timing generator and a driver, converts the optical image into a two-dimensional image signal with the electronic imaging unit 2 every predetermined period, and the electronic imaging unit 2 Image frames are read out from the imaging area for each screen and output to the image data generation unit 4.
When live view image display is performed at the time of shooting, the imaging control unit 3 causes the electronic imaging unit 2 to continuously capture the subject S at a predetermined imaging frame rate, and each image frame from the electronic imaging unit 2. The image data generation unit 4 sequentially outputs the data.

Further, the imaging control unit 3 performs adjustment control of imaging conditions by the lens unit 1 and the electronic imaging unit 2 as imaging condition setting means. For example, the imaging control unit 3 moves the lens unit 1 in the optical axis direction to adjust the focusing condition (automatic focusing process), and adjusts the exposure condition (shutter speed, aperture, amplification factor, etc.). (Automatic exposure processing), AWB (automatic white balance) for adjusting white balance, and the like are performed.
Specifically, in the non-flatness determination process (described later) of the live view image of the first subject extraction image P2a, it is determined that the variation amount of the feature point and the pixel value in the feature point is equal to or less than a predetermined value. After that, when the first subject extraction image P2a (see FIG. 6B) is captured, the imaging control unit 3 fixes the imaging conditions when capturing the first subject extraction image P2a. Maintain the state.
Then, based on the imaging instruction operation of the shutter button 12a by the user, the imaging control unit 3 uses the subject presence image P1a (see FIG. 5) that has passed through the lens unit 1 under the imaging conditions fixed after the first subject extraction image P2a is captured. 7 (a)) is converted into a two-dimensional image signal by the electronic imaging unit 2, and the second subject extraction image P2b (FIG. 6C) is converted from the imaging region of the electronic imaging unit 2. ))) Is read out. That is, the imaging control unit 3 sets the imaging conditions of the first subject extraction image P2a and the second subject extraction image P2b captured as live view images by the lens unit 1 and the electronic imaging unit 2 to the same condition. .

Further, the imaging control unit 3 adjusts the imaging condition of the subject existing image P1a captured by the lens unit 1 and the electronic imaging unit 2 in accordance with the subject S after imaging the image frame of the second subject extraction image P2b. Thereafter, the imaging control unit 3 causes the electronic imaging unit 2 to convert the optical image of the subject existing image P1a that has passed through the lens unit 1 into a two-dimensional image signal, and extracts the second subject from the imaging region of the electronic imaging unit 2. The image frame related to the image P2b is read out.
Note that when taking the subject presence image P1a, considering that it takes time to adjust the lens position of the lens unit 1, for example, AE (automatic exposure processing) is performed without performing AF (automatic focusing processing). ) Or AWB (automatic white balance).

The image data generation unit 4 appropriately adjusts the gain for each RGB color component with respect to the analog value signal of the image frame transferred from the electronic imaging unit 2, and then performs sample holding by a sample hold circuit (not shown). The digital signal is converted into digital data by a / D converter (not shown), color processing including pixel interpolation processing and γ correction processing is performed by a color process circuit (not shown), and then a digital luminance signal Y and color difference signal Cb , Cr (YUV data).
The luminance signal Y and the color difference signals Cb and Cr output from the color process circuit are DMA-transferred to an image memory 5 used as a buffer memory via a DMA controller (not shown).

  A demosaic unit (not shown) for developing the digital data after A / D conversion may be mounted on the custom LSI 1A.

  The image memory 5 is constituted by, for example, a DRAM or the like, and temporarily stores data processed by the resolution conversion unit 6, the non-flatness calculation unit 7, the image processing unit 8, the CPU 13, and the like.

The resolution conversion unit 6 acquires YUV data of an image stored in the image memory 5 and performs resolution conversion processing on the YUV data. Specifically, the resolution conversion unit 6 continuously captures images at a predetermined imaging frame rate by the electronic imaging unit 2 and displays the image frames generated by the image data generation unit 4 when performing live view image display during shooting. Based on YUV data (for example, 3200 × 2400 pixels), YUV data of an image frame for live view image display (for example, 320 × 240 pixels) is sequentially generated.
That is, the resolution conversion unit 6 generates the subject extraction images generated by the image data generation unit 4 (first subject extraction image P2a and second subject extraction image P2b; see FIGS. 6B and 6C). ) YUV data of an image frame (for example, a first live view image, a second live view image, etc.) for displaying a live view image of the subject extraction image is sequentially generated.
Here, the resolution conversion unit 6 constitutes a generation unit that generates an image frame of a subject extraction image captured by the lens unit 1 and the electronic imaging unit 2.

  The non-flatness calculation unit 7 calculates non-flatness of the image frame of the first subject extraction image P2a as calculation means. Specifically, the non-flatness calculation unit 7 includes a feature amount calculation unit 7a and a variation amount calculation unit 7b.

The feature amount calculator 7a performs feature extraction processing for extracting feature points from the image frame of the first subject extraction image P2a as the non-flatness.
Specifically, the feature amount calculation unit 7a generates a high-frequency component based on an image frame (first live view image) for displaying a live view image of the first subject extraction image P2a generated by the resolution conversion unit 6. A block region having a large number of features (for example, a square of 16 × 16 pixels) is extracted as a feature point.

なお、上記式（１）において、ｂ：各ブロック領域の画素値としては、例えば、輝度値が挙げられる。 The variation amount calculation unit 7b calculates the variation amount of the pixel value in the feature point as the non-flatness.
The variation amount calculation unit 7b calculates the pixel value in the feature point (block area) of the image frame (first live view image) for displaying the live view image of the first subject extraction image P2a generated by the resolution conversion unit 6. As the variation amount, for example, a standard deviation is calculated according to the following formula (1).

In the above formula (1), b: a pixel value of each block area includes, for example, a luminance value.

  As described above, the feature amount calculation unit 7a and the variation amount calculation unit 7b constitute a calculation unit that calculates the non-flatness of the image frame of the first subject extraction image P2a.

The image processing unit 8 includes a non-flatness determination unit 8a that determines the non-flatness calculated in the non-flatness calculation process.
The non-flatness determination unit 8a is a feature point of a live view image display image frame (first live view image) of the first subject extraction image P2a generated by the resolution conversion unit 6, that is, a feature amount calculation unit 7a. It is determined whether or not the block region having a high feature of the first live view image extracted by the above is equal to or less than a predetermined value. Further, the non-flatness determination unit 8a determines whether or not the variation amount of the pixel value in the feature point of the first live view image calculated by the variation amount calculation unit 7b is equal to or less than a predetermined value.
Here, the non-flatness determination unit 8a constitutes first determination means for determining whether or not the non-flatness of the image frame of the subject extraction image is equal to or less than a predetermined value.

なお、上記式（２）にあっては、第１ライブビュー画像若しくは抽出用背景画像Ｐ３のＹＵＶデータを「Y」、「U」、「V」で表し、第２ライブビュー画像若しくは被写体存在画像Ｐ１ａのＹＵＶデータを「Yc」、「Uc」、「Vc」で表す。また、Ｇは、色差信号U、Vのゲインを表している。
また、上記した相違度Ｄの算出式（２）は、一例であってこれに限られるものではない。 Further, the image processing unit 8 includes a dissimilarity calculating unit 8b that calculates the dissimilarity between corresponding pixels of a plurality of images.
The dissimilarity calculation unit 8b is configured to display an image frame (first live view image) for displaying a live view image of the first subject extraction image P2a generated by the resolution conversion unit 6 and a live view image of the second subject extraction image P2b. The difference D of each pixel corresponding to the image frame for display (second live view image) is calculated according to the following equation (2). Further, the difference calculation unit 8b calculates the difference D of each corresponding pixel between the extraction background image P3 generated by the background generation process and the subject existing image P1a according to the following equation (2).

In the above formula (2), YUV data of the first live view image or the extraction background image P3 is represented by “Y”, “U”, “V”, and the second live view image or the subject existing image. The YUV data of P1a is represented by “Yc”, “Uc”, “Vc”. G represents the gain of the color difference signals U and V.
Further, the calculation formula (2) for the degree of difference D described above is an example, and the present invention is not limited to this.

In addition, the image processing unit 8 includes a dissimilarity determination unit 8c that determines the dissimilarity D between corresponding pixels of the plurality of images calculated by the dissimilarity calculation unit 8b.
The difference determination unit 8c determines whether or not the difference D of each pixel corresponding to the first live view image and the second live view image calculated by the difference calculation unit 8b is equal to or less than a predetermined value.
Here, the dissimilarity determination unit 8c determines whether or not the dissimilarity D of each corresponding pixel between the image frame of the first subject extraction image P2a and the image frame of the second subject extraction image P2b is equal to or less than a predetermined value. The second determination means for determining the above is configured.

In addition, the image processing unit 8 includes a background generation unit 8d that generates an extraction background image P3 for extracting a subject area using the chroma key technique.
In the background generation unit 8d, the non-flatness determination unit 8a determines that the feature point of the first live view image is equal to or less than a predetermined value and the variation amount of the pixel value in the feature point is equal to or less than the predetermined value. In this case, an extraction background image P3 is generated based on the first subject extraction image P2a. Specifically, when the difference determination unit 8c determines that the difference D of each corresponding pixel between the first live view image and the second live view image is equal to or less than a predetermined value, the background generation unit 8d Then, an extraction background image P3 having the same color as that of the pixel for which the degree of difference D is determined to be equal to or less than a predetermined value is generated. For example, as shown in FIG. 6B, when the subject S is photographed against a plain background with a small feature amount, the feature amount of the first subject extraction image P2a decreases, and the first subject extraction image In a state where P2a is overlapped with the second subject extraction image P2b, an extraction background image P3 having the same color as the pixels in the region where the subject S does not overlap is generated.
Note that the chroma key is a method of cutting out the subject S from one image data using a specific color background. In the chroma key, a blue or green screen having a complementary color relationship with the subject S is usually used as the background. Therefore, by generating the extraction background image P3 having the same color as the pixel whose difference between the first subject extraction image P2a and the second subject extraction image P2b is a predetermined value or less, the extraction background image P3 and the subject exist. The background portion and the subject S portion can be separated based on the color information of the image P1a.

  Here, when the non-flatness determination unit 8a determines that the non-flatness of the first subject extraction image P2a is equal to or less than a predetermined value, the background generation unit 8d performs the image frame of the first subject extraction image P2a. The background generation means for generating the extraction background image P3 based on the above is configured.

The image processing unit 8 also includes a mask generation unit 8e that generates a mask image M for extracting a subject area from the subject presence image P1a.
The mask generation unit 8e generates a dissimilarity map based on the dissimilarity D of each corresponding pixel between the extraction background image P3 and the subject presence image P1a calculated by the dissimilarity calculation unit 8b. Then, the mask generation unit 8e generates a mask image M by binarizing (0, 255) the generated difference map with a predetermined threshold.

Further, the mask generation unit 8e performs a contraction process for removing fine noise to remove a pixel set smaller than a predetermined value, and then performs a expansion process for correcting the contraction. By performing a labeling process for assigning the same numbers to the pixel sets to be configured, a hole area is also filled by replacing an area having a predetermined ratio or less in the number of constituent pixels of the effective area with the effective area. Further, the mask generation unit 8e applies an averaging filter to the region information to add a composite gradation to the edge of the region.
Here, the mask generation unit 8e extracts a subject area from the subject existing image P1a based on the degree of difference (difference information) D between corresponding pixels of the extraction background image P3 and the subject existing image P1a. First extraction information generating means for generating an image (extraction information) M is configured.

The image processing unit 8 combines the image of the subject S with a predetermined single color background image P5 (see FIG. 7D) to generate image data of the subject cutout image P4 (see FIG. 7D). A cut-out image generation unit 8f is provided.
The cut-out image generation unit 8f uses α blend to cut out the subject area from the subject existing image P1a using the mask image M generated by the mask generation unit 8e, and combines it with the single color background image P5 to cut out the subject. Image data of the image P4 is generated. In addition, since the synthetic gradation is given to the edge part of the mask image M, it can be combined with a natural feeling in which the boundary part between the clipped subject area and the single color background image P5 is not clear. .
Here, the cut-out image generation unit 8f constitutes subject extraction means for extracting a subject region from the subject existing image P1a based on the difference information of each corresponding pixel between the extraction background image P3 and the subject existing image P1a. .

The recording medium 9 is composed of, for example, a nonvolatile memory (flash memory) or the like, and stores image data for recording a captured image encoded by a JPEG compression unit (not shown) of the image processing unit 8.
In addition, the recording medium 9 compresses the mask image M generated by the mask generation unit 8e of the image processing unit 8 and the image data of the subject cutout image P4, and associates them with each other to obtain image data of the subject cutout image P4. Save the extension as “.jpe”.

The display control unit 10 performs control to read display image data temporarily stored in the image memory 5 and display the display image data on the display unit 11.
Specifically, the display control unit 10 includes a VRAM, a VRAM controller, a digital video encoder, and the like. The digital video encoder periodically reads the luminance signal Y and the color difference signals Cb and Cr read from the image memory 5 and stored in the VRAM (not shown) under the control of the CPU 13 from the VRAM via the VRAM controller. Are read out, a video signal is generated based on these data, and is output to the display unit 11.

  The display unit 11 is, for example, a liquid crystal display device, and displays an image captured by the electronic imaging unit 2 based on a video signal from the display control unit 10 on a display screen. Specifically, the display unit 11 displays a live view image based on a plurality of image frames generated by imaging the subject S by the lens unit 1 and the electronic imaging unit 2 in the imaging mode, The REC view image picked up as is displayed.

The operation input unit 12 is for performing a predetermined operation of the imaging apparatus 100. Specifically, the operation input unit 12 includes a shutter button 12a related to an instruction to shoot the subject S, a mode button 12b related to an instruction to select an imaging mode or a function on a menu screen, and a zoom button ( (Not shown).
When a predetermined button of the operation input unit 12 is operated by the user, a control instruction corresponding to the button is output to the CPU 13. When a control instruction is input, the CPU 13 performs a process of controlling each unit according to the operated button.

  Further, the shutter button 12a as a first imaging instruction means outputs an imaging instruction of the subject existing image P1a by the electronic imaging unit 2 to the CPU 13 when a predetermined operation is performed by the user.

  The CPU 13 controls each part of the imaging device 100. Specifically, the CPU 13 performs various control operations in accordance with various processing programs (not shown) for the imaging apparatus 100.

Next, the subject clipping process according to the image processing method performed by the imaging apparatus 100 will be described with reference to FIGS.
2 and 3 are flowcharts showing an example of an operation related to the subject clipping process.

The subject clipping process is controlled by the CPU 13 when the subject clipping mode is instructed from a plurality of imaging modes displayed on the menu screen based on a predetermined operation of the mode button 12b of the operation input unit 12 by the user. This process is executed by each unit of the imaging apparatus 100.
As shown in FIG. 2, first, the display control unit 10 displays a live view image based on the image frames sequentially generated by the resolution conversion unit 6 in accordance with the imaging of the subject S by the lens unit 1 and the electronic imaging unit 2. And a background instruction message (for example, “Please turn to the background”) is displayed on the display screen of the display unit 11 (step S1; FIG. 6A). )reference).

Then, the display control unit 10 updates the live view image based on the image frame of the first subject extraction image P2a sequentially generated by the resolution conversion unit 6 by imaging the subject S by the lens unit 1 and the electronic imaging unit 2. Control is performed (step S2). Each time the live view image is updated, the feature amount calculation unit 7a of the non-flatness calculation unit 7 performs YUV of the image frame (first live view image) for displaying the live view image of the first subject extraction image P2a. Based on the data, a block region (feature point) having a high feature is extracted, and the variation amount calculation unit 7b calculates a standard deviation as a variation amount in each block region according to the equation (1) (step S3).

Next, the non-flatness determination unit 8a determines whether or not a feature point (block region) having a high feature extracted by the feature amount calculation unit 7a is equal to or less than a predetermined value, and the block calculated by the variation amount calculation unit 7b. By determining whether or not the variation amount of the pixel value in the region is equal to or less than a predetermined value, it is determined whether or not there is a variation amount of the feature point and the pixel value in the block region (step S4).
Here, when it is determined that there is a variation amount between the feature point and the pixel value in the block area (step S4; NO), the CPU 13 receives an end instruction based on a predetermined operation of the operation input unit 12 by the user. (Step S5).

If it is determined in step S5 that an end instruction has not been input (step S5; NO), the CPU 13 proceeds to step S2 and executes the subsequent processes.
On the other hand, if it is determined that an end instruction has been input (step S5; YES), the CPU 13 ends the subject clipping process.

If it is determined in step S4 that there is no variation amount between the feature point and the pixel value in the block area (step S4; YES), for example, the subject S with a plain background having a small feature amount as a background. (See FIG. 6B), the imaging control unit 3 causes the electronic imaging unit 2 to capture an optical image of the first subject extraction image P2a in which the subject S does not exist in the background under a predetermined imaging condition. Based on the YUV data of the first subject extraction image P2a generated by the image data generation unit 4 by the imaging, the resolution conversion unit 6 displays an image frame (for displaying a live view image of the first subject extraction image P2a). A first live view image) is generated and stored in the image memory 5 (step S6).
Thereafter, the imaging control unit 3 maintains a state in which imaging conditions such as a focus position (AF), an exposure condition (AE), and a white balance (AWB) at the time of imaging the first subject extraction image P2a are fixed. (Step S7).

Next, the display control unit 10 superimposes it on the live view image and displays an imaging instruction message (for example, “Please shoot subject S”) on the display screen of the display unit 11 (step S8; FIG. 6 ( b)).
Thereafter, the display control unit 10 performs control to update the live view image based on the image frames sequentially generated by the resolution conversion unit 6 in accordance with the imaging of the subject S by the lens unit 1 and the electronic imaging unit 2 (step S9). When the live view image is updated, the CPU 13 determines whether the user has operated the shutter button 12a of the operation input unit 12 for a predetermined operation to input an imaging instruction (step S10).

If it is determined in step S10 that an imaging instruction has not been input (step S10; NO), it is determined whether an end instruction has been input based on a predetermined operation of the operation input unit 12 by the user (step S11). ).
If it is determined that the end instruction has not been input (step S11; NO), the CPU 13 shifts the process to step S9 and executes the subsequent processes.
On the other hand, if it is determined that an end instruction has been input (step S11; YES), the CPU 13 ends the subject clipping process.

  Thereafter, the user moves the subject S within the angle of view or waits for the subject S to move (see FIG. 6C), and then determines in step S10 that an imaging instruction has been input ( (Step S10; YES), the imaging control unit 3 causes the electronic imaging unit 2 to capture an optical image of the second subject extraction image P2b in which the subject S is present in the background under predetermined imaging conditions, and the resolution conversion unit 6 Based on the YUV data of the second subject extraction image P2b generated by the image data generation unit 4 by the imaging, an image frame (second live view image) for displaying the live view image of the second subject extraction image P2b is obtained. Generated and stored in the image memory 5 (step S12).

As shown in FIG. 3, the imaging control unit 3 continues to set the imaging conditions such as the exposure condition (AE) and white balance (AWB) when imaging the subject existing image P1a (see FIG. 7A). Adjust according to S (step S13). FIG. 7A schematically shows a state in which the photometric area frame W is aligned with the face position of the subject S.
Thereafter, the imaging control unit 3 causes the electronic imaging unit 2 to capture an optical image of the subject existing image P1a under the adjusted imaging conditions, and based on the image frame of the subject existing image P1a transferred from the electronic imaging unit 2, The YUV data of the existing image P1a is generated by the image data generation unit 4, and the YUV data is temporarily stored in the image memory 5 (step S14).
As a result, a subject existing image P1a in which the subject S is photographed against a background with a small feature amount is generated.

  Next, the CPU 13 causes the non-flatness calculation unit 7 and the image processing unit 8 to perform background generation processing for generating an extraction background image P3 for extracting a subject area from the subject existing image P1a (step S15).

Here, the background generation processing will be described in detail with reference to FIG.
FIG. 4 is a flowchart illustrating an example of an operation related to the background generation process.
As shown in FIG. 4, the dissimilarity calculation unit 8 b of the image processing unit 8 calculates the dissimilarity D of each pixel corresponding to the first live view image and the second live view image stored in the image memory 5 as follows. It calculates according to Formula (2) (step S31).

Next, the dissimilarity determination unit 8c of the image processing unit 8 has the dissimilarity D of each pixel corresponding to the first live view image and the second live view image calculated by the dissimilarity calculation unit 8b equal to or less than a predetermined value. It is determined whether or not there is, and a set of pixels whose dissimilarity D is equal to or less than a predetermined value is specified as a background region (step S32).
Subsequently, the feature amount calculation unit 7a of the non-flatness calculation unit 7 applies the background region specified by the difference determination unit 8c to the image frame of the first subject extraction image P2a, and then performs the first subject extraction use. Based on the image frame of the image P2a, a block area having a high frequency component and a high feature (for example, a square of 16 × 16 pixels) is extracted as a feature point, and the variation amount calculation unit 7b is extracted by the feature amount calculation unit 7a. The standard deviation is calculated according to the following formula (1) as the variation amount of the pixel value in the feature point (block region) (step S33).

Next, the non-flatness determination unit 8a evaluates the feature points extracted by the feature amount calculation unit 7a and the feature points calculated by the variation amount calculation unit 7b (step S34). Specifically, the non-flatness determination unit 8a determines whether or not the variation amount of the pixel value in each feature point (block region) of the first subject extraction image P2a is equal to or less than a predetermined value, and the determination As a result, the block areas in which the variation amount of the pixel value in the block area is determined to be equal to or smaller than the predetermined value are counted to determine whether there are a plurality of block areas. Further, the non-flatness determination unit 8a determines that the variation amount of the pixel value is equal to or less than a predetermined value, and counts the block regions of the same color among the block regions that are regarded as having no variation.
Then, the non-flatness determination unit 8a determines whether there is no variation and the number of block areas of the same color is equal to or greater than a predetermined number (for example, half the number of blocks of the entire image) (step S35).
Here, when it is determined that there is no variation and there are a predetermined number or more of block areas of the same color (step S35; YES), the background generation unit 8d has the same color as the block area of the first subject extraction image P2a. YUV data of the extraction background image P3 having the color as the background color is generated (step S36), and the background generation is successful (step S37).

On the other hand, if it is determined in step S35 that there is no variation and the number of block regions of the same color is less than the predetermined number (step S35; NO), for example, even if there is a stable block region without variation, the same color If there is no block area, the background cannot be specified. Therefore, the extraction background image P3 is not generated, and background generation fails (step S38).
Thereby, the background generation process is terminated.

As illustrated in FIG. 3, the background generation unit 8d then determines whether background generation has been successful (step S16).
If it is determined that the background generation has succeeded (step S16; YES), the CPU 13 causes the image processing unit 8 to perform a first area detection process for detecting a subject area including the subject S from the subject existing image P1a. (Step S17).

Here, the first region detection process will be described in detail with reference to FIG.
FIG. 5 is a flowchart illustrating an example of an operation related to the first region detection process.
As illustrated in FIG. 5, the dissimilarity calculation unit 8b of the image processing unit 8 enlarges (resizes) the extraction background image P3 generated in the background generation process according to the actual imaging size of the subject existing image P1a. (Step S50), the difference D of each corresponding pixel between the enlarged extraction background image P3 and the subject presence image P1a is calculated according to the following equation (2), and the mask generation unit 8e of the image processing unit 8 calculates A difference map is generated based on the difference D that has been made (step S51).

Next, the mask generation unit 8e binarizes the generated difference map with a predetermined threshold value to generate a mask image (step S52). Then, the mask generation unit 8e determines whether or not the binarization is successful by determining whether or not the background (binarization; 0) region is extremely small, for example (step S53).
Here, if it is determined that the binarization has succeeded (step S53; YES), the mask generation unit 8e corrects the remaining part of the periphery of the subject region and removes fine noise, so that the mask image M is removed. After performing contraction processing on the image data to remove a pixel set smaller than a predetermined value (step S54), expansion processing for correcting the contraction is performed (step S55).

Subsequently, when the subject S has a color similar to the background color, the subject region of the mask image M is lost, and thus the mask generation unit 8e performs labeling that assigns the same number to a set of pixels that constitute the same connected component. Filling is performed by replacing the area below a predetermined ratio in the number of constituent pixels of the effective area of the image data of the mask image M with the effective area by the processing (step S56).
Then, the mask generation unit 8e applies an averaging filter to the image data of the mask image M, adds a synthetic gradation to the edge of the subject region (step S57), and sets the region detection success (step S58). .

On the other hand, if it is determined in step S53 that the binarization has not been successful (step S53; NO), for example, if the background (binarization; 0) region is extremely small, the mask generation unit 8e. Is assumed to have failed in binarization, and region detection fails (step S59).
Thus, the first area detection process is terminated.

As shown in FIG. 3, next, the mask generation unit 8e determines whether or not the detection of the subject area has succeeded (step S18).
Here, if it is determined that the detection of the subject region has succeeded (step S18; YES), the cut-out image generation unit 8f uses the α blend to generate the mask image M generated by the first region detection process. Using this, a subject region is cut out from the subject existing image P1a, and is combined with a predetermined single color background image P5 to generate image data of the subject cutout image P4 (see FIG. 7B) (step S19).
Specifically, the cut-out image generation unit 8f reads out the image data of the subject existing image P1a, the single color background image P5, and the mask image M and develops the image data in the image memory 5, and then the mask image M of the subject existing image P1a. The pixels covered by the filled portion of the image data (hatched portion other than the subject S) are filled with a predetermined single color of the single-color background image P5, while the pixels of the subject S portion are predetermined without doing anything. Do not transmit light to a single color. In addition, since the synthetic gradation is given to the edge part of the mask image M, it becomes a natural feeling that the boundary part between the cut out subject area and the single color background image P5 is not clear.
Thereafter, the display control unit 10 displays a subject cutout image P4 in which the subject S is superimposed on a predetermined single color background image P5 based on the image data of the subject cutout image P4 generated by the cutout image generation unit 8f. Is displayed on the display screen (see FIG. 7C).

Next, the CPU 13 associates the image data of the mask image M generated by the mask generation unit 8e of the image processing unit 8 with the image data of the subject cutout image P4 in a predetermined storage area of the recording medium 9, and The extension of the image data of the cutout image P4 is saved as one file with “.jpe” (step S20).
Thereby, the subject clipping process is completed.

On the other hand, when it is determined in step S18 that the detection of the subject area has not been successful (step S18; NO), the CPU 13 determines a predetermined message (for example, “To cut out the subject S” ”And the like” are displayed on the display screen of the display unit 11 (step S21), and the subject clipping process is terminated.
Also, when it is determined in step S16 that the background generation has not been successful (step S16; NO), the CPU 13 shifts the process to step S21 and performs a predetermined message relating to the subject S clipping failure. Is displayed on the display screen of the display unit 11 (step S21), and the subject clipping process is terminated.

As described above, according to the imaging apparatus 100 of the present embodiment, the image frame of the first subject extraction image P2a in which the subject S does not exist in the background is generated, and the image frame of the first subject extraction image P2a is generated. Calculate the non-flatness of. Then, depending on whether or not the non-flatness of the image frame of the first subject extraction image P2a is equal to or less than a predetermined value, it is determined whether or not the shooting environment is suitable for extraction of the subject region in one shooting. Can be determined.
Specifically, a plurality of feature points are extracted from the live view image display image frame of the first subject extraction image P2a, and it is determined whether or not the extracted feature points are equal to or less than a predetermined value. In addition, the amount of variation in pixel values in a plurality of feature points (block regions) is calculated, and it is determined whether or not the amount of variation in pixel values in the plurality of feature points is equal to or less than a predetermined value. When it is determined that the feature point is equal to or smaller than the predetermined value and the variation amount of the pixel value within the characteristic point is determined to be equal to or smaller than the predetermined value, for example, the background ( For example, it can be determined that this is an environment in which the subject S is photographed with a uniform background) and is suitable for extraction of the subject region in one photographing.
If it is determined that the non-flatness of the image frame for displaying the live view image of the first subject extraction image P2a is equal to or less than a predetermined value, the extraction background image P3 is based on the image frame of the first subject extraction image P2a. Is generated. Specifically, the degree of difference D between the corresponding pixels of each image frame of the first subject extraction image P2a where the subject S does not exist in the background and the second subject extraction image P2a where the subject S exists in the background. An extraction background image P3 having the same color as the pixel determined to be equal to or less than a predetermined value is generated. Then, a mask image M is generated based on the difference D between the corresponding pixels of the subject presence image P1a captured by the electronic imaging unit 2 and the extraction background image P3 in accordance with the imaging instruction by the shutter button 12a, and the mask A subject region is extracted from the subject presence image P1a using the image M.
Therefore, it is possible to generate the extraction background image P3 by performing only one shooting of the subject presence image P1a without separately capturing a background image for extracting the subject region. The subject region can be easily extracted from the subject presence image P1a using the mask image M generated by using the mask image M.

In addition, since the imaging conditions of the first subject extraction image P2a captured first and the second subject extraction image P2b captured after capturing the first subject extraction image P2a are set to the same condition, An extraction background image P3 suitable for extracting a subject area can be calculated appropriately by calculating the difference D between the corresponding pixels of the image frames of the one subject extraction image P2a and the second subject extraction image P2b. Can be generated.
Then, after the second subject extraction image P2b is captured, the subject presence image P1a that is the source of the subject region can be appropriately captured by resetting the imaging conditions for capturing the subject presence image P1a. Thus, a higher-quality subject cutout image P4 can be obtained.

[Embodiment 2]
Below, the imaging device 200 of Embodiment 2 is demonstrated with reference to FIGS.
The imaging apparatus 200 according to the second embodiment has the same background as the background of the subject presence image P1b (see FIG. 12A) when it is determined that the non-flatness of the first subject extraction image P2a is not less than a predetermined value. The subject nonexistent image P6 (see FIG. 12B) in which the subject S does not exist is imaged, and the subject exists based on the difference information of each corresponding pixel between the subject nonexistent image P6 and the subject present image P1b. A subject area is extracted from the image P1b.
Note that the imaging apparatus 200 according to the second embodiment has substantially the same configuration as the imaging apparatus 100 according to the first embodiment except for the points described below, and the description thereof is omitted.

  The feature amount calculation unit 7a of the non-flatness calculation unit 7 performs feature points from the subject non-existence image P6 on the basis of the subject non-existence image P6 in the alignment processing of the subject presence image P1b and the subject non-existence image P6. The process which extracts is performed. Specifically, the feature amount calculation unit 7a, based on the YUV data of the subject non-existing image P6, has a predetermined number (or a predetermined number or more) of high feature block regions (or more than a predetermined number) that are convenient for tracking from a large number of candidate blocks. The feature point is selected, and the contents of the block area are extracted as a template (for example, a square of 16 × 16 pixels).

Further, the imaging apparatus 200 according to the second embodiment includes a block matching unit 14 that performs a block matching process for aligning the subject non-existing image P6 and the subject existing image P1b.
For example, the block matching unit 14 uses the subject non-existing image P6 as a reference image and performs alignment with the subject existing image P1b as a target image. Specifically, the block matching unit 14 determines where in the subject existing image P1b the template (for example, a square of 16 × 16 pixels) extracted by the feature extraction process corresponds, that is, in the subject existing image P1b. Thus, a position (corresponding region) where the pixel value of the template is optimally matched is searched. Then, an optimum offset between the subject non-existing image P6 and the subject existing image P1b having the best evaluation value (for example, sum of squared differences (SSD), sum of absolute differences (SAD), etc.) of the pixel value is calculated. Calculated as a template motion vector.

Further, the image processing unit 8 includes an alignment unit 8g that performs alignment between the subject presence image P1b and the subject non-existence image P6.
The alignment unit 8g aligns the subject presence image P1b and the subject non-existence image P6 based on the feature points extracted from the subject non-existence image P6. That is, the alignment unit 8g calculates a coordinate conversion formula (projection conversion matrix) of each pixel of the subject existing image P1b with respect to the subject nonexistent image P6 based on the feature points extracted from the subject nonexistent image P6. The subject presence image P1b is coordinate-transformed according to the conversion formula and aligned with the subject non-existence image P6.
Specifically, the alignment unit 8g calculates the motion vectors of a plurality of templates calculated by the block matching unit 14 by majority, and the motion determined to be statistically greater than or equal to a predetermined percentage (for example, 50%). Using the vector as the overall motion vector, the projection transformation matrix of the subject existing image P1b is calculated using the feature point correspondence related to the motion vector. The alignment unit 8g performs coordinate conversion of the subject existing image P1b according to the projective transformation matrix and performs alignment with the subject non-existing image P6.

なお、上記式（２）にあっては、被写体非存在画像Ｐ６のＹＵＶデータを「Y」、「U」、「V」で表し、被写体存在画像Ｐ１ｂのＹＵＶデータを「Yc」、「Uc」、「Vc」で表す。また、Ｇは、色差信号U、Vのゲインを表している。
ここで、マスク生成部８ｅは、被写体非存在画像Ｐ６と被写体存在画像Ｐ１ｂとの対応する各画素の相違度（差分情報）Ｄに基づいて、被写体存在画像Ｐ１ｂから被写体領域を抽出するためのマスク画像（抽出用情報）Ｍを生成する第２抽出用情報生成手段を構成している。 The mask generating unit 8e generates a dissimilarity map based on the dissimilarity D of each corresponding pixel between the subject non-existing image P6 and the subject existing image P1b calculated by the dissimilarity calculating unit 8b according to Expression (2). Then, the mask generation unit 8e generates a mask image M by binarizing (0, 255) the generated difference map with a predetermined threshold.

In the above formula (2), YUV data of the subject non-existing image P6 is represented by “Y”, “U”, “V”, and YUV data of the subject existing image P1b is “Yc”, “Uc”. , “Vc”. G represents the gain of the color difference signals U and V.
Here, the mask generation unit 8e extracts a subject region from the subject presence image P1b based on the difference (difference information) D between the corresponding pixels of the subject non-existence image P6 and the subject presence image P1b. A second extraction information generating unit that generates an image (extraction information) M is configured.

The shutter button 12a of the operation input unit 12 is the same as the background of the subject existing image P1b when the user performs a predetermined operation after determining that the non-flatness of the first subject extraction image P2a is not less than a predetermined value. The CPU 13 outputs to the CPU 13 an imaging instruction from the electronic imaging unit 2 for a subject non-existing image P6 in which no subject S exists in the background.
That is, the shutter button 12a is a second imaging instruction unit that instructs the electronic imaging unit 2 to capture the non-subject image P6 when it is determined that the non-flatness of the first subject extraction image P2a is not less than a predetermined value. Is configured.

Next, the subject clipping process performed by the imaging apparatus 200 will be described with reference to FIGS.
9 and 10 are flowcharts illustrating an example of operations related to the subject clipping process.
Note that the processing other than the imaging of the subject non-existing image P6 and the second region detection processing using the subject non-existing image P6 and the subject existing image P1b is substantially the same as the subject cropping process of the first embodiment. Description is omitted.

  As shown in FIG. 9, first, the CPU 13 writes a background shooting flag = 0 in a flag table (not shown) of the image memory 5 (step S201), and then performs display control similarly to the subject clipping process of the first embodiment. The unit 10 displays the live view image on the display screen of the display unit 11 based on the image frames sequentially generated by the resolution conversion unit 6 in accordance with the imaging of the subject S, and superimposes the live view image on the live view image to display the background instruction message. Is displayed on the display screen of the display unit 11 (step S1; see FIG. 6A).

Then, similarly to the subject clipping process of the first embodiment, the display control unit 10 performs control to update the live view image based on the image frame of the first subject extraction image P2a (step S2). Each time the live view image is updated, the feature amount calculation unit 7a extracts a block region (feature point) having a high feature based on the YUV data of the first live view image, and the variation amount calculation unit 7b Then, the standard deviation as the variation amount in each block area is calculated according to the equation (1) (step S3).

Next, similarly to the subject clipping process of the first embodiment, the non-flatness determination unit 8a determines whether or not the feature point (block area) is equal to or smaller than a predetermined value and the variation amount of the pixel value in the block area is predetermined. By determining whether or not the value is equal to or less than the value, it is determined whether or not there is a variation amount between the feature point and the pixel value in the block area (step S4).
Here, if it is determined that there is a variation amount between the feature point and the pixel value in the block region (step S4; NO), the CPU 13 instructs the user to perform an imaging instruction in which the shutter button 12a of the operation input unit 12 is operated by a predetermined operation. Whether or not is input is determined (step S202).

If it is determined in step S202 that an imaging instruction has not been input (step S202; NO), the CPU 13 determines whether or not an end instruction has been input based on a predetermined operation of the operation input unit 12 by the user. (Step S5).
If it is determined that the end instruction has not been input (step S5; NO), the CPU 13 shifts the process to step S2 and executes the subsequent processes.
On the other hand, if it is determined that an end instruction has been input (step S5; YES), the CPU 13 ends the subject clipping process.

If it is determined in step S202 that an imaging instruction has been input (step S202; YES), the CPU 13 writes the background shooting flag = 1 in the flag table (not shown) of the image memory 5 (step S203). ), The process proceeds to step S13, and the imaging control unit 3 is set with imaging conditions such as exposure conditions (AE) and white balance (AWB) when imaging the subject existing image P1b (see FIG. 12A). Adjustment is made according to the subject S (step S13). Thereafter, the imaging control unit 3 causes the electronic imaging unit 2 to capture an optical image of the subject existing image P1b under the adjusted imaging condition, and based on the image frame of the subject existing image P1b transferred from the electronic imaging unit 2, The YUV data of the presence image P1b is generated by the image data generation unit 4, and the YUV data is temporarily stored in the image memory 5 (step S14).
As a result, a subject presence image P1b in which the subject S is photographed against a background with a large amount of feature is generated.
The imaging control unit 3 maintains a state in which imaging conditions such as a focus position (AF), an exposure condition (AE), and a white balance (AWB) at the time of imaging the subject existing image P1b are fixed.

On the other hand, if it is determined in step S4 that there is no variation amount between the feature point and the pixel value in the block area (step S4; YES), for example, the subject S with a plain background having a small feature amount as a background. When the camera 13 is photographed (see FIG. 6B), the CPU 13 performs steps S6 to S12 in the same manner as the subject clipping process of the first embodiment.
As a result, the first live view image and the second live view image are stored in the image memory 5 (steps S6 and S12).

Next, as shown in FIG. 10, the CPU 13 determines whether or not the background shooting flag = 0 (step S204).
If it is determined that the background shooting flag = 0 is not satisfied (step S204; NO), that is, if the background shooting flag = 1, the CPU 13 superimposes the live view image on the display control unit 10. Thus, an image of a semi-transparent display mode of the subject presence image P1b and an imaging instruction message of the subject non-existence image P6 (eg, “shoot the background”) are displayed on the display screen 11a of the display unit 11 ( Step S205; see FIG. 12B).

Thereafter, the CPU 13 controls the display control unit 10 to update the live view image based on the image frames sequentially generated by the imaging of the subject S by the lens unit 1 and the electronic imaging unit 2 (step S206), and the user Thus, it is determined whether or not the shutter button 12a of the operation input unit 12 has been operated in a predetermined manner and an imaging instruction has been input (step S207).
Thereafter, after the user moves the subject S out of the angle of view or waits for the subject S to move (see FIG. 8B), when it is determined in step S207 that an imaging instruction has been input. (Step S207; YES), the imaging control unit 3 causes the electronic imaging unit 2 to capture an optical image of the subject nonexistent image P6 in which the subject S does not exist in the background under predetermined imaging conditions, and the image data generation unit 4 Based on the image frame of the subject nonexistent image P6 transferred from the electronic imaging unit 2, YUV data of the subject nonexistent image P6 is generated, and the YUV data is temporarily stored in the image memory 5 (step S208).
As a result, a subject non-existing image P6 in which a background with a large amount of feature without the subject S is captured is generated.

Next, the CPU 13 sends the YUV data of the subject existing image P1b and the subject nonexisting image P6 to the non-flatness calculating unit 7, the block matching unit 14, and the image processing unit 8 with reference to the YUV data of the subject nonexisting image P6. A process for aligning the YUV data is performed (step S209).
Specifically, the feature amount calculation unit 7a selects a predetermined number (or a predetermined number or more) of highly featured block areas (feature points) based on the YUV data of the subject non-existing image P6, and performs this block Extract the contents of the area as a template. Then, the block matching unit 14 searches the subject existing image P1b for a position where the pixel value of the template extracted by the feature extraction process is optimally matched, and the evaluation value of the difference between the pixel values is the best. An optimum offset between the subject non-existing image P6 and the subject existing image P1b is calculated as a motion vector of the template.
Then, the alignment unit 8g of the image processing unit 8 statistically calculates the entire motion vector based on the motion vectors of the plurality of templates calculated by the block matching unit 14, and performs feature point correspondence related to the motion vector. After calculating the projection transformation matrix of the subject existing image P1b, the subject existing image P1b is subjected to projective transformation based on the projective transformation example, so that the YUV data of the subject existing image P1b and the YUV data of the subject nonexisting image P6 are converted. To perform the alignment process.

  Next, the image processing unit 8 determines whether or not the alignment is successful (step S210). In other words, in step S209, the image processing unit 8 can statistically calculate the entire motion vector from the motion vectors of a plurality of templates, and uses the feature point correspondence related to the motion vector to projective transformation matrix of the subject existing image P1b. It is determined whether or not it has been calculated.

  If it is determined that the alignment is successful (step S210; YES), the CPU 13 causes the image processing unit 8 to perform a second area detection process for detecting a subject area including the subject S from the subject existing image P1b. This is performed (step S211).

Here, the second region detection process will be described in detail with reference to FIG.
FIG. 11 is a flowchart illustrating an example of an operation related to the second region detection process.
The second area detection process is substantially the same as the first area detection process except that the object relating to the calculation of the difference degree D by the difference degree calculation unit 8b is different from the first area detection process. Omitted.

That is, as shown in FIG. 11, the dissimilarity calculating unit 8b of the image processing unit 8 calculates the dissimilarity D of each corresponding pixel between the subject non-existing image P6 and the subject existing image P1b according to the following equation (2). The mask generation unit 8e of the image processing unit 8 generates a dissimilarity map based on the calculated dissimilarity D (step S71).

Next, as in the first region detection process, the mask generation unit 8e binarizes the generated dissimilarity map with a predetermined threshold value to generate a mask image M (see FIG. 7B) (step S52). . Then, the mask generation unit 8e determines whether or not the binarization is successful (step S53). Here, if it is determined that the binarization is successful (step S53; YES), the mask generation unit 8e performs a contraction process on the image data of the mask image M to remove a pixel set smaller than a predetermined value. (Step S54), an expansion process for correcting the contraction is performed (Step S55).
Subsequently, the mask generation unit 8e makes an area that is equal to or less than a predetermined ratio in the number of constituent pixels of the effective area of the image data of the mask image M by performing a labeling process that assigns the same number to a set of pixels that form the same connected component. By replacing, hole filling is performed (step S56).
Then, the mask generation unit 8e applies an averaging filter to the image data of the mask image M, adds a synthetic gradation to the edge of the subject region (step S57), and sets the region detection success (step S58). .
On the other hand, when it is determined in step S53 that the binarization has not been successful (step S53; NO), the mask generation unit 8e regards the binarization as failure and sets a region detection failure (step S59). .
Thereby, the second region detection process is terminated.

As shown in FIG. 10, next, similarly to the subject clipping process of the first embodiment, the mask generation unit 8e determines whether or not the subject region has been successfully detected (step S18).
Here, if it is determined that the detection of the subject region has succeeded (step S18; YES), the cut-out image generation unit 8f uses the α blend to generate the mask image M generated by the second region detection process. The subject region is cut out from the subject existing image P1b and synthesized with a predetermined single color background image P5 to generate image data of the subject cutout image P4 (see FIG. 7B) (step S19).
Thereafter, the display control unit 10 displays a subject cutout image P4 in which the subject S is superimposed on a predetermined single color background image P5 based on the image data of the subject cutout image P4 generated by the cutout image generation unit 8f. Is displayed on the display screen (see FIG. 7C).

Next, the CPU 13 associates the image data of the mask image M generated by the mask generation unit 8e of the image processing unit 8 with the image data of the subject cutout image P4 in a predetermined storage area of the recording medium 9, and The extension of the image data of the cutout image P4 is saved as one file with “.jpe” (step S20).
Thereby, the subject clipping process is completed.

  On the other hand, whether it is determined in step S18 that the detection of the subject area has not been successful (step S18; NO), or whether it is determined in step S16 that the background generation has not been successful (step S16; NO). Alternatively, when it is determined in step S210 that the alignment of the subject presence image P1b and the subject non-existence image P6 is not successful (step S210; NO), the CPU 13 fails to cut out the subject S. Is displayed on the display screen of the display unit 11 (step S21), and the subject clipping process is terminated.

If it is determined in step S204 that the background shooting flag = 0 (step S204; YES), the CPU 13 performs each of steps S13 to S20 as in the subject clipping process of the first embodiment.
That is, the subject existing image P1a in which the subject S is photographed with the background having a small feature amount is temporarily stored in the image memory 5 (step S14). In addition, after the background generation process generates the extraction background image P3 based on the first live view image and the second live view image (step S15), the first area detection process performs the extraction background image P3. A mask image M is generated based on the subject presence image P1a (step S17). Thereafter, image data of the subject cutout image P4 (see FIG. 7B) is generated (step S20), and the image data of the subject cutout image P4 and the image data of the mask image M are associated with each other and saved as one file. Is done.
Thereby, the subject clipping process is completed.

As described above, according to the imaging apparatus 200 of the second embodiment, when it is determined that the non-flatness of the image frame for live view image display of the first subject extraction image P2a is not less than or equal to the predetermined value, the subject presence image After the imaging of P1b, a subject non-existing image P6 in which no subject S exists in the same background as the background of the subject existing image P1b is captured. Then, a mask image M is generated based on the difference D between the corresponding pixels of the subject non-existing image P6 and the subject existing image P1b, and the subject region is extracted from the subject existing image P1b using the mask image M. To do.
That is, for example, when the subject S is photographed against a background with a large amount of feature (see FIG. 12A), if it is determined that there are more feature points than a predetermined value, the subject presence image P1b is captured. After the subject nonexistent image P6 in which the subject S does not exist within the same background as the subject present image P1b is imaged, the subject present image P1b and the subject nonexistent image P6 are aligned. The subject region can be extracted from the subject presence image P1b based on the difference D between the corresponding pixels of the subject presence image P1b and the subject non-existence image P6 that are aligned.
That is, even when the subject S is photographed against a background with a large amount of features, the degree of difference between corresponding pixels of the subject present image P1b and the subject nonexistent image P6 is obtained by capturing the subject nonexistent image P6. A mask image M is generated based on D, and a subject area can be extracted from the subject presence image P1b using the mask image M.
As described above, by using the image frame for displaying the live view image, not only when shooting the subject S with the background having a small feature amount, but also when shooting the subject S with the background having a large feature amount. In addition, it is possible to find an optimum environment for clipping the subject S, and to provide an imaging device 200 that is more convenient to use.

The present invention is not limited to the above-described embodiment, and various improvements and design changes may be made without departing from the spirit of the present invention.
For example, in the first and second embodiments, the extraction background image P3 is generated in the background generation process (step S15) of the subject clipping process, and the first region detection process (step S17) Although the mask image M is generated using the extraction background image P3, the method of generating the mask image M is not limited to this.
That is, for example, the mask image M may be generated based on the difference D between the corresponding pixels of the first live view image and the second live view image.

なお、上記式（２）にあっては、第１ライブビュー画像のＹＵＶデータを「Y」、「U」、「V」で表し、第２ライブビュー画像のＹＵＶデータを「Yc」、「Uc」、「Vc」で表す。また、Ｇは、色差信号U、Vのゲインを表している。 Specifically, as shown in FIG. 13, the difference calculation unit 8b calculates the difference D of each pixel corresponding to the first live view image and the second live view image according to the equation (2), and The mask generation unit 8e of the processing unit 8 generates a dissimilarity map based on the calculated dissimilarity D (step S91).

In the above formula (2), YUV data of the first live view image is represented by “Y”, “U”, “V”, and YUV data of the second live view image is represented by “Yc”, “Uc”. ”And“ Vc ”. G represents the gain of the color difference signals U and V.

Next, the mask generator 8e binarizes the generated dissimilarity map with a predetermined threshold to generate image data of the mask image M (see FIG. 7B) (step S52). Then, the mask generation unit 8e determines whether or not the binarization is successful by determining whether or not the background (binarization; 0) region is extremely small, for example (step S53).
Here, if it is determined that the binarization is successful (step S53; YES), the mask generation unit 8e corrects the remaining portion of the periphery of the subject region and removes fine noise. After performing contraction processing on the image data to remove a pixel set smaller than a predetermined value (step S54), expansion processing for correcting the contraction is performed (step S55).

  Subsequently, when the subject S has a color similar to the background color, the subject region of the mask image M is lost, and thus the mask generation unit 8e performs labeling that assigns the same number to a set of pixels that constitute the same connected component. Filling is performed by replacing the area below a predetermined ratio in the number of constituent pixels of the effective area of the image data of the mask image M with the effective area by the processing (step S56).

  Then, the mask generation unit 8e resizes the image data of the mask image M to the actual imaging size of the subject existing image P1a (step S92), and then applies an averaging filter to the image data of the mask image M to apply the subject. A composite gradation is given to the edge of the region (step S57), and the region detection is successful (step S58).

On the other hand, if it is determined in step S53 that the binarization has not been successful (step S53; NO), for example, if the background (binarization; 0) region is extremely small, the mask generation unit 8e. Is assumed to have failed in binarization, and region detection fails (step S59).
Thus, the first area detection process is terminated.

  Therefore, by generating the mask image M as described above, it is not necessary to perform the background generation process in the subject clipping process. That is, for example, the first subject extraction image P2a, the second subject extraction image P2b, and the like are captured in a state where the imaging device 100 (200) is fixed to a tripod, and the first subject extraction image P2a is captured. When the immobile state continues until the second subject extraction image P2b is captured, the first subject extraction image P2a can be applied as the extraction background image. Thereby, the background generation process can be omitted, and the subject clipping process can be speeded up.

In the first and second embodiments, whether or not both the feature point of the first live view image and the variation amount of the pixel value in the feature point are equal to or smaller than the predetermined value in step S4 of the subject clipping process. However, the present invention is not limited to this, and non-flatness is determined by determining the non-flatness of the first subject extraction image P2a by determining either the feature point or the variation amount. When it is determined that the degree is equal to or less than the predetermined value, the subject presence image P1a may be captured.
In the background generation process, it is determined whether both the feature point of the first subject extraction image P2a and the variation amount of the pixel value in the feature point are equal to or less than a predetermined value. When the non-flatness of the first subject extraction image P2a is determined by determining either the feature point or the variation amount, and the non-flatness is determined to be equal to or less than a predetermined value. In addition, the extraction background image P3 may be generated from the first subject extraction image P2a.

Furthermore, the generated subject cutout image P4 and a predetermined background image (not shown) may be combined to generate a subject composite image (not shown).
In the process of generating the subject composite image, for the image data of the subject cut-out image P4, for the pixels covered by the filled portion of the image data of the mask image M (the hatched portion other than the subject S), the color of the background image On the other hand, the pixels of the subject S portion are not transmitted to the color of the predetermined background image without doing anything. As a result, it is possible to generate a subject composite image in which the subject region of the subject cutout image P4 is superimposed on a predetermined background image (not shown).

  In the first and second embodiments, the live view image display of the first subject extraction image P2a and the second subject extraction image P2b is performed by calculating and determining the non-flatness of the subject extraction image. Although the image frame is used, the image size of the first subject extraction image P2a and the second subject extraction image P2b is not limited to this, and the first imaging size of the actual imaging size generated by the image data generation unit 4 is used. One subject extraction image P2a and second subject extraction image P2b may be used. That is, the resolution conversion unit 6 may generate the first subject extraction image P2a and the second subject extraction image P2b having a size different from the size for displaying the live view image, and use the images.

  In the first and second embodiments, when capturing the subject existing image P1a after storing the live view image display image frame (second live view image) of the second subject extraction image P2b, Although AF (automatic focusing process) is not performed, the present invention is not limited to this. For example, in order to obtain a higher-quality subject existing image P1a even if the shooting time is somewhat longer, AF It is preferable to perform (automatic focusing process).

  Furthermore, the mask image M is exemplified as the extraction information for extracting the subject area from the subject existence image P1a (P1b). However, the present invention is not limited to this, and the position of the subject area in the subject existence image P1a is specified. Any object can be used as long as it can extract the subject area.

  In the first and second embodiments, the image data of the mask image M and the image data of the subject cutout image P4 are stored in one file in association with each other. The image data of the image P1a (P1b) may be associated with each other and stored in the recording medium (storage unit) 9 as one file. In this case, for the reproduction of the file, a mode for reproducing the subject existing image P1a and two modes for combining and displaying the subject cutout image P4 by applying the image data of the mask image M at the time of reproduction are prepared. Good.

  Furthermore, the configurations of the imaging apparatuses 100 and 200 are merely examples as illustrated in the above-described embodiment, and are not limited thereto.

In addition, in the above embodiment, the functions as the generation unit, the calculation unit, the determination unit, the background generation unit, and the subject extraction unit are controlled by the resolution conversion unit 6 and the non-flatness calculation unit under the control of the CPU 13. However, the present invention is not limited to this, and a configuration realized by executing a predetermined program or the like by the CPU 13 is also possible.
That is, a program including a generation processing routine, an arithmetic processing routine, a determination processing routine, a background generation processing routine, and a subject extraction processing routine is stored in a program memory (not shown) that stores the program. Then, the CPU 13 may be caused to function as a generation unit that generates an image frame of a subject extraction image (first subject extraction image P2a) for extraction of a subject area captured by the imaging unit by the generation processing routine. . Further, the CPU 13 may be caused to function as a calculation unit that calculates the non-flatness of the image frame of the subject extraction image (first subject extraction image P2a) generated by the generation unit by the calculation processing routine. In addition, the CPU 13 performs determination processing as a determination unit that determines whether the non-flatness of the image frame of the subject extraction image (first subject extraction image P2a) calculated by the calculation unit is equal to or less than a predetermined value. You may make it function. Further, when the background generation processing routine determines that the non-flatness is equal to or less than a predetermined value by the determination unit, the CPU 13 performs extraction based on the image frame of the subject extraction image (first subject extraction image P2a). You may make it function as a background production | generation means which produces | generates the background image P3. Further, the CPU 13 causes the subject extraction processing routine to execute the CPU 13 based on the difference information of each corresponding pixel between the subject existing image P1a imaged by the electronic imaging unit 2 in accordance with the imaging instruction from the operation input unit 12 and the extraction background image P3. You may make it function as a subject extraction means which extracts a subject area | region from the subject presence image P1a.

100, 200 Imaging device 1 Lens unit 2 Electronic imaging unit 3 Imaging control unit 7 Non-flatness calculation unit 7a Feature amount calculation unit 7b Variation amount calculation unit 8 Image processing unit 8a Non-flatness determination unit 8d Background generation unit 8e Mask generation unit 8f Cutout image generator 12a Shutter button 13 CPU

Claims (13)

Imaging means;
Generating means for generating an image frame of a subject extraction image for extraction of a subject area imaged by the imaging means;
A computing means for computing the non-flatness of the image frame of the subject extraction image generated by the generating means;
First determination means for determining whether or not the non-flatness of the image frame of the subject extraction image calculated by the calculation means is equal to or less than a predetermined value;
First imaging instruction means for instructing imaging of a subject existing image in which a subject is present in the background by the imaging means;
A background generation unit that generates an extraction background image based on an image frame of the subject extraction image when the first determination unit determines that the non-flatness is equal to or less than a predetermined value;
The subject area is extracted from the subject presence image based on difference information of each corresponding pixel between the subject presence image captured by the imaging unit and the extraction background image in accordance with an imaging instruction by the first imaging instruction unit. Subject extraction means for
An imaging apparatus comprising: The generating means includes
Each of the first subject extraction image in which no subject is present in the background and the second subject extraction image in which the subject is present in the background is imaged by the imaging means to generate each image frame,
A second determination means for determining whether or not a difference between corresponding pixels of the image frame of the first subject extraction image and the image frame of the second subject extraction image is equal to or less than a predetermined value;
The background generation means includes
The imaging apparatus according to claim 1, wherein the extraction background image is generated based on a pixel whose degree of difference is determined to be equal to or less than a predetermined value by the second determination unit. The background generation means includes
The imaging apparatus according to claim 2, wherein the background image for extraction having the same color as a pixel whose degree of difference is determined to be equal to or less than a predetermined value by the second determination unit is generated.   Imaging conditions for setting the imaging conditions of the first subject extraction image first captured by the imaging unit and the second subject extraction image captured after the first subject extraction image is captured to the same condition The imaging apparatus according to claim 2, further comprising a setting unit. The imaging condition setting means further includes:
The imaging apparatus according to claim 4, wherein after the second subject extraction image is captured, an imaging condition for capturing the subject existing image by the imaging unit is set. First extraction information generating means for generating extraction information for extracting the subject region from the subject existing image based on difference information of corresponding pixels between the subject existing image and the extraction background image; In addition,
The subject extracting means includes
The imaging apparatus according to claim 1, wherein the subject area is extracted from the subject presence image using the extraction information generated by the first extraction information generation unit. The computing means is
As the non-flatness, a plurality of feature points of the subject extraction image are extracted,
The first determination means includes
Furthermore, it is determined whether or not the plurality of feature points extracted by the calculation means are equal to or less than a predetermined value,
The background generation means includes
The imaging according to any one of claims 1 to 6, wherein the extraction background image is generated when the first determination unit determines that the feature point is equal to or less than a predetermined value. apparatus. The computing means is
As the non-flatness, a variation amount of pixel values in a plurality of image regions of the subject extraction image is calculated,
The first determination means includes
Further, it is determined whether or not the variation amount of the pixel value in the plurality of image areas calculated by the calculation means is equal to or less than a predetermined value,
The background generation means includes
The extraction background image is generated when the first determination unit determines that the variation amount of the pixel value is equal to or less than a predetermined value. The imaging device described. The imaging means includes
9. The subject existence image is picked up when the non-flatness of the subject extraction image is judged to be a predetermined value or less by the first judgment means. The imaging device described in 1. When the first determination unit determines that the non-flatness is not equal to or less than a predetermined value, the imaging unit instructs the imaging unit to capture a subject nonexistent image in the same background as the background of the subject existing image. A second imaging instruction means;
The subject extracting means includes
The subject region is extracted from the subject presence image based on the difference information of each corresponding pixel between the subject non-existence image captured by the imaging unit and the subject presence image according to the imaging instruction by the second imaging instruction unit. The imaging apparatus according to any one of claims 1 to 9, wherein Second extraction information generating means for generating extraction information for extracting the subject region from the subject presence image based on difference information of corresponding pixels between the subject non-existence image and the subject presence image. In addition,
The subject extracting means includes
The imaging apparatus according to claim 10, wherein the subject area is extracted from the subject presence image using the extraction information generated by the second extraction information generation unit. An imaging apparatus comprising imaging means and imaging instruction means for instructing imaging of a subject existing image in which a subject is present in the background by the imaging means,
Generating an image frame of a subject extraction image for extraction of a subject region imaged by the imaging means;
A calculation step of calculating the non-flatness of the image frame of the subject extraction image generated by the generation step;
A determination step of determining whether or not the non-flatness of the image frame of the subject extraction image calculated by the calculation step is equal to or less than a predetermined value;
A background generation step of generating an extraction background image based on an image frame of the subject extraction image when the non-flatness is determined to be equal to or less than a predetermined value by the determination step;
Extracting the subject region from the subject existing image based on the difference information of each corresponding pixel between the subject existing image captured by the imaging unit and the background image for extraction according to the imaging instruction by the imaging instruction unit. When,
An image processing method characterized in that A computer of an imaging apparatus comprising imaging means and imaging instruction means for instructing imaging of a subject existing image in which a subject is present in the background by the imaging means,
Generating means for generating an image frame of a subject extraction image for extraction of a subject area imaged by the imaging means;
Arithmetic means for calculating the non-flatness of the image frame of the subject extraction image generated by the generating means;
Determination means for determining whether or not the non-flatness of the image frame of the subject extraction image calculated by the calculation means is equal to or less than a predetermined value;
Background generation means for generating an extraction background image based on an image frame of the subject extraction image when the determination means determines that the non-flatness is equal to or less than a predetermined value;
A subject for extracting the subject area from the subject existing image based on difference information of each corresponding pixel between the subject existing image captured by the imaging unit and the background image for extraction according to the imaging instruction by the imaging instruction unit Extraction means,
A program characterized by functioning as

Images