JP5565227B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  Conventionally, an application that uses an imaging device to capture an image with a subject in the background and a background image without the subject, generate difference information from the background image and the image with the subject, and extract only the subject It is known (see, for example, Patent Document 1).

Japanese Patent Laid-Open No. 10-21408

  However, since the subject image is automatically extracted by the image analysis process in the application disclosed in Patent Document 1, when an unintended image area is extracted, the process is canceled and the process is not performed again from the beginning. There was a problem not to be.

  Therefore, an object of the present invention is to provide an image processing device, an image processing method, and a program that can correct a subject image even after the subject image is extracted.

In order to solve the above problem, an image processing apparatus according to claim 1 is provided.
Based on the corresponding differential information of each pixel with the non-existing object absence image of the subject on the subject-including image and the previous xenon in Jing the presence of an object in the background, include the subject from the subject existing image Extraction means for extracting a subject image, first generation means for generating an alpha map of a predetermined gradation indicating the position of the subject image extracted by the extraction means in the subject presence image, and generated by the first generation means A second generation unit that generates a graph of an alpha value in a predetermined axis direction based on the generated alpha map, and a determination as to whether or not the subject image is generated by the graph generated by the second generation unit setting means for setting the predetermined threshold value, based on the predetermined threshold set by the setting means, modified in the extracted object image by the extraction unit Specifying means for specific areas to, that this area should be corrected specified by the specifying means, with a, a correction means for correcting, based on the region corresponding to the region to be the modification in the subject-including image It is characterized by.

The invention according to claim 2 is the image processing apparatus according to claim 1 ,
The correction means sequentially corrects the alpha map generated by the first generation means every time the predetermined threshold is set by the setting means.

The invention according to claim 3 is the image processing apparatus according to claim 2 ,
A third generating unit that sequentially generates a subject cut-out image obtained by cutting out the subject image from the subject-existing image for display based on the alpha map each time the alpha map is corrected by the correcting unit; 3 is further provided with display control means for sequentially displaying the subject clipped image on the display unit each time the subject clipped image is generated by the generating means.

According to a fourth aspect of the present invention, in the image processing apparatus according to the first aspect,
The specifying means specifies a region specified based on a predetermined operation of the operation unit in the subject image as the region to be corrected.

The image processing method of the invention according to claim 5
An image processing method using the image processing apparatus, the difference information of each pixel corresponding with the non-existent object absence image of the subject on the subject-including image and the previous xenon in Jing the presence of an object in the background And a process for extracting a subject image including a subject from the subject existing image, and a process for generating an alpha map of a predetermined gradation indicating a position of the extracted subject image in the subject existing image. Processing for generating a graph of an alpha value in a predetermined axis direction based on the determined alpha map, processing for setting a predetermined threshold for determining whether or not the subject image is the generated graph, and setting based on the predetermined threshold which is a process of identify the region to be corrected in the extracted object image, the region to be corrected identified, put the subject-including image It is characterized by comprising a process of modified based on a region corresponding to the modified region to be.

The program of the invention described in claim 6 is:
The computer of the image processing apparatus, based on the corresponding differential information of each pixel with the non-existing object absence image of the subject on the subject-including image and the previous xenon in Jing the presence of an object in the background, the object Extraction means for extracting a subject image including a subject from a presence image; first generation means for generating an alpha map of a predetermined gradation indicating a position of the subject image extracted by the extraction means in the subject presence image; A second generation unit that generates a graph of an alpha value in a predetermined axis direction based on the alpha map generated by the first generation unit; whether the subject image is the graph generated by the second generation unit; setting means for setting a predetermined threshold value according to the determination of, based on the set predetermined threshold by the setting means, extracted by the extraction means Specifying means, correction means the region to be corrected, which is specified by the specifying means, modified based on the region corresponding to the region to be the modification in the subject-including image that identify the region to be corrected in Utsushitai image, It is characterized by making it function as.

  According to the present invention, the subject image can be corrected even after the subject image is extracted.

It is a block diagram which shows schematic structure of the imaging device of one Embodiment to which this invention is applied. 3 is a flowchart illustrating an example of an operation related to subject clipping processing by the imaging apparatus of FIG. 1. It is a figure which shows typically an example of the image for demonstrating the object clipping process of FIG. 3 is a flowchart illustrating an example of an operation related to subject correction processing by the imaging apparatus of FIG. 1. FIG. 5 is a diagram schematically illustrating an example of an image for explaining subject correction processing in FIG. 4.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.
FIG. 1 is a block diagram illustrating a schematic configuration of an imaging apparatus 100 according to an embodiment to which the present invention is applied.

The imaging apparatus 100 according to the present embodiment obtains a subject image Pg including a subject from the subject presence image P1 based on difference information of each pixel corresponding to the subject presence image P1 and a subject non-existence image (not shown). Extraction is performed, the region to be corrected in the subject image Pg is specified according to a predetermined condition, and the specified region to be corrected is corrected based on the corresponding region in the subject presence image P1.
Specifically, as illustrated in FIG. 1, the imaging apparatus 100 includes a lens unit 1, an electronic imaging unit 2, an imaging control unit 3, an image data generation unit 4, an image memory 5, and a feature amount calculation unit. 6, a block matching unit 7, an image processing unit 8, a display control unit 9, a display unit 10, a recording medium control unit 11, an operation input unit 12, and a central control unit 13.
Further, the imaging control unit 3, the feature amount calculation unit 6, the block matching unit 7, the image processing unit 8, and the central control unit 13 are designed as a custom LSI 1A, for example.

The lens unit 1 includes a plurality of lenses and includes a zoom lens, a focus lens, and the like.
Although not shown, the lens unit 1 includes a zoom drive unit that moves the zoom lens in the optical axis direction and a focus drive unit that moves the focus lens in the optical axis direction when imaging a subject. May be.

  The electronic imaging unit 2 is composed of an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-oxide Semiconductor), for example, and converts an optical image that has passed through various lenses of the lens unit 1 into a two-dimensional image signal. To do.

Although not shown, the imaging control unit 3 includes a timing generator, a driver, and the like. Then, the imaging control unit 3 scans and drives the electronic imaging unit 2 with a timing generator and a driver, converts the optical image into a two-dimensional image signal with the electronic imaging unit 2 every predetermined period, and the electronic imaging unit 2 Image frames are read out from the imaging area for each screen and output to the image data generation unit 4.
The imaging control unit 3 performs AF (automatic focusing processing), AE (automatic exposure processing), AWB (automatic white balance), and the like as adjustment control of the imaging conditions of the subject.

  The lens unit 1, the electronic image capturing unit 2, and the image capturing control unit 3 configured in this way are used as an image capturing unit, such as a subject existing image P 1 (see FIG. 3A) or the like, A non-existing subject image (not shown) or the like in which no subject exists is captured in a background that is substantially the same as the background of the image P1 (preferably completely the same).

The image data generation unit 4 appropriately adjusts the gain for each RGB color component with respect to the analog value signal of the image frame transferred from the electronic imaging unit 2, and then performs sample holding by a sample hold circuit (not shown). The digital signal is converted into digital data by a / D converter (not shown), color processing including pixel interpolation processing and γ correction processing is performed by a color process circuit (not shown), and then a digital luminance signal Y and color difference signal Cb , Cr (YUV data).
The luminance signal Y and the color difference signals Cb and Cr output from the color process circuit are DMA-transferred to an image memory 5 used as a buffer memory via a DMA controller (not shown).

  The image memory 5 is configured by, for example, a DRAM or the like, and temporarily stores data processed by the feature amount calculation unit 6, the block matching unit 7, the image processing unit 8, the central control unit 13, and the like.

The feature amount calculation unit 6 performs feature extraction processing for extracting feature points from the subject non-existing image with reference to the subject non-existing image. Specifically, the feature amount calculation unit 6 selects, for example, a predetermined number (or a predetermined number or more) of highly featured block areas (feature points) based on the YUV data of the subject non-existing image, The contents of the block are extracted as a template (for example, a square of 16 × 16 pixels).
Here, the feature extraction process is a process of selecting a feature having a high characteristic convenient for tracking from a large number of candidate blocks.

  When generating the subject cutout image P2 (see FIG. 3C), the block matching unit 7 performs block matching processing for aligning the subject nonexistent image and the subject present image P1. Specifically, the block matching unit 7 optimally matches where the template extracted by the feature extraction processing corresponds to in the subject existing image P1, that is, the template pixel value in the subject existing image P1. Search for a position (corresponding region). Then, an optimum offset between the subject non-existing image and the subject existing image P1 having the best evaluation value (for example, sum of squared differences (SSD), sum of absolute differences (SAD), etc.) of the pixel value is determined as the template. As a motion vector.

  The image processing unit 8 includes an image acquisition unit 8a, a registration unit 8b, an image extraction unit 8c, a position information generation unit 8d, an image generation unit 8e, a graph generation unit 8f, a correction region specification unit 8g, And an image correction unit 8h.

The image acquisition unit 8a acquires the image data of the subject presence image P1 and the subject non-existence image.
That is, the image acquisition unit 8a reads and acquires the YUV data of the subject existing image P1 and the YUV data of the subject non-existing image temporarily stored in the image memory 5.

The alignment unit 8b performs alignment between the subject presence image P1 and the subject non-existence image.
That is, the alignment unit 8b calculates a coordinate conversion formula (projection conversion matrix) of each pixel of the subject presence image P1 with respect to the subject non-existence image based on the feature points extracted from the subject non-existence image. The subject presence image P1 is subjected to coordinate transformation to align with the subject non-existence image.

The image extraction unit 8c extracts a subject image Pg including a subject from the subject presence image P1.
That is, the image extraction unit 8c generates difference information of each corresponding pixel between the subject presence image P1 and the subject non-existence image aligned by the alignment unit 8b, and the subject presence image is based on the difference information. A subject image Pg including the subject is extracted from P1.
Here, the image extraction unit 8c extracts the subject image Pg including the subject from the subject presence image P1 based on the difference information of each pixel corresponding to the subject presence image P1 and the subject non-existence image. Is configured.

The position information generation unit 8d generates position information indicating the position of the subject image Pg in the subject presence image P1.
That is, the position information generation unit 8d specifies the position of the subject image Pg extracted from the subject existence image P1, and indicates a predetermined gradation (for example, 256 gradations: 8 bits) indicating the position of the subject image Pg in the subject existence image P1. ) Alpha map M (see FIG. 3B).
Here, the alpha map M represents a weight when alpha blending the subject image Pg with respect to a predetermined background for each pixel of the subject existing image P1 as an alpha value (α: 0 to 255). . In FIG. 3B and FIGS. 5D to 5F described later, an area where the subject image Pg having an alpha value of “0” does not exist is represented in black, and the alpha value is “255”. The area where the subject image Pg exists is expressed in white, and the intermediate gradation area where the alpha value is “1 to 254” is indicated by an intermediate color between white and black with dots.
As described above, the position information generation unit 8d constitutes a first generation unit that generates an alpha map M having a predetermined gradation indicating the position of the subject image Pg extracted by the image extraction unit 8c in the subject existing image P1. .

The image generation unit 8e generates image data of the subject cutout image P2.
In other words, based on the generated alpha map M, the image generation unit 8e transmits a pixel having an alpha value of “255” among the pixels of the subject existing image P1 to a predetermined single color image (not shown). The subject image Pg is synthesized with the predetermined single color image so that the pixels with the alpha value of “0” are transmitted and the pixels with the alpha value of 1 ≦ α ≦ 254 are blended with the predetermined single color. Image data of the cutout image P2 is generated.
Here, the subject presence image P1 may be high-resolution image data recorded on the recording medium 11a for recording, or a low-resolution reduced image for live view display generated by the image data generation unit 4. It may be data.

Further, the image generation unit 8e cuts the subject image Pg from the reduced image data for live view display of the subject existing image P1 based on the alpha map M after the alpha map M is corrected by the image correction unit 8h. An image P2 is generated. That is, the image generation unit 8e cuts out the subject image Pg from the display subject existing image P1 based on the alpha map M every time the alpha map M is corrected by the image correction unit 8h as the third generation unit. The subject cutout images P2 are sequentially generated.
Note that the method of generating the subject cutout image P2 is the same as described above, and a detailed description thereof is omitted.

The graph generation unit 8 f generates a graph G of the alpha value of the alpha map M.
That is, the graph generation unit 8f generates a graph G of an alpha value in the x-axis (predetermined axis) direction based on the alpha map M generated by the position information generation unit 8d. Specifically, the graph generation unit 8f fixes the y coordinate of the alpha map M to a predetermined coordinate, acquires an alpha value along the x-axis (predetermined axis) direction, and displays the alpha value graphs Ga to Gc. (See FIGS. 5A to 5C).
The fixed coordinates in the y-axis direction of the alpha map M are, for example, the center coordinates of the alpha map M, the continuity and frequency of pixels whose alpha value along the direction substantially equal to the x-axis is not “0”, the x-axis Automatically considering the continuity and frequency of pixels with an alpha value of “255” along the direction that is almost equal to the vertical axis, and the continuity and frequency of pixels with an alpha value of “0” along the direction that is almost equal to the x axis. The predetermined y coordinate may be specified based on a predetermined operation of the operation input unit 12 by the user.

The graph generation unit 8f fixes the y coordinate of the alpha map M to a predetermined coordinate and generates the graph G of the alpha value in the x-axis direction, but is not limited to this example. For example, a graph of an alpha value in the y-axis direction may be generated by fixing the x coordinate of the alpha map M to a predetermined coordinate.
As described above, the graph generation unit 8f constitutes a second generation unit that generates the graph G of the alpha value in the predetermined axis direction based on the alpha map M generated by the position information generation unit 8d.

The correction area specifying unit 8g specifies an area to be corrected in the subject image Pg.
That is, the correction area specifying unit 8g specifies an area to be corrected in the subject image Pg extracted by the image extraction unit 8c. Specifically, the correction area specifying unit 8g is based on a predetermined threshold value for determining whether or not the subject image Pg is set by the central control unit 13 based on a predetermined operation of the operation input unit 12 by the user. Identify the correction area. For example, based on the fact that a straight line indicating a predetermined threshold value is set to a predetermined alpha value on the y-axis based on a predetermined operation of the operation input unit 12 by the user, the correction area specifying unit 8g An area composed of pixels having a value equal to or greater than the alpha value (predetermined threshold value) is specified. Then, the correction area specifying unit 8g specifies an area corresponding to the area specified by the alpha map M in the subject image Pg as the correction area.
Here, the correction area specifying unit 8g constitutes specifying means for specifying the area to be corrected in the subject image Pg extracted by the image extracting unit 8c according to a predetermined condition.

The image correction unit 8h corrects the correction area in the subject image Pg.
That is, the image correcting unit 8h corrects the region to be corrected specified by the correction region specifying unit 8g based on the corresponding region in the subject existing image P1. Specifically, the image correction unit 8h corrects an area composed of pixels having a value equal to or higher than the alpha value in the alpha map M based on a predetermined threshold (alpha value) set by the central control unit 13. More specifically, the image correction unit 8h sets the alpha value of an intermediate gradation pixel having a value equal to or larger than a predetermined threshold (alpha value) set by the central control unit 13 in the alpha map M to the maximum value ( For example, it is changed to “255” for 256 gradations. The image correcting unit 8h sequentially corrects the alpha map M every time a predetermined threshold is set by the central control unit 13.
Thereafter, the image generation unit 8e generates a subject cutout image P2 obtained by cutting out the subject image Pg from the reduced image data for live view display of the subject existing image P1 based on the alpha map M corrected by the image correction unit 8h. Thus, the subject image Pg of the subject cutout image P2 is corrected.
Note that the alpha map M is used to correct the subject image Pg of the subject cutout image P2, but this is an example, and the present invention is not limited to this. For example, pixels in the correction area of the subject image Pg are set as the subject existing image P1. It is possible to arbitrarily change the method of replacing with the pixel in the corresponding region.
As described above, the image correcting unit 8h and the image generating unit 8e constitute correcting means for correcting the region to be corrected specified by the correction region specifying unit 8g based on the corresponding region in the subject existing image P1. .

The display control unit 9 performs control to read display image data temporarily stored in the image memory 5 and display the data on the display unit 10.
Specifically, the display control unit 9 includes a VRAM, a VRAM controller, a digital video encoder, and the like. The digital video encoder reads the luminance signal Y and the color difference signals Cb and Cr read from the image memory 5 and stored in the VRAM (not shown) under the control of the central control unit 13 via the VRAM controller. The data is periodically read from the VRAM, a video signal is generated based on these data, and is output to the display unit 10.

  The display unit 10 is, for example, a liquid crystal display device, and displays an image captured by the electronic imaging unit 2 on the display screen based on a video signal from the display control unit 9. Specifically, the display unit 10 displays a live view image based on a plurality of image frames generated by imaging the subject by the lens unit 1, the electronic imaging unit 2, and the imaging control unit 3 in the imaging mode. The REC view image captured as the actual captured image is displayed.

Further, the display unit 10 displays the subject cutout image P2 generated by the image generation unit 8e under the control of the display control unit 9. Specifically, the display control unit 9 causes the display unit 10 to display the subject cutout image P2 every time the subject cutout image P2 is generated by the image generation unit 8e.
Here, the display control unit 9 constitutes a display control means for sequentially displaying the subject cutout image P2 on the display unit 10 every time the subject cutout image P2 is generated by the image generation unit 8e.

The recording medium control unit 11 is configured such that the recording medium 11a is detachable, and controls reading of data from the loaded recording medium 11a and writing of data to the recording medium 11a. That is, the recording medium control unit 11 records the image data for recording the captured image encoded by the JPEG compression unit (not shown) of the image processing unit 8 on the recording medium 11a.
The image data of the subject image Pg is associated with the alpha map M and the subject presence image P1 generated by the position information generation unit 8d of the image generation unit 8e, and the extension of the image data of the subject cutout image P2 is “. jpe ".

  The recording medium 11a is composed of, for example, a non-volatile memory (flash memory) or the like. However, the recording medium 11a is an example and is not limited thereto, and can be arbitrarily changed as appropriate.

  The operation input unit 12 is for performing a predetermined operation of the imaging apparatus 100. Specifically, the operation input unit 12 includes a shutter button 12a related to a subject shooting instruction, a selection determination button 12b related to a selection instruction for an imaging mode, a function, and the like, and a zoom button (not shown) related to a zoom amount adjustment instruction. And a predetermined operation signal is output to the central control unit 13 in accordance with the operation of these buttons.

In addition, the selection determination button 12b of the operation input unit 12 is generated based on the alpha value in the predetermined axis direction of the alpha map M, and is a subject image Pg in the graph G displayed on the display unit 10. A setting instruction for a predetermined threshold related to the determination (for example, an arbitrary value from “0” to “255” when the alpha value is 256 gradations) is input. Specifically, for example, based on a predetermined operation of the selection decision button 12b by the user, a straight line indicating a predetermined threshold extending along a direction substantially equal to the x axis is moved along the y axis direction. Then, a predetermined threshold setting instruction is input. Then, the selection determination button 12 b outputs a predetermined instruction signal corresponding to the operation of the operation input unit 12 to the central control unit 13.
The central control unit 13 sets a predetermined value indicated by the input instruction signal as a predetermined threshold for determining whether or not the image is the subject image Pg.
Here, the operation input unit 12 and the central control unit 13 constitute setting means for setting a predetermined threshold for determining whether or not the image is the subject image Pg in the graph G generated by the graph generation unit 8f. ing.

  The central control unit 13 controls each unit of the imaging apparatus 100. Specifically, the central control unit 13 includes a CPU, a RAM, and a ROM (all not shown), and performs various control operations according to various processing programs (not shown) for the imaging apparatus 100.

Next, the subject clipping process performed by the imaging apparatus 100 will be described with reference to FIGS.
FIG. 2 is a flowchart illustrating an example of an operation related to the subject clipping process.

The subject clipping process is executed when a subject clipping mode is selected from a plurality of imaging modes displayed on the menu screen based on a predetermined operation of the selection determination button 12b of the operation input unit 12 by the user. It is processing.
In the subject cutout process described below, for example, a train toy is imaged as a subject.

As shown in FIG. 2, first, the display control unit 9 displays a live view image based on a plurality of image frames generated by imaging a subject by the lens unit 1, the electronic imaging unit 2, and the imaging control unit 3. And the image capturing instruction message of the subject existing image P1 is displayed on the display screen of the display unit 10 (step S1).
Thereafter, the central control unit 13 determines whether or not an imaging instruction has been input based on a predetermined operation of the shutter button 12a of the operation input unit 12 by the user (step S2). If it is determined that an imaging instruction has been input (step S2; YES), the imaging control unit 3 determines the focus lens focus position, exposure conditions (shutter speed, aperture, amplification factor, etc.), white balance, and the like. The imaging condition is adjusted, and an optical image of the subject presence image P1 (see FIG. 3A) is captured by the electronic imaging unit 2 under a predetermined condition (step S3). Then, the image data generation unit 4 generates YUV data of the subject existing image P1 transferred from the electronic imaging unit 2. The YUV data of the subject existing image P1 is temporarily stored in the image memory 5.
Further, the imaging control unit 3 maintains a state in which conditions such as a focus position, an exposure condition, and a white balance at the time of imaging the subject presence image P1 are fixed.

Next, the display control unit 9 displays a live view image on the display screen of the display unit 10 based on a plurality of image frames generated by imaging the subject by the lens unit 1, the electronic imaging unit 2, and the imaging control unit 3. At the same time, the image of the semi-transparent display mode of the subject existing image P1 and the imaging instruction message of the subject non-existing image are displayed on the display screen of the display unit 10 so as to be superimposed on the live view image (step S4).
Thereafter, the central control unit 13 determines whether or not an imaging instruction has been input based on a predetermined operation of the shutter button 12a of the operation input unit 12 by the user (step S5). Then, after the user moves the subject out of the angle of view, the camera position is adjusted by the user so that the subject non-existing image overlaps the semi-transparent image of the subject existing image P1, and the shutter button of the operation input unit 12 is moved. If it is determined that the image capturing instruction is input by performing a predetermined operation on 12a (step S5; YES), the image capturing control unit 3 captures the optical image of the subject non-existing image under the image capturing condition fixed after capturing the subject existing image P1. An image is picked up by the electronic image pickup unit 2 (step S6). Then, the image data generation unit 4 generates YUV data of the subject non-existence image transferred from the electronic imaging unit 2. The YUV data of the subject existing image P1 is temporarily stored in the image memory 5.

Next, the central control unit 13 uses the feature existence calculation unit 6, the block matching unit 7, and the image processing unit 8 as a reference to the subject presence image P 1 based on the YUV data of the subject non-existence image temporarily stored in the image memory 5. A projection transformation matrix for projective transformation of the YUV data is calculated using a predetermined image transformation model (for example, a similarity transformation model or a joint transformation model) (step S7).
Specifically, the feature amount calculation unit 6 selects a predetermined number (or a predetermined number or more) of highly featured block regions (feature points) based on the YUV data of the subject nonexistent image, and Extract the contents as a template. Then, the block matching unit 7 searches the subject existing image P1 for a position where the pixel value of the template extracted by the feature extraction process is optimally matched, and the evaluation value of the difference between the pixel values is the best. An optimum offset between the subject non-existing image and the subject existing image P1 is calculated as a motion vector of the template. Then, after the image acquisition unit 8a of the image processing unit 8 acquires the YUV data of the subject existing image P1 and the YUV data of the subject nonexistent image from the image memory 5, the alignment unit 8b is calculated by the block matching unit 7. The overall motion vector is statistically calculated based on the motion vectors of the plurality of templates, and the projective transformation matrix of the subject existing image P1 is calculated using the feature point correspondence related to the motion vector.

  Next, the alignment unit 8b performs the projective transformation on the subject existing image P1 based on the calculated projective transformation example, thereby aligning the YUV data of the subject existing image P1 and the YUV data of the subject nonexistent image. Processing is performed (step S8).

Then, the image extraction unit 8c performs a process of extracting the subject image Pg including the subject from the subject existing image P1 (step S9).
Specifically, the image extraction unit 8c applies a low-pass filter to each of the YUV data of the subject existing image P1 and the YUV data of the subject nonexistent image to remove high frequency components of each image. Thereafter, the image extraction unit 8c calculates a difference for each corresponding pixel between the subject existing image P1 and the subject non-existing image subjected to the low-pass filter, and generates a difference map. Subsequently, the image extraction unit 8c binarizes the dissimilarity map relating to each pixel with a predetermined threshold value, and then performs a contraction process in order to remove a region in which the dissimilarity is caused by fine noise or camera shake from the dissimilarity map. . After that, the image extraction unit 8c performs a labeling process, removes areas below a predetermined value and areas other than the maximum area, identifies the largest island pattern as the subject image Pg, and corrects the shrinkage. The expansion process is performed.

  Next, the position information generation unit 8d displays an alpha map M (see FIG. 3B) having a predetermined gradation (for example, 256 gradations) indicating the position of the extracted subject image Pg in the subject existing image P1. Generate (step S10).

Thereafter, the image generation unit 8e generates image data of a subject cutout image P2 (see FIG. 3C) obtained by combining the subject image Pg with a predetermined single color image based on the subject existing image P1 for live view display. Is performed (step S11).
Specifically, the image generation unit 8e reads out the subject existing image P1, the single color image, and the alpha map M, develops them in the image memory 5, and then reduces the subject existing image P1 at a predetermined magnification both in the horizontal and vertical directions. Low-resolution reduced image data for view display is generated. Further, the image generation unit 8e reduces the alpha map M in accordance with the resolution of the subject existing image P1. Then, the image generation unit 8e transmits all the pixels of the low-resolution subject existing image P1 for live view display with respect to the pixels with the alpha value “0” and the pixels with the alpha value 1 ≦ α <254. Performs blending with a predetermined single color, and does not transmit the pixel with an alpha value of “255” to the predetermined single color without doing anything.

  Thereafter, the recording medium control unit 11 stores the image data of the subject existing image P1, the alpha map M generated by the position information generation unit 8d of the image processing unit 8, and the subject clipped image P2 in a predetermined area of the recording medium 11a. The image data is associated and recorded in one file (step S12).

Thereby, the subject clipping process is completed.
As a result, for example, image data of a subject cutout image P2 in which a train toy is extracted as a subject from within a predetermined background is generated, but the alpha value is `` in the color portion equal to the background color of the subject image Pg ''. Instead of 255 ”(white), an intermediate tone area (white and black intermediate color) with an alpha value of“ 1 to 254 ”occurs.

Next, the subject correction process will be described with reference to FIGS.
FIG. 4 is a flowchart illustrating an example of an operation related to the subject correction process.

In the subject correction process, the subject correction mode is selected from a plurality of operation modes displayed on the menu screen based on a predetermined operation of the selection decision button 12b of the operation input unit 12 by the user after the subject clipping process. This process is executed when
In the subject correction processing described below, for example, a subject image Pg from which a train toy is extracted is set as a correction target.

As shown in FIG. 4, the display control unit 9 is a predetermined subject specified in accordance with a predetermined operation of the operation input unit 12 by the user among the image data of at least one subject cutout image P2 recorded on the recording medium 11a. Based on the image data of the cutout image P2, the subject image Pg is displayed on the display screen of the display unit 10 (step S21).
Subsequently, the central control unit 13 determines whether or not a correction instruction for the subject image Pg has been input based on a predetermined operation of the operation input unit 12 by the user (step S22).

If it is determined in step S22 that a correction instruction has been input (step S22; YES), the graph generation unit 8f fixes the y coordinate of the alpha map M to a predetermined coordinate and follows the x axis (predetermined axis) direction. Thus, the alpha value is acquired, and a graph G of the alpha value (for example, graph Ga or the like; see FIG. 5A) is generated. And the display control part 9 displays the graph G (for example, graph Ga etc.) produced | generated by the graph production | generation part 8f on the display screen of the display part 10 (step S23).
On the other hand, if it is determined in step S22 that no correction instruction has been input (step S22; NO), the central control unit 13 ends the subject correction process.

Next, the central control unit 13 determines whether or not a predetermined threshold value related to determination of whether or not the subject image Pg is input based on a predetermined operation of the operation input unit 12 by the user (step S24). .
Here, if it is determined that a predetermined threshold is input (step S24; YES), the correction area specifying unit 8g is based on an alpha value in which a straight line indicating the predetermined threshold is set on the y-axis. An area composed of pixels having a value equal to or higher than the alpha value (predetermined threshold) in the alpha map M is specified as a correction area (step S25). Then, the image correction unit 8h sets the alpha value of all the pixels in the correction area in the alpha map M, that is, the intermediate gradation having a value equal to or larger than a predetermined threshold (alpha value) to a maximum value (for example, in 256 gradations, By changing to “255”), a corrected alpha map M (for example, alpha map Ma etc .; see FIG. 5D) is generated (step S26).

Subsequently, the image generating unit 8e combines the subject image Pg with a predetermined single color image based on the subject existing image P1 for live view display and the corrected alpha map M (for example, alpha map Ma). Processing for generating image data of the cutout image P2 (for example, the subject cutout image P2a; see FIG. 5G) is performed (step S27).
Specifically, the image generation unit 8e reduces the subject existing image P1 at a predetermined magnification both horizontally and vertically to generate low-resolution reduced image data for live view display, and matches the resolution of the subject existing image P1. Thus, the corrected alpha map M (for example, alpha map Ma) is reduced. Then, the image generation unit 8e transmits all the pixels of the low-resolution subject existing image P1 for live view display with respect to the pixels with the alpha value “0” and the pixels with the alpha value 1 ≦ α <254. Performs blending with a predetermined single color, and does not transmit the pixel with an alpha value of “255” to the predetermined single color without doing anything.
Thereafter, the display control unit 9 performs the subject cut-out image in which the subject is superimposed on a predetermined single color image based on the image data of the subject cut-out image P2 (for example, the subject cut-out image P2a) generated by the image generation unit 8e. P2 is displayed on the display screen of the display unit 10 (step S28).

  Subsequently, the central control unit 13 shifts the process to step S23, and the graph generation unit 8f, based on the corrected alpha map M (for example, alpha map Ma), displays an alpha value graph G (for example, The graphs Gb, Gc, etc .; see FIGS. 5B, 5C, etc.) are generated again, and the display control unit 9 generates the graph G generated by the graph generation unit 8f (for example, the graphs Gb, Gc, etc.). ) Is again displayed on the display screen of the display unit 10 (step S23).

Next, in the same manner as described above, the central control unit 13 determines whether or not a predetermined threshold value relating to determination of whether or not the subject image Pg is newly input based on a predetermined operation of the operation input unit 12 by the user. Is determined (step S24). That is, if the user determines that the correction of the subject image Pg of the subject clipped image P2 is insufficient even when the corrected alpha map M (for example, the alpha map Ma) is used, a new threshold value is input. Will be.
In step S24, when a predetermined threshold value for determining whether or not the subject image is Pg is newly input (step S24; YES), the correction area specifying unit 8g causes the alpha map M (for example, alpha) to be input. An area composed of pixels having a value equal to or greater than a predetermined threshold in the map Ma or the like is specified again as a correction area (step S25). Then, the image correction unit 8h sets the alpha value of the correction region in the alpha map M (for example, the alpha map Ma), that is, the alpha value of all pixels of the intermediate gradation having a value equal to or greater than a predetermined threshold (alpha value) to the maximum value. (For example, “255” in 256 gradations) By changing to the modified alpha map M (for example, alpha map Mb, Mc, etc .; see FIG. 5E, FIG. 5F, etc.) (Step S26).

Next, in the same manner as described above, the image generation unit 8e generates a subject image Pg based on the subject existing image P1 for live view display and the corrected alpha map M (for example, alpha map Mb, Mc, etc.). Processing is performed to generate image data of a subject cutout image P2 (for example, subject cutout images P2b, P2c, etc .; see FIG. 5H, FIG. 5I, etc.) combined with the single color image (step S27).
Thereafter, as described above, the display control unit 9 converts the subject into a predetermined single color image based on the image data of the subject cutout image P2 (for example, the subject cutout images P2b, P2c, etc.) generated by the image generation unit 8e. Is displayed on the display screen of the display unit 10 (step S28).

  Subsequently, the central control unit 13 shifts the process to step S23 to display a graph G (for example, graphs Gb, Gc, etc.) of the alpha value of the corrected alpha map M (for example, alpha map Mb, Mc, etc.). The display control unit 9 again generates and displays the graph G (for example, graphs Gb, Gc, etc.) generated by the graph generation unit 8f on the display screen of the display unit 10 (step S23).

Next, in the same manner as described above, the central control unit 13 determines whether or not a predetermined threshold value relating to determination of whether or not the subject image Pg is newly input based on a predetermined operation of the operation input unit 12 by the user. Is determined (step S24). That is, if the user determines that the correction of the subject image Pg of the subject cutout image P2 is sufficient by using the corrected alpha map M (for example, the alpha map Mc), a new threshold value is input. Never happen.
If it is determined in step S24 that a predetermined threshold value for determining whether or not the subject image is Pg is not input (step S24; NO), the central control unit 13 determines that the subject clipped image P2 It is determined whether or not the subject image Pg has been corrected (step S29).

When it is determined in step S29 that the subject image Pg has been corrected (step S29; YES), the recording medium control unit 11 reads the image data of the subject existing image P1 in the predetermined area of the recording medium 11a and the correction. The subsequent alpha map M (for example, the alpha map Mc) is associated with the image data of the subject cutout image P2 and recorded in one file (step S30).
On the other hand, if it is determined that the subject image Pg has not been corrected (step S29; NO), the central control unit 13 skips step S30 and ends the subject correction process.

  As described above, according to the imaging apparatus 100 of the present embodiment, the region to be corrected in the subject image Pg is specified, and the specified region to be corrected is based on the corresponding region in the subject existing image P1. Since the correction is made, the subject image Pg can be appropriately corrected even after the subject image Pg is once extracted from the subject existing image P1.

Specifically, a graph G of an alpha value in the direction of a predetermined axis (for example, the x axis) of an alpha map M having a predetermined gradation indicating the position of the subject image Pg in the subject existing image P1 is generated. Thus, an area to be corrected in the subject image Pg is specified based on setting a predetermined threshold value (alpha value) relating to the determination as to whether or not the subject image is Pg. Further, every time a predetermined threshold value is set, by changing the alpha value of the intermediate gradation pixel having a value equal to or higher than the predetermined threshold value to the maximum value (for example, “255” in 256 gradations), Modify alpha map M sequentially.
Then, each time the alpha map M is sequentially corrected, the subject clipping image P2 is generated by clipping the subject image Pg from the subject existing image P1 for live view display based on the alpha map M. The subject image Pg of the image P2 can be corrected. That is, instead of generating the subject cutout image P2 from the high resolution subject existing image P1 recorded for recording, the subject cutout image P2 is generated from the low resolution subject existing image P1 for display. The subject presence image P1 can be recorded without being processed as an original image. Further, by using the low-resolution subject existence image P1 for display, it is possible to increase the processing speed related to the generation of the subject image Pg compared to the case where the high-resolution subject existence image P1 is used.
At this time, each time the subject cutout image P2 is generated (every time the subject image Pg is corrected), the subject image Pg of the subject cutout image P2 is displayed, so that the user can confirm the degree of correction of the subject image Pg. It is possible to determine whether or not further correction is necessary for the subject image Pg.

The present invention is not limited to the above-described embodiment, and various improvements and design changes may be made without departing from the spirit of the present invention.
For example, in the above embodiment, the predetermined threshold (alpha value) relating to the determination of whether or not the subject image Pg is set based on the predetermined operation of the operation input unit 12 by the user. For example, the predetermined threshold value may be automatically set. That is, a predetermined threshold value is set by automatically moving a straight line indicating a predetermined threshold value on the y axis in the graph G in a predetermined direction (for example, a direction in which an alpha value is changed from “255” to “0”). Then, the alpha map M may be automatically corrected based on the threshold value.

In the above embodiment, the correction region in the subject image Pg is specified based on the alpha value in which a straight line indicating a predetermined threshold is set on the y axis in the graph G of the alpha map M. However, this is only an example, and the present invention is not limited to this, and the method of specifying the correction region can be arbitrarily changed as appropriate.
That is, the operation input unit 12 includes a touch panel (not shown), and the correction area specifying unit 8g is designated by the user specifying an area to be corrected on the subject image Pg displayed on the display unit 10. The correction area may be specified. Specifically, when an area to be corrected is specified on the subject image Pg displayed on the display unit 10, the correction area specifying unit 8g specifies the specified area as an area to be corrected. To do. Then, the image correction unit 8h changes the alpha value of the pixel corresponding to the correction area in the alpha map M (the alpha value of the intermediate gradation pixel) to the maximum value (for example, “255” in 256 gradations). That will fix the alpha map M.

  As described above, since the region to be corrected in the subject image Pg designated based on the predetermined operation of the operation input unit 12 by the user is specified, the correction region is specified to the user by the subject image Pg displayed on the display unit 10. It can be specified intuitively, and the subject image Pg can be easily corrected.

  Further, in the above embodiment, the alpha map M is corrected based on the fact that the central control unit 13 sets a predetermined threshold value (alpha value). However, the alpha map M correction method is an example. However, the present invention is not limited to this. For example, even if a predetermined threshold value (alpha value) is not set, an alpha value equal to or higher than a predetermined intermediate gradation value is automatically set to a maximum value (for example, 256 gradations). , “255”), or an alpha value less than a predetermined intermediate gradation value may be changed to a minimum value (for example, “0” for 256 gradations).

  Further, the corrected alpha map M (for example, alpha map Mc) recorded on the recording medium 11a may be subjected to an averaging filter to apply a synthetic gradation to the edge of the subject area. When the subject image Pg is combined with the background image, a natural feeling that the boundary portion with the subject region is not clear can be obtained.

  In addition, the configuration of the imaging apparatus 100 is merely an example illustrated in the above embodiment, and is not limited thereto. That is, the imaging apparatus 100 is illustrated as the image processing apparatus, but is not limited thereto. For example, the subject presence image P1 and the subject non-existence image are picked up by an image pickup device (not shown) different from the image pickup device 100, and only the image data transferred from the image pickup device is recorded. A configuration in which clipping processing, subject correction processing, or the like is executed may be employed.

In addition, in the above-described embodiment, the functions of the extraction unit, the specification unit, and the correction unit are the image extraction unit 8c, the image generation unit 8e, the correction region specification unit 8g, and the image correction unit 8h of the image processing unit 8. The configuration realized by driving is not limited to this, and may be realized by executing a predetermined program or the like by the CPU of the central control unit 13.
That is, a program memory (not shown) that stores the program of the imaging apparatus 100 stores a program including an extraction processing routine, a specific processing routine, and a correction processing routine. Then, by the extraction processing routine, the CPU of the central control unit 13 causes each pixel corresponding between the subject existing image P1 in which the subject exists in the background and the subject nonexistent image in which the subject does not exist in the same background as the background. Based on this difference information, it may be made to function as an extracting means for extracting the subject image Pg including the subject from the subject existing image P1. Further, the CPU of the central control unit 13 may function as a specifying unit that specifies a region to be corrected in the subject image Pg extracted by the extracting unit by the specifying process routine. Further, the CPU of the central control unit 13 may function as a correction unit that corrects the area to be corrected specified by the specifying unit based on the corresponding area in the subject existing image P1 by the correction processing routine.

  Similarly, the first generation unit, the second generation unit, the setting unit, the third generation unit, and the display control unit may be realized by executing a predetermined program or the like by the CPU of the central control unit 13. good.

  Furthermore, as a computer-readable medium storing a program for executing each of the above processes, a non-volatile memory such as a flash memory or a portable recording medium such as a CD-ROM is applied in addition to a ROM or a hard disk. Is also possible. A carrier wave is also used as a medium for providing program data via a predetermined communication line.

DESCRIPTION OF SYMBOLS 100 Imaging device 1 Lens part 2 Electronic imaging part 3 Imaging control part 8 Image processing part 8c Image extraction part 8d Position information generation part 8e Image generation part 8f Graph generation part 8g Correction area specification part 8h Image correction part 12 Operation input part 13 Center Control unit

Claims (6)

Based on the corresponding differential information of each pixel with the non-existing object absence image of the subject on the subject-including image and the previous xenon in Jing the presence of an object in the background, include the subject from the subject existing image Extraction means for extracting a subject image;
First generation means for generating an alpha map of a predetermined gradation indicating a position of the subject image extracted by the extraction means in the subject existing image;
Second generation means for generating a graph of an alpha value in a predetermined axis direction based on the alpha map generated by the first generation means;
Setting means for setting a predetermined threshold value for determining whether or not the image is the subject image in the graph generated by the second generation means;
Based on this setting the predetermined threshold value set by means specifying means for identify the region to be corrected in the extracted object image by said extraction means,
Correction means for correcting the area to be corrected specified by the specifying means based on an area corresponding to the area to be corrected in the subject existing image;
An image processing apparatus comprising: The image processing apparatus according to claim 1 , wherein the correction unit sequentially corrects the alpha map generated by the first generation unit each time the predetermined threshold is set by the setting unit. . Third generation means for sequentially generating a subject cutout image obtained by cutting out the subject image from the subject existing image for display based on the alphamap each time the alphamap is corrected by the correction means;
3. The image processing apparatus according to claim 2 , further comprising: a display control unit configured to sequentially display the subject clipped image on the display unit each time the subject clipped image is generated by the third generation unit. .   The image processing apparatus according to claim 1, wherein the specifying unit specifies a region specified based on a predetermined operation of the operation unit in the subject image as the region to be corrected. An image processing method using an image processing apparatus,
Based on the corresponding differential information of each pixel with the non-existing object absence image of the subject on the subject-including image and the previous xenon in Jing the presence of an object in the background, include the subject from the subject existing image Processing to extract a subject image;
A process for generating an alpha map of a predetermined gradation indicating a position of the extracted subject image in the subject existing image;
Based on the generated alpha map, a process of generating a graph of alpha values in a predetermined axis direction,
A process of setting a predetermined threshold value for determining whether or not the image is the subject image in the generated graph;
Based on the set the predetermined threshold, the process of identify the region to be corrected in the extracted object image,
A process of correcting the identified area to be corrected based on an area corresponding to the area to be corrected in the subject existing image;
An image processing method comprising: The computer of the image processing device
Based on the corresponding differential information of each pixel with the non-existing object absence image of the subject on the subject-including image and the previous xenon in Jing the presence of an object in the background, include the subject from the subject existing image Extraction means for extracting a subject image;
First generation means for generating an alpha map of a predetermined gradation indicating a position of the subject image extracted by the extraction means in the subject existing image;
Second generation means for generating a graph of alpha values in a predetermined axial direction based on the alpha map generated by the first generation means;
Setting means for setting a predetermined threshold value for determining whether or not the image is the subject image in the graph generated by the second generation means;
Based on the predetermined threshold set by the setting means, identification means for identify the area to be corrected in the subject image extracted by the extraction means,
Correction means for correcting the area to be corrected specified by the specifying means based on an area corresponding to the area to be corrected in the subject existing image;
A program characterized by functioning as