JP2010226291A - Image synthesis apparatus, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an apparatus and a program for generating a synthetic image reducing a feeling of wrongness. <P>SOLUTION: An imaging apparatus 100 includes: a clip image generating portion 8d for acquiring a subject clip image obtained by clipping an area including a subject from a subject existing image in which a background and the subject exist; a segment detection portion 8e for detecting a segment extending along an edge portion of the subject image in the subject clip image; and a synthesis control portion 8g for restricting a synthesizing position of the subject clip image with a background image so that the segment of the detected subject clip image approximately coincides with the approximately straight edge portion of the background image. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image composition apparatus and a program for synthesizing a plurality of images.

  Conventionally, an application that takes an image with a subject in the background, then takes a background image without the subject in the background, generates difference information from the background image and the image with the subject, and extracts only the subject (For example, refer to Patent Document 1).

JP 2004-159158 A

  However, when the subject image extracted by the above technique is cut from the angle of view, there is a problem that when the image is combined with a background image, the subject image becomes a noticeable unnatural image. .

  Therefore, an object of the present invention is to be able to generate a composite image with little uncomfortable feeling.

In order to solve the above-described problem, an image composition device according to a first aspect of the present invention provides:
First detection means for detecting a substantially straight line portion having a predetermined length or more from the contour of the subject image, and second detection means for detecting a substantially straight line portion having a predetermined length or more from the background image to be combined with the subject image. And the background image and the substantially linear portion of the subject image detected by the first detection means and the substantially straight line portion of the background image detected by the second detection means substantially match. And a synthesizing unit for synthesizing the subject images.

According to a second aspect of the present invention, in the image composition device according to the first aspect,
Determining means for determining whether or not a substantially straight line segment having a predetermined length or more has been detected by the first detection unit; and determining that a substantially straight line segment having a predetermined length or more has been detected by the determining unit, It further comprises synthesis control means for controlling the synthesis means to perform synthesis.

According to a third aspect of the present invention, in the image composition device according to the first or second aspect,
The synthesizing unit is configured to detect a substantially straight line segment of the subject image detected by the first detecting unit and a background image when the second detecting unit does not detect a substantially straight line segment from the background image. The subject image is synthesized with the background image so that the edge portion substantially matches.

According to a fourth aspect of the present invention, in the image synthesizing device according to any one of the first to third aspects,
The synthesizing unit rotates the subject image in a predetermined direction so that the substantially straight line segment detected by the second detection unit and the substantially straight line segment detected by the first detection unit substantially coincide with each other. It is characterized by combining with images.

The invention according to claim 5 is the image composition device according to any one of claims 1 to 4,
A determination unit for comparing and determining a length of the substantially straight line detected from the background image by the second detection unit and a length of the substantially straight line of the subject image detected by the first detection unit; The synthesizing unit adjusts the size of the subject image based on the determination result by the determining unit, and synthesizes it with the background image.

The invention according to claim 6 is the image synthesizing apparatus according to any one of claims 1 to 5,
An image pickup means for picking up the subject presence image, and an image generation means for extracting the region including the subject from the subject presence image picked up by the image pickup means and generating the subject image. Yes.

The program of the invention described in claim 7 is:
A first detection unit configured to detect a substantially straight line portion having a predetermined length or more from the contour of the subject image; and a substantially straight line portion having a predetermined length or more from the background image to be combined with the subject image. Second detecting means for detecting, so that the substantially straight line portion of the subject image detected by the first detecting means and the substantially straight line portion of the background image detected by the second detecting means substantially match. It is characterized by functioning as a synthesis means for synthesizing the background image and the subject image.

  According to the present invention, it is possible to generate a composite image with little discomfort in which the edge portions of the subject image and the background image are substantially matched.

It is a block diagram which shows schematic structure of the imaging device of Embodiment 1 to which this invention is applied. 3 is a flowchart illustrating an example of an operation related to subject clipping processing by the imaging apparatus of FIG. 1. It is a figure which shows typically an example of the image which concerns on the to-be-photographed object cutting process of FIG. 3 is a flowchart illustrating an example of an operation related to background image generation processing by the imaging apparatus of FIG. 1. 3 is a flowchart illustrating an example of an operation related to a composite image generation process by the imaging apparatus of FIG. 1. It is a flowchart which shows an example of the operation | movement which concerns on the image composition process in the composite image generation process of FIG. It is a figure which shows typically an example of the image which concerns on the synthesized image generation process of FIG. It is a figure which shows typically an example of the to-be-photographed object image which concerns on the to-be-photographed object process by the imaging device of Embodiment 2 to which this invention is applied. It is a flowchart which shows an example of the operation | movement which concerns on the composite image generation process by the imaging device of FIG. It is a figure which shows typically an example of the image which concerns on the synthesized image generation process of FIG. It is a figure which shows typically an example of the image which concerns on a synthetic image generation process.

  Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.

[Embodiment 1]
FIG. 1 is a block diagram illustrating a schematic configuration of an imaging apparatus 100 according to the first embodiment to which the present invention is applied.
The imaging apparatus 100 according to the first embodiment detects a line segment L1 extending along the edge of the subject cutout image P1 (see FIG. 7B), and uses the line segment L1 of the subject cutout image P1 and the background. The composition position of the subject cutout image P1 with respect to the background image P2 is limited so that the peripheral portion E of the image P2 (see FIG. 7A) substantially matches.
Specifically, as illustrated in FIG. 1, the imaging apparatus 100 includes a lens unit 1, an electronic imaging unit 2, an imaging control unit 3, an image data generation unit 4, an image memory 5, and a feature amount calculation unit. 6, a block matching unit 7, an image processing unit 8, a recording medium 9, a display control unit 10, a display unit 11, an operation input unit 12, and a CPU 13.
In addition, the imaging control unit 3, the feature amount calculation unit 6, the block matching unit 7, the image processing unit 8, and the CPU 13 are designed as, for example, a custom LSI 1A.

The lens unit 1 includes a plurality of lenses and includes a zoom lens, a focus lens, and the like.
Although not shown, the lens unit 1 includes a zoom drive unit that moves the zoom lens in the optical axis direction and a focus drive unit that moves the focus lens in the optical axis direction when imaging a subject. May be.

  The electronic imaging unit 2 is composed of an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-oxide Semiconductor), for example, and converts an optical image that has passed through various lenses of the lens unit 1 into a two-dimensional image signal. To do.

Although not shown, the imaging control unit 3 includes a timing generator, a driver, and the like. Then, the imaging control unit 3 scans and drives the electronic imaging unit 2 with a timing generator and a driver, converts the optical image into a two-dimensional image signal with the electronic imaging unit 2 every predetermined period, and the electronic imaging unit 2 Image frames are read out from the imaging area for each screen and output to the image data generation unit 4.
In addition, the imaging control unit 3 performs adjustment control of conditions when imaging a subject such as AF (automatic focusing process), AE (automatic exposure process), and AWB (automatic white balance).

The imaging lens unit 1, the electronic imaging unit 2, and the imaging control unit 3 configured as described above serve as an imaging unit as a subject presence image P <b> 3 (see FIG. 3B) and a background image P <b> 2 (FIG. 7 (a)).
In addition, the imaging lens unit 1, the electronic imaging unit 2, and the imaging control unit 3 generate the subject cutout image P1 after fixing the subject presence image P3 and fixing the imaging conditions when capturing the subject presence image P3. A non-existing subject image P4 (see FIG. 3B) is captured.

The image data generation unit 4 appropriately adjusts the gain for each RGB color component with respect to the analog value signal of the image frame transferred from the electronic imaging unit 2, and then performs sample holding by a sample hold circuit (not shown). The digital signal is converted into digital data by a / D converter (not shown), color processing including pixel interpolation processing and γ correction processing is performed by a color process circuit (not shown), and then a digital luminance signal Y and color difference signal Cb , Cr (YUV data).
The luminance signal Y and the color difference signals Cb and Cr output from the color process circuit are DMA-transferred to an image memory 5 used as a buffer memory via a DMA controller (not shown).

  The image memory 5 is composed of, for example, a DRAM or the like, and temporarily stores data processed by the feature amount calculation unit 6, the block matching unit 7, the image processing unit 8, the CPU 13, and the like.

The feature amount calculation unit 6 performs feature extraction processing for extracting feature points from the subject non-existing image P4 with reference to the subject non-existing image P4. Specifically, the feature amount calculation unit 6 selects a predetermined number (or a predetermined number or more) of highly featured block regions (feature points) based on the YUV data of the subject non-existing image P4, and Is extracted as a template (for example, a square of 16 × 16 pixels).
Here, the feature extraction process is a process of selecting a feature having a high characteristic convenient for tracking from a large number of candidate blocks.

  The block matching unit 7 performs block matching processing for aligning the subject non-existing image P4 and the subject existing image P3. Specifically, the block matching unit 7 determines where the template extracted by the feature extraction process corresponds in the subject existing image P3, that is, the template pixel value optimally matches in the subject existing image P3. Search for a position (corresponding region). Then, the optimum offset between the subject non-existing image P4 and the subject existing image P3 having the best evaluation value (for example, the sum of squared differences (SSD), the sum of absolute differences (SAD), etc.) is calculated. Calculated as a template motion vector.

The image processing unit 8 includes an alignment unit 8a that performs alignment between the subject presence image P3 and the subject non-existence image P4.
The alignment unit 8a calculates a coordinate conversion formula (projection conversion matrix) of each pixel of the subject presence image P3 with respect to the subject non-existence image P4 based on the feature points extracted from the subject non-existence image P4. Accordingly, the subject presence image P3 is coordinate-converted to align with the subject non-existence image P4.

  Further, the image processing unit 8 generates difference information of corresponding pixels between the subject presence image P3 and the subject non-existence image P4 aligned by the alignment unit 8a, and the subject exists based on the difference information. A subject image extraction unit 8b that extracts a subject image A including a subject from the image P3 is provided.

Further, the image processing unit 8 specifies a position of the subject image A extracted in the subject presence image P3, and generates a position information generation unit 8c that generates position information indicating the position of the subject image A in the subject presence image P3. It has.
Here, as the position information, for example, an alpha map can be cited. The alpha map is a weight for alpha blending the image of the subject image A with respect to a predetermined background for each pixel of the subject existing image P3. It is expressed as a value (0 ≦ α ≦ 1).

Further, the image processing unit 8 does not transmit a pixel having an alpha value of 1 to a predetermined single color image (not shown) among the pixels of the subject existing image P3 based on the generated alpha map. In addition, a cut-out image generation unit that generates the image data of the cut-out subject image P1 (see FIG. 3C) by synthesizing the subject image A with a predetermined single-color image so that pixels with an alpha value of 0 are transmitted. 8d.
Thereby, the cut-out image generation unit 8d acquires a subject cut-out image P1 obtained by cutting out a region including the subject from the subject-existing image P3 in which the background and the subject exist.

In addition, the image processing unit 8 includes a line segment detection unit 8e that detects a line segment L1 extending in a predetermined direction of the subject image A.
The line segment detection unit 8e detects, based on the image data of the subject cutout image P1, whether or not the subject image A including the subject exists in the upper, lower, left and right peripheral portions of the image. Specifically, in the line segment detection unit 8e, pixels having an alpha value of 1 in the alpha map (that is, a portion other than a single color in a single color image) are set to a predetermined length at the upper, lower, left and right peripheral portions of the subject cutout image P1. It is determined whether or not the images are continuously arranged, and a portion (for example, the right end portion of the subject image A in FIG. 7B) continuously arranged for a predetermined length or longer is an edge portion of the subject cutout image P1. Is detected as a line segment L1 of the subject image A extending along the line.

In addition, the image processing unit 8 includes an image composition unit 8f that synthesizes the subject image A and the background image P2. Specifically, the image composition unit 8f acquires a user-desired background image P2 from the recording medium, and transmits a pixel having an alpha value of 0 among the pixels of the background image P2, so that the alpha value is The 1 pixel is overwritten with the pixel value of the corresponding pixel of the subject cutout image P1, and among the pixels of the background image P2, the pixel whose alpha value is 0 <α <1 is a 1's complement (1-α). Is used to generate an image (background image × (1−α)) obtained by cutting out the subject image A, and then when the subject cutout image P1 is generated using the one's complement (1−α) in the alpha map. A value blended with one background color is calculated, and the value is subtracted from the subject cutout image P1 and is combined with an image obtained by cutting out the subject image A (background image × (1-α)).
The composition position of the subject image A with respect to the background image P2 will be described later.

Further, the image processing unit 8 includes a synthesis control unit 8g that limits the synthesis position of the subject image A with respect to the background image P2.
The composition control unit 8g uses the background image of the subject image A so that the line segment L1 of the subject image A detected by the line segment detection unit 8e substantially coincides with the vertical and horizontal straight edge portions E of the background image P2. A restriction is added to the synthesis position for the image P2. Here, the position where the line segment L1 of the subject image A and the peripheral portion E of the background image P2 substantially match is the state where the line segment L1 of the subject image A and the peripheral portion E of the background image P2 match, or The line segment L1 of the subject image A and the peripheral edge portion E of the background image P2 are arranged with a predetermined number of pixels apart from each other, but can be regarded as substantially matching.
For example, as shown in FIGS. 7A to 7C, the composition control unit 8g has a position where the right end line segment L1 of the subject image A substantially matches the right edge portion E of the background image P2. The subject image A is moved only when the right edge portion E of the background image P2 and the line segment L1 at the right end of the subject image A substantially match so that the subject image A can be combined with the background image P2. It can be placed. Then, based on a predetermined operation of the selection determination button 12b of the operation input unit 12 by the user, the composite position of the subject image A is adjusted to determine the composite position, or the right end line segment L of the subject image A is determined. When the composition position is automatically determined from the ratio of the background image P2 to the right edge portion E, the image composition unit 8f composites the subject image A and the background image P2 at the determined composition position. Thus, the subject composite image P5 is generated.

The recording medium 9 is composed of, for example, a nonvolatile memory (flash memory) or the like, and stores image data of the background image P2 and the subject cutout image P1 encoded by the JPEG compression unit (not shown) of the image processing unit 8. To do.
The image data of the subject image A is associated with the alpha map generated by the position information generation unit 8c of the image processing unit 8, and the extension of the image data of the subject cutout image P1 is saved as “.jpe”. Has been.

The display control unit 10 performs control for reading display image data temporarily stored in the image memory 5 and displaying the read image data on the display unit 11.
Specifically, the display control unit 10 includes a VRAM, a VRAM controller, a digital video encoder, and the like. The digital video encoder periodically reads the luminance signal Y and the color difference signals Cb and Cr read from the image memory 5 and stored in the VRAM (not shown) under the control of the CPU 13 from the VRAM via the VRAM controller. Are read out, a video signal is generated based on these data, and is output to the display unit 11.

  The display unit 11 is, for example, a liquid crystal display device, and displays an image captured by the electronic imaging unit 2 based on a video signal from the display control unit 10 on a display screen. Specifically, the display unit 11 displays a live view image based on a plurality of image frames generated by imaging an object by the imaging lens unit 1, the electronic imaging unit 2, and the imaging control unit 3 in the imaging mode. Or a REC view image captured as the actual captured image.

  The operation input unit 12 is for performing a predetermined operation of the imaging apparatus 100. Specifically, the operation input unit 12 includes a shutter button 12a related to a subject shooting instruction, a selection determination button 12b related to a selection instruction such as an imaging mode and a function, and a setting position of a composition position of the subject image A, and a zoom amount adjustment. A zoom button (not shown) or the like according to instructions is provided, and a predetermined operation signal is output to the CPU 13 in accordance with the operation of these buttons.

  The CPU 13 controls each part of the imaging device 100. Specifically, the CPU 13 performs various control operations in accordance with various processing programs (not shown) for the imaging apparatus 100.

Next, the subject clipping process performed by the imaging apparatus 100 will be described with reference to FIGS.
FIG. 2 is a flowchart illustrating an example of an operation related to the subject clipping process. 3A to 3C are diagrams schematically illustrating an example of an image related to the subject clipping process.
In the subject clipping process described below, it is assumed that the subject presence image P3 in which the subject is completely cut off at the right end of the angle of view is captured.

The subject clipping process is executed when a subject clipping mode is selected from a plurality of imaging modes displayed on the menu screen based on a predetermined operation of the selection determination button 12b of the operation input unit 12 by the user. It is.
As shown in FIG. 2, first, the CPU 13 causes the display control unit 10 to display a live view image based on a plurality of image frames generated by imaging an object by the imaging lens unit 1, the electronic imaging unit 2, and the imaging control unit 3. Is displayed on the display screen of the display unit 11 and superimposed on the live view image, an imaging instruction message of the subject existing image P3 is displayed on the display screen of the display unit 11 (step S1).

Next, the CPU 13 determines whether or not the user has performed an imaging instruction operation on the shutter button 12a of the operation input unit 12 (step S2). If it is determined that the shutter button 12a has been operated to instruct imaging (step S2; YES), the CPU 13 instructs the imaging controller 3 to determine the focus lens focus position and exposure conditions (shutter speed, aperture, amplification factor, etc.). ) And white balance are adjusted, and an optical image of the subject existing image P3 (see FIG. 3A) is picked up by the electronic image pickup unit 2 under a predetermined condition, and the image data generation unit 4 makes the electronic image pickup. YUV data of the image frame of the subject existing image P3 transferred from the unit 2 is generated (step S3). Note that the YUV data of the subject existing image P3 is temporarily stored in the image memory 5.
Further, the CPU 13 controls the imaging control unit 3 to maintain a state in which conditions such as a focus position, an exposure condition, and a white balance at the time of imaging the subject presence image P3 are fixed.

Then, the CPU 13 causes the display control unit 10 to display a live view image on the display unit 11 based on a plurality of image frames generated by imaging the subject by the imaging lens unit 1, the electronic imaging unit 2, and the imaging control unit 3. And the image of the semi-transparent display mode of the subject presence image P3 and the imaging instruction message of the subject non-existence image P4 are superimposed on the live view image and displayed on the display screen of the display unit 11 (step S4). .
Thereafter, the CPU 13 determines whether or not the user has performed an imaging instruction operation on the shutter button 12a of the operation input unit 12 (step S5). Then, the user moves the subject outside the angle of view or waits for the subject to move, and then adjusts the camera position so that the subject non-existing image P4 overlaps the semi-transparent image of the subject existing image P3. When it is determined that the shutter button 12a of the operation input unit 12 has been operated to instruct imaging (step S5; YES), the CPU 13 causes the imaging control unit 3 to display the subject nonexistent image P4 (see FIG. 3B). Based on the image frame of the subject non-existence image P4 transferred from the electronic imaging unit 2 to the image data generation unit 4. Thus, the YUV data of the subject non-existing image P4 is generated (step S6). Note that the YUV data of the subject existing image P3 is temporarily stored in the image memory 5.

Next, the CPU 13 uses the YUV data of the subject non-existing image P4 temporarily stored in the image memory 5 in the feature amount calculating unit 6, the block matching unit 7, and the image processing unit 8 as a reference to YUV of the subject existing image P3. A projection transformation matrix for projectively transforming data is calculated using a predetermined image transformation model (for example, a similarity transformation model or a joint transformation model) (step S7).
Specifically, the feature amount calculation unit 6 selects a predetermined number (or a predetermined number or more) of highly featured block regions (feature points) based on the YUV data of the subject non-existing image P4, and Is extracted as a template. Then, the block matching unit 7 searches the subject existing image P3 for a position where the pixel value of the template extracted by the feature extraction process is optimally matched, and the evaluation value of the difference between the pixel values is the best. An optimum offset between the subject non-existing image P4 and the subject existing image P3 is calculated as a motion vector of the template. Then, the alignment unit 8a of the image processing unit 8 statistically calculates the entire motion vector based on the motion vectors of the plurality of templates calculated by the block matching unit 7, and performs feature point correspondence related to the motion vector. The projection transformation matrix of the subject existing image P3 is calculated by using it.

  Next, the CPU 13 causes the alignment unit 8a to perform projective transformation on the subject presence image P3 based on the calculated projective transformation row example, so that the YUV data of the subject presence image P3 and the YUV data of the subject non-existence image P4 Is performed (step S8).

Then, the CPU 13 causes the subject image extraction unit 8b of the image processing unit 8 to perform processing for extracting the subject image A including the subject from the subject presence image P3 (step S9).
Specifically, the subject image extraction unit 8b applies a low-pass filter to each of the YUV data of the subject presence image P3 and the YUV data of the subject non-existence image P4 to remove high frequency components of each image. Thereafter, the subject image extraction unit 8b calculates a difference for each corresponding pixel between the subject existing image P3 and the subject non-existing image P4 subjected to the low-pass filter, and generates a difference map. Subsequently, the subject image extraction unit 8b binarizes the dissimilarity map relating to each pixel with a predetermined threshold, and then performs a contraction process in order to remove an area in which the difference is caused by fine noise or camera shake from the dissimilarity map. Do. After that, the subject image extraction unit 8b performs a labeling process, removes a region below a predetermined value or a region other than the maximum region, identifies the largest island pattern as the subject image A, and corrects the shrinkage. Expansion processing is performed.

  Next, the CPU 13 causes the position information generation unit 8c of the image processing unit 8 to generate an alpha map indicating the position of the extracted subject image A in the subject presence image P3 (step S10).

Thereafter, the CPU 13 performs a process of generating image data of a subject cutout image P1 (see FIG. 3C) obtained by combining the subject image A with a predetermined single color image in the cutout image generation unit 8d of the image processing unit 8. (Step S11).
Specifically, the cut-out image generation unit 8d reads out the subject existing image P3, the single color image, and the alpha map and develops them in the image memory 5, and thereafter, for all the pixels of the subject existing image P3, the pixels having an alpha value of 0. (Α = 0) for transmission, and for pixels with an alpha value of 0 <α <1, (0 <α <1), blending with a predetermined single color is performed, and for pixels with an alpha value of 1 (α = 1) Do nothing and do not transmit light to a predetermined single color.

Thereafter, the CPU 13 associates the alpha map generated by the position information generation unit 8c of the image processing unit 8 with the image data of the subject clipped image P1 in a predetermined storage area of the recording medium 9, and stores the clipped image P1 of the subject clipped image P1. The extension of the image data is saved as one file with “.jpe” (step S12).
Thereby, the subject clipping process is completed. As a result, the image data of the subject cutout image P1 including the subject image A extracted in a state where it is completely cut off at the right end of the angle of view is generated.

Next, background image generation processing by the imaging apparatus 100 will be described with reference to FIG.
FIG. 4 is a flowchart illustrating an example of an operation related to the background image generation processing.

The background image generation process is a normal still image imaging process, and a still image is selected from a plurality of imaging modes displayed on the menu screen based on a predetermined operation of the selection determination button 12b of the operation input unit 12 by the user. This is a process executed when an imaging mode selection instruction is given.
As shown in FIG. 4, first, the CPU 13 causes the display control unit 10 based on a plurality of image frames generated by imaging the background image P <b> 2 by the imaging lens unit 1, the electronic imaging unit 2, and the imaging control unit 3. The live view image is displayed on the display screen of the display unit 11 (step S21).

  Next, the CPU 13 determines whether or not the user has performed an imaging instruction operation on the shutter button 12a of the operation input unit 12 (step S22). If it is determined that the shutter button 12a has been operated to instruct imaging (step S22; YES), the CPU 13 instructs the imaging controller 3 to determine the focus lens focus position and exposure conditions (shutter speed, aperture, amplification factor, etc.). ) And white balance are adjusted, and the optical image of the background image P2 (see FIG. 6B) is picked up by the electronic image pickup unit 2 under a predetermined condition (step S23).

Subsequently, the CPU 13 causes the image data generation unit 4 to generate the YUV data of the image frame of the background image P2 transferred from the electronic imaging unit 2, and then the background image is stored in a predetermined storage area of the recording medium 9. The YUV data of P2 is stored as an Exif format image file (step S24).
Thus, the background image generation process is terminated.

Next, the composite image generation process will be described in detail with reference to FIGS.
FIG. 5 is a flowchart illustrating an example of an operation related to the composite image generation process. FIG. 6 is a flowchart illustrating an example of an operation related to the image composition processing in the composite image generation processing. FIGS. 7A to 7C are diagrams schematically illustrating an example of an image related to the composite image generation process.

The composite image generation processing is executed when an image combination mode is selected from a plurality of imaging modes displayed on the menu screen based on a predetermined operation of the selection determination button 12b of the operation input unit 12 by the user. It is processing.
As shown in the flowchart of FIG. 5, based on a predetermined operation of the operation input unit 12 by the user, a desired background image P <b> 2 that becomes the background of the composite image among a plurality of images recorded on the recording medium 9 (FIG. 7 (a)) is selected and designated, the image processing unit 8 reads out the image data of the designated background image P2 and develops it in the image memory 5 (step S31).

Next, based on a predetermined operation of the operation input unit 12 by the user, a desired subject cutout image P1 (see FIG. 7B) is selected and specified from among a plurality of images recorded on the recording medium 9. Then, the image processing unit 8 reads out the image data of the designated subject cutout image P1 and develops it in the image memory 5 (step S32).
Subsequently, based on the image data of the subject cutout image P1, the line segment detection unit 8e of the image processing unit 8 is a line composed of pixels other than the above-described single color (subject image A) from the upper, lower, left and right peripheral portions of the image. By detecting the segment L1 and determining whether or not the line segment L1 of the subject image A is equal to or longer than a predetermined length, it is determined whether or not the segment L1 exists in the peripheral portion of the subject clipped image P1 (step S1). S33).

Here, when it is determined that the line segment L1 exists in the peripheral portion of the subject cutout image P1 (subject image A) (step S33; YES), the composition control unit 8g combines the subject image A with the background image P2. (Step S34). For example, when the line segment L1 is present at the right edge portion of the subject image A (see FIG. 7B), the right edge portion of the subject image A and the right edge portion E of the background image P2 substantially match. So that the subject image A can be synthesized with the background image P2 only at the position where the subject image A is in a state where the right edge portion E of the background image P2 and the right end line segment L1 of the subject image A substantially match. Can be moved and arranged (see FIG. 7C).
Thereafter, when the composition position of the subject image A is determined (step S35), the image composition unit 8f performs image composition processing for compositing the background image P2 and the subject cutout image P1 (step S36).
In step S34, the ratio of the rightmost line segment L of the subject image A to the right edge portion E of the background image P2, etc., without restricting the composition position of the subject image A with respect to the background image P2. Alternatively, the synthesis position may be automatically determined.

  On the other hand, if it is determined in step S33 that there is no line segment L1 in the edge portion of the subject cutout image P1 (step S33; NO), based on a predetermined operation of the selection determination button 12b of the operation input unit 12 by the user, After specifying and determining the synthesis position of the user-described subject clipped image P1 (step S37), the CPU 13 shifts the processing to step S36 to synthesize the background image P2 and the subject clipped image P1. I do.

Here, the image composition processing will be described in detail with reference to FIG.
As shown in FIG. 6, the image composition unit 8f reads an alpha map stored in association with the subject cutout image P1 and develops it in the image memory 5 (step S41).
Note that when the synthesis position of the subject cutout image P1 in the background image P2 is determined in step S35 in FIG. 5, the background image P2 and the alpha map are shifted, and the region is outside the alpha map range. , Α = 0, so that an area where no alpha value exists is not generated.

  Next, the image composition unit 8f designates any one pixel (for example, the pixel at the upper left corner) of the background image P2 (step S42), and processes the pixel based on the alpha value of the alpha map. Is branched (step S43). Specifically, the image composition unit 8f, for any one of the pixels of the background image P2, for the pixel having an alpha value of 1 (step S43; α = 1), the corresponding pixel of the subject cutout image P1 The pixel value is overwritten (step S44), and the subject image A is cut out using a one's complement (1-α) for pixels with an alpha value of 0 <α <1 (step S43; 0 <α <1). After the image (background image × (1-α)) is generated, the blended value with the single background color is calculated when the subject cutout image P1 is generated using the one's complement (1-α) in the alpha map. Then, the value is subtracted from the subject cutout image P1 and is combined with an image obtained by cutting out the subject image A (background image × (1−α)) (step S45, for pixels with an alpha value of 0 (step S43; α = 0), what So that monkey through the background image P2 without.

Subsequently, the image composition unit 8f determines whether or not all the pixels of the background image P2 have been processed (step S46).
If it is determined that all the pixels have not been processed (step S46; NO), the image composition unit 8f designates the next pixel as a processing target and moves the processing target to the pixel (step S46). S47), the process proceeds to step S43.
By repeating the above process until it is determined that all pixels have been processed in step S46 (step S46; YES), the image composition unit 8f composites the subject cutout image P1 and the background image P2. Image data of the subject composite image P5 is generated.
Thereby, the image composition process is terminated.

As shown in FIG. 5, thereafter, the CPU 13 causes the display control unit 10 to combine the subject on the background image P2 based on the image data of the subject composite image P5 generated by the image composition unit 8f. P5 (see FIG. 7C) is displayed on the display screen of the display unit 11 (step S38).
Thus, the composite image generation process is finished.

As described above, according to the imaging apparatus 100 of the first embodiment, the subject presence image P3 is captured, and the subject cutout image P1 obtained by extracting the subject image A including the subject from the subject presence image P3 is generated. Then, a line segment L1 extending along the peripheral portion in the subject cutout image P1 of the subject image A is detected, and the line segment L1 of the subject image A and the straight peripheral portion E of the background image P2 are approximately. A restriction is imposed on the synthesis position of the subject image A with respect to the background image P2 so as to match. Specifically, when a line segment L1 having a predetermined length or more is detected from the subject cutout image P1, the composition position of the subject image A with respect to the background image P2 is limited.
As a result, for example, when a subject existing image P3 in which the subject is completely viewed at an angle of view is captured, a subject cutout image P1 is generated from the subject existing image P3, and the subject image A is combined with the background image P2, the subject image Depending on the composition position of A, the line segment L1 of the edge portion of the image becomes conspicuous, but by limiting the composition position of the subject image A with the background image P2, the line segment L1 of the subject image A and the background image It is possible to generate a natural subject composite image P5 without any sense of incongruity in which the straight peripheral portion E of P2 substantially matches.

[Embodiment 2]
Hereinafter, the imaging apparatus 100 according to the second embodiment will be described with reference to FIGS.
The imaging apparatus 100 according to the second embodiment detects a line segment L2 extending along a predetermined direction from the background image P2, and the line segment L2 in the background image P2 and the line segment L1 of the subject cutout image P1 are detected. Restrict the synthesis position so that it almost matches.
Note that the imaging apparatus 100 of the second embodiment has substantially the same configuration as the imaging apparatus 100 of the first embodiment except that the configuration of the line segment detection unit 8e and the composition control unit 8g is different, and the description thereof is omitted. .

Based on the image data of the subject cutout image P1, the line segment detection unit 8e detects whether or not the line segment L1 is present at the edge portion of the subject image A including the subject in the image. For example, when a person whose body part is hidden behind a column in the angle of view is photographed as a subject presence image P3 (see FIG. 8A), the subject presence image P3 and the subject non-existence image P4 (FIG. 8 ( When the subject image A is detected by using (b)) and the subject cut-out image P1 is generated, as shown in FIG. 8C, the subject cut-out image P1 is formed along the vertical direction substantially at the center other than the peripheral portion. A line segment extending along the edge of the subject image A is generated.
In such a case, the line segment detection unit 8e has pixels having an alpha value of 1 in the alpha map (that is, a portion other than a single color in a single color image) along a predetermined direction at the edge portion of the subject image A. It is determined whether or not a predetermined length or more is continuously arranged, and a portion (for example, the right end portion of the subject image A in FIG. 10B) that is continuously arranged for a predetermined length or more is detected as a line segment L1. .

  Further, the line segment detection unit 8e detects a line segment L2 extending along a predetermined direction in the background image P2. Specifically, the line segment detection unit 8e performs a Hough transform on the image data of the background image P2, thereby detecting a line segment L2 extending along a predetermined direction in the image. For example, as illustrated in FIG. 10A, the line segment detection unit 8e detects a line segment L2 extending along the outer surface of the building image in the background image P2.

The composition control unit 8g restricts the composition position of the background image P2 and the subject cutout image P1 so that the line segment L2 in the background image P2 and the line segment L1 of the edge portion of the subject image A substantially match each other. Add.
For example, as shown in FIG. 10A to FIG. 10C, the composition control unit 8g displays the left end of the building in the line segment L1 of the right end portion of the subject image A of the subject cutout image P1 and the background image P2. The line segment L2 of the left outer side surface portion of the building of the background image P2 and the subject image A so that the subject clipped image P1 can be combined with the background image P2 only at a position where the line segment L2 of the side surface portion substantially matches The subject cut-out image P1 can be moved and arranged only in a state where the line segment L1 of the right end portion of the image substantially matches. Then, based on a predetermined operation of the selection determination button 12b of the operation input unit 12 by the user, the composite position of the subject cutout image P1 is adjusted to determine the composite position, or the line segment of the right end portion of the subject image A When the composition position is automatically determined based on the ratio of the background image P2 of L1 to the left side portion of the building, the image composition unit 8f and the subject clipping image P1 at the determined composition position of the subject clipping image P1. The subject image P5 is generated by synthesizing the background image P2.

Next, the composite image generation process will be described in detail with reference to FIGS.
FIG. 9 is a flowchart illustrating an example of an operation related to the composite image generation process. FIGS. 10A to 10C are diagrams schematically illustrating an example of an image related to the composite image generation process.
In the composite image generation process described below, it is assumed that a subject cutout image P1 having a line segment L1 extending in the vertical direction at the substantially central portion at the edge portion of the subject image A is used.

  As shown in the flowchart of FIG. 9, as in the first embodiment, based on a predetermined operation of the operation input unit 12 by the user, a desired background that is a background of a composite image among a plurality of images recorded on the recording medium 9 When the background image P2 (see FIG. 10A) is selected and designated, the image processing unit 8 reads out the image data of the designated background image P2 and develops it in the image memory 5 (step S31). ).

Next, as in the first embodiment, a desired subject cutout image P1 (see FIG. 10B) among a plurality of images recorded on the recording medium 9 based on a predetermined operation of the operation input unit 12 by the user. ) Is selected and designated, the image processing unit 8 reads out the image data of the designated subject cutout image P1 and develops it in the image memory 5 (step S32).
Subsequently, the line segment detection unit 8e of the image processing unit 8 determines whether or not there is a line segment L1 of a predetermined length or more based on the image data of the subject cutout image P1 (step S51). Specifically, the line segment detection unit 8e detects a line segment L1 including pixels (subject image A) other than a single color from the upper, lower, left, and right edge portions of the subject cutout image P1, and also detects the subject image A in the image. By detecting whether or not the line segment L1 exists at the edge portion, and determining whether or not the detected line segment L1 of the subject image A is longer than a predetermined length, the line segment L1 in the subject cutout image P1 is determined. Judge whether there is.

If it is determined that the subject cutout image P1 has the line segment L1 (step S51; YES), the line segment detection unit 8e extends the detected line segment L1 along the edge portion of the subject cutout image P1. It is determined whether or not it is to be performed (step S52).
Here, when it is determined that the line segment L1 extends along the edge portion of the subject cutout image P1 (step S52; YES), the composition control unit 8g, as in the first embodiment, A restriction is applied so that the combined position of the cut-out image P1 with the background image P2 is the peripheral edge portion E of the background image P2 (step S53).

  On the other hand, if it is determined in step S52 that the line segment L1 does not extend along the edge of the subject cutout image P1 (step S52; NO), that is, the subject image A in the subject cutout image P1. When the line segment L1 is present at the edge portion of the image (see FIG. 10A), the line segment detection unit 8e performs Hough transform on the image data of the background image P2, thereby A line segment L2 extending along a predetermined direction is detected, and it is determined whether or not there is a line segment L2 in the background image P2 (step S54). Note that the line segment detection unit 8e may detect only the line segment L2 extending along the same direction as the line segment L1 of the subject image A of the subject cutout image P1.

Here, if it is determined that the line segment L2 is present in the background image P2 (step S54; YES), for example, as illustrated in FIG. 10A, the line segment detection unit 8e includes the line segment L2 in the background image P2. When the line segment L2 extending along the outer surface of the building image is detected, the composition control unit 8g sets the composition position of the subject cutout image P1 to the background image P2 to be the line segment L2 in the background image P2. Restrictions are added as described above (step S55). For example, the composition control unit 8g applies the subject image A to the background image P2 only at a position where the right end line segment L1 of the subject image A of the subject cutout image P1 substantially matches the line segment L2 in the background image P2. So that the subject clipped image P1 can be moved and arranged only when the line segment L2 in the background image P2 and the rightmost line segment L1 of the subject image A of the subject clipped image P1 substantially match each other ( (Refer FIG.10 (c)).
On the other hand, if it is determined in step S54 that there is no line segment L2 in the background image P2 (step S54; NO), the CPU 13 shifts the processing to step S53, and the composition control unit 8g Is limited so that the combination position with respect to the background image P2 becomes the edge portion of the background image P2 (step S53).
Thereafter, as in the first embodiment, when the synthesis position of the subject image A is determined (step S56), the image synthesis unit 8f performs an image synthesis process for synthesizing the background image P2 and the subject image A (step S56). S36).

  If it is determined in step S51 that the subject cutout image P1 does not have the line segment L1 (step S51; NO), the user's desired operation is determined based on a predetermined operation of the selection determination button 12b of the operation input unit 12 by the user. After designating and determining the synthesis position of the subject cutout image P1 (step S37), the CPU 13 shifts the processing to step S36 and performs image synthesis processing for synthesizing the background image P2 and the subject cutout image P1.

After that, as in the first embodiment, the CPU 13 causes the display control unit 10 to combine the subject on the background image P21 based on the image data of the subject composite image P5 generated by the image composition unit 8f. P5 (see FIG. 10C) is displayed on the display screen of the display unit 11 (step S38).
Thus, the composite image generation process is finished.

As described above, according to the imaging apparatus 100 of Embodiment 2, the line segment L2 extending along the predetermined direction from the background image P2 is detected, and the line segment L2 in the background image P2 and the subject cutout image are detected. The synthesis position is limited so that the line segment L1 in P1 substantially matches. In addition, when the line segment L2 is not detected from the background image P2, the composition position is limited so that the line segment L1 of the subject image A and the peripheral portion E of the background image P2 substantially match.
As a result, the degree of freedom of composition of the subject image A with the background image P2 can be increased, and not only the peripheral portion E of the background image P2 but also the line segment L2 detected from the background image P2. In addition, it is possible to generate a natural subject composite image P5 that is substantially in agreement with the line segment L1 of the subject cutout image P1 and has no sense of incongruity.

Further, when the line segment L1 having a predetermined length or more is detected from the subject image A of the subject cutout image P1, the line segment L2 is detected from the background image P2, so that no subject exists for extracting the subject image A. When the subject cutout image P1 is generated from the subject existence image P3 that has been cut off by the shooting environment of the image P4, the line segment L2 is detected from the background image P2 and the line segment L1 of the subject cutout image P1 and the line of the background image P2 are detected. It is possible to generate a natural subject composite image P5 that does not have a sense of incongruity that substantially matches the minute L2.
Therefore, not only the subject existence image P3 in which the subject is completely viewed at the angle of view, but also the composition of the subject presence image P3 that has been observed in the shooting environment of the subject nonexistence image P4 for extracting the subject image A can be more appropriately performed.

The present invention is not limited to the first and second embodiments, and various improvements and design changes may be made without departing from the spirit of the present invention.
Below, the modification of the imaging device 100 is demonstrated.

For example, in the second embodiment, when the line segment L1 of the edge portion of the subject image A extends in a direction different from the line segment L2 in the background image P2, that is, as shown in FIG. The extending direction of the line segments L1 and L2 is substantially the same as when the line segment L1 of the edge portion of the image A extends obliquely while the line segment L2 in the background image P2 extends in the vertical direction. If they do not match, the composition control unit 8g rotates the subject image A in a predetermined direction so that the extending direction of the line segment L1 of the edge portion of the subject image A and the line segment L2 in the background image P2 substantially match. Processing may be performed (see FIG. 11B).
Thus, even when the extending direction of the line segment L1 of the edge portion of the subject image A and the line segment L2 in the background image P2 does not coincide with each other, the subject cut-out image P1 is rotated in a predetermined direction, and the subject image It is possible to generate a natural subject composite image P5 without a sense of incongruity in which the extending direction of the line segment L1 of the edge portion A and the line segment L2 in the background image P2 substantially match.

Further, in the first and second embodiments, the composition control unit 8g includes the line segment L in the background image P2 and the straight edge portion E of the background image P2, and the edge portion detected from the subject cutout image P1. The length of the line segment (including the line segment L1 of the edge portion of the subject image A) may be compared and determined, and processing for adjusting the size of the subject cutout image P1 may be performed based on the determination result. .
For example, with respect to the line segment L2 in the background image P2 and the straight line peripheral portion E of the background image P2, the line segment of the edge portion detected from the subject cutout image P1 (the line segment of the edge portion of the subject image A) (Including L1) is long, the subject cutout image P1 is reduced at a predetermined magnification.
As a result, even if the size of the subject image A is inappropriate for synthesis with the background image P2, the size of the subject image A is reduced by reducing the subject cutout image P1 at a predetermined magnification. Therefore, it is possible to obtain a natural subject composite image P5 without any sense of incongruity.

Further, in the first and second embodiments, when the subject cutout image P1 has two or more line segments L1, among these line segments L1, more line segments L1 are peripheral portions of the background image P2. It is preferable to synthesize the images so as to be in a state that substantially matches a predetermined line segment L2 in E and the background image P2.
In the first and second embodiments, one subject cutout image P1 is synthesized with the background image P2. However, the number of subject cutout images P1 can be arbitrarily changed as appropriate. It may be. In such a case, it is preferable to perform predetermined image processing, rotation processing, or the like so that all the subject cutout images P1,...
Furthermore, after performing the subject clipping process, the background image generation process is performed, but the order of these processes may be reversed.

  In addition, the configuration of the imaging apparatus 100 is merely an example illustrated in the first and second embodiments, and is not limited thereto. In other words, the image pickup apparatus is exemplified as the image composition apparatus, but is not limited thereto. For example, the subject cutout image P1 and the background image P2 are generated by an imaging device different from the imaging device 100, and only the image data transferred from the imaging device is recorded, and only the composite image generation processing is executed. An image synthesizing apparatus may be used.

In addition, in the above embodiment, the functions of the first detection unit, the second detection unit, and the synthesis unit are controlled under the control of the CPU 13 by the line segment detection unit 8e and the image synthesis unit 8f of the image processing unit 8. However, the present invention is not limited to this, and may be realized by executing a predetermined program or the like by the CPU 13.
In other words, a program including a first detection processing routine, a second detection processing routine, and a synthesis processing routine is stored in a program memory (not shown) that stores the program. Then, the first detection processing routine may cause the CPU 13 to detect a substantially straight line portion having a predetermined length or more from the contour of the subject image A. Further, the second detection processing routine may cause the CPU 13 to detect a substantially straight line portion having a predetermined length or more from the background image P2 to be combined with the subject image A. Further, the CPU 13 synthesizes the background image P2 and the subject image A so that the substantially straight line portion of the detected subject image A and the detected substantially straight line portion of the background image P2 substantially coincide with each other by the composition processing routine. You may do it.

DESCRIPTION OF SYMBOLS 100 Imaging device 1 Lens part 2 Electronic imaging part 3 Imaging control part 8 Image processing part 8d Cutout image generation part 8e Line segment detection part 8f Image composition part 8g Composition control part 9 Recording medium 11 Display part 13 CPU

Claims (7)

First detection means for detecting a substantially straight line segment having a predetermined length or more from the contour of the subject image;
Second detection means for detecting a substantially straight line portion having a predetermined length or more from a background image to be combined with the subject image;
The background image and the subject image so that a substantially straight line segment of the subject image detected by the first detection unit and a substantially straight line portion of the background image detected by the second detection unit substantially match. A synthesis means for synthesizing
An image composition device comprising: Determining means for determining whether or not a substantially straight line portion having a predetermined length or more has been detected by the first detecting means;
If it is determined by this determination means that a substantially straight line segment having a length equal to or greater than the predetermined length has been detected, a composition control means for controlling the composition means to perform composition;
The image synthesizing apparatus according to claim 1, further comprising:   The synthesizing unit is configured to detect a substantially straight line segment of the subject image detected by the first detecting unit and a background image when the second detecting unit does not detect a substantially straight line segment from the background image. The image synthesizing apparatus according to claim 1, wherein the subject image is synthesized with the background image so that an edge portion substantially matches. The synthesis means includes
The subject image is rotated in a predetermined direction so that the substantially straight line segment detected by the second detector and the substantially straight line segment detected by the first detector substantially coincide with each other and synthesized with the background image. The image composition device according to any one of claims 1 to 3. A determination unit for comparing and determining a length of the substantially straight line detected from the background image by the second detection unit and a length of the substantially straight line of the subject image detected by the first detection unit;
The synthesis means includes
5. The image synthesizing apparatus according to claim 1, wherein a size of the subject image is adjusted based on a determination result by the determination unit and is synthesized with the background image. 6. Imaging means for imaging the subject presence image;
6. The image generating device according to claim 1, further comprising: an image generating unit configured to extract a region including a subject from the subject existing image captured by the imaging unit and generate the subject image. The image composition device described in 1. The computer of the image synthesizer
First detecting means for detecting a substantially straight line portion having a predetermined length or more from the contour of the subject image;
Second detection means for detecting a substantially straight line portion having a predetermined length or more from the background image to be synthesized with the subject image;
The background image and the subject image so that a substantially straight line segment of the subject image detected by the first detection unit and a substantially straight line portion of the background image detected by the second detection unit substantially match. A program that functions as a synthesis means for synthesizing.

Images