JP5092958B2 - Image processing apparatus and program - Google Patents

Description

The present invention relates to an image processing apparatus, and relates to a program.

2. Description of the Related Art Conventionally, an imaging device that extracts a subject from a captured image and synthesizes the subject and a background image to obtain a composite photograph is known (see, for example, Patent Document 1). Specifically, at least two images are acquired by changing the aperture value, the subject is extracted from the subject extraction image, and the subject is combined with the background image.
JP 2006-128754 A

  However, in Patent Document 1, since at least two subjects must be photographed and recorded, the photographer must photograph at least two images with different photographing conditions for the same photographing angle of view. There was a problem.

Accordingly, an object of the present invention is to provide an image processing apparatus and a program capable of determining an image region including a main subject from one subject image and combining the images in consideration of the position of the image region. It is.

In order to solve the above-mentioned problem, the invention described in claim 1
Material image storage means for storing a material image, position specifying means for specifying the position of a first image area corresponding to a main subject included in one subject image, and the one subject specified by the position specifying means A composite position input means for inputting the composite position of the material image in the second image area other than the position of the first image area of the image by detecting an external operation, and the composite position input means A composite position that is closer to the position of the first image area specified by the position specifying means than the composite position is acquired, and the material image stored in the material image storage means is acquired at the acquired composite position. And a synthesis processing means for performing a process of synthesis.

The invention according to claim 2 is the invention according to claim 1 ,
The main subject is drawn by excluding an outline of a pixel set that constitutes the subject image and an outline of a pixel set that is smaller than a predetermined value among the outlines extracted by the extraction means. Contour specifying means for specifying the main contour line, wherein the position specifying means determines the position of the image area surrounded by the main contour line specified by the contour specifying means as the main image area. It is characterized by specifying as the position of.

The invention according to claim 3 is the invention according to claim 1 ,
A division unit that divides the one subject image into a plurality of regions; and an information acquisition unit that acquires frequency component information for each of the regions divided by the division unit. Based on the frequency component information acquired by the information acquisition means, an area where the high frequency component is larger than a predetermined value is determined as an image area constituting the first image area.

According to a fourth aspect of the invention, there is provided the method according to any one of the first to third aspects, further comprising an imaging unit that images the subject image.

According to the fifth aspect of the present invention, the computer specifies a position of a first image region corresponding to a main subject included in one subject image, and the one subject specified by the position specifying unit. Composite position input means for inputting the composite position of the material image in the second image area other than the position of the first image area of the image by detecting an external operation, and the composite position input by the composite position input means Rather than a composite processing unit that acquires a composite position that is closer to the position of the first image area specified by the position specifying unit, and performs a process of combining the material image with the acquired composite position . It is characterized by that.

According to the present invention, it is possible to discriminate an image region including a main subject from one subject image, and to combine the images in consideration of the position of the image region .

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.
FIG. 1 is a diagram schematically illustrating an appearance of an imaging apparatus 100 according to an embodiment to which the present invention is applied. FIG. 1A illustrates a front view of the imaging apparatus 100, and FIG. A rear view of the imaging device 100 is shown, and FIG. 1C shows a plan view of the imaging device 100. FIG. 2 is a block diagram illustrating a schematic configuration of the imaging apparatus 100.

The imaging apparatus 100 of the present embodiment specifies the position of the main image area (first image area) A1 corresponding to the main subject S in one monitor display image or recording image, and Predetermined processing is performed on the background area (second image area) A2 other than the position of the main image area A1 in the recording image.
Specifically, as illustrated in FIGS. 1A to 1C, the imaging device 100 has an outer shape formed in a substantially rectangular parallelepiped shape, and the imaging lens 1 a of the imaging unit 1 is exposed on the front side thereof. The strobe light emission window 2 is disposed on the upper right side of the imaging lens 1a (see FIG. 1A). Further, as shown in FIG. 1B, the display device 3 is disposed on the rear surface of the imaging device 100 so as to be exposed, and on the operation input unit 4 on the right side of the display screen 3a. A menu key 4a, a cursor key 4b and a set key 4c are provided. A shutter button 4d and a power button 4e provided in the operation input unit 4 are arranged side by side on the upper surface of the imaging apparatus 100.
Although illustration is omitted, for example, a USB connection terminal 5 (see FIG. 2) for connecting to an external device such as a PC via a USB cable (not shown) on either the left or right side of the imaging apparatus 100. Is arranged.

  As shown in FIG. 2, the imaging apparatus 100 includes an imaging unit 1, a DMA control unit 6, a work memory 7, an image engine 8, a display unit 3, a display control unit 9, a VRAM 10, and an operation. An input unit 4, a USB control unit 11, a CPU 12, and a flash memory 13 are provided.

  The imaging unit 1, as an imaging unit, continuously captures a subject and generates a plurality of image frames. Specifically, the imaging unit 1 includes an imaging lens 1a, an electronic imaging unit 1b, an ISP 1c, and the like.

The imaging lens 1a is composed of, for example, a plurality of lenses and includes a focusing lens, a zoom lens, and the like.
The electronic imaging unit 1b includes a CCD (Charge Coupled Device), a CMOS (Complementary Metal-Oxide Semiconductor), or the like that converts a subject image that has passed through the imaging lens 1a into a two-dimensional image signal. Further, the electronic imaging unit 1b performs A / D conversion on the photoelectrically converted image signal and outputs it as digital data to the ISP 1c.
The digital data output from the electronic imaging unit 1b has a format according to an array (for example, Bayer array) of color filters (not shown).
An ISP (Image Signal Processor) 1c performs color adjustment, data format conversion (for example, conversion from a Bayer arrangement to an 8-bit luminance signal Y and color difference signals Cb and Cr) based on the input digital data. .

  Data (luminance signal Y, color difference signals Cb, Cr) output from the imaging unit 1 is DMA-transferred and stored in the work memory 7 under the control of the DMA control unit 6.

  The work memory 7 is composed of, for example, a DRAM or the like, and temporarily stores a system program, various programs, various data, various parameters, and the like read from the flash memory 13.

  The image engine 8 converts the luminance signal Y and the color difference signals Cb and Cr in the work memory 7 into RGB data and outputs them to the VRAM 10 or converts them into a predetermined file format such as JPEG at the time of image shooting to the flash memory 13. Output.

Further, the image engine 8 performs a display image CG synthesis process or a captured image CG synthesis process (described later) under the control of the CPU 12 with a desired image stored in the flash memory 13 for the monitor display image and the recording image. The CG image 50 is synthesized.
Here, the image engine 8 and the CPU 12 function as image composition means.

The display control unit 9 performs control to read out display data temporarily stored in the VRAM 10 and display it on the display unit 3.
The display unit 3 displays an image captured by the imaging unit 1 on the display screen 3a based on an output signal from the display control unit 9. Specifically, the display unit 3 displays a through image, a main image (recorded image) recorded in the flash memory 13 based on a plurality of image frames generated by imaging the subject by the imaging unit 1, and CG synthesis processing. The CG composite image I6 (see FIG. 10) and the like generated in the above are displayed.

The operation input unit 4 is for performing a predetermined operation of the imaging apparatus 100, and includes a menu key 4a, a cursor key 4b, a set key 4c, a shutter button 4d, a power button 4e, and the like. A predetermined operation signal is output to the CPU 12 according to the operation.
The menu key 4a instructs display on the display unit 3 of a menu screen related to function selection, display setting, and the like based on an operation by the user.

  The cursor key 4b includes, for example, up / down / left / right cursor keys 4b, and instructs selection of various items displayed on the menu screen based on an operation by the user. Specifically, as an operation mode of the imaging apparatus 100, for example, a normal shooting mode for shooting a still image of a predetermined size, a CG composite shooting mode for combining the CG image 50 with a still image, and a recorded image are reproduced and displayed. Instructs selection of playback mode and the like. The cursor key 4b instructs selection of an image displayed on the display unit 3 in the reproduction mode. Furthermore, the cursor key 4b instructs selection of the CG image 50 to be combined with the still image in the CG composite shooting mode.

The shutter button 4d instructs imaging of the subject by the imaging unit 1 based on an operation by the user.
The power button 4e instructs to turn on / off the power of the imaging apparatus 100 based on an operation by the user.

  The USB control unit 11 controls transmission / reception of various data to / from an external device such as a PC connected to the USB connection terminal 5 via a USB cable.

  The CPU 12 performs various control operations in accordance with various processing programs for the imaging device 100 stored in the flash memory 13.

  The flash memory 13 stores various programs and data necessary for the operation of the CPU 12. Specifically, as shown in FIG. 2, the flash memory 13 stores an extraction program 13a, an outline specifying program 13b, a CG synthesis control program 13c, and the like.

The extraction program 13a causes the CPU 12 to function as extraction means. In other words, the extraction program 13a is a program for causing the CPU 12 to realize a function related to the extraction processing for extracting the outline of the pixel set constituting the subject image.
Specifically, when the CPU 12 executes the extraction program 13a, a monitor display image or a recording image captured and acquired by the imaging unit 1 is subjected to smoothing processing (described later). A first-order differentiation process is performed by a 3 × 3 matrix operation using a bell filter to extract a contour line (edge portion) of the pixel set (see FIG. 8B).
In the extraction process, for example, a differential filter, a Prewitt filter, a Laplacian filter, or the like may be used instead of the Sobel filter, or any two or more of these may be used.

The contour specifying program 13b causes the CPU 12 to function as contour specifying means. That is, the contour specifying program 13b removes the contour of the pixel set smaller than the predetermined value from the contour extracted by the extraction process, thereby creating the outer peripheral contour (main contour) L1 depicting the main subject S. This is a program for causing the CPU 12 to realize the function related to the contour line specifying process for specifying the.
Specifically, the CPU 12 executes the contour line specifying program 13b, so that an expansion process for improving edge connectivity with respect to the image I2 after the extraction process (see FIG. 8B), and a predetermined number In addition, the outer contour of the main subject S is specified by performing small area deletion processing excluding connected components (image sets) having a small number of connected pixels.

The CG composition control program 13c is a program for causing the CPU 12 to implement control related to the CG composition processing for causing the image engine 8 to compose the CG image 50 with the background area A2 (second image area) of the subject image.
Specifically, the subject image extracted by the background extraction process (described later) in the image engine 8 by the display image CG synthesis process or the captured image CG synthesis process by the CPU 12 executing the CG synthesis control program 13c. The CG image 50 is synthesized with the CG synthesis coordinates in the background area (second image area) A2.

Further, the flash memory 13 stores a plurality of CG images (material images) 50 to be combined with the subject image in the CG combining process as a material image storage unit.
Furthermore, the flash memory 13 stores a main image (captured image) captured by the imaging unit 1, a CG composite image I6 (see FIG. 10) after the CG composite processing, and the like.

Next, CG composition processing by the imaging apparatus 100 will be described with reference to FIGS.
In the following description, it is assumed that the CG composite shooting mode is set in advance based on a predetermined operation of the operation input unit 4 by the user.
As the CG synthesis process, there are a display image CG synthesis process for synthesizing the CG image 50 with the through image, and a captured image CG synthesis process for synthesizing the CG image 50 with the main image (recorded image).

First, the display image CG composition processing will be described with reference to FIG.
FIG. 3 is a flowchart illustrating an example of an operation related to the display image CG composition process.

When the imaging apparatus 100 is set to the CG composite shooting mode, as shown in FIG. 3, the CPU 12 executes a process of capturing and acquiring a monitor display image by the imaging unit 1 (step S1A).
Next, based on the acquired monitor display image, the CPU 12 executes CG composite coordinate calculation processing for calculating CG composite coordinates in the monitor display image (step S2A: details will be described later).
Subsequently, the CPU 12 executes the CG composition control program 13c in the flash memory 13, and the image engine 8 performs a through image (monitor display image) based on the CG composition coordinates calculated in the CG composition coordinate calculation processing. The CG image 50 is synthesized (step S3A), and the display unit 3 displays the synthesized CG synthesized image I6 (see FIG. 10) on the display screen 3a (step S4A).

Next, the captured image CG composition processing will be described with reference to FIG.
FIG. 4 is a flowchart illustrating an example of an operation related to the captured image CG composition process.

When the user operates the shutter button 4d for a predetermined time while displaying a through image on the display screen 3a or displaying the CG composite image I6 synthesized by the display image CG synthesis process, the CPU 12 drives the optical system. The control unit (not shown) moves the focusing lens in the optical axis direction, and executes a process of capturing and acquiring a recording image focused on the main subject S by the imaging unit 1 (step S1B).
Next, the CPU 12 executes CG composite coordinate calculation processing for calculating CG composite coordinates in the recording image based on the acquired recording image (step S2B: details will be described later).
Subsequently, the CPU 12 executes the CG composition control program 13c in the flash memory 13, and the image engine 8 adds the CG image 50 to the recording image based on the CG composition coordinates calculated in the CG composition coordinate calculation processing. After combining (step S3B), the DMA control unit 6 transfers the combined CG combined image I6 (see FIG. 10) to the flash memory 13 and stores it (step S4B).

Next, the CG composite coordinate calculation process will be described with reference to FIGS.
FIG. 5 is a flowchart illustrating an example of an operation related to the CG composite coordinate calculation process. FIG. 6 is a flowchart illustrating an example of an operation related to the small area deletion process in the CG composite coordinate calculation process, and FIG. 7 is a flowchart illustrating an example of an operation related to the background extraction process in the G composite coordinate calculation process. . FIGS. 8A and 8B and FIGS. 9A to 9C are diagrams schematically showing images for explaining the CG composite coordinate calculation processing.

The CG composite coordinate calculation process (step S2A) in the display image CG composite process and the CG composite coordinate calculation process (step S2B) in the captured image CG composite process are substantially the same processes, and will be described together below.
In the CG composite coordinate calculation process described below, it is assumed that a predetermined CG image 50 to be combined with the subject image is set in advance based on a predetermined operation of the operation input unit 4 by the user.

In the CG composite coordinate calculation process, as shown in FIG. 5, first, the CPU 12 first displays the luminance signal Y of the monitor display image (recording image) I1 acquired by the imaging unit 1 (see FIG. 8A). Then, smoothing processing is performed using a Gaussian filter to remove noise (step S21).
Next, the CPU 12 executes the extraction program 13a in the flash memory 13, performs a first-order differentiation process on the smoothed image by a 3 × 3 matrix operation using a Sobel filter, and generates a pixel. The outline (edge portion) of the set is extracted (step S22; see FIG. 8B).

  Next, the CPU 12 executes the contour line specifying program 13b in the flash memory 13 and performs an expansion process on the extracted image I2 in order to enhance edge connectivity (step S23; FIG. 9). (Refer to (a)), the small area deletion process is performed on the image I3 after the expansion process to remove an isolated connected component (image set) whose number of connected pixels is smaller than a predetermined number (step S24; FIG. 9B). )reference)).

Hereinafter, the small area deletion process (step S24) will be described in detail with reference to FIG.
As shown in FIG. 6, first, the CPU 12 performs labeling for assigning the same number (label) to the pixel sets constituting the same connected component on the image (binarized image) I3 after the expansion processing. The number of labels is stored in “LN” (step S241).

Next, after initializing the loop variable n to 1 (step S242), the CPU 12 stores the number of connected n (for example, n = 1) th labels in “LS” (step S243). Then, the CPU 12 compares the predetermined threshold value DEF_LS set in advance with the number of connections of the nth label stored in “LS”, and determines whether the value of “LS” is larger than the predetermined threshold value DEF_LS. It is determined whether or not (step S244).
Here, when it is determined that the value of “LS” is equal to or smaller than the predetermined threshold DEF_LS (step S244; NO), the CPU 12 determines that the nth label related to the “LS” is a small area and deletes it. (Step S245).

Thereafter, the CPU 12 increments “n” related to the label number by +1 (n = n + 1) (step S246). If it is determined in step S244 that the value of “LS” is greater than the predetermined threshold DEF_LS, the process proceeds to step S246.
Thereafter, the CPU 12 compares the incremented “n” with the number of labels stored in “LN”, and determines whether “n” is larger than the value of “LN” (step S247). .
Here, when it is determined that “n” is equal to or less than the value of “LN” (step S247; NO), the process returns to step S243.

The above process is repeated until it is determined in step S247 that “n” is larger than the value of “LN” (step S247; YES). The deletion process ends. FIG. 9B shows the image I4 after the small area deletion process.
In this way, the CPU 12 serves as the position specifying means for the position of the image region surrounded by the outer peripheral contour of the main subject S in the monitor display image and the recording image as the main subject corresponding to the main subject S included in the subject image. The position is specified as the position of the image area (first image area) A1 (see FIG. 9B).

Thereafter, as shown in FIG. 5, the CPU 12 performs background extraction processing (step S25).
The background extraction process will be described in detail below with reference to FIG.
As shown in FIG. 7, first, the process branches according to the combination position of the CG image 50 designated in advance based on a predetermined operation of the operation input unit 4 by the user (step S251).
Here, when the synthesis position of the CG image 50 is designated on the left side of the main subject S, scanning is performed from the left end of the image toward the right side to track the boundary (contour line) of the connected components (step S252). On the other hand, when the synthesis position of the CG image 50 is designated on the right side of the main subject S, the boundary of the connected component is traced by scanning from the right end to the left side of the image (step S253).

  Then, the CPU 12 extracts a region until the main image region A1 specified by the small region deletion process is reached (from the black pixel to the white pixel) as a background region (second image region) A2 (step S1). S254).

  Thereafter, as shown in FIG. 5, the CPU 12 performs a substantially equal amount of contraction processing to correct the expansion due to the expansion processing (step S223) (step S26; see FIG. 9C).

Next, the CPU 12 calculates CG composite coordinates in the background area A2. Specifically, first, a branch is made according to the combination position of the CG image 50 designated in advance based on a predetermined operation of the operation input unit 4 by the user (step S27).
Here, when the synthesis position of the CG image 50 is designated on the left side of the main subject S, scanning is performed from the left end of the image I5 toward the right side, and the CG image 50 is placed on the right side as much as possible in the background area A2. CG composite coordinates are calculated so that they can be combined (step S28). On the other hand, when the composite position of the CG image 50 is designated on the right side of the main subject S, scanning is performed from the right end of the image I5 toward the left side. Then, the CG synthesis coordinates are calculated so that the CG image 50 can be synthesized as much as possible in the background area A2 (step S29; see FIG. 9C).
The vertical position of the CG composite coordinates is set, for example, approximately in the center in advance, but can be arbitrarily changed as appropriate based on a predetermined operation of the operation input unit 4 by the user.

Thereafter, in the display image CG composition process, as shown in FIG. 3, the CPU 12 executes the CG composition control program 13c in the flash memory 13, and the image engine 8 is calculated by the CG composition coordinate calculation process. Based on the CG composite coordinates, the through image and the CG image 50 are combined to generate a CG composite image I6 (see FIG. 10) (step S3A), and then the CG composite image I6 is displayed on the display screen of the display unit 3. It is displayed on 3a (step S4A).
Further, in the captured image CG composition process, as shown in FIG. 4, the CPU 12 executes the CG composition control program 13c in the flash memory 13, and the image engine 8 is calculated by the CG composition coordinate calculation process. Based on the CG composite coordinates, the recording image and the CG image 50 are combined to generate a CG composite image I6 (see FIG. 10) (step S3B), and then the CG composite image I6 is transferred to the flash memory 13. (Step S4B).
In this way, the CPU 12 and the image engine 8 serve as a processing unit as a predetermined processing for the background image A2 other than the main image region A1 of the monitor display image or the recording image. A captured image CG synthesis process is performed.

As described above, according to the imaging apparatus 100 of the present embodiment, the position of the main image area A1 corresponding to the main subject S in one monitor display image or recording image is specified, and the monitor display image or Predetermined processing such as CG synthesis processing can be performed on the background area A2 other than the position of the main image area A1 in the recording image. Specifically, by extracting the contour lines of the pixel set constituting the monitor display image and the recording image, and removing the contour lines of the pixel set smaller than a predetermined value from the extracted contour lines, Since the outer peripheral contour of the subject S is specified, an image region including the main subject S can be determined from one subject image without taking two subject images as in the conventional art.
Since the CG image 50 stored in the flash memory 13 is synthesized in the background area A2 other than the specified position of the main image area A1, the background area A2 excluding the main subject S can be easily processed. Can do.

  In the above embodiment, after performing the expansion process on the image I2 after the extraction process (step S23), the contraction process for correcting the expansion due to the expansion process is performed (step S26). However, the present invention is not limited to this. When the subject image has little noise and the main image area A1 corresponding to the main subject S can be properly specified, the above expansion processing and contraction processing are necessarily performed. There is no need.

The present invention is not limited to the above-described embodiment, and various improvements and design changes may be made without departing from the spirit of the present invention.
Below, the modifications 1-3 of an imaging device are demonstrated.

<Modification 1>
As shown in FIGS. 11 to 16, the imaging apparatus 200 of Modification 1 divides the monitor display image and the recording image into a plurality of image areas B,... For the image region B, frequency component information is acquired, and based on the frequency component information, a process of determining an image region B having a high frequency component larger than a predetermined value as an image region constituting the main image region A1 is performed.
Note that the imaging apparatus 200 of Modification 1 has substantially the same configuration as the imaging apparatus 100 of the above-described embodiment except for the CG composite coordinate calculation process described below, and a description thereof is omitted.

  That is, as shown in FIG. 11, the flash memory 13 of the imaging apparatus 200 according to the first modification includes a division program 13d and an information acquisition program 13e in addition to the extraction program 13a, the contour specifying program 13b, and the CG synthesis control program 13c. I remember it.

The dividing program 13d causes the CPU 12 to function as a dividing unit. That is, the division program 13d is a program for causing the CPU 12 to realize a function related to the division processing for dividing the monitor display image (recording image) I7 into a plurality of image regions B,.
Specifically, based on the execution of the division program 13d by the CPU 12, the luminance signal Y (see FIG. 14A) of the monitor display image (recording image) I7 is converted into a plurality of image regions B,. “YS” × horizontal “XS” is divided into 48 regions of 6 × 8).

The information acquisition program 13e causes the CPU 12 to function as information acquisition means. That is, the information acquisition program 13e is a program for causing the CPU 12 to realize a function related to the information acquisition process for acquiring the frequency component information for each image region B divided by the division process.
Specifically, based on the execution of the information acquisition program 13e by the CPU 12, a two-dimensional Fourier transform is performed on each of a plurality of image areas B,. The frequency amplitude information is acquired by taking this.

Next, CG composite coordinate calculation processing will be described with reference to FIGS.
FIG. 12 is a flowchart illustrating an example of an operation related to the CG composite coordinate calculation process. FIG. 13 is a diagram illustrating the sorting result of the amplitude values of the plurality of image regions B,... According to the CG composite coordinate calculation process. FIGS. 14A and 14B and FIGS. 15A to 15C are diagrams schematically showing images for explaining the CG composite coordinate calculation processing.

In the CG composite coordinate calculation process, as shown in FIG. 12, the CPU 12 executes the division program 13 d in the flash memory 13 to obtain the horizontal division number “XS” = 8 and the vertical division number “ Assuming that “YS” = 6, the luminance signal Y (see FIG. 14A) of the monitor display image (recording image) I7 is divided into a plurality of image areas B,... (For example, 48 areas) (step S201; FIG. 14 (b)).
Thus, for each of the plurality of image regions B,..., The horizontal size (XStep) becomes a value obtained by dividing the horizontal size (XSize) of the luminance signal Y by “XS”, and the vertical size (YStep). Is a value obtained by dividing the vertical size (YSize) of the luminance signal Y by “YS” (step S202). For example, when the image size of the luminance signal Y is 400 dots in the horizontal direction and 264 dots in the vertical direction, the XStep of each image region B is 50 (400 ÷ 8) dots and the YStep is 44 (264 ÷ 6) dots.

Next, an information acquisition process for acquiring frequency amplitude information for all image regions B is performed.
Of the plurality of image regions B,..., X = 1, y = 1, and the lower right corner region are x = XS, y using the index of x in the horizontal direction and y in the vertical direction. Let y = YS.

  First, after executing the information acquisition program 13e in the flash memory 13 and setting the loop variable y to 1 (step S203), the CPU 12 compares the value of y with the number of vertical divisions “YS”. It is determined whether or not the value of y is larger than the vertical division number “YS” (step S204).

If it is determined in step S204 that the value of y is less than or equal to the vertical division number “YS” (step S204; NO), the CPU 12 sets the loop variable x to 1 (step S205), and then x And the horizontal division number “XS” are compared to determine whether the x value is larger than the horizontal division number “XS” (step S206).
Here, if it is determined that the value of x is equal to or smaller than the horizontal division number “XS” (step S206; NO), the CPU 12 sets the image area B where x = 1 and y = 1 among the plurality of areas. Is stored in I {x, y} (step S207).

Subsequently, the CPU 12 obtains frequency component information including amplitude information and phase information by performing a two-dimensional Fourier transform on I {x, y}, and then calculates frequency amplitude information by taking an absolute value. And stored in FFT_I {x, y} (step S208).
Here, for each image area B, the origin of the frequency amplitude information moved to the center is shown in FIG. 15A, and the whiter the peripheral part is, the higher the frequency component is.
Then, the CPU 12 accumulates frequency amplitude information stored in FFT_I {x, y} that is 1 / K (for example, K = 2) times or more of the maximum frequency, and FFT_IS {x, y}. (Step S209).
Thereafter, the CPU 12 increments the loop variable x by +1 (x = x + 1) (step S210), and proceeds to step S206.

Then, until it is determined in step S206 that the value of x is larger than the horizontal division number “XS” (step S206; YES), the processes of steps S206 to S210 are repeatedly executed.
If it is determined that the value of x is larger than the horizontal division number “XS” (step S206; YES), the CPU 12 increments the loop variable y by +1 (y = y + 1) (step S211), and step S204. Migrate to

Then, until it is determined in step S204 that the value of y is larger than the vertical division number “YS” (step S204; YES), the processes in steps S204 to S210 are repeatedly executed.
If it is determined that the value of y is larger than the number of divisions “YS” in the vertical direction (step S204; YES), the CPU 12 makes a determination for setting a portion with a high frequency component as a subject to be photographed and the other as a background region A2. I do.

First, for a plurality of image regions B,..., The amplitude value (for example, X period; for example) of a high frequency component having a period of 1 / R (for example, R = 4) or more of the maximum frequency obtained after FFT from discrete points 4 / XStep, Y cycle; 4 / YStep) is added and substituted into FFT_IS {x, y} (step S212). As a result, the amplitude value of the high frequency component from which the low frequency is removed is substituted for FFT_IS {x, y}. The larger the value of FFT_IS {x, y}, the larger the spatial frequency of the image region B. Will be expensive.
Subsequently, the CPU 12 sorts all the FFT_IS {x, y} relating to the plurality of image regions B,..., And sets the threshold value TH as a point where the amplitude value changes sharply (step S213). For example, as shown in FIG. 13, when the amplitude values of 48 image regions B in which the horizontal division number “XS” = 8 and the vertical division number “YS” = 6 are sorted, 100000 is the boundary. Therefore, the threshold value TH is set to 100,000.

Next, the CPU 12 compares the threshold TH with all the FFT_IS {x, y} related to the image area B for a plurality of image areas B,... And the FFT_IS {x, y} smaller than the threshold TH. The image area B is set as the background area A2, and the other area is set as the main subject S (photographing target) (step S214).
In this way, the CPU 12 determines, as the position specifying means, the main image area A1 (first image area) corresponding to the main subject S from the image area B having a high frequency component larger than the predetermined value based on the frequency component information. It is determined that the image area is to be configured (see FIG. 15B).
In FIG. 15B, the image area B constituting the background area A2 is shown in black, and the other image areas B correspond to the main image area A1.

Next, the CPU 12 performs substantially the same processing as steps S27 to S29 in the imaging device 100 of the above embodiment, and calculates the CG composite coordinates in the background area A2. Specifically, first, branching is performed according to the combination position of the CG image 50 designated in advance based on a predetermined operation of the operation input unit 4 by the user (step S215).
Here, when the synthesis position of the CG image 50 is designated on the left side of the main subject S, scanning is performed from the left end of the image I8 toward the right side, and the CG image 50 is placed on the right side as much as possible in the background area A2. CG composite coordinates are calculated so that they can be combined (step S216). On the other hand, when the composite position of the CG image 50 is designated on the right side of the main subject S, scanning is performed from the right end of the image I8 toward the left side. Then, the CG synthesis coordinates are calculated so that the CG image 50 can be synthesized as far as possible in the background area A2 (step S217: see FIG. 15C).
In FIG. 15C, the CG composite coordinates are represented as a region C surrounded by a broken line.

  Thereafter, in step S3A in FIG. 3 (or step S4A in FIG. 4), the CPU 12 executes the CG synthesis control program 13c in the flash memory 13, and the image engine 8 is calculated by CG synthesis coordinate calculation processing. Based on the CG synthesized coordinates, the through image (or recording image) and the CG image 50 are synthesized to generate a CG synthesized image I9 (see FIG. 16), and then in step S4A, the CG synthesized image. I6 is displayed on the display screen 3a of the display unit 3 (or, in step S4B, the CG composite image I9 is transferred to the flash memory 13 and stored).

  Therefore, according to the imaging apparatus 200 of Modification 1, the monitor display image and the recording image are divided into a plurality of image areas B,... Component information is acquired, and based on the frequency component information, an image region B having a high frequency component larger than a predetermined value is determined as an image region constituting the main image region A1, so that it is not affected by noise. The image area including the main subject S can be determined from the subject images.

<Modification 2>
As shown in FIGS. 17 (a), 17 (b), 18 (a), and 18 (b), the imaging apparatus according to the second modification includes a CG image 50 in the background area A2 of the monitor display image I10. Is input based on a predetermined operation of the operation input unit 4 by the user, and the CG image 50 is combined with the input combining position.
Note that the imaging apparatus of Modification 2 has substantially the same configuration as that of the imaging apparatus 200 of Modification 1 except for the CG composite position determination process described below, and a description thereof will be omitted.

The cursor key 4b, as a composite position input means, inputs an instruction for the composite position of the CG image 50 with respect to the subject image.
Then, the CPU 12 performs CG combination position determination processing for determining the combination position of the CG image 50 in the monitor display image I10 based on a predetermined operation signal input in response to the operation of the cursor key 4b by the user.
Specifically, every time the left cursor key (right cursor key) 4b is operated by the user, the CPU 12 determines the synthesis position of the CG image 50 in the monitor display image I10, and the monitor display image I10 The CG image 50 moves counterclockwise (clockwise) with respect to the main subject S in the background area A2 (see FIGS. 17A and 18A). Then, on the display screen 3a of the display unit 3, the CG image 50 is synthesized with the monitor display image I10 by the image engine 8 based on the synthesis position of the CG image 50 set in the CG synthesis position determination process. The image I11 (see FIGS. 17B and 18B) is displayed.
As described above, the CPU 12 functions as a composite position determination unit that determines the composite position of the CG image 50.

  The CG image 50 may be moved according to the number of operations of the left cursor key (right cursor key) 4b, or may be moved only for the time during which the pressing operation is performed.

  Therefore, according to the imaging apparatus of the second modification, the position where the CG image 50 is synthesized within the background area A2 of the monitor display image I10 is input based on a predetermined operation of the operation input unit 4 by the user. Since the CG image 50 is synthesized at the synthesis position, the CG image 50 can be synthesized at an arbitrary position in the monitor display image, and a CG composite image I11 desired by the user can be obtained.

<Modification 3>
As shown in FIGS. 19A and 19B, the imaging apparatus according to the third modification has a CG image 50 based on the size of the area around the main subject S in the background area A2 of the subject image I12. The composite position of is automatically set.
Specifically, the CPU 12 determines the size of the area around the main subject S (for example, the upper side, the left side, and the right side) in the background area A2 of the subject image I12, and the CG image 50 in the largest area. Set the composite position. For example, the CPU 12 calculates the distance D1 from the main subject S to the upper edge, the distance D2 to the left edge, and the distance D3 to the right edge (see FIG. 19A), and the right area having the longest distance D3. Is designated as an area where the CG image 50 is synthesized (see FIG. 19B).

  Therefore, according to the imaging apparatus of the third modification, since the synthesis position of the CG image 50 is automatically set based on the size of the area around the main subject S in the background area A2 of the subject image I12, the user However, the CG image 50 can be synthesized at an appropriate position in the background area A2 without performing a predetermined operation on the operation input unit 4 or the like.

  In the above embodiment, the CG composition processing is exemplified as the predetermined processing for the subject image. However, the present invention is not limited to this, and any processing processing for the background region other than the main image region in the subject image may be used. It can be anything.

  In the above embodiment, as a process for specifying the position of the main image region A1, the contour line of the pixel set constituting the subject image is extracted, and is smaller than a predetermined value of the extracted contour lines. The outer contour of the main subject S is specified by removing the contour line of the pixel set. However, the processing content is an example, and the present invention is not limited to this.

  Furthermore, although the CG image 50 is illustrated as the material image, the present invention is not limited to this, and any material image can be used as long as it can be combined with the subject image. For example, although illustration is omitted, it may be an image of a person or an animal photographed in advance.

Further, the configurations of the imaging apparatuses 100 and 200 are merely examples as illustrated in the above-described embodiment, and are not limited thereto.
Furthermore, in the above embodiment, the CPU 12 executes a function as a position specifying unit, a processing unit, an extracting unit, an outline specifying unit, a dividing unit, an information acquiring unit, and a composite position determining unit by a predetermined program or the like. However, the present invention is not limited to this. For example, a configuration including a logic circuit for realizing various functions may be used.

  In addition, the image capturing apparatuses 100 and 200 are illustrated as the image synthesizing apparatus. However, the present invention is not limited to this, and an external device such as a computer connected with the plurality of image data generated by the image capturing unit 1 via the USB connection terminal 5 is not limited thereto. The information may be output to a device, and the external device may perform position specifying processing, CG combining processing, extraction processing, contour specifying processing, division processing, information acquisition processing, combining position determination processing, and the like.

1 is a diagram schematically illustrating an appearance of an imaging apparatus according to an embodiment to which the present invention is applied. It is a block diagram which shows schematic structure of the imaging device of FIG. 3 is a flowchart illustrating an example of an operation related to a display image CG composition process by the imaging apparatus of FIG. 1. 3 is a flowchart illustrating an example of an operation related to a captured image CG synthesis process performed by the imaging apparatus of FIG. 1. 3 is a flowchart illustrating an example of an operation related to a CG composite coordinate calculation process by the imaging apparatus of FIG. 1. It is a flowchart which shows an example of the operation | movement which concerns on the small area deletion process in the CG synthetic | combination coordinate calculation process of FIG. It is a flowchart which shows an example of the operation | movement which concerns on the background extraction process in the CG synthetic | combination coordinate calculation process of FIG. It is a figure which shows typically the image for demonstrating the CG synthetic | combination coordinate calculation process of FIG. It is a figure which shows typically the image for demonstrating the CG synthetic | combination coordinate calculation process of FIG. It is a figure which shows typically the image after the synthesis process of the CG image by the imaging device of FIG. It is a block diagram which shows schematic structure of the imaging device of the modification 1. 12 is a flowchart illustrating an example of an operation related to a CG composite coordinate calculation process by the imaging apparatus of FIG. 11. It is a figure which shows the sorting result of the amplitude value of the several image area which concerns on the CG synthetic | combination coordinate calculation process of FIG. It is a figure which shows typically the image for demonstrating the CG synthetic | combination coordinate calculation process of FIG. It is a figure which shows typically the image for demonstrating the CG synthetic | combination coordinate calculation process of FIG. It is a figure which shows typically the image after the synthesis process of the CG image by the imaging device of FIG. FIG. 10 is a diagram schematically illustrating an image related to CG synthesis processing by an imaging apparatus according to Modification 2. It is a figure which shows typically the image which concerns on CG synthetic | combination processing by the imaging device of FIG. FIG. 10 is a diagram schematically showing an image related to CG composition processing by an imaging apparatus according to Modification 3.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Imaging device 1 Imaging part 4 Operation input part 4b Cursor key 8 Image engine 12 CPU
13 Flash memory 50 CG image I6, I9, I11 CG composite image

Claims (5)

Material image storage means for storing material images;
Position specifying means for specifying the position of the first image area corresponding to the main subject included in one subject image;
A composite position input for inputting a composite position of the material image in a second image area other than the position of the first image area of the one subject image specified by the position specifying means by detecting an external operation Means,
A composite position that is closer to the position of the first image area specified by the position specifying means than the composite position input by the composite position input means is acquired, and the material image storage means is acquired at the acquired composite position. Combining processing means for performing processing for combining the material images stored in
An image processing apparatus comprising: Extracting means for extracting a contour line of a pixel group constituting the subject image;
An outline specifying means for specifying a main outline on which the main subject is drawn by excluding an outline of a pixel set smaller than a predetermined value from the outline extracted by the extracting means;
Further comprising
The position specifying means includes
The image processing apparatus according to claim 1 , wherein the position of the image area surrounded by the main outline specified by the outline specifying unit is specified as the position of the main image area . Dividing means for dividing the one subject image into a plurality of regions;
Information acquisition means for acquiring frequency component information for each region divided by the dividing means;
Further comprising
The position specifying means includes
Based on the frequency component information acquired by the information acquiring unit, according to claim 1 in which the high-frequency component is characterized in that it is determined that the image areas constituting the first image region more area than a predetermined value Image processing apparatus. The image processing apparatus according to claim 1 , further comprising an imaging unit that captures the subject image . Computer
Position specifying means for specifying the position of the first image area corresponding to the main subject included in one subject image;
  Composite position input means for inputting the composite position of the material image in the second image area other than the position of the first image area of the one subject image specified by the position specifying means by detecting an external operation. ,
  A composite position that is closer to the position of the first image area specified by the position specifying means than the composite position input by the composite position input means is acquired, and the material image is combined with the acquired composite position. Synthesis processing means for performing processing,
A program characterized by functioning as