JP2012060384A - Image processing apparatus, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a composite image in which subjects imaged at different times are present on a substantially common background.SOLUTION: A pickup image acquisition section 61 acquires pickup image data. A reference image acquisition section 62 acquires Kth reference image data. A display control section 51 controls a display section 19 to display the pickup image, which is a through image, with the reference image superimposed thereon. A differential image generation section 63 differentiates the pickup image data acquired for Kth recording pickup from the reference image data acquired at the beginning to generate differential image data. A composition section 64 composites the differential image data with the Kth reference image data to generate Kth composite image data. A setting section 52 sets the Kth composite image data as new (K+1)th reference image data.

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  As a conventional technique, after extracting the contour of the subject image at the time of the current imaging, the contour of the subject image captured at the previous time is extracted, so that it is superimposed on the contour at the time of the current imaging, and the angle of view at the time of the current imaging Has been disclosed (see Patent Document 1).

JP 2005-26874 A

  However, in the above-mentioned Patent Document 1, even if the current imaging is performed so as to be substantially the same as the composition of the previous captured image, for example, the background is substantially common, but there is a subject in which each subject is different at the time of imaging. It is not supported until an image is generated.

  The present invention has been made in view of such a situation, and an object of the present invention is to be able to generate a composite image in which subjects at different times of imaging exist in a substantially common background.

  In order to achieve the above object, the invention of claim 1 includes a first image acquisition unit that acquires image data, a display unit that displays an image of the image data acquired by the first image acquisition unit, and a composition reference. A second image acquisition unit that acquires reference image data, and an image of the reference image data acquired by the second image acquisition unit superimposed on an image of the image data acquired by the first image acquisition unit Display control means for controlling the display means to display, difference image data between the reference image data acquired by the second image acquisition means and the image data acquired by the first image acquisition means The difference image generation means to be generated, the reference image data acquired by the second image acquisition means, and the difference image data generated by the difference image generation means are combined. Synthesizing means for generating a composite image data, the composite image data generated by the synthesizing means, characterized in that it comprises a setting means for setting a new reference image data.

  The invention according to claim 2 is the invention according to claim 1, further comprising an imaging unit that captures an image and outputs image data of the image, wherein the first image acquisition unit includes the imaging unit. The image data output from the means is acquired.

  In order to achieve the above object, the invention of claim 3 is an image processing method of an image processing apparatus comprising display means for displaying an image, the first image acquisition step for acquiring image data, and a composition reference A second image acquisition step for acquiring the reference image data, and the image of the reference image data acquired in the second image acquisition step is superimposed on the image of the image data acquired in the first image acquisition step And a difference between the display control step for controlling the display means to display, the reference image data acquired in the second image acquisition step, and the image data acquired in the first image acquisition step. The difference image generation step for generating image data, the reference image data acquired in the second image acquisition step, and the difference image generation step A combination step of combining the difference image data and generating combined image data, and a setting step of setting the combined image data generated in the combining step as new reference image data. Features.

  In order to achieve the above object, a fourth aspect of the present invention provides a computer of an image processing apparatus including display means for displaying an image, first image acquisition means for acquiring image data, reference image data serving as a reference for synthesis. A second image acquisition unit that acquires the image of the reference image data acquired by the second image acquisition unit and superimposing the image of the image data acquired by the first image acquisition unit on the display unit The display control means for controlling the difference between the reference image data first acquired by the second image acquisition means and the image data acquired by the first image acquisition means, thereby generating difference image data. The reference image data acquired by the difference image generation unit and the second image acquisition unit and the difference image data generated by the difference image generation unit are synthesized. Synthesizing means for generating a composite image data, the composite image data generated by the synthesizing means, characterized in that it is a program to function setting means as, for setting a new reference image data.

  According to the present invention, it is possible to generate a composite image in which subjects that are different at the time of imaging exist in a substantially common background.

1 is a block diagram illustrating a hardware configuration of an image processing apparatus according to an embodiment of the present invention. It is a functional block diagram which shows the functional structure for performing an image synthesis process. 3 is a flowchart for explaining a flow of image composition processing executed by the image processing apparatus of FIG. 2. It is a figure which shows an example of the specific process result of an image synthesis process. It is a figure which shows an example of the specific process result of an image synthesis process.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a hardware configuration of an imaging apparatus 1 as an embodiment of an image processing apparatus according to the present invention. The imaging device 1 can be configured by a digital camera, for example.

  The imaging apparatus 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an image processing unit 14, a bus 15, an input / output interface 16, and an input / output. An interface 16, an imaging unit 17, an operation unit 18, a display unit 19, a storage unit 20, a communication unit 21, and a drive 22 are provided.

The CPU 11 executes various processes according to a program recorded in the ROM 12 or a program loaded from the storage unit 20 to the RAM 13. The display control unit 51 and the setting unit 52 provided in the CPU 11 will be described later with reference to FIG.
The RAM 13 also stores data necessary for the CPU 11 to execute various processes as appropriate.

The image processing unit 14 is configured by a DSP (Digital Signal Processor), a VRAM (Video Random Access Memory), and the like, and performs various image processing on image data in cooperation with the CPU 11.
Here, in this embodiment, the processing unit of the image by the imaging apparatus 1 is one still image, and the still image as such a processing unit is referred to as a “frame” in this specification. Hereinafter, unless otherwise specified, an image means a frame.
For example, the image processing unit 14 performs image processing such as noise reduction, white balance, camera shake correction, and the like on captured image data output from the imaging unit 17 described later.
In the present embodiment, the image processing unit 14 further includes a difference image generation unit 63 and a synthesis unit 64. However, details of the captured image acquisition unit 61 to the synthesis unit 64 will be described later with reference to FIG.

  The CPU 11, ROM 12, RAM 13, and image processing unit 14 are connected to each other via a bus 15. An input / output interface 16 is also connected to the bus 15. An imaging unit 17, an operation unit 18, a display unit 19, a storage unit 20, a communication unit 21, and a drive 22 are connected to the input / output interface 16.

  Although not shown, the imaging unit 17 includes an optical lens unit and an image sensor.

The optical lens unit is configured by a lens that collects light, for example, a focus lens or a zoom lens, in order to photograph a subject.
The focus lens is a lens that forms a subject image on the light receiving surface of the image sensor. The zoom lens is a lens that freely changes the focal length within a certain range.
The optical lens unit is also provided with a peripheral circuit for adjusting setting parameters such as focus, exposure, and white balance as necessary.

The image sensor includes a photoelectric conversion element, AFE (Analog Front End), and the like.
The photoelectric conversion element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element or the like. A subject image is incident on the photoelectric conversion element from the optical lens unit. Therefore, the photoelectric conversion element photoelectrically converts (captures) the subject image, accumulates the image signal for a predetermined time, and sequentially supplies the accumulated image signal as an analog signal to the AFE.
The AFE performs various signal processing such as A / D (Analog / Digital) conversion processing on the analog image signal. A digital signal is generated by various signal processing and output as an output signal of the imaging unit 17.
Hereinafter, it is assumed that the output signal of the imaging unit 17 is “data of a captured image”. Accordingly, captured image data is output from the imaging unit 17 and is appropriately supplied to the CPU 11, the image processing unit 14, and the like.

The operation unit 18 includes various buttons and the like, and accepts user instruction operations.
The display unit 19 is composed of a liquid crystal display or the like and displays various images.
The storage unit 20 is configured by a DRAM (Dynamic Random Access Memory) or the like, and temporarily stores image data output from the image processing unit 14 or the like. The storage unit 20 also stores various data necessary for processing by the image processing unit 14 and the like.
The communication unit 21 controls communication with other devices (not shown) via a network including the Internet.

  A removable medium 31 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached to the drive 22. The program read from the removable medium 31 by the drive 22 is installed in the storage unit 20 as necessary. The removable medium 31 can also store various data such as image data stored in the storage unit 20 in the same manner as the storage unit 20.

The imaging apparatus 1 having such a configuration starts through imaging processing and through display processing before and after imaging.
That is, the imaging device 1 continues the imaging operation by the imaging unit 17, and during that time, captured image (frame) data sequentially output from the image processing unit 14 via the imaging unit 17 is stored in a memory (in this embodiment). The data is temporarily stored in the storage unit 20). Such a series of processes is the “through imaging process” referred to herein.
The imaging apparatus 1 sequentially reads data of each captured image (frame) temporarily recorded in the memory (the storage unit 20 in the present embodiment) during through imaging, and displays the corresponding captured image on the display unit 19. Are displayed sequentially. Such a series of control processes is the “through display process” here. The captured image displayed on the display unit 19 by the through display process is hereinafter referred to as a “through image”.

After determining the composition while viewing the through image, the user can instruct the recording of the captured image data by pressing a shutter switch (not shown) of the operation unit 18 to the lower limit. The operation of pressing the shutter switch to the lower limit in this way is hereinafter referred to as “full press operation” or simply “full press”.
Hereinafter, in order to distinguish from imaging for through imaging processing and through display processing, imaging when a full-press operation is performed and a recording instruction is issued is referred to as “recording imaging”.

Furthermore, in the present embodiment, the imaging apparatus 1 can execute the following series of processes when generating composite image data in which a plurality of subjects captured at different times are combined.
The imaging device 1 performs the 0th recording imaging as preparation in advance for generation of data of the composite image. Here, since the first recording imaging for generating the composite image data is the first time, the preliminary preparation is the 0th time.
The imaging apparatus 1 holds the captured image data obtained by the 0-th recording imaging as the first reference image data, and causes the display unit 19 to display the reference image semi-transparently.
That is, the imaging device 1 causes the display unit 19 to display the first reference image in a semi-transmissive state on the through image before the first recording imaging.
In this case, the photographer can visually recognize a state in which the background is translucently displayed on the through image. Therefore, the imager moves the imaging device 1 before the first recording imaging, so that the position of the first subject becomes appropriate based on the semi-transparent background. You can compose.
Then, the user can instruct the first recording imaging by performing a full-press operation when an appropriate composition is achieved.
In response to such an instruction, the imaging apparatus 1 performs the first recording imaging, and calculates the difference between the captured image data obtained as a result and the first reference image (background image described later) data as a pixel. By taking the unit, data of an image in which each difference value becomes each pixel value (hereinafter referred to as “difference image”) is generated.
And the imaging device 1 produces | generates the data of the 1st synthesized image by synthesize | combining each data of a difference image and a reference | standard image.
Furthermore, the imaging apparatus 1 sets and holds the data of the first composite image as the data of the next (second) reference image.
The imaging device 1 causes the display unit 19 to display the second reference image in a semi-transmissive state superimposed on the through image before the second recording imaging.
In this case, since the data of the second reference image is data of the composite image generated at the time of the previous (first) recording for recording, the second reference image includes the background (1) The subject at the time of recording for the second time) is also included. Therefore, the photographer can visually recognize not only the background but also the state in which the subject at the time of recording imaging for the first time is displayed semi-transparently on the through image. Therefore, the imager moves the image capturing apparatus 1 to change the semi-transparent background and the subject at the time of the previous recording (first time) for recording (second time) of the subject (second time). Composition can be made so that the position is appropriate. That is, the photographer can instruct the current (second) recording for recording with a composition in which the subject is arranged at an appropriate position in the composite image.
That is, the photographer can instruct the current (second) recording for recording by performing a full-press operation when an appropriate composition is achieved.
In response to such an instruction, the imaging apparatus 1 performs the current (second) recording for imaging, and obtains the data of the captured image obtained as a result and the data of the first reference image (background image described later). By using this, the data of the difference image is generated.
Then, the imaging apparatus 1 generates data of the second combined image by combining the data of the difference image and the second reference image.
Furthermore, the imaging apparatus 1 sets and holds the data of the second combined image as the data of the next (third) reference image.
Thereafter, the above-described series of processing is repeatedly executed every time recording for recording is performed three times or more.
Note that each recording imaging may be performed immediately after the previous recording imaging is performed, or may be performed after a long period of time (for example, in units of years).
Such a repetition of a series of processing is hereinafter referred to as “image composition processing”.

FIG. 2 is a functional block diagram showing a functional configuration for the image capturing apparatus 1 of FIG. 1 to execute image composition processing.
In FIG. 2, only the CPU 11, the image processing unit 14, the imaging unit 17, the display unit 19, and the storage unit 20 are illustrated in the configuration of the imaging device 1 in FIG. 1.
In the CPU 11, a display control unit 51 and a setting unit 52 are provided.
In the present embodiment, the display control unit 51 and the setting unit 52 are configured by a combination of the CPU 11 and a program executed by the CPU 11, that is, a combination of hardware and software. However, this is merely an example, and it is naturally possible to transfer at least some of the functions of the display control unit 51 and the setting unit 52 to other components (such as the image processing unit 14) other than the CPU 11.
The image processing unit 14 is provided with a captured image acquisition unit 61 to a synthesis unit 64. Note that the captured image acquisition unit 61 to the combining unit 64 are provided in the image processing unit 14, and at least a part of the captured image acquisition unit 61 to the combining unit 64 is other than the image processing unit 14. Of course, it is also possible to transfer it to the constituent elements (CPU 11 or the like).
As one area of the storage unit 20, a background image storage unit 41 and a reference image storage unit 42 are provided. Note that the background image storage unit 41 and the reference image storage unit 42 are provided as one area of the storage unit 20, and may be provided as another area of the removable media 31, for example.

  The background image storage unit 41 acquires captured image data obtained by the 0th recording imaging (recording imaging as a preliminary preparation for generating a composite image) as background image data.

The reference image storage unit 42 stores data of one or more reference images.
Specifically, in the 0th recording imaging (recording imaging as a preliminary preparation), captured image data is stored in the reference image storage unit 42 as reference image data. That is, in the 0-th recording imaging, the background image data stored in the background image storage unit 41 is stored in the reference image storage unit 42 as reference image data.
Thereafter, for each recording imaging, the composite image data sequentially output from the composition unit 64 described later is stored in the reference image storage unit 42 as the next reference image data.
In this manner, the reference image data stored in the reference image storage unit 42 is updated each time the number of times of recording imaging advances.

  The display control unit 51 performs control for causing the display unit 19 to display captured images represented by data sequentially supplied from the captured image acquisition unit 61 as through images. Further, the display control unit 51 executes control for making the reference image represented by the data supplied from the reference image acquisition unit 62 a semi-transparent image and superimposing it on the through image.

The setting unit 52 sets, as new (next) reference image data, combined image data sequentially generated by a combining unit 64 described later for each recording imaging.
The new reference image data set by the setting unit 52 in this way is output from the combining unit 64 and stored in the reference image storage unit 42.

The captured image acquisition unit 61 acquires captured image data sequentially output from the imaging unit 17.
The captured image acquisition unit 61 supplies sequentially acquired captured image data to the display control unit 51 as through image data. The captured image acquisition unit 61 supplies the captured image data to the difference image generation unit 63 when the acquired captured image data is output during recording imaging. Furthermore, when the acquired captured image data is output at the time of the 0th recording imaging (at the time of recording imaging as a preliminary preparation), the captured image acquisition unit 61 outputs the captured image data. The data is supplied to the background image storage unit 41 as background image data.

  The reference image acquisition unit 62 acquires reference image data from the reference image storage unit 42 and supplies the data to the display control unit 51 and the synthesis unit 64.

The difference image generation unit 63 captures the captured image data obtained during the first and subsequent recording imaging and the background image data stored in the background image storage unit 41 (obtained during the 0th recording imaging as a preliminary preparation). The difference image data is generated using the captured image data, that is, the first reference image data).
A specific example of a method for generating difference image data will be described later with reference to FIGS. 4 and 5.
The difference image data generated by the difference image generation unit 63 is supplied to the synthesis unit 64.

The combining unit 64 generates combined image data by combining the reference image data supplied from the reference image acquiring unit 62 and the difference image data supplied from the difference image generating unit 63.
The synthesizing unit 64 stores the generated synthesized image data in the reference image storage unit 42 as new (next) reference image data set by the setting unit 52.

Next, with reference to FIG. 3, image compositing processing realized by the functional configuration of FIG. 2 will be described in the processing of the imaging apparatus 1.
FIG. 3 is a flowchart for explaining the flow of the image composition process.

The image composition process in FIG. 3 is a process that is executed when the K-th image capturing for recording (K is an integer value of 1 or more) is performed.
That is, at least the 0th recording imaging as the preliminary preparation is completed, and the captured image data (first reference image data) obtained by the 0th recording imaging is used as the background image data. It is assumed that it is stored in the image storage unit 41.
Further, it is assumed that the through imaging process and the through display process are continuously executed before and after the execution of the image processing of FIG. That is, it is assumed that a through image is always displayed on the display unit 19.
Under such a premise, the first image composition processing for K = 1 is started when the recording for the 0th recording is completed, and then the second and subsequent image composition processing is performed for the previous K−. The first image compositing process is started when a week is considered. That is, as long as the recording imaging is repeatedly performed, the image composition processing in FIG. 3 is repeatedly performed.

  In step S <b> 11, the reference image acquisition unit 62 acquires data of the Kth reference image from the reference image storage unit 42.

  In step S <b> 12, the captured image acquisition unit 61 acquires captured image data output from the imaging unit 17.

  In step S <b> 13, the display control unit 53 causes the display unit 19 to superimpose a semi-transmissive reference image with the captured image as a through image.

  In step S14, the captured image acquisition unit 61 determines whether or not a full-press operation has been performed.

  If the full pressing operation has not been performed, it is determined as NO in Step S14, and the process returns to Step S12. That is, the loop processing of steps S12 to S14 is repeatedly executed until the full pressing operation is performed. During this time, the captured image represented by the data sequentially output from the imaging unit 17 becomes a through image, that is, the through image is sequentially updated, and the semi-transparent reference image is superimposed on the sequentially updated through image. Is displayed.

Here, in the K (> 1) -th image compositing process, the current (K-th) reference image data acquired in the process of step S11 is generated during the previous (K-1) recording imaging. This is the synthesized image data. Therefore, the K-th reference image includes not only the background but also the respective subjects at the time of recording imaging for the first time (1st to (K-1) th). Therefore, the photographer can visually recognize not only the background but also each subject at the time of the first (Kth to (K-1) th) recording images being displayed semi-transparently on the through image. .
Therefore, the imager moves the image capturing apparatus 1 to change the semi-transparent background and the subject at the time of recording for the previous (1st to (K-1)) recordings (this time ( The composition can be made so that the position of the subject in the (Kth) time becomes appropriate. That is, the imager can instruct the current (Kth) recording for recording with a composition in which the subject is arranged at an appropriate position in the composite image.
That is, the photographer can instruct the current (Kth) recording for recording by performing a full-press operation when an appropriate composition is achieved.

  In this case, it is determined as YES in Step S14, and the process proceeds to Step S15.

  In step S15, the captured image acquisition unit 61 temporarily stores captured image data in the storage unit 20 or the like.

  In step S <b> 16, the difference image generation unit 63 uses the background image data stored in the background image storage unit 41 and the captured image data temporarily stored in the process of step S <b> 15 to generate the difference image. Generate data.

  In step S <b> 17, the combining unit 64 generates combined image data by combining the reference image data acquired in step S <b> 11 and the difference image data generated in step S <b> 16.

  In step S18, the setting unit 52 sets the data of the Kth composite image generated in the process of step S17 as new (next K + 1) reference image data.

In step S19, the composition unit 64 overwrites and saves the new reference image data (Kth composite image data) set in step S18 in the reference image storage unit.
As a result, the image composition process ends.

Furthermore, with reference to FIG. 4 and FIG. 5, the image composition processing will be specifically described.
FIG. 4 is a diagram for explaining a specific processing result of the image composition processing.

FIG. 4A illustrates an example of captured image data obtained by the 0th recording imaging, that is, background image data, as advance preparation for the image composition process.
That is, in the example of FIG. 4, the data of the background image 101 shown in FIG. 4A is stored in the background image storage unit 41 before the K = 1st image composition process is executed. .
The background image 101 includes a house 91 as a part of the background.

  FIG. 4B shows a specific example of the result of K = 1 first image composition processing (including the intermediate result).

  3, the data of the background image 101 shown in FIG. 4A is acquired as the data of the reference image 111 for the first time K = 1, as shown in B-1 of FIG. 4B. The

Thereafter, the loop process of steps S12 to S14 is repeatedly executed until the full-press operation is performed. During this time, as shown in B-2 of FIG. 4B, an image 112 in which the reference image 111 in a semi-transparent state is superimposed on the through image is displayed on the display unit 19.
In the image 112, a house 91 and a subject 92 indicated by a solid line are included in the through image, and a house 91 indicated by a dotted line is included in the semi-transparent reference image 111.
In this case, the imager moves the imaging device 1 before the first recording imaging, so that the first subject 92 is captured with reference to the house 91 of the semi-transmissive display (dotted line in the figure). Composition can be made so that the position is appropriate. Specifically, the photographer may compose the semi-transparent display house 91 (dotted line in the figure) and the through-image house 91 (solid line in the figure) so as to substantially coincide with each other.

  Then, the user can instruct the first recording imaging by performing a full-press operation when an appropriate composition is achieved.

  In this case, it is determined as YES in step S14 of FIG. 3, and the data of the captured image 113 as shown in B-3 of FIG. 4B is temporarily stored in the storage unit 20 or the like in the process of step S15. Remembered.

In the process of step S16, the difference image generation unit 63 uses the data of the background image 101 (K = first reference image 111) of FIG. ) Of the difference image 114 as shown in B-4 of FIG.
In this case, the difference image 114 includes only the subject 92 that did not exist in the background image 101 of FIG. In other words, it can be understood that the subject 92 has been cut out.

In the process of step S17, the data of the reference image 111 (background image 101) for the first time K = 1 and the data of the difference image 114 are combined, as shown in B-5 of FIG. Data of K = first composite image 115 is generated.
That is, at the first time K = 1, data of the composite image 115 is generated in which the clipped subject 92 is pasted on the background (house 91 or the like) included in the reference image 111.

  In the process of step S18, the data of the composite image 115 for the first time K = 1 is set as the data for the next (= K + 1) th reference image, and is overwritten and saved in the reference image storage unit 42 in the process of step S19. .

  As a result, the first image composition processing of K = 1 is completed. Thereafter, when the recording for the second recording is performed, the second image processing starts. Note that the start timing of the second image processing for K = 2 is sufficient if it is after the end of the first image compositing process, and may be immediately after the end, or for a long period of time (even yearly). Yes) It may be after it has passed.

  FIG. 4C shows a specific example of the result (including the intermediate result) of the image synthesis process for the second time K = 2.

  As a result of the process of step S11 in FIG. 3, as shown in C-1 of FIG. 4C, as the data of the reference image 121 for the K = 2 time, the composite image 115 for the K = 1 time shown in FIG. What was the data of is acquired.

Thereafter, the loop process of steps S12 to S14 is repeatedly executed until the full-press operation is performed. During this time, as shown by C-2 in FIG. 4C, an image 122 in which the semi-transmissive reference image 121 is superimposed on the through image is displayed on the display unit 19.
In the image 122, a house 91 and a subject 93 indicated by a solid line are included in the through image, and a house 91 and a subject 92 indicated by a dotted line are included in the semi-transparent reference image 121.
In this case, the imager moves the imaging device 1 before the second recording imaging, so that the second time with reference to the house 91 and the subject 92 in the semi-transparent display (dotted line in the figure). It is possible to compose the subject 93 so that the position of the subject 93 is appropriate. Specifically, the photographer substantially matches the semi-transparent display house 91 (dotted line in the figure) with the through-image house 91 (solid line in the figure) so that the subject 92 and the subject 93 do not overlap. You just have to compose it.

  Then, the user can instruct the second recording imaging by performing a full-press operation when an appropriate composition is achieved.

  In this case, it is determined as YES in step S14 of FIG. 3, and the data of the captured image 123 as shown in C-3 of FIG. 4C is temporarily stored in the storage unit 20 or the like in the process of step S15. Remembered.

In the process of step S <b> 16, the difference image generation unit 63 uses the data of the background image 101 (K = first reference image 111) and the captured image 123 of FIG. ) Of the difference image 124 as shown in C-4 of FIG.
In this case, the difference image 124 includes only the subject 93 that did not exist in the background image 101 of FIG. In other words, it can be understood that the subject 93 has been cut out.

In the process of step S17, the data of the K = 2th reference image 121 (first synthesized image 115) and the data of the difference image 124 are synthesized, which is shown in B-5 of FIG. 4B. In this way, data of the composite image 125 for the second time K = 2 is generated.
That is, at the second time of K = 2, data of the composite image 125 in which the clipped subject 93 is pasted on the background (house 91 and the like) and the subject 92 included in the reference image 121 is generated. .

  In the process of step S18, the data of the composite image 125 for the K = 2th time is set as the data for the next 3 (= K + 2) reference image, and is overwritten and saved in the reference image storage unit 42 in the process of step S19. .

FIG. 5 is a diagram for explaining more specific processing results (including intermediate progress) of the second image composition processing of K = 2 shown in FIG.
FIG. 5A shows the specific processing result (including the progress) of step S16 in FIG.

As described above, when the full-press operation is performed, the data of the background image 101 (K = first reference image 111 shown in FIG. 4B) and the data of the captured image 123 are processed in step S16. By using this, data of the difference image 124 is obtained.
As described above, the difference image 124 can be said to be an image in which the subject 93 is cut out.

Therefore, in the present embodiment, the difference image generation unit 63 further generates the data of the mask image 131 by performing a binarization process of transmission or non-transmission on the difference image 124 that is a clipped image. In the example of FIG. 5, a mask image in which the subject 93 that is a cut-out portion (portion other than the background) becomes a non-transparent pixel (hereinafter, referred to as white) and the background becomes a transparent pixel (hereinafter, referred to as black). 131 data are obtained. If each pixel constituting the image is represented by RGB values, all values of RGB are 0 for black, and all values of RGB values are 1 for white.
Here, two processes related to the present invention, ie, an AND process and an OR process when one of the two pixels is black (the other is a pixel other than black) will be described.
Now, since the RGB values of black pixels are all 0, when AND processing is performed on two pixels, the pixel after AND processing becomes black. On the other hand, when the OR process is performed on the two pixels, the RGB values of the black pixels are all 0, so that the pixel after the OR process has a color other than black.
In the present embodiment, these AND processing and OR processing are performed on all the pixels of two images to be processed.

The difference image generation unit 63 converts the data of the mask image 131 into the data of the inverted image 132 by performing the inversion process. In the example of FIG. 5, data of a reversed image 132 is obtained in which the subject 93 that is a cut-out portion (portion other than the background) is reversed to black and the background is reversed to white.
That is, the data of the reverse image 132 is the final processing result of step S16. Accordingly, when such inverted image 132 data is obtained, the process proceeds to step S17.

  FIG. 5B shows the specific processing result (including the intermediate progress) of step S17 in FIG.

As the first process of the process of step S <b> 17, the synthesizer 64 performs the K = 2 second reference image 121 (first synthesized image 115 shown in FIG. 4B) data and the inverted image 132 based on the difference image 124. AND processing is performed on the data.
Thereby, data of the AND image 141 as shown in the center of FIG. 5B is generated. That is, the image after the AND process is an image in which the reverse image 132 is superimposed on the reference image 121. In the AND image 141, the portion of the subject 93 remains the black portion in the inverted image 132.
Therefore, the combining unit 64 further performs an OR process on the data of the AND image 141 and the data of the difference image 124.
As a result, data of the composite image 125 as shown in B-8 of FIG. 5B is generated. That is, data of the composite image 125 in which the clipped subject 93 is pasted on the background (the house 91 and the like) and the subject 92 included in the reference image 121 is generated. That is, the image after OR processing is an image in which the difference image 124 is superimposed on the AND image 141.

  Thereafter, in the process of step S18, the data of the composite image 125 of the K = 2th time is set as the data of the next 3 (= K + 2) reference image, and is overwritten and saved in the reference image storage unit 42 in the process of step S19. Then, the K = 2 second image composition processing is completed.

As described above, the imaging apparatus 1 according to the present embodiment includes the captured image acquisition unit 61, the reference image acquisition unit 62, the display control unit 51, the setting unit 52, the difference image generation unit 63, and the synthesis unit 64. And.
The captured image acquisition unit 61 acquires captured image data. The reference image acquisition unit 62 acquires Kth reference image data.
The display control unit 51 controls the display unit 19 to display the reference image superimposed on the captured image that is a through image.
The difference image generation unit 63 obtains the difference by taking the difference between the reference image (that is, the background image referred to in the present embodiment) data acquired for the first time and the captured image data obtained by the K-th recording imaging. Generate image data.
The synthesizing unit 64 generates K-th synthesized image data by synthesizing the K-th reference image data and the difference image data.
The setting unit 52 sets the Kth composite image data as new (K + 1) th reference image data.
Thus, in generating composite image data in which a plurality of subjects captured by each of the 1 to K recording images is combined, the subject is arranged at an appropriate position in the composite image, and each time It is possible to easily take an image for recording.

  In addition, this invention is not limited to the above-mentioned embodiment, The deformation | transformation in the range which can achieve the objective of this invention, improvement, etc. are included in this invention.

  For example, in the above-described embodiment, the display control unit 53 semi-transparently displays the reference image on the display unit 19. However, the present invention is not limited to this, and it is sufficient that the reference image and the captured image can be displayed differentially. Specifically, it is possible to adopt a display form in which only the outline of the reference image is displayed, only the outline is a dotted line, or the subject of the reference image is highlighted in red.

  For example, in the above-described embodiment, the image processing apparatus to which the present invention is applied has been described as an example configured as an image processing apparatus such as a digital camera. However, the present invention is not particularly limited to an image processing apparatus, and in general, an electronic apparatus that can execute the above-described image processing regardless of the presence or absence of an imaging function (data of captured images may be acquired from another apparatus) Can be applied. Specifically, for example, the present invention can be applied to a personal computer, a video camera, a portable navigation device, a portable game machine, and the like.

  The series of processes described above can be executed by hardware or can be executed by software.

  When a series of processing is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium. The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

  The recording medium including such a program is not only constituted by the removable medium 31 of FIG. 1 distributed separately from the apparatus main body in order to provide the program to the user, but also in a state of being incorporated in the apparatus main body in advance. It is comprised with the recording medium etc. which are provided in this. The removable medium is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), or the like. The magneto-optical disk is configured by an MD (Mini-Disk) or the like. In addition, the recording medium provided to the user in a state of being preliminarily incorporated in the apparatus main body includes, for example, the ROM 12 in FIG. 1 in which a program is recorded, the hard disk included in the storage unit 20 in FIG.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series along the order, but is not necessarily performed in time series, either in parallel or individually. The process to be executed is also included.

  DESCRIPTION OF SYMBOLS 1 ... Imaging device, 11 ... CPU, 12 ... ROM, 13 ... RAM, 14 ... Image processing part, 15 ... Bus, 16 ... Input / output interface, 17 ... Image pickup unit, 18 ... operation unit, 19 ... display unit, 20 ... storage unit, 21 ... communication unit, 22 ... drive, 31 ... removable media, 41 ... background Image storage unit 42 ... Reference image storage unit 51 ... Display control unit 52 ... Setting unit 61 ... Captured image acquisition unit 62 ... Reference image acquisition unit 63 ... Difference image generation unit, 64... Synthesis unit

Claims (4)

First image acquisition means for acquiring image data;
Display means for displaying an image of the image data acquired by the first image acquisition means;
Second image acquisition means for acquiring reference image data as a reference for synthesis;
Display control means for controlling the image of the reference image data acquired by the second image acquisition means to be superimposed on the image of the image data acquired by the first image acquisition means and displayed on the display means;
Difference image generation means for generating difference image data between the reference image data acquired by the second image acquisition means and the image data acquired by the first image acquisition means;
A combining unit that generates combined image data by combining the reference image data acquired by the second image acquiring unit and the difference image data generated by the difference image generating unit;
Setting means for setting the combined image data generated by the combining means as new reference image data;
An image processing apparatus comprising: The image processing apparatus further includes an imaging unit that captures an image and outputs image data of the image,
The first image acquisition means acquires the image data output from the imaging means;
The image processing apparatus according to claim 1. An image processing method of an image processing apparatus including display means for displaying an image,
A first image acquisition step of acquiring image data;
A second image acquisition step of acquiring reference image data as a reference for synthesis;
A display control step for controlling the image of the reference image data acquired in the second image acquisition step to be displayed on the display unit in a manner superimposed on the image of the image data acquired in the first image acquisition step; ,
A difference image generation step for generating difference image data between the reference image data acquired in the second image acquisition step and the image data acquired in the first image acquisition step;
Combining the reference image data acquired in the second image acquisition step and the difference image data generated in the difference image generation step to generate combined image data;
A setting step for setting the combined image data generated in the combining step as new reference image data;
An image processing method comprising: A computer of an image processing apparatus comprising display means for displaying an image,
First image acquisition means for acquiring image data;
Second image acquisition means for acquiring reference image data that is a reference for synthesis;
Display control means for controlling the image of the reference image data acquired by the second image acquisition means to be superimposed on the image of the image data acquired by the first image acquisition means and displayed on the display means;
Difference image generation means for generating difference image data by taking a difference between the reference image data first acquired by the second image acquisition means and the image data acquired by the first image acquisition means;
A combining unit configured to combine the reference image data acquired by the second image acquiring unit and the difference image data generated by the difference image generating unit to generate combined image data;
Setting means for setting the combined image data generated by the combining means as new reference image data;
A program characterized by functioning as

Images