JP6718762B2 - Image processing device, imaging device, image processing method, and program - Google Patents

Description

The present invention relates to an image processing device, an imaging device, an image processing method, and a program that combine a plurality of image data arranged in time series.

2. Description of the Related Art Conventionally, a technique of synthesizing a plurality of image data to generate one image data in an image pickup apparatus such as a digital camera is known. For example, in Japanese Patent Application Laid-Open No. 2004-242242, image data generated sequentially while shooting is combined to generate combined image data, and the output of a predetermined area in the combined image corresponding to this combined image data reaches the target output. A technique for sometimes ending shooting is disclosed.

In addition, Patent Document 2 discloses a technique of combining images by a different combining method for each image region. In this technique, a composite image subjected to a different composition method for each area is generated by selecting and specifying a composition method such as comparative bright composition or comparative dark composition for each area of the specified image.

JP, 2014-135609, A Japanese Unexamined Patent Publication No. 2015-73199

By the way, when performing image synthesis, it is desirable to be able to freely set various settings for image synthesis in order to reflect the user's creative intention. However, in the above-mentioned Patent Documents 1 and 2, it is not possible to freely set various settings for image composition, and it is not possible to reflect the user's original intention in the composite image.

The present invention has been made in view of the above, and an object thereof is to provide an image processing device, an imaging device, an image processing method, and a program that can generate a composite image that reflects a user's creative intention. ..

In order to solve the above problems and achieve the object, an image processing apparatus according to the present invention synthesizes each of a plurality of images in a predetermined area of each of a plurality of images corresponding to a plurality of image data. For synthesizing at least two synthesizing methods, an image information extracting unit for sequentially extracting image information included in the predetermined area of each of the plurality of images, and an image information extracting unit for sequentially extracting the image information. A combination processing unit configured to sequentially combine the plurality of images and generate one combined data while switching the at least two combining methods specified in the predetermined region based on the image information. Characterize.

Further, the image information apparatus according to the present invention is characterized in that, in the above-mentioned invention, the image information apparatus further includes a composite area designating unit for designating one or more of the predetermined areas for each of the plurality of images.

Further, in the image information device according to the present invention, in the above invention, a threshold value designating unit that designates a threshold value for switching the at least two synthesis methods designated by the synthesis method designating unit in the predetermined area, and the image information. A threshold determination unit that determines whether or not the value determined according to the image information extracted by the extraction unit is equal to or greater than the threshold, and the combining processing unit is based on the determination result of the threshold determination unit. Then, the synthetic data is generated.

Further, in the image information device according to the present invention, in the above invention, the threshold value designation unit sets a value determined according to the image information extracted by the image information extraction unit for the oldest image in the plurality of images. It is characterized in that the threshold value is designated.

Further, in the image information device according to the present invention, in the above invention, when two or more of the predetermined areas are designated by the synthesis area designating unit, an overlapping area in which at least some of the areas overlap each other occurs. Further comprising a priority designating unit for designating the priority of the synthesizing method for the overlapping area, and the synthesizing processing unit is configured to convert the synthetic data by the synthesizing method having the high priority designated by the priority designating unit. It is characterized by generating.

Further, in the image information device according to the present invention, in the above invention, a subject designation unit that designates a specific subject captured in each of the plurality of images, and the subject designated by the subject designation unit is in the predetermined region. And a subject tracking unit for sequentially tracking a subject region including a position where the subject is captured in each of the plurality of images, wherein the combined region designating unit sequentially tracks the subject tracking unit. The specified subject area is designated as the predetermined area.

Further, in the image information device according to the present invention, in the above invention, the synthesizing method is any one of a comparative bright synthesizing process, a comparative dark synthesizing process, an additive synthesizing process, an average synthesizing process, a replacing synthesizing process, and a non-synthesizing process. It is characterized by

Further, in the image information device according to the present invention, in the above invention, the image information is any one of a luminance, a color component, a change amount of a pixel value between image data, and a movement amount of a subject between image data. Characterize.

In the image information device according to the present invention, in the above invention, a display unit that can display each of the plurality of images, and a display control unit that displays the image information extracted by the image information extraction unit on the display unit. And are further provided.

Further, the image information device according to the present invention further comprises a display unit capable of displaying each of the plurality of images, and a display control unit for displaying the combining method and the threshold value on the display unit. It is characterized by

Further, an image pickup apparatus according to the present invention sequentially records the above image processing apparatus, an image pickup section capable of continuously picking up an image of a subject and sequentially generating image data, and the combined data generated by the combining processing section. A recording unit, and the synthesizing processing unit generates new synthetic data by synthesizing with the synthetic data immediately before being recorded by the recording unit each time the imaging unit generates the image data. And updating the combined data recorded by the recording unit.

An image processing method according to the present invention is an image processing method executed by an image processing apparatus, wherein each of the plurality of images is combined with a predetermined area of each of the plurality of images corresponding to the plurality of image data. In order to specify at least two synthesizing methods, an image information extracting step for sequentially extracting image information included in the predetermined area of each of the plurality of images, and an image information extracting step A composition processing step of composing the plurality of images in sequence while switching the at least two composition methods specified in the predetermined area based on the extracted image information to generate one composite data. Is characterized by.

Further, the program according to the present invention specifies, in the image processing device, at least two combining methods for combining each of the plurality of images in a predetermined area of each of the plurality of images corresponding to the plurality of image data. A synthesizing method specifying step, an image information extracting step of sequentially extracting image information included in the predetermined area of each of the plurality of images, and based on the image information sequentially extracted in the image information extracting step, And a synthesizing step of sequentially synthesizing the plurality of images to generate one synthesized data while switching the at least two synthesizing methods designated in a predetermined area.

According to the present invention, it is possible to generate a composite image that reflects the user's original intention.

FIG. 1 is a block diagram showing a functional configuration of the image pickup apparatus according to Embodiment 1 of the present invention. FIG. 2 is a flowchart showing an outline of processing executed by the image pickup apparatus according to Embodiment 1 of the present invention. FIG. 3 is a diagram showing an example of a live view image displayed by the display unit of the image pickup apparatus according to Embodiment 1 of the present invention. FIG. 4 is a flowchart showing an outline of the image information extraction processing of FIG. FIG. 5 is a diagram showing an example of an extraction area displayed by the display unit of the image pickup apparatus according to Embodiment 1 of the present invention. FIG. 6 is a diagram showing an example of image information displayed by the display unit of the image pickup apparatus according to Embodiment 1 of the present invention. FIG. 7 is a flowchart showing an outline of the synthesis setting process of FIG. FIG. 8 is a diagram showing an example of an image displayed by the display unit of the image pickup apparatus according to Embodiment 1 of the present invention during the synthesis setting process. FIG. 9 is a diagram showing an example of an image displayed by the display unit of the image pickup apparatus according to Embodiment 1 of the present invention during the synthesis setting process. FIG. 10 is a flowchart showing an outline of the combining process of FIG. FIG. 11 is a diagram schematically illustrating an example of a time-series image group that is combined by the combining processing unit of the image pickup apparatus according to Embodiment 1 of the present invention. FIG. 12 is a diagram schematically showing transition of a combined image when the images in FIG. 11 are combined by the conventional comparative bright combination processing. FIG. 13 is a diagram schematically showing a transition of a combined image generated by the combining processing by the combining processing unit of the image pickup apparatus according to Embodiment 1 of the present invention. FIG. 14 is a flowchart showing an outline of the composition setting process executed by the imaging device according to the first modification of the first embodiment of the present invention. FIG. 15 is a diagram showing an example of an image displayed by the display unit of the imaging device according to the first modification of the first embodiment of the present invention. FIG. 16 is a diagram showing an example of an image generated at time T by the imaging device according to the second modification of the first embodiment of the present invention. FIG. 17 is a diagram showing an example of an image generated by the imaging device according to the second modification of the first embodiment of the present invention after a predetermined time t has elapsed from time T. FIG. 18 is a diagram schematically illustrating processing performed by the image information processing unit of the imaging device according to the second modification of the first embodiment of the present invention. FIG. 19 is a diagram schematically illustrating an outline of the synthesis setting process of the imaging device according to the second modification of the first embodiment of the present invention. FIG. 20 is a diagram schematically showing a transition of a combined image when combined by the conventional comparative light combination processing. FIG. 21 is a diagram schematically showing a transition of a combined image generated by the combining processing by the combining processing unit of the imaging device according to the second modification of the first embodiment of the present invention. FIG. 22 is a flowchart showing an outline of processing of the image pickup apparatus according to the third modification of the first embodiment of the present invention. FIG. 23 is a block diagram showing a functional configuration of the image pickup apparatus according to Embodiment 2 of the present invention. FIG. 24 is a flowchart showing an outline of the composition setting process executed by the imaging device according to the second embodiment of the present invention. FIG. 25 is a diagram showing an example of an image displayed by the display unit of the image pickup apparatus according to Embodiment 2 of the present invention. FIG. 26 is a block diagram showing a functional configuration of the image pickup apparatus according to Embodiment 3 of the present invention. FIG. 27 is a flowchart showing an outline of the combination setting process executed by the imaging device according to the third embodiment of the present invention. FIG. 28 is a diagram showing an example of an image displayed by the display unit of the image pickup apparatus according to Embodiment 3 of the present invention. FIG. 29 is a diagram showing an example of an image group continuously generated by the imaging device according to the third embodiment of the present invention. FIG. 30 is a diagram schematically showing a transition of a combined image generated by the image combining unit of the image pickup apparatus according to Embodiment 3 of the present invention. FIG. 31 is a block diagram showing a functional configuration of the image pickup apparatus according to Embodiment 4 of the present invention. FIG. 32 is a diagram schematically showing an example of the composite information recorded by the composite information recording unit of the image pickup apparatus according to Embodiment 4 of the present invention. FIG. 33 is a diagram showing an example of an image generated by the image pickup apparatus according to Embodiment 4 of the present invention. FIG. 34 is a flowchart showing an outline of the combination setting process executed by the fourth embodiment of the present invention. FIG. 35 is a diagram showing an example of a menu image displayed by the display unit of the image pickup apparatus according to Embodiment 4 of the present invention.

Hereinafter, modes for carrying out the present invention (hereinafter, referred to as “embodiments”) will be described with reference to the drawings. The present invention is not limited to the embodiments described below. In the description of the drawings, the same parts will be denoted by the same reference numerals.

(Embodiment 1)
[Structure of imaging device]
FIG. 1 is a block diagram showing a functional configuration of the image pickup apparatus according to Embodiment 1 of the present invention. The imaging device 1 shown in FIG. 1 includes an imaging unit 10, an image processing unit 20, a recording unit 21, a memory I/F 22, a recording medium 23, a display unit 24, an input unit 25, a bus 26, and And a control unit 27.

The imaging unit 10 images a subject and generates image data of this subject. The imaging unit 10 includes a lens unit 11, a diaphragm 12, a shutter 13, an image sensor 14, an A/D conversion unit 15, and a drive control unit 16.

The lens unit 11 forms an optical image (subject image) on the light receiving surface of the image sensor 14. The lens unit 11 is configured by using one or a plurality of lenses. For example, the lens unit 11 is configured by using a focus lens, a zoom lens, and the like. The lens unit 11 moves along the optical axis under the control of the drive control unit 16.

Under the control of the drive control unit 16, the diaphragm 12 adjusts the exposure by limiting the incident amount of the light condensed by the lens unit 11.

Under the control of the drive control unit 16, the shutter 13 switches the state of the image sensor 14 to the exposure state or the light shielding state by performing the opening/closing operation. Under the control of the drive control unit 16, the shutter 13 adjusts the shutter speed, which is the incident time of light that enters the image sensor 14.

The image sensor 14 receives a optical image collected by the lens unit 11 and performs photoelectric conversion to output an electric signal, thereby forming a CMOS (Complementary Metal Oxide Semiconductor) and a CCD (Complementary Metal Oxide Semiconductor). It is configured using an image sensor such as a Charge Coupled Device). Under the control of the drive controller 16, the image sensor 14 continuously generates analog image data at a predetermined frame rate, for example, 30 fps or 60 fps, and outputs the analog image data to the A/D converter 15.

Under the control of the drive control unit 16, the A/D conversion unit 15 performs A/D conversion on the analog image data input from the image sensor 14 to generate digital image data. Output.

The drive control unit 16 controls the entire imaging unit 10 under the control of the control unit 27. Specifically, the drive control unit 16 controls the drive of the lens unit 11 to control the zoom position and the focus position and the open/close state of the diaphragm 12 and the shutter 13. The drive control unit 16 also controls the light exposure time (light reception time) of the image sensor 14 and the image data read timing.

The image processing unit 20 performs various kinds of image processing on the image data generated by the imaging unit 10 and outputs the image data. The detailed configuration of the image processing unit 20 will be described later.

The recording unit 21 is configured by using a ROM (Read Only Memory), a RAM (Random Access Memory), or the like, and has image data input from the A/D conversion unit 15 via the bus 26 and processing being performed by the imaging device 1. Record information temporarily. The recording unit 21 also has a program recording unit 21a. The program recording unit 21a records various programs for operating the imaging apparatus 1, the program according to the first embodiment, and various data used during execution of the programs.

The recording medium 23 is configured by using a memory card or the like mounted from the outside of the imaging device 1. The recording medium 23 is detachably attached to the imaging device 1 via the memory I/F 22. The recording medium 23 is recorded by writing image data that has been subjected to image processing by the image processing unit 20 and digital image data (RAW data) generated by the A/D conversion unit 15. Further, the recording medium 23 reads the image data recorded via the memory I/F 22.

Under the control of the control unit 27, the display unit 24 displays an image corresponding to the image data recorded in the recording unit 21 or the recording medium 23 via the bus 26. Here, as the image display, a confirmation display (REC view display) for displaying image data for a predetermined time (for example, 3 seconds) immediately after shooting, and a reproduction for reproducing an image corresponding to the image data recorded in the recording medium 23 Display, and live view display for sequentially displaying live view images corresponding to image data continuously generated by the imaging unit 10 in a time series are included. The display unit 24 is configured by using a display panel or the like such as liquid crystal or organic EL (Electro Luminescence).

The input unit 25 is configured by using physical switches such as buttons and switches, a touch panel that is provided so as to be superimposed on the display area of the display unit 24, and the like. The input unit 25 receives an input of an operation related to the imaging device 1. Specifically, the input unit 25 inputs an instruction signal for switching the power supply state of the image pickup apparatus 1 to an on state or an off state, a release signal for giving an image pickup instruction to the image pickup apparatus 1, and a designation signal for designating various operations. Accept. The touch panel detects a touch position of an object from the outside and outputs a position signal corresponding to the detected touch position to the control unit 27 via the bus 26.

The bus 26 is configured by using a transmission line or the like that connects the respective components of the image pickup apparatus 1. The bus 26 transfers various data generated inside the imaging device 1 to each component of the imaging device 1.

The control unit 27 is configured by using a CPU (Central Processing Unit) or the like, and controls the operation of the image pickup apparatus 1 by giving instructions and data to each unit included in the image pickup apparatus 1 and the like.

[Configuration of image processing unit]
Next, a detailed configuration of the image processing unit 20 will be described.
The image processing unit 20 includes an image composition unit 30, an image information processing unit 40, and a development processing unit 50.

The image synthesis unit 30 synthesizes the plurality of image data generated by the imaging unit 10 to generate synthetic image data. The image synthesizing unit 30 includes a synthesizing area designating unit 301, a synthesizing method designating unit 302, a threshold designating unit 303, a threshold determining unit 304, and a synthesizing processing unit 305.

The composition area designating unit 301 defines a predetermined area (hereinafter, referred to as “composite area”) in which two composition methods can be designated by the composition method designating unit 302, from among a plurality of images corresponding to a plurality of image data arranged in chronological order. Specify one or more for each. Specifically, the synthesis area designating unit 301 designates a synthesis area for each of the plurality of images arranged in chronological order according to the instruction signal input from the input unit 25.

The synthesizing method designation unit 302 designates two synthesizing methods for synthesizing each of the plurality of images in a predetermined area of each of the plurality of images corresponding to the plurality of image data arranged in time series. Specifically, the synthesis method designating unit 302 designates, by the synthesis region designating unit 301, two synthesis methods that can be executed by the synthesis processing unit 305, which will be described later, according to an instruction signal input from the input unit 25. Specify in the composition area. For example, the synthesizing method designating unit 302, in accordance with the instruction signal input from the input unit 25, adds a comparative bright synthesizing process and a comparative dark synthesizing process to the synthesizing region designated by the synthesizing region designating unit 301 by the synthesizing processing unit 305 described later. Specify the process.

The threshold value specifying unit 303 specifies a threshold value for switching between the two combining methods specified in the combining area by the combining method specifying unit 302. Specifically, the threshold value is designated according to the instruction signal input from the input unit 25.

The threshold value determination unit 304 determines whether or not the value determined according to the image information extracted by the image information extraction unit 402 of the image information processing unit 40, which will be described later, is a threshold value or more. For example, the threshold determination unit 304 determines whether or not the brightness component included in the image information is equal to or greater than the threshold.

The combining processing unit 305 combines at least two pieces of image data based on the combining area specified by the combining area specifying unit 301 and the combining method specified by the combining method specifying unit 302 to generate combined image data. .. Specifically, the synthesis processing unit 305 sequentially synthesizes a plurality of images arranged in chronological order while sequentially switching the two synthesis methods designated in the synthesis area by the synthesis method designation unit 302 to generate one synthesized data. .. The composition processing unit 305 includes a comparative light composition processing unit 305a, a comparative dark composition processing unit 305b, an addition composition processing unit 305c, an average composition processing unit 305d, and a replacement composition processing unit 305e.

When synthesizing the image data of at least two frames or more, the comparative light composition processing unit 305a compares the pixel values of the same pixel address (coordinate position), and the largest pixel value is the pixel value (output value) of the composite image. To generate composite image data.

The comparative dark composition processing unit 305b compares the pixel values of the same pixel address when combining the image data of at least two frames or more, and generates the composite image data with the smallest pixel value as the pixel value of the composite image.

When synthesizing image data of at least two frames or more, the addition synthesizing processing unit 305c adds pixel values of pixels having the same pixel address, and generates synthetic image data by using the added value obtained as a pixel value of the synthetic image. ..

When synthesizing image data of at least two frames, the average synthesizing processing unit 305d averages pixel values of the same pixel address, and generates synthetic image data by using the averaged average value as a pixel value of the synthetic image.

The replacement composition processing unit 305e generates composite image data by using one of the pixel values of the same pixel address as the pixel value of the composite image when combining the images of the two frames corresponding to the image data of the two frames. To do.

The image information processing unit 40 extracts image information included in a predetermined area from the image data generated by the imaging unit 10, and causes the display unit 24 to display the extracted information. The image information processing unit 40 includes an information area designation unit 401, an image information extraction unit 402, and a display control unit 403.

The information area designation unit 401 designates an information area for the image information extraction unit 402 to extract image information. Specifically, according to the instruction signal input from the input unit 25, the image information extraction unit 402 specifies the information area for extracting the image information.

The image information extraction unit 402 extracts the image information included in the information area designated by the information area designation unit 401. Here, the image information is any one or more of the luminance, the color component, the change amount of the pixel value between the image data, and the movement amount of the subject.

The display control unit 403 causes the display unit 24 to display the image information extracted by the image information extraction unit 402.

The development processing unit 50 performs predetermined image processing on the image data output from the imaging unit 10 or the combined image data generated by the image combining unit 30 to develop the image data or the combined image data and display the image data. 24 or the recording medium 23. Here, the predetermined image processing includes demosaicing processing, noise reduction processing, white balance processing, resizing processing, compression processing, and the like.

[Processing of imaging device]
Next, the processing executed by the imaging device 1 will be described. FIG. 2 is a flowchart showing an outline of processing executed by the image pickup apparatus 1.

As shown in FIG. 2, the control unit 27 first causes the image capturing unit 10 to capture an image, and causes the display unit 24 to display a live view image corresponding to image data obtained by performing image processing on the image data by the image processing unit 20 ( Step S101). Specifically, as shown in FIG. 3, the display unit 24 displays the live view image LV1 generated by the imaging unit 10.

Subsequently, the image information processing unit 40 executes an image information extraction process of extracting and displaying information such as brightness and RGB components from the image corresponding to the image data generated by the imaging unit 10 (step S102).

[Image information extraction processing]
FIG. 4 is a flowchart showing an outline of the image information extraction processing in step S102 of FIG.

As illustrated in FIG. 4, the display control unit 403 causes the live view image LV1 displayed by the display unit 24 to be superimposed and displayed with an extraction region for extracting image information (step S201). Specifically, as shown in FIG. 5, the display control unit 403 causes the display unit 24 to display the extraction region Z1 superimposed on the live view image LV1. This allows the user to intuitively understand the current extraction area Z1.

Subsequently, the information area designating unit 401 designates an extraction area from which image information is extracted according to the instruction signal from the input unit 25 (step S202).

After that, the image information extraction unit 402 extracts the image information of the extraction area designated by the information area designation unit 401 (step S203).

Subsequently, the display control unit 403 superimposes the image information extracted by the image information extraction unit 402 on the live view image LV1 and displays it on the display unit 24 (step S204). Specifically, as shown in FIG. 6, the display control unit 403 causes the display unit 24 to display the image information A1 superimposed on the live view image LV1. This allows the user to intuitively understand the image information of the area designated via the input unit 25. In FIG. 6, as image information, the luminance component (Y:22), the R component (R:22), the G component (G:35), and the B component (B:15) of the average value of each component in the extraction region are included. However, in addition to this, for example, saturation, histogram, etc. may be displayed. Of course, only a predetermined component, for example, the luminance component may be displayed. Further, the image information extraction unit 402 calculates the brightness component (R component, G component, B component) of the average value of each frame for a plurality of live view images, and the average value, maximum value, and minimum value of the values. May be displayed. Further, although the image information extraction unit 402 extracts the image information from the live view image LV1, the image information may be extracted from an image in the process of combining, which will be described later.

After that, when the instruction signal for additionally designating the extraction region for extracting the image information is input from the input unit 25 (step S205: Yes), the imaging device 1 returns to step S202 described above. On the other hand, when the instruction signal for adding the extraction region for extracting the image information is not input from the input unit 25 (step S205: No), the imaging device 1 returns to the main routine of FIG.

Returning to FIG. 2, the description of step S103 and subsequent steps will be continued.
In step S103, the image composition unit 30 executes composition setting processing for setting the composition method specified for each area specified by the user.

[Composition setting process]
FIG. 7 is a flowchart showing an outline of the synthesis setting process of step S103 of FIG. FIG. 8 is a diagram schematically illustrating the combination setting process.

As shown in FIG. 7, the synthesis area designating unit 301 designates a synthesis area for performing different synthesizing methods when synthesizing images of two or more frames according to an instruction signal from the input unit 25 ( Step S301). Specifically, as shown in FIG. 8, the composition area designating unit 301 composes different composition methods when composing images of two or more frames according to an instruction signal from the input unit 25. The region R1 is designated. As a result, the display control unit 403 causes the display unit 24 to display the combined area R1 specified by the combined area specifying unit 301 in a superimposed manner on the live view image LV1. In this case, the display control unit 403 causes the display unit 24 to display the icon A12 related to the combining method in the combining mode for the area other than the specified combining area R1 and the information A121 of the combining method. Further, the display control unit 403 causes the display unit 24 to display the information A10 regarding the threshold value of the combining method specified for the combining area R1 described later and the icon A11 regarding the combining method for the combining area R1. Furthermore, the display control unit 403 causes the display unit 24 to display information A111 and information A112 related to the combining method set according to a threshold value described later.

Subsequently, the synthesis method designating unit 302 designates a synthesis method to be synthesized by the synthesis processing unit 305 with respect to the synthesis area designated by the synthesis area designating unit 301 in accordance with the instruction signal from the input unit 25 (step S302). Specifically, as shown in FIG. 9, when the user selects the information A111 of the icon A11 via the input unit 25, the display control unit 403 causes the display unit 24 to display a plurality of synthesizing processes that can be combined. .. The user selects two from the displayed plurality of combining processes via the input unit 25. As a result, the synthesizing method designating unit 302 designates, in accordance with the instruction signal from the input unit 25, two synthesizing methods to be synthesized by the synthesizing processing unit 305 with respect to the synthesizing area designated by the synthesizing area designating unit 301. .. The information A111 includes a comparative light composition process, a comparative dark composition process, an additive composition process, an average composition process, a replacement composition process, and a non-composition.

After that, the threshold designation unit 303 designates a threshold for switching between the two synthesis methods designated by the synthesis method designation unit 302 for the synthesis area in response to the instruction signal from the input unit 25 (step S303). Specifically, as shown in FIG. 8, the user selects each image information displayed in the information A100 by the display unit 24 via the input unit 25, and inputs the threshold value when performing the combining process on the combining region R1. To do. Accordingly, the threshold value specifying unit 303 can specify a threshold value for switching between the two combining methods specified by the combining method specifying unit 302 for the combining area in accordance with the instruction signal from the input unit 25.

Subsequently, when the instruction signal for additionally designating the composite area is input from the input unit 25 (step S304: Yes), the imaging device 1 returns to step S301 described above. On the other hand, when the instruction signal for additionally designating the composite area is not input from the input unit 25 (step S304: No), the imaging device 1 returns to the main routine of FIG.

Returning to FIG. 2, the description of step S104 and subsequent steps will be continued.
In step S104, when the instruction signal for instructing the imaging is input from the input unit 25 (step S104: Yes), the imaging unit 10 executes the imaging (step S105). On the other hand, when the instruction signal for instructing the imaging is not input from the input unit 25 (step S104: No), the imaging device 1 continues this determination.

Subsequently, when the image data captured and generated by the image capturing unit 10 is the first frame (step S106: Yes), the image capturing apparatus 1 proceeds to step S108 described below. On the other hand, when the image data captured and generated by the image capturing unit 10 is not the first frame (step S106: No), the image synthesizing unit 30 executes a synthesizing process of synthesizing image data of at least two frames or more. Yes (step S107). The details of the combining process will be described later. After step S107, the imaging device 1 moves to step S108 described later.

In step S108, the control unit 27 causes the recording unit 21 to temporarily record the image data generated by the imaging unit 10.

Subsequently, the development processing unit 50 performs the development processing on the image data temporarily recorded in the recording unit 21 (step S109). Specifically, the development processing unit 50 develops image data by performing demosaicing processing, noise reduction processing, white balance processing, resizing processing, and compression processing on the image data.

After that, the control unit 27 causes the display unit 24 to display the shooting result for a predetermined time, for example, 3 seconds (step S110).

Then, when the instruction signal for ending the combining process is input from the input unit 25 (step S111: Yes), the imaging device 1 proceeds to step S112 described below. On the other hand, when the instruction signal to end the combining process is not input from the input unit 25 (step S111: No), the imaging device 1 returns to step S105 described above.

In step S112, the control unit 27 records the image data generated by the development processing unit 50 on the recording medium 23 via the memory I/F 22 (step S112).

Subsequently, the control unit 27 causes the display unit 24 to display an image corresponding to the image data generated by the development processing unit 50 (step S113). After step S113, the imaging device 1 ends this process.

[Synthesis processing]
Next, details of the combining process described in step S107 of FIG. 2 described above will be described. FIG. 10 is a flowchart showing an outline of the combining process executed by the image combining unit 30.

As shown in FIG. 10, the synthesis area designating unit 301 selects the synthesis area designated in step S301 of FIG. 7 described above for the image corresponding to the image data generated by the imaging unit 10 (step S401).

Subsequently, the image information extraction unit 402 extracts the image information of the combined area specified by the combined area specifying unit 301 (step S402).

Then, the threshold determination unit 304 determines whether the image information extracted by the image information extraction unit 402 is equal to or more than the threshold specified by the threshold specification unit 303 (step S403). When the threshold value determination unit 304 determines that the image information extracted by the image information extraction unit 402 is equal to or greater than the threshold value (step S403: Yes), the imaging device 1 proceeds to step S404 described below. On the other hand, when the threshold determination unit 304 determines that the image information extracted by the image information extraction unit 402 is not greater than or equal to the threshold (step S403: No), the imaging device 1 proceeds to step S405 described below.

In step S404, the combining processing unit 305, in the combining area specified by the combining area specifying unit 301, the combining method specified by the combining method specifying unit 302 when the image information is equal to or larger than the threshold (the first specified by the threshold or more). New image data is combined by the (1 combining method). After step S404, the imaging device 1 moves to step S406 described later.

In step S405, the combining processing unit 305 determines that the combining method specified by the combining area specifying unit 301 when the image information is not greater than or equal to the threshold value (less than the threshold value) by the combining method specifying unit 302 (less than the threshold value). New image data is synthesized by the designated second synthesis method). After step S405, the imaging device 1 moves to step S406 described later.

FIG. 11 is a diagram schematically illustrating an example of a time-series image group that is synthesized by the synthesis processing unit 305. FIG. 12 is a diagram schematically showing transition of a combined image when the images in FIG. 11 are combined by the conventional comparative bright combination processing. FIG. 13 is a diagram schematically showing the transition of the combined image generated by the combining processing unit 305 by the combining process. In addition, FIG. 13 schematically illustrates transitions of a combined image that is combined by the combining method specified by the combining method specifying unit 302 in the combining area specified by the combining area specifying unit 301. Further, in FIG. 13, the comparative bright combination process is specified for an area other than the combination area specified by the combination area specification unit 301, and no combination (non-combination) is specified for the combination area when the comparison area is equal to or more than the threshold value. A case where the comparative bright combination process is designated when the value is less than the threshold will be described. 12 and 13, the case where the comparative bright combination is specified by the combining method specifying unit 302 will be described, but the same applies to other combining methods.

As shown in FIG. 11, in the case where the image capturing apparatus 1 is capturing a star trajectory as a main subject, when the vehicle K1 passes through during capturing, the light of the headlight of the vehicle K1 is also captured (image P1→ Image P2→image P3→image P4). For this reason, as shown in FIG. 12, in the conventional comparative bright combination, when the images P1 to P4 are sequentially combined in a time series, the light of the headlight of the vehicle K1 is also gradually combined (image P11). → image P12 → image P13 → image P14). As a result, in the conventional comparative bright combination, there is a problem that the locus of the headlight of the car K1 becomes an obstacle and an image P14 (composite image) that does not meet the photographer's intention of creation is generated.

On the other hand, as shown in FIG. 13, the combining processing unit 305 specifies the threshold value and the combining method specified by the threshold value specifying unit 303 in the combining area R1 specified by the combining area specifying unit 301 in FIG. 7 described above. When performing a combining process of sequentially combining the images P2 to P4 with the image P1 in a time series by the combining method specified by the unit 302, image information (for example, brightness) in the combining region R1 of the images P2 to P4 is When the threshold value is equal to or more than the threshold value, the combination method is not combined (non-combined), and thus the images P2 to P4 are not combined in the combined area R1 of the image P1 and the image information (for example, brightness) in the combined area R1 of the images P2 to P4 is set. When it is less than the threshold value, the images P2 to P4 are sequentially compared and brightly combined in the combination area R1 of the image P1. Further, the combination processing unit 305 performs comparative bright combination on the area other than the combination area R1. As a result, in the first embodiment, when the images P1 to P4 are sequentially combined in a time series, there is no headlight trajectory of the car K1 and the photographer's original intention that the trajectory of the star appears is in line. A composite image (image P21→image P22→image P23→image P24) can be generated.

Returning to FIG. 10, the description of step S406 and subsequent steps will be continued.
In step S406, when there is an additional area where the images are combined (step S406: Yes), the imaging device 1 returns to step S401 described above. On the other hand, when there is no additional area for combining images (step S406: No), the imaging device 1 returns to the main routine of FIG.

According to the first embodiment of the present invention described above, the synthesizing method designating unit 302 designates two synthesizing methods for synthesizing each of a plurality of images in each synthesizing region area of the plurality of images, and The information extraction unit 402 sequentially extracts the image information included in a predetermined area of each of the plurality of images, and the synthesis processing unit 305 specifies the synthesis area based on the image information sequentially extracted by the image information extraction unit 402. Since a plurality of images are sequentially combined while one of the two combining methods is switched to generate one combined data, it is possible to generate a combined image that reflects the user's original intention.

Further, according to the first embodiment of the present invention, the threshold designating unit 303 designates a threshold for switching between the two synthesis methods designated in the synthesis area by the synthesis method designating unit 302, and the threshold determining unit 304 sets the image information. Since the value determined according to the image information extracted by the extraction unit 402 is greater than or equal to a threshold value, and the combination processing unit 305 generates combined data based on the determination result of the threshold value determination unit 304, the user Can generate a composite image that better reflects the original intention.

(Modification 1 of Embodiment 1)
Next, a first modification of the first embodiment of the present invention will be described. The first modification according to the first embodiment has the same configuration as that of the image pickup apparatus 1 according to the above-described first embodiment, and the composition setting process executed by the image composition unit is different. Specifically, in the above-described first embodiment, the threshold for switching the content of the composition process specified in the composition area is manually specified according to the instruction signal input from the input unit 25. In the first modification of the first embodiment, the threshold for switching the content of the composition process specified in the composition area is automatically specified based on the image information extracted by the image information extraction unit 402. In the following, a combination setting process executed by the image combining unit according to the first modification of the first embodiment will be described. The same components as those of the image pickup apparatus 1 according to the first embodiment described above are designated by the same reference numerals, and description thereof will be omitted.

[Composition setting process]
FIG. 14 is a flowchart showing an outline of the combination setting process executed by the image pickup apparatus 1 according to the first modification of the first embodiment of the present invention. 14, steps S501 and S503 correspond to steps S302 and S304 of FIG. 7 described above, respectively.

In step S502, the image composition unit 30 specifies the extraction region in which the image information is extracted by the image information processing unit 40 as the composition region, and sets the image information as the threshold value. Specifically, as shown in FIG. 15, the synthesis area designating unit 301 designates the extraction area designated by the information area designating unit 401 as the synthesis area R1. Further, the threshold value specifying unit 303 specifies the image information extracted by the image information extracting unit 402 as a threshold value for switching the content of the combining method specified in the combining area. In this case, the threshold value specifying unit 303 may specify a value obtained by adding any one of predetermined values to the image information extracted by the image information extracting unit 402, subtraction, multiplication, and division as the threshold value. .. For example, in the case shown in FIG. 15, the threshold designation unit 303 designates the threshold by adding +10 as a predetermined value to each value included in the image information extracted by the image information extraction unit 402. The threshold value specifying unit 303 adds, subtracts, multiplies, or divides the image information extracted by the image information extracting unit 402 by a predetermined value according to the content of the combining process specified by the combining method specifying unit 302. A value obtained by performing either one may be designated as the threshold value. Further, the threshold value designation unit 303 designates a value (luminance of the extraction region or an average value of color components) determined in accordance with the image information of the oldest image among the plurality of images extracted by the image information extraction unit 402, as the synthesis region. The threshold value for switching the contents of the combined method is set. After step S502, the imaging device 1 moves to step S503.

According to the first modification of the first embodiment of the present invention described above, the threshold value specifying unit 303 sets a threshold value that is determined according to the image information extracted by the image information extracting unit 402 for the oldest image among the plurality of images. Therefore, by specifying only the extraction area without performing complicated settings, it is possible to generate a composite image that reflects the user's original intention.

(Modification 2 of Embodiment 1)
Next, a second modification of the first embodiment of the present invention will be described. In the above-described first embodiment, YRGB (luminance component, R component, G component, and B component) is used as the image information, but in the second modification of the first embodiment, images arranged temporally as image information. The movement amount of the moving subject in the group is used. Further, since the image pickup apparatus according to the second modification of the first embodiment performs the same processing as the image pickup apparatus 1 according to the above-described first embodiment, the image pickup apparatus according to the second modification of the first embodiment is The details of the processing to be executed are omitted, and the outline of the processing executed by the imaging device according to the second modification of the first embodiment will be described. In the following, the same components as those of the image pickup apparatus 1 according to the first embodiment described above are designated by the same reference numerals, and description thereof will be omitted.

[Outline of image information extraction]
FIG. 16 is a diagram showing an example of an image generated by the imaging device 1 at time T. FIG. 17 is a diagram illustrating an example of an image generated by the imaging device 1 after a predetermined time t has elapsed from the time T. FIG. 18 is a diagram schematically illustrating the process performed by the image information processing unit 40. Since FIG. 17 is an image generated at the same position where the image pickup device 1 generated the image P _T after a predetermined time t has elapsed, the star of the subject shown in the image P _T+t has moved from the image P _T.

As shown in FIGS. 16 to 18, when the extraction region R10 for extracting the image information is specified according to the instruction signal from the input unit 25, the image information extraction unit 402 determines that the image P _T and the image P _T+t are included. Based on this, the movement amount of the subject (star) that has moved in the extraction region R10 is extracted. Specifically, the image information extraction unit 402 digitizes the movement amount of the subject (star) that has moved in the extraction region R10 by the number of pixels (number of pixels), and extracts the digitized value as image information. For example, as shown in FIG. 16 to FIG. 18, the image information extraction unit 402 determines that the average of the movement amounts of the plurality of subjects moving in the extraction region R10 is 100 at time T+t after a predetermined time t has elapsed from time T. If it is a pixel, 100 pixels are extracted as the image information in the extraction region R10. In this case, the image information extraction unit 402 extracts the image information by calculating the movement amount of the pixel or the region having the brightness equal to or higher than the predetermined value in the extraction region R10 based on the image P _T and the image P _T+t . Then, the display control unit 403 superimposes the image information A20 extracted by the image information extraction unit 402 on the image P _T+t and displays it on the display unit 24. As a result, the user can intuitively grasp the amount of movement of the subject.

[Outline of synthesis setting processing]
Next, an outline of the combination setting process will be described. FIG. 19 is a diagram schematically illustrating the outline of the synthesis setting process.

As shown in FIG. 19, the user specifies, via the input unit 25, each of the synthesizing method for the synthesizing area R1, the threshold for switching the content of the synthesizing method, and the synthesizing method other than the synthesizing area R1. More specifically, the user specifies the comparative bright combination process in the non-selected area other than the combination area R1 and also specifies the combination area R1 via the input unit 25. In this case, the synthesis area designating unit 301 designates the synthesis area R1 according to the instruction signal input from the input unit 25.

Subsequently, the user specifies the composition method in the composition area R1 via the input unit 25. In this case, the synthesizing method designating unit 302 designates the synthesizing method in the synthesizing region R1 according to the instruction signal input from the input unit 25. For example, the composition method designation unit 302 designates the comparative light composition and the comparative dark composition for the composition region R1.

After that, the user specifies, via the input unit 25, a threshold for switching the contents of the combining method specified in the combining area R1. For example, the user specifies a threshold for switching the content of the composition method specified in the composition area R1 based on the image information displayed in FIG. For example, when the subject moves 100 pixels in FIG. 18, 200 pixels are designated as the threshold value. In this case, the threshold value specifying unit 303 specifies a threshold value for switching the content of the combining method specified in the combining area R1 by the combining method specifying unit 302 according to the instruction signal input from the input unit 25. As a result, an object that moves significantly by 200 pixels or more in the composition area R1 (for example, an object that moves faster than a star) is subjected to comparative dark composition by the composition processing unit 305, while it moves small by less than 200 pixels in the composition area R1. The subject (for example, a subject that moves slowly including a star) is subjected to comparative bright combination by the combination processing unit 305.

[Outline of synthesis processing]
Next, an outline of the combining process performed by the combining processing unit 305 will be described. FIG. 20 is a diagram schematically showing a transition of a combined image when combined by the conventional comparative light combination processing. FIG. 21 is a diagram schematically showing the transition of the combined image generated by the combining processing unit 305 by the combining process.

As shown in FIG. 20, in the conventional comparative bright combination processing, when the airplane K10 is reflected in the middle of shooting by the imaging device 1, the lights L1 of the airplane K10 are sequentially combined (image P31→image P32→image P33). →Image P34). Since the user photographs the star as the main subject, the trajectory of the light L1 of the airplane K10 becomes an image P34 (composite image) that does not match the original intention, and the photographing fails.

On the other hand, as shown in FIG. 21, the combination processing unit 305 specifies the threshold value and the combination method specified by the threshold value specifying unit 303 in the combination area R1 specified by the combination area specifying unit 301 in FIG. The images are sequentially combined by the combining method designated by the unit 302 (image P41→image P42→image P43→image P44). As shown in FIG. 21, the moving speed of the airplane K10 is faster than that of a star, so that the moving amount of each frame is 200 pixels or more. Therefore, the synthesis processing unit 305 performs comparative dark synthesis on the trajectory of the light L1 of the airplane K10. In this case, the synthesizing unit 305 does not synthesize the light L1 of the airplane K10, because the background starry sky of the synthesis region R1 is darker than the light L1 of the airplane K10 (the brightness is low). As a result, the synthesis processing unit 305 generates an image P44 (synthetic image) in which the trajectory of the light of the star is reflected and the trajectory of the light L1 of the unnecessary airplane K10 is not reflected. As a result, it is possible to generate the image P44 that is in line with the photographer's original intention.

According to the second modification of the first embodiment of the present invention described above, it is possible to generate a composite image that reflects the user's creative intention.

In the second modification of the first embodiment of the present invention, the YRGB components may be added together with the movement amount designated by the threshold designation unit 303. For example, the lights of the airplane K10 have various colors, but the colors of stars are limited. Therefore, by specifying a threshold value for the colors (RGB components), the airplane K10 flies in the distance and moves. Even in the case of an airplane whose amount is the same as or smaller than that of a star, it is possible to generate a composite image in which the trajectory of the light of the airplane is excluded.

(Modification 3 of Embodiment 1)
Next, a third modification of the first embodiment of the present invention will be described. The modified example 3 of the first embodiment is different in the process executed by the imaging device. Specifically, in the third modification of the first embodiment, when the display unit is displaying the combined image for which the combining processing has been completed by the combining processing unit, an instruction signal for instructing to change the combining method is input from the input unit. When is input, the specified combined area, combining method, and threshold value are changed to generate a combined image again. In the following, a process executed by the imaging device according to the third modification of the first embodiment will be described. The same components as those of the image pickup apparatus 1 according to the first embodiment described above are designated by the same reference numerals, and description thereof will be omitted.

[Processing of imaging device]
FIG. 22 is a flowchart showing an outline of processing executed by the image pickup apparatus 1 according to the third modification of the first embodiment. 22, steps S601 to S610 correspond to steps S101 to S110 of FIG. 2 described above, respectively.

In step S611, when the instruction signal for changing the combining method is input from the input unit 25 (step S611: Yes), the imaging device 1 proceeds to step S612 described below. On the other hand, when the instruction signal for changing the composition method is not input from the input unit 25 (step S611: No), the imaging device 1 proceeds to step S614 described below.

Steps S612 and S613 correspond to steps S102 and S103 of FIG. 2 described above, respectively. In this case, the user can change the setting of the combined image and the combining method while viewing the combined image that is the shooting result. After step S613, the imaging device 1 moves to step S614 described later. Steps S614 to S616 correspond to steps S111 to S113 of FIG. 2 described above, respectively.

In step S617, when the instruction signal for changing the composition method is input via the input unit 25 (step S617: Yes), the imaging device 1 proceeds to step S618 described below. On the other hand, when the instruction signal for changing the combining method is not input via the input unit 25 (step S617: No), the imaging device 1 ends this processing.

Steps S618 and S619 correspond to steps S102 and S103 of FIG. 2 described above, respectively. Step S620 corresponds to step S107 of FIG. 2 described above. Steps S621 and S622 correspond to steps S112 and S113 of FIG. 2 described above, respectively.

In step S623, when the instruction signal for changing the combining method is input via the input unit 25 (step S623: Yes), the imaging device 1 proceeds to step S618 described above. On the other hand, when the instruction signal for changing the synthesizing method is not input via the input unit 25 (step S623: No), the imaging device 1 ends this processing.

According to the modified example 3 of the first embodiment of the present invention described above, the combining method and the threshold value can be changed via the input unit 25 even during shooting, so that the user's original intention is reflected. A synthetic image can be generated.

Further, according to the third modification of the first embodiment of the present invention, the composition method and the threshold value can be changed via the input unit 25 even after photographing, so that the composition reflecting the user's creative intention is performed. Images can be generated.

(Embodiment 2)
Next, a second embodiment of the present invention will be described. The image pickup apparatus according to the second embodiment has a different configuration from the image pickup apparatus 1 according to the above-described first embodiment, and also has different composition setting processing to be executed. Specifically, in the imaging apparatus according to the second embodiment, when a plurality of composite areas designated by the user are designated, when the plurality of designated composite areas overlap, the priority assigned to the composite area is set. A synthetic image is generated by a synthesizing method according to. In the following, the configuration of the imaging device according to the second embodiment will be described, and then the synthesis setting process executed by the imaging device according to the second embodiment will be described. The same components as those of the image pickup apparatus 1 according to the first embodiment described above are designated by the same reference numerals, and description thereof will be omitted.

[Structure of imaging device]
FIG. 23 is a block diagram showing a functional configuration of the image pickup apparatus according to Embodiment 2 of the present invention. The image pickup apparatus 1a illustrated in FIG. 23 includes an image processing unit 20a instead of the image processing unit 20 of the image pickup apparatus 1 according to the first embodiment described above. The image processing unit 20a has a priority designation unit 306 in addition to the configuration of the image processing unit 20 according to the first embodiment described above.

The priority designating section 306 designates the priority of each of the plurality of composite areas designated by the composite area designating section 301. Specifically, when at least a part of the plurality of combined areas specified by the combined area specifying unit 301 overlap, the priority specifying unit 306 specifies the priority of the overlapping overlapping areas. ..

[Composition setting process]
Next, the combination setting process executed by the image pickup apparatus 1a will be described. FIG. 24 is a flowchart showing an outline of the combination setting process executed by the image pickup apparatus 1a. 24, steps S701 to S704 correspond to steps S301 to S304 of FIG. 7 described above.

In step S705, when there is an overlap in the combined area specified by the combined area specifying unit 301 (step S705: Yes), the imaging device 1a moves to step S706 described below. On the other hand, when there is no overlap of the combined areas specified by the combined area specifying unit 301 (step S705: No), the imaging device 1a returns to the main routine of FIG.

In step S<b>706, the priority designation unit 306 designates the priority of each of the plurality of synthesis areas designated by the synthesis area designation unit 301. Specifically, as shown in FIG. 25, in the case where the combination area R1 and the combination area R2 specified by the combination area specification section 301 overlap, the priority specifying section 306 overlaps this overlap. The priority for the region R3 is designated. For example, when the user specifies the priority of the combined area R1 as “1” and the priority of the combined area R2 as “2” via the input unit 25, the priority specifying unit 306 receives the input from the input unit 25. In response to the instruction signal of 1, the priority a1 of the synthesis area R1 is designated as "1" and the priority a2 of the synthesis area R2 is designated as "2". The display control unit 403 causes the display unit 24 to display the information A40 regarding the threshold value, the combination method, and the priority set in each of the combination area R1 and the combination area R2 on the live view image LV1 in a superimposed manner. After that, the image composition unit 30a performs composition on the overlapping area R3 with the threshold value and the composition method set in the composition area R1.

According to the second embodiment of the present invention described above, in the case where two or more synthesis areas are designated, even when overlapping areas in which at least some areas of the synthesis areas overlap with each other are generated, the priority designation section 306. Since the synthesizing unit 305 performs synthesizing according to the priority of, the synthetic image that reflects the user's original intention can be generated.

(Embodiment 3)
Next, a third embodiment of the invention will be described. The third embodiment is different from the configuration of the image pickup apparatus 1a according to the second embodiment described above, and is different in the combination setting process. Specifically, in the third embodiment, a synthetic image is generated while tracking a moving subject. In the following, the configuration of the image pickup apparatus according to the third embodiment will be described, and then the combination setting process executed by the image pickup apparatus according to the third embodiment will be described. The same components as those of the image pickup apparatus 1a according to the second embodiment described above are designated by the same reference numerals, and description thereof will be omitted.

[Structure of imaging device]
FIG. 26 is a block diagram showing a functional configuration of the image pickup apparatus according to Embodiment 3 of the present invention. The imaging device 1b shown in FIG. 26 includes an image processing unit 20b instead of the image processing unit 20a of the imaging device 1a according to the second embodiment described above. The image processing unit 20b includes a subject specifying unit 60 and a subject tracking unit 70 in addition to the configuration of the image processing unit 20a according to the second embodiment described above.

The subject designation unit 60 designates a subject including a subject in the image according to the instruction signal input from the input unit 25 as a tracking subject.

The subject tracking unit 70 extracts the feature amount of the subject designated by the subject designation unit 60, and based on the feature amount, each of the plurality of images corresponding to the plurality of image data continuously generated by the imaging unit 10 The subject area including the position where the subject is captured is sequentially tracked. Here, the feature amount is brightness, color, edge, contrast, or the like. Further, the display control unit 403 may superimpose the tracking result by the subject tracking unit 70 on the image and display it on the display unit 24.

[Composition setting process]
Next, the combination setting process executed by the image pickup apparatus 1b will be described. FIG. 27 is a flowchart showing an outline of the combination setting process executed by the image pickup apparatus 1b. 27, steps S801 to S804 correspond to steps S301 to S304 of FIG. 7 described above.

In step S805, when there is an instruction signal instructing the tracking designation of the combined area designated by the combined area designation section 301 from the input section 25 (step S805: Yes), the imaging device 1b proceeds to step S806 described later. .. On the other hand, when there is no instruction signal from the input unit 25 to instruct the tracking designation of the synthesis area designated by the synthesis area designation unit 301 (step S805: No), the imaging device 1b proceeds to the main routine of FIG. Return.

In step S806, the subject designation unit 60 designates a subject to be tracked in the image group continuously generated by the imaging unit 10 according to the instruction signal input from the input unit 25. Specifically, as shown in FIG. 28, when the synthesis area designating unit 301 designates the synthesis area R100 for the subject K200, an instruction signal for designating the subject K200 as the tracking subject is input from the input unit 25. At this time, the subject specifying unit 60 specifies the subject K200 as a tracking subject. In this case, the synthesis area designating unit 301 moves the synthesis area R100 including the subject K200 for each image continuously generated by the imaging unit 10 based on the tracking result tracked by the subject tracking unit 70. However, the position of the composite area R100 (subject K200) is tracked and designated. Further, the display control unit 403 causes the display unit 24 to display the composite area including the subject designated by the subject designation unit 60 on the image P100. Furthermore, the display control unit 403 causes the display unit 24 to display the information related to the combining method other than the combining area A200, the threshold value for the combining area R100, the combining method, and the information A200 regarding the presence or absence of tracking on the image P100.

Then, when there is an instruction signal from the input unit 25 to instruct the tracking designation of the synthesis area designated by the synthesis area designation unit 301 (step S807: Yes), the imaging device 1b proceeds to step S808 described later. On the other hand, when there is no instruction signal from the input unit 25 to instruct the tracking designation of the synthesis area designated by the synthesis area designation unit 301 (step S807: No), the imaging apparatus 1b proceeds to the main routine of FIG. Return.

In step S808, the subject designation unit 60 designates a subject to be tracked in the image group continuously generated by the imaging unit 10 according to the instruction signal input from the input unit 25. Specifically, as shown in FIG. 28, when the combined area R100 and the combined area R101 are specified by the combined area specifying unit 301 for each of the object K200 and the object K201, the object K201 is tracked from the input unit 25. When an instruction signal for further specifying is input, the subject specifying unit 60 further specifies the subject K201 as a tracking subject. In this case, the combination area designating unit 301 includes the combination area including the subject K200 and the subject K201 for each image continuously generated by the imaging unit 10 based on the tracking result tracked by the object tracking unit 70. The positions of the combination area R100 (subject K200) and the combination area R101 (subject K201) are tracked and designated while moving R100 and the combination area R101. Further, the display control unit 403 causes the display unit 24 to display the composite area including the subject designated by the subject designation unit 60 on the image P100. Furthermore, the display control unit 403 provides information regarding a synthesizing method other than the synthesizing area as A100, a threshold for the synthesizing area R100, synthesizing method, and information regarding presence/absence of tracking A200 and a threshold for synthesizing area R101, regarding a synthesizing method, and presence/absence of tracking. The information A201 is superimposed on the image P100 and displayed on the display unit 24.

Then, if there is an instruction signal from the input unit 25 to instruct tracking designation of the synthesis area designated by the synthesis area designation unit 301 (step S809: Yes), the imaging device 1b returns to step S808 described above. On the other hand, if there is no instruction signal from the input unit 25 to instruct the tracking designation of the synthesis area designated by the synthesis area designation unit 301 (step S809: No), the imaging device 1b proceeds to step S810 described below. To do.

In step S810, the priority designating unit 306 designates the priority of each of the plurality of composite areas designated by the composite area designating unit 301. After step S807, the imaging device 1b returns to the main routine of FIG.

[Outline of synthesis processing]
Next, an outline of the combining process performed by the image combining unit 30a set by the above-described combining setting process will be described. FIG. 29 is a diagram showing an example of an image group continuously generated by the imaging device 1b. FIG. 30 is a diagram schematically showing the transition of the composite image generated by the image composition unit 30a. Note that, in FIG. 30, the replacement method is specified by the combining method designating unit 302 in the combining areas R100 and R101, and the average combining process is specified in areas other than the combining areas R100 and R101.

As shown in FIG. 29, the composite area designation unit 301 moves the composite area for each frame based on the tracking result of the subject tracking unit 70 (image P100→image P101→image P102). Then, as shown in FIG. 30, the combining processing unit 305 performs the replacement combining process on the combining regions R100 and R101, and performs the average combining process on the regions other than the combining regions R100 and R101. As a result, the composition processing unit 305 replaces the images of the composition areas R100 and R101 with the images of the Ferris wheel, and composes the areas other than the composition areas R100 and R101 by the average composition processing. For example, a pillar of an airplane K100 or a ferris wheel that moves between images appears thin (image P200→image P201). In this case, when the airplane K100 and the combination area R101 overlap, the combination processing unit 305 gives priority to the combination area R101. Therefore, the synthesizing unit 305 replaces the image of the synthesizing region R101 with the image of the Ferris wheel.

After that, when the combining area R100 and the combining area R101 occur, the combining processing unit 305 specifies the content of the combining processing for the overlapping area where the combining area R100 and the combining area R101 overlap each other by the priority specifying unit 306. The synthesizing process is performed by the synthesizing process of the synthesizing region R100 having high priority. Specifically, as shown in FIG. 30, the combining processing unit 305 performs the combining processing by giving priority to the contents of the combining processing specified in the combining area R100. As a result, the subject K200 (ferris wheel) is combined, and the subject K201 (ferris wheel) is not combined into a combined image P202.

According to the third embodiment of the present invention described above, when the subject tracking unit 70 includes the subject designated by the subject designation unit 60 in the composite area, the position where the subject is captured in each of the plurality of images is determined. While sequentially tracking the included subject regions, the combining region designating unit 301 designates the subject regions sequentially tracked by the subject tracking unit 70 as the combining region, and the combining processing unit 305 switches between the two combining methods specified in the combining region. Since a plurality of images are sequentially combined to generate one combined data, it is possible to generate a combined image that reflects the user's original intention.

(Embodiment 4)
Next, a fourth embodiment of the invention will be described. The fourth embodiment has a different configuration from the above-described image pickup apparatus according to the first embodiment, and also has a different combination setting process executed by the image pickup apparatus. Specifically, in the fourth embodiment, each of the combination area, the combination method, and the threshold value is automatically designated. In the following, the configuration of the image pickup apparatus according to the fourth embodiment will be described, and then the combination setting process executed by the image pickup apparatus according to the fourth embodiment will be described. The same components as those of the image pickup apparatus 1 according to the first embodiment described above are designated by the same reference numerals, and description thereof will be omitted.

[Structure of imaging device]
FIG. 31 is a block diagram showing a functional configuration of the image pickup apparatus according to Embodiment 4 of the present invention. The imaging device 1d according to FIG. 31 includes a recording unit 21d in place of the recording unit 21 of the imaging device 1 according to the first embodiment described above. The recording unit 21d has a combined information recording unit 21b in addition to the configuration of the recording unit 21 according to the first embodiment described above.

The combination information recording unit 21b specifies a combination mode set based on the image information extracted by the image information extraction unit 402, a combination area to be combined by the image combination unit 30, a combination method, and a combination value. Record the information.

FIG. 32 is a diagram schematically showing an example of the composite information recorded by the composite information recording unit 21b. As shown in FIG. 32, each combination mode, combination range, image information (movement amount), and combination method are recorded in the combination information T1 in association with each other. Specifically, when the synthesis mode is the starry sky mode, each of the dark area, the small movement amount, and the comparative bright synthesis is described in the synthesis range. In the starry sky mode, in the case of the image P300 in FIG. 33, the composition range is set to the dark part (black-painted area), the subject is a star, and the amount of movement is small, so the amount of movement as a threshold is small (for example, 200 Pixel) and the composition legal is set to comparative light composition. Further, in the firework mode, in the case of the image P300 in FIG. 33, the synthesis range is set to the entire area (entire area), the subject is fireworks, and the movement amount is large, so the movement amount as the threshold value is large (for example, 50). Pixel) and the compositing method is set to comparative bright compositing. Further, in the night view mode, in the case of the image P300 of FIG. 33, the composition range is set to the bright part (white area), and the subject is a building and does not move, so the movement amount as a threshold value is set to 0, The composition method is set to comparative light composition. Note that, in FIG. 32, the starry sky mode, the fireworks mode, and the night view mode are described as an example, but the present invention is not limited to this, and for example, a night landscape ferris wheel mode or a night view train mode may be provided.

[Composition setting process]
Next, the combination setting process executed by the image pickup apparatus 1d will be described. FIG. 34 is a flowchart showing an outline of the combination setting process executed by the image pickup apparatus 1d.

As shown in FIG. 34, the display control unit 403 displays the recommended mode in the current shooting scene on the display unit 24 based on the image information extraction processing of FIG. 4 described above and the composition information recorded by the composition information recording unit 21b. It is displayed (step S901). Specifically, as shown in FIG. 35, the display control unit 403 causes the display unit 24 to display a menu image G1 showing a list of recommended modes. The menu image G1 includes a starry sky mode M11, a fireworks mode M12, and a night view mode M13.

Subsequently, when any mode of the menu image G1 displayed on the display unit 24 is selected via the input unit 25 (step S902: Yes), the imaging device 1d proceeds to step S903. On the other hand, when any mode of the menu image G1 displayed on the display unit 24 is not selected via the input unit 25 (step S902: No), the imaging device 1d returns to step S901.

In step S903, the synthesis area designating unit 301 designates an area corresponding to the mode selected via the input unit 25 as a synthesis area (step S903).

Subsequently, the threshold value specifying unit 303 specifies a threshold value corresponding to the mode selected via the input unit 25 in the synthesis area (step S904).

After that, the synthesis method designating unit 302 designates a synthesis method according to the mode selected via the input unit 25 in the synthesis area (step S905). After step S905, the imaging device 1d returns to the main routine of FIG.

According to the fourth embodiment of the present invention described above, it is possible to generate a composite image that reflects the user's original intention.

(Other embodiments)
In addition to the digital still camera, the image pickup apparatus and the image processing apparatus according to the present invention are picked up by an electronic device such as a digital video camera, a surveillance camera, a tablet type portable device having an image pickup function, an endoscope or a microscope. It can also be applied to a display device or the like that displays an image corresponding to image data in the medical and industrial fields.

Further, a program to be executed by the image pickup apparatus and the image processing apparatus according to the present invention is a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk) in the form of installable or executable file data. ), a USB medium, a flash memory, or other computer-readable recording medium.

The program executed by the image pickup apparatus and the image processing apparatus according to the present invention may be stored in a computer connected to a network such as the Internet and may be provided by being downloaded via the network. Further, the program to be executed by the image pickup apparatus according to the present invention may be provided or distributed via a network such as the Internet.

In the description of the flow charts in this specification, expressions such as “first”, “after”, and “continue” are used to clearly indicate the context of processing between steps. However, in order to implement the present invention, The order of processing required is not uniquely defined by their representation. That is, the order of the processes in the flow charts described in this specification can be changed within a range without contradiction.

Although some of the embodiments of the present application have been described in detail with reference to the drawings, these are merely examples, and various modifications and improvements are made based on the knowledge of those skilled in the art, including the modes described in the section of the disclosure of the invention. It is possible to carry out the present invention in other forms.

As described above, the present invention can include various embodiments not described here, and various design changes and the like can be made within the scope of the technical idea specified by the claims. Is.

1, 1a, 1b, 1d... Imaging device; 10... Imaging unit; 11... Lens unit; 12... Aperture; 13... Shutter; 14... Imaging device; 15... A/D conversion unit; 16... Drive control unit; 20, 20a, 20b... Image processing unit; 21, 21d... Recording unit; 21a... Program recording unit; 21b... Composite information recording Part: 22... Memory I/F; 23... Recording medium; 24... Display unit; 25... Input unit; 26... Bus; 27... Control unit; 30, 30a... Image synthesis unit; 40... Image information processing unit; 50... Development processing unit; 60... Subject designation unit; 70... Subject tracking unit; 301... Synthesis region designation unit; 302...・Composition method designation unit; 303... Threshold designation unit; 304... Threshold determination unit; 305... Synthesis processing unit; 305a... Comparative bright synthesis processing unit; 305b... Comparative dark synthesis processing unit 305c... Addition synthesis processing unit; 305d... Average synthesis processing unit; 305e... Replacement synthesis processing unit; 306... Priority designation unit; 401... Information area designation unit; 402... Image Information extraction unit; 403... Display control unit

Claims (11)

A synthesizing method designating unit that designates at least two synthesizing methods for synthesizing each of the plurality of images in a predetermined area of each of the plurality of images corresponding to the plurality of image data;
An image information extraction unit that sequentially extracts image information included in the predetermined area of each of the plurality of images,
Based on the image information sequentially extracted by the image information extraction unit, the plurality of images are sequentially combined while switching the at least two combining methods specified in the predetermined area, and one combined data is generated. A processing unit,
A composite area designating unit that designates one or more of the predetermined area for each of the plurality of images;
When two or more of the predetermined areas are designated by the synthesis area designating unit and an overlapping area in which at least some of the areas overlap with each other is generated, priority for designating the priority of the combining method for the overlapping area. Degree specification section,
Equipped with
The image processing apparatus , wherein the synthesizing processing unit generates the synthetic data by the synthesizing method having the high priority designated by the priority designating unit . A threshold value specifying section for specifying a threshold value for switching the at least two combining methods specified by the combining method specifying section in the predetermined area;
A threshold determination unit that determines whether or not a value determined according to the image information extracted by the image information extraction unit is equal to or greater than the threshold,
Further equipped with,
The image processing apparatus according to claim 1 , wherein the composition processing unit generates the composition data based on a determination result of the threshold value determination unit. The threshold specifying unit, according to claim 2, wherein the specifying the value determined in accordance with the image information by the image information extracting unit has extracted for oldest the image in the plurality of images to the threshold Image processing device. A subject designation unit that designates a specific subject that appears in each of the plurality of images,
When the subject designated by the subject designation unit is included in the predetermined region, a subject tracking unit that sequentially tracks a subject region including a position where the subject is captured in each of the plurality of images,
Further equipped with,
The synthetic region specifying unit, an image processing apparatus according to any one of claims 1 to 3, characterized in that specifying the subject region in which the subject tracking unit has successively tracking the predetermined region. The method of synthesis, lighten compositing process, relatively dark combination treatment, additive synthesis process, the average composition processing, any one of claims 1-4, characterized in that either the displacement synthesis processing and non-synthetic process The image processing device according to 1. The image information, luminance, color components, according to any one of claims 1-5, characterized in that either the moving amount of the subject between the change amount and the image data of the pixel values between the image data Image processing device. A display unit capable of displaying each of the plurality of images,
A display control unit for displaying the image information extracted by the image information extraction unit on the display unit;
The image processing apparatus according to any one of claims 1-6, characterized in further comprising a. A display unit capable of displaying each of the plurality of images,
A display control unit for displaying the combination method and the threshold value on the display unit;
The image processing apparatus according to claim 2 or 3, further comprising a. An image processing apparatus according to any one of claims 1-7,
An image pickup unit capable of continuously picking up an image of a subject and sequentially generating image data,
A recording unit that sequentially records the combined data generated by the combining processing unit;
Equipped with
Each time the imaging unit generates the image data, the composition processing unit combines the composition data with the composition data immediately before being recorded by the recording unit to generate new composition data and records it by the recording unit. An image pickup apparatus characterized by updating the composite data. An image processing method executed by an image processing apparatus, comprising:
A synthesizing method designating step of designating at least two synthesizing methods for synthesizing each of the plurality of images in a predetermined area of each of the plurality of images corresponding to the plurality of image data;
An image information extracting step of sequentially extracting image information included in the predetermined area of each of the plurality of images;
A synthesis for sequentially synthesizing the plurality of images while switching the at least two synthesizing methods designated in the predetermined area based on the image information sequentially extracted in the image information extracting step to generate one synthetic data. Processing steps,
A composite area designating step of designating one or more of the predetermined area for each of the plurality of images;
When two or more of the predetermined areas are designated by the synthesizing area designating step, and when an overlapping area in which at least a part of the areas overlap with each other occurs, priority of designating the priority of the synthesizing method for the overlapping area is given. Degree specification step,
Only including,
The image processing method , wherein in the combining processing step, the combined data is generated by the combining method having the high priority specified by the priority specifying step . Image processing device,
A synthesizing method designating step of designating at least two synthesizing methods for synthesizing each of the plurality of images in a predetermined area of each of the plurality of images corresponding to the plurality of image data;
An image information extracting step of sequentially extracting image information included in the predetermined area of each of the plurality of images;
A synthesis for sequentially synthesizing the plurality of images while switching the at least two synthesizing methods designated in the predetermined area based on the image information sequentially extracted in the image information extracting step to generate one synthetic data. Processing steps,
A composite area designating step of designating one or more of the predetermined area for each of the plurality of images;
When two or more of the predetermined areas are designated by the synthesizing area designating step, and when an overlapping area in which at least a part of the areas overlap with each other occurs, priority of designating the priority of the synthesizing method for the overlapping area is given. Degree specification step,
Was executed,
The synthesizing step generates the synthesized data by the synthesizing method having the high priority designated by the priority designating step .