JP5200820B2 - Imaging apparatus, imaging method, and image processing program - Google Patents

Description

  The present invention relates to an imaging apparatus and an imaging method that acquire a captured image by performing strobe light emission, and an image processing apparatus that performs image processing on the acquired captured image.

  2. Description of the Related Art Conventionally, as an imaging mode of an imaging apparatus, a slow sync shooting mode in which strobe light emission is performed and a predetermined long time exposure is performed to acquire a captured image is known. In this slow sync shooting, strobe light emission and long exposure are performed during one shooting operation. Specifically, first, exposure of the imaging apparatus is started in a state in which the subject is brightly illuminated by performing strobe light emission, and exposure is performed for a long time as it is to expose a low-illuminance background. According to this slow sync photography, it is possible to acquire a captured image that brightly captures both a subject with high illuminance by a strobe and a background with low illuminance.

Conventionally, as described in Patent Document 1 below, as a technique for capturing an image of a subject and a background brightly, image processing using an image irradiated with strobe light and an image not irradiated with strobe light is performed. By doing so, a technique for acquiring a natural and beautiful night view photographed image at the time of night view photography is known.
JP 2003-87645 A

  In the above-described slow sync photography and the technique described in Patent Document 1, it is necessary to perform long-time exposure in order to capture a subject (background) with low surface illuminance without strobe light during night photography. However, if the subject moves during long exposure or the imaging device moves due to camera shake, the incident position of incident light fluctuates and blurring occurs in the subject in the captured image. .

  Specifically, the captured image 100 obtained by irradiating a person who is a subject with strobe light as shown in FIG. 10A has a short exposure time, so both the person image 101 and the background object image 102 are both. Although there is no image blur, the brightness of the background object image 102 is lowered. Therefore, by increasing the exposure time, it is possible to obtain a captured image 110 including a background object 102 'having high brightness as shown in FIG. However, if the person who is the subject moves during the exposure time, a blur component 111 is generated as a result of the person image 101 being blurred. Such blur component 111 is erased in the process of superimposing the image irradiated with the strobe light and the image not irradiated with the strobe light or the process of replacing the person part as described in Patent Document 1 described above. I can't.

  Therefore, the present invention has been proposed in view of the above-described circumstances, and an imaging apparatus, an imaging method, and an image that can acquire a captured image that clearly captures a main subject and a background even during night imaging. An object is to provide a processing program.

According to a first aspect of the present invention, there is provided an imaging unit that detects incident light and obtains a captured image, a light emitting unit that performs strobe light emission on an area including an imaging range of the imaging unit, and controls the imaging unit and the light emitting unit. and control means for, and image processing means for performing image processing on the obtained captured image by the imaging means, before Symbol control means, predetermined by the imaging means to perform a flash light emission by the light emitting means together to obtain the short-time exposure image by performing a short-time exposure, a long time to acquire the exposure image by performing a predetermined long-time exposure by said image pickup means, said image processing means, the main included in the short-time exposure image Contour detection means for detecting the contour of a subject, ratio calculation means for calculating a scaling ratio between the short-time exposure image and the long-time exposure image, and a variable calculated by the ratio calculation means. Enlarging / reducing means for performing at least one of enlargement of the short-time exposure image or reduction of the long-time exposure image according to the ratio, and a main subject included in the short-time exposure image based on the contour detected by the contour detection means Combining the extracted main subject image with the long-time exposure image to create a composite image, and the ratio calculation unit is configured to capture the main subject in the short-time exposure image. based on, it characterized that you determine uniquely set scaling ratio to the imaging condition.
According to a second aspect of the present invention, there is provided an imaging unit that detects incident light and acquires a captured image, a light emitting unit that performs strobe light emission on an area including an imaging range of the imaging unit, and controls the imaging unit and the light emitting unit. Control means for performing image processing on the captured image obtained by the imaging means, and the control means causes the light emitting means to perform strobe light emission and causes the imaging means to perform a predetermined short time. A time exposure is performed to acquire a short exposure image, a predetermined long exposure is performed by the imaging unit to acquire a long exposure image, and the image processing unit includes a main subject included in the short exposure image. A contour detecting means for detecting the contour of the image, a ratio calculating means for calculating a magnification ratio between the short-time exposure image and the long-time exposure image, and a variable calculated by the ratio calculating means. Enlarging / reducing means for performing at least one of enlargement of the short-time exposure image or reduction of the long-time exposure image according to the ratio, and a main subject included in the short-time exposure image based on the contour detected by the contour detection means Combining the extracted main subject image with the long-time exposure image to generate a composite image, and the ratio calculating unit includes a blur amount in the long-time exposure image of the main subject. The zoom ratio is calculated so as to cancel.

The third invention is the first invention, the ratio calculation means is characterized in that the ratio of main subject included in the short-time exposure image is increased higher magnification ratio.

The fourth invention is the first invention or second invention, the ratio calculation means, the zooming ratio, and limits to be within a predetermined range.

Fifth invention, in the first invention or second invention, the image processing means, when the magnification ratio was calculated by the ratio calculation means exceeds a within a predetermined range, the scaling means The image enlargement or reduction process is prohibited.

According to a sixth aspect of the invention, in response to an imaging instruction, a light emission unit performs strobe light emission to perform a predetermined short-time exposure to obtain a short-time exposure image, and to perform a predetermined long-time exposure without performing strobe light emission. Performing an acquisition step for obtaining a long-exposure image, a detection step for detecting a contour of a main subject included in the short-exposure image, and a magnification ratio between the short-exposure image and the long-exposure image. a calculating step of calculating, the computed and line intends scaling step at least one of reduction of enlargement or the long exposure image of the short-time exposure image according to zoom ratio, on the basis of the said detected contour short the main object image extracted from the time exposure image, and synthesis with respect to the long-time exposure image, and a synthesis step of generating a composite image, wherein the computing step, the short Based on the imaging state of the main subject included in the light image, and determines the unique set scaling ratio to the imaging condition.
According to a seventh aspect of the invention, in response to an imaging instruction, the light emission unit performs strobe light emission to perform a predetermined short-time exposure to obtain a short-time exposure image, and to perform a predetermined long-time exposure without performing strobe light emission. Performing an acquisition step for obtaining a long-exposure image, a detection step for detecting a contour of a main subject included in the short-exposure image, and a magnification ratio between the short-exposure image and the long-exposure image. An operation step for calculating, an enlargement / reduction step for performing at least one of enlargement of the short-time exposure image or reduction of the long-time exposure image according to the calculated zoom ratio, and the short time based on the detected contour Combining a main subject image extracted from an exposure image with the long-time exposure image to create a composite image, and the calculation step includes the main object image. Characterized by calculating a scaling ratio so as to cancel the shake amount of the at long exposure image of the body.

An eighth invention is an image processing program executed by a computer, and is a short-exposure image captured by a predetermined short-time exposure at the time of flash emission, and a predetermined long-time exposure simultaneously with the short-exposure image. An image acquisition process for acquiring a long exposure image, a contour detection process for detecting a contour of a main subject included in the short exposure image, and a scaling ratio between the short exposure image and the long exposure image A ratio calculation process that calculates the above, an enlargement / reduction process that performs at least one of enlargement of the short-exposure image or reduction of the long-exposure image according to the scaling ratio calculated by the ratio calculation process, and the contour detection process the main object image obtained by extracting a main subject included in the short-time exposure image on the basis of the detected contour, by combining with respect to the long-time exposure image, To execute the synthesizing process of creating a formed image to the computer, scaling the ratio calculation process, based on the imaging condition of the main subject included in the short-time exposure image, which is uniquely set for the imaging condition characterized that you determining the ratio.
A ninth aspect of the invention is an image processing program executed by a computer, the short-exposure image captured by a predetermined short-time exposure at the time of flash emission, and the predetermined long-time exposure simultaneously with the short-exposure image. An image acquisition process for acquiring a long exposure image, a contour detection process for detecting a contour of a main subject included in the short exposure image, and a scaling ratio between the short exposure image and the long exposure image A ratio calculation process that calculates the above, an enlargement / reduction process that performs at least one of enlargement of the short-exposure image or reduction of the long-exposure image according to the scaling ratio calculated by the ratio calculation process, and the contour detection process A main subject image obtained by extracting a main subject included in the short-time exposure image based on the detected contour is synthesized with the long-time exposure image, To execute the synthesizing process of creating a formed image to the computer, the ratio calculation means and calculates the magnification ratio so as to cancel the shake amount in the in the long exposure image of the main subject.

  According to the present invention, a short-exposure image captured by a predetermined short-time exposure during strobe light emission and a long-exposure image captured by a predetermined long-exposure at the same time as the short-exposure image are acquired. The main subject image included in the short-time exposure image is subjected to at least one of enlargement of the short-time exposure image or reduction of the long-time exposure image according to the magnification ratio of the time-exposure image and the long-time exposure image. Therefore, even when shooting at night, a captured image obtained by clearly capturing the main subject and the background can be acquired.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[Configuration of imaging device]
The present invention is applied to an imaging apparatus configured as shown in FIGS. 1 and 2, for example. FIG. 1 shows the minimum functions necessary for the image pickup apparatus according to the present invention, and FIG. 2 is a block diagram showing an example of hardware constituting the image pickup apparatus according to the present invention.

  The imaging apparatus shown in FIG. 1 includes a CCD (Charge Coupled Device) 1, an imaging processing unit 2, a DRAM 3, an image processing unit 4, and an external memory 5. Such an imaging apparatus generates an imaging signal based on incident light received by the CCD 1 and performs predetermined video processing by the imaging processing unit 2. This imaging processing unit 2 performs noise removal by a CDS circuit (correlated double sampling circuit), sample and hold processing, amplification by an AGC (automatic gain control) amplifier, ADC (A / D converter) processing, etc. Generate captured image data of the method. The captured image data is once supplied to the DRAM 3.

  The DRAM 3 has a short exposure (strobe shooting) image captured with a short exposure time during strobe shooting, a subject extraction image obtained by extracting a subject from the short exposure image, and a predetermined length longer than the short exposure described above during strobe shooting. A long exposure image captured with a long exposure time and a composite image as a result of image processing are recorded.

  The image processing unit 4 performs predetermined image processing using the image recorded in the DRAM 3. The image processing by the image processing unit 4 includes contour detection processing (1) for detecting the contour of the main subject included in the short-time exposure image, and ratio calculation for calculating a magnification ratio between the short-time exposure image and the long-time exposure image. Processing, enlargement / reduction processing for performing at least one of enlargement of the short-exposure image and reduction of the long-exposure image in accordance with the magnification ratio calculated by the ratio calculation processing, and shortening based on the contour detected by the contour detection processing A process (2) for extracting a main subject included in the time-exposure image and a synthesis process (3) for creating a composite image by synthesizing the extracted main subject image with the long-time exposure image are included. The composite image obtained by such image processing is recorded in the DRAM 3 as a composite result (4). Then, the composite image as the composite result is subjected to display processing for confirmation to the user, for example, and is subjected to predetermined compression and encoding processing by the image processing unit 4 (5) and recorded in the external memory 5. (6).

  The imaging apparatus shown in FIG. 2 is a computer that performs various processes performed by the imaging optical system 11, the imaging unit 12 including the CCD 1, the imaging processing unit 13, the memory card 14, and the image processing unit 4. A control unit 15, a key input unit 16, a program memory 17 for storing an image processing program for various processes performed by the control unit 15, a data memory 18 corresponding to the above-described DRAM 3, an image compression unit 19, A main display processing unit 20, a main display unit 21, a light emission control unit 22, a built-in light emission drive unit 23, and a built-in light emission unit 24.

  The imaging optical system 11 mainly includes a motor, a lens unit, and the like. This lens unit is composed of a plurality of lens groups for entering external light. The lens unit is an optical system for forming an image of outside light in an imaging region within a predetermined range on the CCD 1. The lens unit includes a shutter unit and an aperture unit (not shown). These shutter units and diaphragm units are moved in focus and diaphragm positions by driving a motor. In particular, the exposure time of the shutter unit is adjusted according to control by the control unit 15. The lens unit also includes a zoom lens that can freely zoom out and zoom in under the control of the control unit 15 to change the shooting range.

  The imaging unit 12 includes a controller that operates according to control by the CCD 1 and the control unit 15, an A / D conversion processing circuit, and the like. The CCD 1 forms an RGB primary color filter, for example. The CCD 1 is scanned and driven by a timing generator and a vertical driver (not shown). The CCD 1 outputs one screen of photoelectric conversion output corresponding to the light image formed at regular intervals.

  The operation of the lens unit and the CCD 1 in the imaging optical system 11 is controlled according to the controller in the imaging unit 12 and the control of the control unit 15 described later. The A / D conversion processing circuit includes a sample hold circuit and an A / D converter. The A / D conversion processing circuit performs desired processing on the photoelectric conversion output output from the CCD 1. Specifically, the gain is appropriately adjusted for each primary color component of RGB in the state of an analog signal with respect to the photoelectric conversion output from the CCD 1. Thereafter, the sample and hold circuit performs sample and hold, converts the image into digital captured image data by an A / D converter, and outputs the digital captured image data to the imaging processing unit 13. Such an imaging optical system 11 and the imaging unit 12 function as an imaging unit that acquires a captured image.

  The imaging processing unit 13 performs predetermined image processing on the captured image data supplied from the imaging unit 12. For example, the imaging processing unit 13 converts captured image data supplied as RGB data into luminance data (Y) and color difference data (Cr, Cb). The luminance data (Y), color difference data (Cr), and color difference data (Cb) are output to the data memory 18 by the imaging processing unit 13 and stored in the data memory 18. In addition to the above processing, the imaging processing unit 13 may perform pixel interpolation processing, γ correction processing, and matrix calculation as necessary.

  The memory card 14 includes a memory card recording / reproducing mechanism provided in the imaging apparatus. Under the control of the control unit 15, the memory card 14 records captured image data captured by the imaging device and captured image data subjected to image processing described later on the memory card. Further, the memory card 14 reproduces and outputs the captured image data recorded on the memory card under the control of the control unit 15.

  The control unit 15 controls the imaging apparatus in an integrated manner. The control unit 15 performs various types of control necessary for the imaging apparatus by operating according to a program stored in the program memory 17.

  The key input unit 16 is operated by the user of the imaging apparatus. The key input unit 16 is a key group including a shutter button 16a, a menu button 16b, a movement button 16c such as a cross key, and an enter button 16d. The key input unit 16 is connected to the control unit 15, and an operation signal indicating the operation content of each button is read by the control unit 15.

  Under the control of the control unit 15, the image compression unit 19 applies ADCT (Adaptive Discrete Cosine Transform) in accordance with JPEG (Joint Photograph Coding Experts Group) standard to the captured image data stored in the data memory 18. (Conversion) processing and data compression.

  The main display processing unit 20 periodically reads luminance data (Y) and color difference data (Cr, Cb) included in the captured image data, generates a video signal based on these data, and outputs the video signal to the main display unit 21. To do.

  Although not shown, the main display unit 21 is provided on the back side of the housing of the imaging device. The main display unit 21 functions as a monitor display unit (electronic finder) in the shooting mode. In this shooting mode, the main display unit 21 performs display based on the video signal from the main display processing unit 20. As a result, the imaging apparatus displays an image captured from the main display processing unit 20 at that time in real time.

  In this way, the imaging device is in a state where it can capture an image in a so-called through image display state in which the captured image at that time is displayed in real time on the main display unit 21 as a monitor image. When a shutter button 16a, which is one of the key input units 16, is operated at a timing at which shooting is desired, a trigger signal is generated for the control unit 15. In response to the trigger signal, the control unit 15 converts (separates) the RGB data for one screen captured from the imaging optical system 11 at that time by luminance data (Y) and color difference data (Cb, Cr). And stored in the data memory 18.

  The light emission control unit 22 causes the built-in light emission driving unit 23 to perform the strobe light emission operation according to the control of the control unit 15.

  The built-in light-emitting drive unit 23 charges a large-capacity strobe capacitor (not shown) during still image shooting, and causes the built-in light-emitting unit 24 to emit strobe light in accordance with control from the control unit 15 via the light emission control unit 22. Further, at the time of shooting a moving image, when the first shutter button 16a is operated, the above-described still image captured image data is compressed by the image compression unit 19 and recording into the data memory 18 is started. Thereafter, this processing is continuously executed at a predetermined frame rate, for example, 30 (frames / second), and when the shutter key is operated for the second time or when a predetermined time limit, for example, 30 seconds elapses. These series of still image data files are collectively set as a motion JPEG data file (AVI file).

  In the above-described imaging apparatus, the control unit 15 performs processing according to the program stored in the program memory 17, so that captured image data including a clear main subject and background can be obtained even during night imaging. . In particular, the control unit 15 performs image processing on the captured image data acquired by the imaging optical system 11, the imaging unit 12, and the imaging processing unit 13 (imaging unit) (image processing unit). At this time, the control unit 15 causes the light emission control unit 22, the built-in light emission driving unit 23, and the built-in light emission unit 24 (light emitting unit) to perform strobe light emission, and the imaging optical system 11, the imaging unit 12, and the imaging processing unit 13 (imaging unit). ) To obtain a short-time exposure image by performing a predetermined short-time exposure, and to perform a predetermined long-time exposure by the imaging optical system 11, the imaging unit 12, and the imaging processing unit 13 (imaging means) Get it. Then, the control unit 15 executes the image processing program to obtain the contour detection process for detecting the contour of the main subject included in the short-time exposure image, and the magnification ratio between the short-time exposure image and the long-time exposure image. Detected by ratio calculation processing to be calculated, enlargement / reduction processing for performing at least one of enlargement of the short-exposure image and reduction of the long-exposure image according to the magnification ratio calculated by the ratio calculation means, and contour detection processing The main subject image obtained by extracting the main subject included in the short-exposure image based on the contour that has been extracted is combined with the long-exposure image to perform a compositing process for creating a composite image.

  Specifically, a short-exposure image 100 obtained by imaging processing with strobe light emission and a predetermined short exposure time (for example, 1/60 seconds) is a person who is a main subject as shown in FIG. The brightness of the image 101 is high, and the brightness of the background object image 102 of the person image 101 is low. On the other hand, the imaging apparatus performs a predetermined long exposure to acquire a long exposure image. Although not illustrated, the long exposure image includes a background object image having a brightness higher than that of the background object image 102 appearing in the short exposure image 100.

  Then, the control unit 15 detects the outline of the person image 101 included in the short-time exposure image 100 and extracts the person image 101 from the short-time exposure image 100 in FIG. 110 is created. Next, the control unit 15 calculates a magnification ratio between the short-time exposure image and the long-time exposure image, and performs an image enlargement process on the short-time exposure image 110 according to the calculated magnification ratio. As shown in FIG. 3C, a short-time exposure image 110 ′ including the enlarged human image 101 ′ is created. Next, as illustrated in FIG. 3D, the control unit 15 synthesizes the person image 101 ′ in an enlarged manner at the contour position detected by the contour detection process in the long-time exposure image, and combines the composite image 120. And the compositing process to create. Here, since the long-time exposure image includes the background object 102 ′ having high brightness, the composite image 120 is a video image including the person image 101 ′ and background object image 102 ′ having both high brightness. In this way, the control unit 15 can acquire a captured image that clearly captures the main subject and the background even during night imaging.

  Further, the composite image 120 enlarges the person image 101 even when the person moves or the hand holding the imaging device shakes when the long exposure image is captured, and the person included in the long exposure image is blurred. Therefore, the blur does not appear. That is, the control unit 15 calculates the magnification ratio so as to cancel out the amount of blurring in the long exposure image of the main subject (person) and enlarges the person image, so that it is included in the long exposure image. It is possible to cancel out image components caused by blurring.

  The imaging apparatus configured as described above acquires captured image data by performing the processing illustrated in FIGS. 4 to 6 when the imaging apparatus is gripped by the user and an operation for imaging a subject is performed.

  As illustrated in FIG. 4, when the imaging apparatus is powered on, the imaging apparatus performs initialization processing in step S <b> 1. The control unit 15 sets the imaging condition of the imaging device, the storage condition of the captured image data, and the like to predetermined initial values by the initialization process.

  In the next step S2, the imaging device is set to the normal shooting mode in response to the completion of the initialization process. This normal shooting mode is a shooting mode in which image pickup processing is performed in accordance with the image pickup conditions in the initial state, and the picked-up image data is stored in the memory card 14.

  In the next step S3, the imaging device detects the operation signal according to the operation on the menu button 16b, the movement button 16c, and the decision button 16d in the key input unit 16, and the shooting mode of the imaging device is changed. Switch to modified shooting mode. This night subject blur correction shooting mode is a process of creating a composite image by acquiring a short exposure image and a long exposure image during strobe shooting at night. Note that the processing in step S3 will be described later with reference to the subroutine shown in FIG. Then, when the imaging device is switched to the normal shooting mode by other key processing, the processing returns to step S2.

  The night subject blur correction shooting mode shown in FIG. 5 is executed every predetermined period. First, in step S11, the control unit 15 determines whether or not the night subject blur correction shooting mode is on. If the night subject blur correction shooting mode is on, the process proceeds to step S12. If the night subject blur correction shooting mode is not on, the subroutine is terminated.

  In step S12, the control unit 15 determines whether or not the shutter button 16a has been pressed. If the control unit 15 receives a trigger signal generated by operating the shutter button 16a, the control unit 15 proceeds to step S13 and can receive the trigger signal for a predetermined period of time in which the night subject blur correction shooting mode is executed. If not, the process ends.

  In step S <b> 13, the control unit 15 controls the drive motor to move the focus lens in the imaging optical system 11 and executes AF processing for focusing on the main subject.

  In the next step S14, the control unit 15 performs darkness measurement (EV measurement) in the shooting environment of the imaging apparatus. At this time, the control unit 15 may measure the darkness by detecting an output signal of a brightness sensor (not shown), and analyze the luminance signal of the pixel included in the captured image data displayed as a through image. You may measure darkness.

  In the next step S <b> 15, the imaging device performs flash emission that emits strobe light from the built-in light emitting unit 24. At this time, the control unit 15 supplies a control signal to the light emission control unit 22 to drive the built-in light emission drive unit 23 to emit strobe light from the built-in light emission unit 24. Further, the imaging device opens the shutter mechanism in the imaging optical system 11 simultaneously with the flash emission, thereby bringing the CCD 1 in the imaging unit 12 into an exposure state. The control unit 15 records the captured image data (short-time exposure image) generated by the imaging unit 12 and the imaging processing unit 13 by exposure for a predetermined short exposure time in the data memory 18, thereby An exposure image can be acquired. This predetermined short exposure time is, for example, 1/60 [sec].

  In the next step S16, the control unit 15 acquires captured image data (long-time exposure image) as a result of exposure over a long exposure time corresponding to the darkness measured in step S14. Here, the longer the exposure time is, the darker the imaging environment is. This is because the background object existing in the background of the main subject is clearly imaged. Through such steps S15 and S16, the imaging apparatus stores two images of the short exposure image and the long exposure image in the data memory 18 by one exposure operation.

  In the next step S17, the control unit 15 recognizes the outline of the main subject included in the short-exposure image acquired in step S15. At this time, the control unit 15 performs edge extraction processing on the short-time exposure image. Thereby, the control unit 15 separates the main subject image with high brightness and the background image with low brightness, and acquires coordinate information representing the outline of the main subject image.

  In the next step S18, the control unit 15 performs a combining process for combining the main subject image included in the short-time exposure image with the long-time exposure image to create a combined image. In step S19, the control unit 15 proceeds to step S18. Then, the long-exposure image synthesized with the main subject is stored in the memory card 14 as a captured image, and the process ends.

  In the synthesizing process in step S18, either the synthesizing process shown in FIG. 6 or the synthesizing process shown in FIG. 9 is performed. In the composition process shown in FIG. 6, taking into account the blurring of the main subject included in the long-time exposure image, the magnification ratio of the short-time exposure image or the reduction of the long-time exposure image is determined for composition. On the other hand, the composition processing shown in FIG. 9 performs composition by uniquely determining the magnification ratio of enlargement of the short-time exposure image or reduction of the long-time exposure image regardless of the blur of the main subject included in the long-exposure image. .

  In the combining process shown in FIG. 6, first, in step S21, the contour of the main subject on the long-time exposure image is recognized using the contour of the main subject recognized on the short-time exposure image in step S17. Here, the image size of the short exposure image and the image size of the long exposure image are the same. Therefore, the vicinity of the coordinates on the long-time exposure image corresponding to the coordinate information representing the outline of the main subject acquired from the short-time exposure image is the contour of the main subject included in the long-time exposure image.

  In the next step S22, the control unit 15 determines the main subject image in the short exposure image and the main subject image in the long exposure image from the image information in the outline of the main subject in the long exposure image recognized in step S21. A magnification ratio that is an enlargement or reduction magnification in which and becomes the same size is obtained. Here, the color information in the outline of the main subject in the short-exposure image and the color information in the outline of the main subject in the long-exposure image are almost the same. From this, the control unit 15 obtains a scaling ratio between the contour of the main subject included in the short-time exposure image and the contour including the blur width of the main subject included in the long-time exposure image.

  For example, assume that a short-time exposure image 100 shown in FIG. 8A and a long-time exposure image 110 shown in FIG. 8B are acquired. Here, the main subject image 101 of the short-exposure image 100 has no blur component, and the main subject image of the long-exposure image 110 includes a blur component indicated by a range of 101′-1 to 101′-2. ing. Further, the control unit 15 acquires image information 101 a indicating the outline of the main subject image 101 included in the short-time exposure image 100. Image information 101 a ′ in the long exposure image 110 exists at coordinates in the long exposure image 110 corresponding to the coordinates of the image information 101 a. Then, the control unit 15 determines whether or not the color information scanned from the image 101a ′ in the long-time exposure image 110 toward the outside of the main subject image is that of the background image. At this time, the control unit 15 scans the color information toward the outside of the main subject image, and sets the coordinate position at which the color information is determined to have low brightness for the first time as the contour of the main subject image in the long-time exposure image 110. It is determined that Then, the control unit 15 changes the ratio between the size determined by the outline of the main subject included in the short-exposure image 100 and the size determined by the outline of the main subject included in the long-exposure image 110 for composition processing. Double ratio. Therefore, the larger the blur component of the main subject image included in the long-exposure image 110, the larger the zoom ratio. Here, the magnification ratio is an enlargement magnification when the main subject image in the short-time exposure image is enlarged, and a reduction magnification when the long-time exposure image is reduced.

  The control unit 15 may determine the scaling ratio according to the size difference between the contours of the faces of the main subject image, instead of setting the scaling ratio based on the size of the entire main subject image. For example, when only the limb portion of the person who is the main subject moves greatly within the long exposure time, the face portion of the main subject has no blur but the limb portion has a large blur. Using such a short-exposure image and a long-exposure image that captures the main subject, the zoom ratio is calculated according to the large blurring of the limbs. The synthesized image becomes unnatural. For this reason, the control unit 15 can set a zoom ratio in an appropriate range by setting the zoom ratio from the contours of the face portions of the short-time exposure image and the long-time exposure image.

  In the next step S23 and step S24, the control unit 15 determines whether or not the zoom ratio determined in step S22 is within a predetermined threshold range. As this threshold value, as shown in FIG. 7, a first threshold value Th1 and a second threshold value Th2 are set. The threshold value to be compared with the obtained zoom ratio may be the first threshold value Th1 and the second threshold value Th2 when the main subject image included in the short-time exposure image is enlarged. It may be the first threshold Th1 and the second threshold Th2 when the exposure image is reduced. As described above, the magnification ratio for enlarging the main subject image in the short-time exposure image or reducing the long-time exposure image is limited to be within a predetermined range, and the composite image has an unnatural pattern. Avoid that.

  When the first threshold value Th1 and the second threshold value Th2 for enlarging the short-time exposure image are used, in step S23, the enlargement ratio acquired in step S22 is compared with the first threshold value Th1. In step S24, the enlargement ratio acquired in step S22 is compared with the second threshold value Th2. If the enlargement ratio is greater than the first threshold value Th1, the combining process is terminated. Even when the enlargement ratio is not larger than the first threshold value Th1 but smaller than the second threshold value T2, the synthesis process is also ended. When the enlargement ratio is not larger than the first threshold value Th1 and not smaller than the second threshold value Th2, that is, when the enlargement ratio is an intermediate value between the first threshold value Th1 and the second threshold value Th2. Advances the process to step S25.

  When the first threshold value Th1 and the second threshold value Th2 for reducing the long-exposure image are used, as shown in parentheses in FIG. 6, in step S23, the reduction ratio acquired in step S22. Are compared with the first threshold value Th1, and in step S24, the reduction ratio obtained in step S22 is compared with the second threshold value Th2. When the reduction ratio is smaller than the first threshold value Th1, the synthesis process is terminated. Even when the reduction ratio is not smaller than the first threshold value Th1 but larger than the second threshold value T2, the combining process is also terminated. When the reduction ratio is not smaller than the first threshold value Th1 and not larger than the second threshold value Th2, that is, when the reduction ratio is an intermediate value between the first threshold value Th1 and the second threshold value Th2. Advances the process to step S25.

  If the determination result in step S23 is affirmative, the blur component of the main subject image in the long-time exposure image is too large, and the main subject image in the short-time exposure image and the main subject image in the long-time exposure image There is a big difference in size. Accordingly, when the calculated zoom ratio exceeds the predetermined threshold value range, the control unit 15 prohibits image enlargement or reduction processing and ends the composition processing. Thereby, it can be prohibited to create a composite image in which the main subject is too large in the composite image.

  Similarly, when the determination result in step S24 is affirmative, there is almost no blur component of the main subject image in the long-time exposure image. Therefore, the control unit 15 does not need to perform processing for combining the main subject image in the short-exposure image with the long-exposure image, and thus can end the combining processing.

  In step S25, the control unit 15 enlarges the main subject image in the short-time exposure image according to the magnification ratio acquired in step S22. In the next step S26, the control unit 15 performs a composition process so that the main subject image in the short-time exposure image enlarged in step S25 is overwritten on the main subject image portion in the long-time exposure image. At this time, the control unit 15 may enlarge the main subject image after extracting the main subject image in the short-time exposure image, or may extract the main subject image after enlarging the short-time exposure image itself.

  When the main subject is a person and the person is enlarged, it is desirable to enlarge the person equally from left to right and up and down from the center of the person. Further, when synthesizing in step S26, the background image portion of the short-time exposure image is made transparent or long-time instead of the process of extracting the main subject from the short-time exposure image or extracting the background image from the long-time exposure image. Processing for making the main subject portion of the exposure image transparent may be performed.

  When reducing the long-exposure image, in step S25, the control unit 15 reduces the long-exposure image according to the zoom ratio acquired in step S22. In the next step S26, the control unit 15 performs a composition process so that the background image portion in the short-time exposure image is overwritten on the long-time exposure image reduced in step S25. If the background of the long-exposure image can be reduced at a reduction ratio corresponding to the blur amount of the main subject of the long-exposure image, the control unit 15 reduces the long-exposure image reduced in step S25 for a short time. Processing for overwriting the background image portion in the exposure image may be performed.

  In this step S25 and step S26, the processing for enlarging or reducing only one of the short-exposure image and the long-exposure image has been described. The background image of the exposure image may be reduced. For example, when the ratio of the main subject image in the short exposure image to the main subject image in the long exposure image is 1: 1.2, the main subject image in the short exposure image is enlarged to 110%. At the same time, the background image of the long exposure image is reduced to 90%.

  As described above, according to the imaging apparatus that performs the combining process illustrated in FIG. 6, it is possible to acquire a captured image that clearly captures the main subject and the background even during night imaging.

  Next, referring to FIG. 9, a composition process for performing composition by uniquely determining the zoom ratio of the short-time exposure image and / or the long-time exposure image regardless of the blur of the main subject included in the long-time exposure image. Will be described.

  In this synthesis process, first, in step S31, the control unit 15 determines the size of the main subject using the outline of the main subject recognized in the short-time exposure image in step S17.

  In the next steps S32 to S39, the control unit 15 determines a zoom ratio that is uniquely set for the imaging state based on the imaging state of the main subject included in the short-time exposure image. Specifically, the zoom ratio is increased as the ratio of the main subject image included in the short-time exposure image is higher. That is, the higher the ratio of the main subject image included in the short-time exposure image, the higher the enlargement magnification or the lower the reduction ratio. Even in this case, the zoom ratio is set to be within a predetermined range. For example, the control unit 15 determines whether or not the person who is the main subject included in the short-exposure image is an image of the whole body captured at a small size in the short-exposure image (step S32). Whether or not only the upper body has been imaged (step S36), and only the upper shoulder has been imaged (step S38). judge.

  If the person who is the main subject included in the short-exposure image is a whole body imaged in a small size in the short-exposure image, the magnification of the main subject image in the short-exposure image in step S33 Is 105%. If the person who is the main subject included in the short-exposure image is an image of the whole body captured in a large size, the magnification of the main subject image in the short-exposure image is set to 110% in step S34. If the person who is the main subject included in the short-exposure image is an image of only the upper body, the magnification of the main subject image in the short-exposure image is set to 120% in step S36. When the person who is the main subject included in the short-exposure image is an image of only the shoulder, the magnification of the main subject image in the short-exposure image is set to 130% in step S38.

  When reducing the long-exposure image, as shown in the parentheses in the figure, the person who is the main subject included in the short-exposure image is taken from the whole body with a small size in the short-exposure image. If there is, the reduction ratio of the long-time exposure image is set to 95% in step S33. If the person who is the main subject included in the short-exposure image is a large image of the whole body, the reduction ratio of the background image of the long-exposure image is set to 90% in step S34. If the person who is the main subject included in the short-exposure image is an image of only the upper body, the reduction ratio of the background image of the long-exposure image is set to 80% in step S36. If the person who is the main subject included in the short-exposure image has been captured only on the shoulder, the reduction ratio of the background image of the long-exposure image is set to 70% in step S38.

  As described above, the control unit 15 increases the magnification of the main subject image in the long-time exposure image because the blur amount of the main subject image in the long-time exposure image increases as the main subject image in the short-time exposure image increases. For example, if a person who is the main subject moves the face by 5 cm within a long exposure time, the amount of blur is small when the whole body is imaged, but the amount of blur is large when only the upper body or the shoulder is imaged. It depends. Note that the magnification was changed depending on the size of the whole body, only the upper body, and only the shoulder, but if the person in the short exposure image is very small or too large, The main subject image in the time exposure image is not enlarged or the background image of the long exposure image is not reduced.

  In the next step S40, the control unit 15 enlarges the main subject image in the short-exposure image according to the acquired zoom ratio. In the next step S41, the control unit 15 performs a composition process so that the main subject image in the short-time exposure image enlarged in step S40 is overwritten on the main subject image portion in the long-time exposure image. At this time, the control unit 15 may enlarge the main subject image after extracting the main subject image in the short-time exposure image, or may extract the main subject image after enlarging the short-time exposure image itself.

  When the long-exposure image is reduced, in step S40, the control unit 15 reduces the long-exposure image according to the acquired zoom ratio. In the next step S41, the control unit 15 performs a composition process so that the background image portion in the short-time exposure image is overwritten on the long-time exposure image reduced in step S40. The control unit 15 may superimpose the background of the reduced long exposure image on the short exposure image. In this case, when the reduction amount of the long exposure image is larger than the blurring amount of the main subject of the short exposure image, the background of the reduced long exposure image is superimposed on the short exposure image. The background covers the main subject of the short-time exposure image. Therefore, when the background of the reduced long exposure image is superimposed on the short exposure image, it is necessary to consider the reduction ratio of the long exposure image so as not to perform such a composition process.

  In step S40 and step S41, the processing for enlarging or reducing only one of the short-time exposure image and the long-time exposure image has been described. The background image of the exposure image may be reduced. For example, when the ratio of the main subject image in the short exposure image to the main subject image in the long exposure image is 1: 1.2, the main subject image in the short exposure image is enlarged to 110%. At the same time, the background image of the long exposure image is reduced to 90%.

  According to the imaging apparatus that performs the synthesis process illustrated in FIG. 9, it is possible to acquire a captured image that clearly captures the main subject and the background even during night imaging. Further, since it is not necessary to perform processing such as comparing the sizes of the main subject image in the short exposure image and the main subject image in the long exposure image, the processing burden can be reduced.

  In the above-described combining process shown in FIGS. 6 and 9, when there are a plurality of persons in the short-time exposure image, it is desirable to set a zoom ratio for each person. At this time, the amount of blur in the long-time exposure image may differ greatly for each person. However, by limiting the difference in magnification ratio between the persons to a predetermined range, it is possible to reduce the uncomfortable feeling in the composite image. . Thereby, even when there are a plurality of main subjects that move within the long exposure time, a captured image including a clear main subject and background can be acquired.

  As described above, according to this imaging apparatus, the short-time exposure image captured by the predetermined short-time exposure at the time of flash emission and the long-time exposure captured by the predetermined long-time exposure simultaneously with the short-time exposure image. A main subject image included in the short-time exposure image by performing at least one of enlargement of the short-time exposure image or reduction of the long-time exposure image according to a magnification ratio of the short-time exposure image and the long-time exposure image. Are combined with the long-exposure image to create a composite image, so that a captured image obtained by clearly capturing the main subject and the background can be acquired even during night imaging.

  Further, according to this imaging apparatus, since the main subject included in the short-time exposure image is enlarged based on the calculated zoom ratio, the blur component can be canceled out by the enlargement of the main subject, and the image is clear. Captured images of the main subject and background can be acquired.

  Furthermore, according to this captured image, the long-exposure image is reduced based on the calculated zoom ratio, so that the blur component of the main subject can be canceled out by the reduction of the background.

  Furthermore, according to this imaging apparatus, the contour of the main subject included in the same long exposure image as the main subject is detected together with the contour of the main subject included in the short exposure image, and the main subject included in the short exposure image is detected. Since the zoom ratio is determined according to the difference between the contour size of the main subject and the contour size of the main subject included in the long-time exposure image, the blur component having the same size as the difference can be canceled.

  Furthermore, according to this imaging apparatus, since the magnification ratio is determined according to the difference in the size of the face contour of the main subject, the main subject in the short-exposure image according to the shake component of the limb with much movement It is possible to avoid the synthesized image from becoming an unnatural pattern without enlarging the image or reducing the long-exposure image.

  Furthermore, according to this imaging apparatus, the magnification ratio uniquely set for the imaging state is determined based on the imaging state of the main subject included in the short-time exposure image. A composite image can be created so as to cancel out different blur amounts. In addition, the amount of processing for calculating the zoom ratio can be reduced.

  Furthermore, according to this imaging apparatus, the higher the ratio of the main subject included in the short-time exposure image, the higher the zoom ratio. Therefore, the higher the ratio of the main subject, the higher the ratio of the main subject. Can be created.

  Furthermore, according to this imaging apparatus, since the zoom ratio is limited to be within a predetermined range, the amount of blurring of the main subject is large, and the composite image is unnatural without excessively increasing the zoom ratio. It can avoid becoming a pattern.

  Furthermore, according to this imaging apparatus, when the calculated zoom ratio exceeds a predetermined range, enlargement of the short-exposure image or reduction of the long-exposure image is prohibited. It is possible to avoid that the composite image has an unnatural pattern without increasing the magnification ratio too much.

  Furthermore, according to this imaging apparatus, since the zoom ratio is calculated so as to cancel out the blur amount in the long exposure image of the main subject, it is possible to create a composite image that reliably cancels out the blur amount. A composite image including a clear main subject and background can be surely created.

  The above-described embodiment is an example of the present invention. For this reason, the present invention is not limited to the above-described embodiment, and various modifications can be made depending on the design and the like as long as the technical idea according to the present invention is not deviated from this embodiment. Of course, it is possible to change.

It is a block diagram which shows the functional structure of the imaging device to which this invention is applied. It is a block diagram which shows the hardware-like structure of the imaging device to which this invention is applied. It is a figure explaining the image processing by the imaging device to which this invention is applied, (a) is a short exposure image, (b) is an extraction image of a main subject, (c) is an enlarged image of a main subject, (d) is It is a composite image. It is a flowchart which shows the whole process in the imaging device to which this invention is applied. It is a flowchart which shows the process sequence of night subject blur correction | amendment imaging | photography mode by the imaging device to which this invention is applied. It is a flowchart which shows the process sequence of the compositing process by the imaging device to which this invention is applied. It is a figure which shows the threshold value set in the imaging device to which this invention is applied. It is a figure which shows (a) short-time exposure image and (b) long-time exposure image imaged with the imaging device to which this invention is applied. It is a flowchart which shows the process sequence of the other synthetic | combination process in the imaging device to which this invention is applied. It is a figure explaining the conventional image processing.

Explanation of symbols

1 CCD
2 Imaging processing unit 3 DRAM
DESCRIPTION OF SYMBOLS 4 Image processing part 5 External memory 11 Imaging optical system 12 Imaging part 13 Imaging processing part 14 Memory card 15 Control part 16 Key input part 16a Shutter button 16b Menu button 16c Move button 16d Determination button 17 Program memory 18 Data memory 19 Image compression part 20 Main display processing unit 21 Main display unit 22 Light emission control unit 23 Built-in light emission drive unit 24 Built-in light emission unit

Claims (9)

Imaging means for detecting incident light and obtaining a captured image;
A light emitting means for performing strobe light emission on an area including an imaging range of the imaging means;
Control means for controlling the imaging means and the light emitting means;
Image processing means for performing image processing on the captured image acquired by the imaging means,
The control means causes the light emission means to perform strobe light emission, performs a predetermined short-time exposure by the imaging means, acquires a short-time exposure image, and performs a predetermined long-time exposure by the imaging means for a long time. Get an exposure image,
The image processing means includes a contour detection means for detecting a contour of a main subject included in the short exposure image, a ratio calculation means for calculating a magnification ratio between the short exposure image and the long exposure image, Based on the contour detected by the contour detecting means, the scaling means for performing at least one of the enlargement of the short-time exposure image or the reduction of the long-time exposure image according to the magnification ratio calculated by the ratio calculation means Combining a main subject image obtained by extracting a main subject included in the short-time exposure image with the long-time exposure image to create a composite image ,
Said ratio calculating means, based on the imaging state of the main subject included in the short-time exposure image, an imaging apparatus characterized that you determine uniquely set scaling ratio to the imaging condition. Imaging means for detecting incident light and obtaining a captured image;
A light emitting means for performing strobe light emission on an area including an imaging range of the imaging means;
Control means for controlling the imaging means and the light emitting means;
Image processing means for performing image processing on the captured image acquired by the imaging means,
The control means causes the light emission means to perform strobe light emission, performs a predetermined short-time exposure by the imaging means, acquires a short-time exposure image, and performs a predetermined long-time exposure by the imaging means for a long time. Get an exposure image,
The image processing means includes a contour detection means for detecting a contour of a main subject included in the short exposure image, a ratio calculation means for calculating a magnification ratio between the short exposure image and the long exposure image, Based on the contour detected by the contour detecting means, the scaling means for performing at least one of the enlargement of the short-time exposure image or the reduction of the long-time exposure image according to the magnification ratio calculated by the ratio calculation means Combining a main subject image obtained by extracting a main subject included in the short-time exposure image with the long-time exposure image to create a composite image,
Said ratio calculating means, the imaging device, characterized by calculating a scaling ratio so as to cancel the shake amount within said long exposure image of the main subject. The imaging apparatus according to claim 1 , wherein the ratio calculation unit increases the zoom ratio as the ratio of the main subject included in the short-time exposure image increases. It said ratio calculating means, the image pickup apparatus according to the magnification ratio, to claim 1 or claim 2, characterized in that to limit to be within a predetermined range. The image processing means prohibits enlargement / reduction processing of an image by the enlargement / reduction means when a scaling ratio calculated by the ratio calculation means exceeds a predetermined range. The imaging device according to claim 1 or 2 . In response to the imaging instruction, the flash unit emits strobe light to perform a predetermined short-time exposure to obtain a short-time exposure image, and at the same time to perform a predetermined long-time exposure without performing strobe light emission. An acquisition step of acquiring an image;
A detecting step for detecting a contour of a main subject included in the short-time exposure image;
A calculation step of calculating a magnification ratio between the short-time exposure image and the long-time exposure image;
And line intends scaling step at least one of reduction of enlargement or the long exposure image of the short-time exposure image according to the computed scaling ratio,
Combining a main subject image extracted from the short-exposure image based on the detected contour with the long-exposure image to create a composite image , and
The imaging method is characterized in that, based on an imaging state of a main subject included in the short-time exposure image, a scaling ratio uniquely set for the imaging state is determined . In response to the imaging instruction, the flash unit emits strobe light to perform a predetermined short-time exposure to obtain a short-time exposure image, and at the same time to perform a predetermined long-time exposure without performing strobe light emission. An acquisition step of acquiring an image;
A detecting step for detecting a contour of a main subject included in the short-time exposure image;
A calculation step of calculating a magnification ratio between the short-time exposure image and the long-time exposure image;
An enlargement / reduction step for performing at least one of enlargement of the short-exposure image or reduction of the long-exposure image according to the calculated magnification ratio;
Combining a main subject image extracted from the short-exposure image based on the detected contour with the long-exposure image to create a composite image, and
The imaging method according to claim 1, wherein the calculating step calculates a zoom ratio so as to cancel out a blur amount of the main subject in the long-time exposure image. An image processing program executed by a computer,
An image acquisition process for acquiring a short-time exposure image captured by a predetermined short-time exposure at the time of strobe light emission, and a long-time exposure image captured by a predetermined long-time exposure simultaneously with the short-time exposure image;
A contour detection process for detecting a contour of a main subject included in the short-time exposure image;
A ratio calculation process for calculating a magnification ratio between the short-time exposure image and the long-time exposure image;
An enlargement / reduction process for performing at least one of the enlargement of the short-time exposure image or the reduction of the long-time exposure image according to a magnification ratio calculated by the ratio calculation process;
A synthesizing process for synthesizing a main subject image obtained by extracting a main subject included in the short-time exposure image based on the contour detected by the contour detection process with the long-time exposure image to create a composite image; To the computer ,
The ratio calculation processing on the basis of the imaging condition of the main subject included in the short-time exposure image, an image processing program characterized that you determine uniquely set scaling ratio to the imaging condition. An image processing program executed by a computer,
An image acquisition process for acquiring a short-time exposure image captured by a predetermined short-time exposure at the time of strobe light emission, and a long-time exposure image captured by a predetermined long-time exposure simultaneously with the short-time exposure image;
A contour detection process for detecting a contour of a main subject included in the short-time exposure image;
A ratio calculation process for calculating a magnification ratio between the short-time exposure image and the long-time exposure image;
An enlargement / reduction process for performing at least one of the enlargement of the short-time exposure image or the reduction of the long-time exposure image according to a magnification ratio calculated by the ratio calculation process;
A synthesizing process for synthesizing a main subject image obtained by extracting a main subject included in the short-time exposure image based on the contour detected by the contour detection process with the long-time exposure image to create a composite image; To the computer,
The image processing program characterized in that the ratio calculation means calculates a zoom ratio so as to cancel out a blur amount in the long exposure image of the main subject.

Images