JP2019096995A - Image processing system, image processing method, and program - Google Patents

Abstract

To generate an image in which lightness of both main subject region and other region becomes appropriate.SOLUTION: A digital camera 100 calculates a prediction vector 905 for predicting movement of a moving subject, by using the differential regions 901, 902 of non-flash images 320, 330, 340. The digital camera 100 sets the region excepting the differential region 906 corresponding to the prediction vector 905, out of the differential region of a flash image 410 and a non-flash image 320, as a main subject replacement region 908. The digital camera 100 replaces the image of a region corresponding to the main subject replacement region 908, out of the background image 420, by a flash image 310 as the final image.SELECTED DRAWING: Figure 9

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program, and is particularly suitable for use in processing a captured image by emitting strobe light.

In photographing a scene including a night scene and a main subject such as a human subject, various techniques have been proposed so that images in which both the night scene and the main subject stand out can be recorded.
In the technology disclosed in Patent Document 1, flash-on shooting with flash light emission and flash-off shooting without flash light emission are performed in succession. At the time of flash-off shooting, the shooting sensitivity is set high so that the night scene is properly exposed. On the other hand, the image at the time of flash-on shooting (flash image) is an image in which the brightness of the main subject is appropriate. Therefore, by combining the non-flash image (image obtained at flash-off shooting) and the flash image, it is possible to record an image in which the brightness of both the night scene and the main subject is appropriate.

JP 2012-44378 A JP, 2008-118555, A Japanese Patent Application Laid-Open No. 7-318793 JP, 2012-169753, A Unexamined-Japanese-Patent No. 2004-104425

However, in the above-described technology, the determination of the area in which the brightness of the main subject is appropriate is performed by comparing the difference value between the flash image and the non-flash image. Therefore, when there is a subject moving to an area other than the main subject not exposed to the strobe light, the area of the moving subject may be erroneously determined as the area of the main subject. Therefore, the area of the main subject and the other areas can not be properly synthesized. As a result, there is a possibility that the brightness of both the area of the main subject and the other areas can not be made appropriate.
The present invention has been made in view of such problems, and it is an object of the present invention to generate an image in which the brightness of both the area of the main subject and the other areas is appropriate.

  An image processing apparatus according to the present invention includes an acquisition unit that acquires a light emission image captured in a state in which flash light is emitted and a non-emission image captured in a state in which flash light is not emitted; And detecting means for detecting an area of a subject having a motion in the luminescent image based on a change in pixel information of the luminescent image.

  According to the present invention, it is possible to generate an image in which the brightness of both the area of the main subject and the other areas is appropriate.

It is a figure showing composition of a digital camera. It is a flowchart which shows the 1st example of the whole process of a digital camera. It is a figure explaining the picture picturized with a digital camera. It is a figure explaining the captured image by which image processing was carried out. It is a figure explaining the method to synthesize | combine a flash image and a background image. It is a flowchart explaining background image generation processing. It is a flowchart explaining the process which calculates difference information. It is a flow chart explaining main subject substitution field detection processing. It is a figure explaining the 1st example of the method of detecting the main subject substitution field. It is a figure explaining the method to predict a motion area. It is a figure explaining the modification of the method of performing prediction of a motion area. It is a flowchart which shows the 2nd example of the whole process of a digital camera. FIG. 6 is a diagram for explaining an image for screen display and an actual photographed image. It is a figure explaining the 2nd example of the method of detecting the main subject substitution field. It is a flowchart which shows the 3rd example of the whole process of a digital camera. It is a figure explaining the 3rd example of the method of detecting a main subject substitution field.

  Hereinafter, embodiments will be described with reference to the drawings. In each of the following embodiments, an example will be described in which the imaging device that captures an object and records moving image and still image data on various recording media is a digital camera. However, the imaging device is not limited to a digital camera, and may be a video camera or may be included in various portable devices such as a smartphone and a tablet. Also, the imaging device may be applied to an industrial camera, an on-vehicle camera, or a medical camera. As the recording medium, for example, a tape, a solid memory, an optical disc, a magnetic disc and the like can be mentioned. In each of the following embodiments, the case where the image processing apparatus is included in the imaging apparatus will be described as an example. However, the image processing apparatus may be an apparatus different from the apparatus provided with the imaging means.

First Embodiment
First, the first embodiment will be described.
FIG. 1 is a block diagram showing an example of a functional configuration of the digital camera 100. As shown in FIG.
The system control unit 101 is, for example, a CPU. The system control unit 101 controls the operation of each block included in the digital camera 100. Specifically, the system control unit 101 reads an operation program of each block included in the digital camera 100, and develops and executes the program in the system memory 103 to control the operation of each block. The operation program of each block provided in the digital camera 100 is stored, for example, in the non-volatile memory 102.

  The nonvolatile memory 102 is, for example, an electrically erasable / recordable memory such as an EEPROM. The non-volatile memory 102 stores, in addition to the operation program of each block included in the digital camera 100, parameters and the like necessary for the operation of each block. The system memory 103 is a volatile memory such as, for example, a RAM. Further, in the present embodiment, the non-volatile memory 102 stores processing information necessary to obtain a suitable image. The system memory 103 is used not only as a development area for the operation program of each block but also as a storage area for temporarily storing intermediate data and the like output in the operation of each block.

The system timer 105 is a timer built in the digital camera 100. The system timer 105 is used for measuring an elapsed time in each program or process executed by the system control unit 101 and for time stamp.
The imaging unit 120 is a unit that images a subject and outputs image data. The imaging unit 120 includes a photographing lens 121, a shutter 122, an imaging element 123, and an A / D converter 124. The imaging lens 121 is an imaging lens group including a focus lens, a zoom lens, a color filter, and the like. The photographing lens 121 forms an optical image on the image pickup element 123 through a shutter 122 having an iris and an ND filter function. The imaging device 123 is an imaging device such as a CCD or a CMOS sensor, converts an optical image formed through the imaging lens 121 into an analog image signal, and outputs the analog image signal to the A / D conversion unit 124. The A / D conversion unit 124 converts the analog image signal into a digital image signal (image data / frame) by applying an A / D conversion process to the analog image signal input from the imaging element 123.

  The image processing unit 104 applies various image conversion processing to the image data output from the imaging unit 120 or the image data read from the memory 107 by the memory control unit 106 described later. Various image conversion processing includes predetermined pixel interpolation processing, resizing processing such as reduction processing, color conversion processing, and the like. Further, the image processing unit 104 executes arithmetic processing related to exposure control and distance control using the digital image signal input from the A / D converter 124, and outputs the result of the arithmetic processing to the system control unit 101. Do. The system control unit 101 operates the photographing lens 121 and the shutter 122 by a drive system (not shown) based on the result of the arithmetic processing to start operations such as exposure control, distance measurement control, and EF (flash pre-emission) processing. Let

  The memory control unit 106 is a block that controls reading of information from the memory 107 and writing of information to the memory 107. The memory control unit 106 writes, in the memory 107, the digital image signal input from the A / D conversion unit 124 or an image signal to which various processes are applied by the image processing unit 104 and output. In the memory 107, in addition to the image group relating to the frame of the image being photographed, information of the sound being photographed is also written. For this reason, the memory 107 is designed to have a sufficient storage capacity to store the information.

  The flash 113 performs flash light emission operation at the time of shooting. The flash control unit 114 controls the light emission operation of the flash 113. The input operation unit 115 is a user interface having operation members (a power button, a mode switching SW, an imaging button, and the like) included in the digital camera 100. When the input operation unit 115 detects that the user operates each operation member, the input operation unit 115 transmits a control signal corresponding to the operation to the system control unit 101. The I / F 111 is an interface for connecting the recording medium 112 and the digital camera 100 to each other. Image data and audio data encoded by the encoding unit 110 according to a predetermined encoding format such as MPEG are recorded on the recording medium 112 via, for example, the recording medium I / F 111. The recording medium 112 is, for example, a recording device detachably connected to the digital camera 100, such as a memory card or an HDD.

An example of processing of the digital camera 100 according to the present embodiment having such a configuration will be described using the flowchart of FIG. Note that the flowchart (whole process) in FIG. 2 is started, for example, when an instruction for photographing is given in the digital camera 100.
In S200, the system control unit 101 performs an imaging process. Specifically, the system control unit 101 acquires the result of the arithmetic processing by the image processing unit 104, and determines the exposure value. The exposure value is a control amount related to the imaging unit 120 including the setting of the shutter speed, the aperture amount of the aperture, and the ISO sensitivity. The system control unit 101 sets the determined exposure value in the imaging unit 120. The system control unit 101 acquires a flash image and a plurality of non-flash images continuously captured by the imaging unit 120 in time series. As described above, in this embodiment, for example, in S200, an example of obtaining a light emission image captured in a state in which flash light is emitted and a non-emission image captured in a state in which flash light is not emitted Is realized.

  Here, an example of an image captured by the digital camera 100 of the present embodiment will be described with reference to FIG. FIG. 3A shows an example of the flash image 310. FIGS. 3B, 3C, and 3D show the first non-flash image 320, the second non-flash image 330, and the third non-flash image 340, respectively. The flash image 310 is an image captured by emitting a strobe light, and the non-flash images 320, 330, and 340 are images captured without emitting a strobe light. 3A to 3D, the flash image 310, the first non-flash image 320, the second non-flash image 330, and the third non-flash image 340 are sequentially captured in this order. Image. The flash image 310 may be continuously captured after the first to third non-flash images 320, 330, and 340.

  The flash image 310 has a darker background than the non-flash images 320, 330, 340. This is because when the flash image 310 is captured, the exposure is lowered by the amount of flash light emission compared to the non-flash images 320, 330, and 340. The system control unit 101 lowers the exposure, for example, by one stage so that the main subject 302 has appropriate brightness. Then, the system control unit 101 controls the flash control unit 114 so that the flash light is emitted from the flash 113 so as to match the appropriate exposure brightness of the non-flash image, and imaging is performed in the state where the flash light is emitted. The unit 120 performs imaging.

  Therefore, the brightness of the non-flash images 320, 330, and 340 is one step higher than that of the flash image 310 in the area 303 (area corresponding to the area 301 of the flash image 310) to which the strobe light does not reach. In addition, when the non-flash images 320, 330, and 340 are captured, the light amount is low because the strobe light is not emitted. Therefore, the non-flash images 320, 330, and 340 become images in which a large amount of noise is included in the entire image. Further, the area 301 of the flash image 310 not exposed to strobe light is an image that is one step worse than the non-flash images 320, 330, and 340. The flash image 310 and the non-flash images 320, 330, and 340 are images that have been captured on the basis of a capture instruction.

  Subsequently, an example of an image generated by performing image processing on an image captured by the digital camera 100 according to the present embodiment will be described with reference to FIG. 4. FIG. 4A shows an example of the flash image 410 after gain adjustment. The flash image 410 is an image obtained by multiplying the flash image 310 shown in FIG. 3A by a digital gain so that the brightness is brightened by one step. The brightness of the background area 401 of the flash image 410 matches the brightness of the non-flash images 320, 330, and 340 shown in FIGS. 3 (b) to 3 (d). Further, FIG. 4B shows an example of the background image 420. The background image 420 is an image obtained by combining the non-flash images 320, 330, and 340 and the flash image 410 by the process of S202 of the flowchart shown in FIG. 2 described later. As described above, in the present embodiment, for example, by generating the flash image 410 after gain adjustment, the brightness of the light emission image is made so that the brightness of the background area of the light emission image approaches the brightness of the non-light emission image. An example of adjusting the height is realized. Further, the light emission image whose brightness has been adjusted is realized, for example, by the flash image 410 after gain adjustment.

  Returning to the explanation of FIG. 2, in step S201, the system control unit 101 performs alignment processing. Specifically, the system control unit 101 uses these images to combine the flash image 410 shown in FIG. 4A and the non-flash images 320, 330, and 340 shown in FIGS. 3B to 3D. Each image is aligned based on a predetermined one of the images. By performing this process, the influence of camera shake or the like on the flash image 410 and the plurality of non-flash images 320, 330, and 340 can be improved. As described above, in the present embodiment, for example, an example of aligning the positions of a plurality of non-light emitting images is realized by S201. In the present embodiment, an example in which the flash image 310 and the plurality of non-flash images 320, 330, and 340 are imaged in order will be described.

  As an example of the image alignment method, there is a method of obtaining a motion vector, calculating a transform coefficient based on the direction and magnitude of the motion vector, and deforming the image based on the transform coefficient. For example, a template image of a predetermined size is created from the reference image, and a motion vector is calculated by performing matching processing (template matching processing) between the alignment target image and the template image. In addition, about the position alignment which used such a motion vector, since it describes in patent document 2, the detailed description is abbreviate | omitted here. Further, the method of alignment is not limited to the one described above, and there are various methods, and therefore the alignment process may be performed using these methods.

  Next, in step S202, the system control unit 101 performs background image generation processing. The background image 420 shown in FIG. 4B is to combine the non-flash images 320, 330 and 340 shown in FIG. 3B to FIG. 3D with the flash image 410 shown in FIG. Created by The background image generation process will be described in detail later. In the present embodiment, for example, an example of generating an image obtained by combining the light emitting image and the plurality of non-light emitting images as an image based on the non-light emitting image is realized by S202. Also, an image based on the non-emission image is realized by, for example, a background image 420.

  Next, in step S203, the system control unit 101 performs main subject replacement area detection processing. Specifically, the system control unit 101 detects a main subject replacement area for combining the flash image 310 shown in FIG. 3A and the background image 420 shown in FIG. 4B. The main subject replacement area is an area of the flash image 310 and an area of the main subject to be substituted for the background image 420. In S203, the area of the moving object is removed using the flash image 410 shown in FIG. 4A and the non-flash images 320, 330, and 340 shown in FIGS. Detect the area that is hit. The main subject replacement area detection process will be described in detail later. In the present embodiment, an example of detecting an area of a subject having a motion in the light emission image is realized based on changes in pixel information of the plurality of non-light emission images, for example, in S203. Also, the area of the subject without motion is realized by, for example, the area of the main subject.

  Next, in step S204, the system control unit 101 in FIG. 1 combines the flash image 310 and the background image 420 based on the detection result of the main subject replacement area obtained in the process of step S203. In the present embodiment, for example, one example of combining the light emission image and the image based on the non-light emission image is realized by S204. An example of a method of combining the flash image 310 and the background image 420 will be described with reference to FIG.

5A shows an example of the flash image 310, and FIG. 5B shows an example of the background image 420. FIG. 5C shows an example of the detection result 530 of the main subject replacement area obtained in the process of S203. FIG. 5D is the final image 540 of this embodiment. In FIG. 5C, an area 501 indicates an area (that is, a main subject replacement area) to be combined with the background image 420 in the flash image 310 shown in FIG. 5A. An area 502 indicates an area where the background image 420 shown in FIG. 5B is to be synthesized. Therefore, the system control unit 101 combines the flash image 310 and the background image 420 according to the detection result 530 of the main subject replacement area shown in FIG. 5C, and generates the final image 540.
In the above, an example of the entire process of combining a flash image and a non-flash image has been described with reference to the flowchart of FIG.

  Subsequently, an example of the background image generation process of S202 of FIG. 2 will be described in detail with reference to the flowchart of FIG. Here, as an example, a process of combining the single flash image 410 (after gain adjustment) and the three non-flash images 320, 330, and 340 to generate the background image 420 will be described.

In step S600, the system control unit 101 initializes a value i of a register that records the number of times of synthesis processing (sets “1” to the value i of the register).
Next, in step S601, the system control unit 101 determines whether the value i of the register exceeds the number N of times of synthesis (here, N = 3). As a result of this determination, if the result is true (if the register value i does not exceed the number N of times of combining), the process proceeds to the step S602. On the other hand, in the case of false (when the register value i exceeds the number N of times of composition), the system control unit 101 ends the background image generation processing, and uses the image obtained at this time as the background image 420. (See FIG. 4 (b)).

Next, in step S602, the system control unit 101 calculates difference information. An example of the process of calculating the difference information will be described with reference to FIGS. 3, 4 and 7.
First, the system control unit 101 calculates inter-frame difference information between the combined reference image and the image to be combined. The system control unit 101 sets, for example, the first non-flash image 320 shown in FIG. 3B as a composite reference image. The difference information between the frames of the first non-flash image 320 and the second non-flash image 330 is the difference information 710 shown in FIG. 7A. The difference information is represented, for example, by a pixel-by-pixel difference value between lightness or color difference frames of each image. In the example illustrated in FIG. 7, a region where the absolute value of the difference value exceeds the threshold is represented by a white region such as a region 701. The system control unit 101 determines that the area indicated by the white area 701 is an area in which the subject is moving. The system control unit 101 determines that the black area 702 is an area in which the subject is not moving. By performing the same process as the above process, as shown in FIG. 7B, difference information 720 between the frames of the first non-flash image 320 and the third non-flash image 340 can be obtained. .

When calculating difference information between frames of the combined reference image and the flash image, the flash image 410 shown in FIG. 4A is adjusted without using the flash image 310 shown in FIG. Use. By matching the brightness of the area other than the main subject with the flash image and the non-flash image, it is possible to extract the area (difference) in which the subject is moving. FIG. 7C illustrates the difference information 730 between the first non-flash image 320 and the flash image 410.
Note that a flash image may be used as a composite reference image. In this case, instead of using the flash image 310 shown in FIG. 3A, the flash image 410 whose gain is adjusted as shown in FIG. 4A is used as a composite reference image.

  Referring back to FIG. 6, in step S603, the system control unit 101 performs the following processing based on the difference information between the frames calculated in step S602. That is, the system control unit 101 calculates an image that has been synthesized up to i-1 times of processing, and an image obtained by calculating a difference value (difference information) between frames with respect to the synthesized reference image in the i-th process (S602). Do the synthesis of In the first process, the image synthesized by i-1 times is the initial synthesized reference image (here, the first non-flash image 320). Also, if the composite reference image and the image to be combined first are the second non-flash image 330, for example, the difference value (difference information) between the frames with respect to the composite reference image in the first process (S602) The calculated image is the second non-flash image 330. Thus, in the present embodiment, for example, in S603, an image serving as a reference among the light emission image and the plurality of light emission images, and an image different from the image serving as the reference among the light emission image and the plurality of light emission images An example of combining is realized.

  Based on the difference information calculated in the i-th process, the system control unit 101 calculates the difference between the frames compared to the image synthesized up to the i-1th process and the synthesized reference image in the i-th process. It is preferable to change the composition ratio with the image for which information is calculated. Specifically, the system control unit 101 increases the ratio of the images synthesized by the process of i-1 times as the area in which the difference information (difference value) calculated in the i-th process (S602) is larger. An image in which an inter-frame difference value (difference information) is calculated with respect to a combined reference image in an area where the value of difference information (difference value) calculated in the i-th process (S602) is extremely large Set the composition ratio of to 0 (zero). In this case, the area is an image synthesized by i-1 times of processing. For example, the system control unit 101 can determine the combining ratio by setting in advance the relationship between the range of difference information (difference value) and the combining ratio. In this embodiment, in this way, an image which has already been synthesized according to the difference between the pixel information of the reference image and the pixel information of the image different from the reference image, and the reference An example of determining the proportion of combining an image with a different image is realized.

Next, in step S604, the system control unit 101 increments the value i of the register. Then, the process returns to the step of S601.
The above processing is repeated, and the non-flash images 320, 330, and 340 shown in FIGS. 3B to 3D and the flash image 410 shown in FIG. A background image 420 shown in FIG.

  By using the difference information 710, 720, and 730 shown in FIGS. 7A to 7C, for example, the following can be performed. That is, for an area where the difference information (difference value) is large, it is possible to obtain an image synthesized by the (i-1) -th process (see areas 403 and 404 shown in FIG. 4B). For the other areas, a weighted average of the image synthesized by the i-1th process and the image for which the inter-frame difference value (difference information) is calculated with respect to the synthesized reference image in the i-th process is taken. An image can be obtained (see region 405 shown in FIG. 4 (b)). That is, in S603 of each time (each i), while changing the weight (composition ratio) for each area, the image synthesized by the process of i-1 times and the frame between the synthesized reference image in the i-th process It is possible to combine an image for which a difference value (difference information) has been calculated. Therefore, the background image 420 illustrated in FIG. 4B is an image in which noise in the area 405 is reduced as compared to the non-flash image 320 illustrated in FIG. 3B.

Subsequently, an example of the main subject replacement area detection process of S203 of FIG. 2 will be described in detail with reference to FIGS. 8 and 9. FIG. 8 is a flowchart for explaining an example of the main subject replacement area detection process. FIG. 9 is a diagram for explaining an example of a method of detecting the main subject replacement area.
In step S800, the system control unit 101 extracts a motion vector of an area in which the subject is moving. In the following description, the area in which the subject is moving is referred to as a moving area as necessary. The system control unit 101 extracts a motion vector (a vector indicating a moving direction and a motion amount of a motion area) using the flash image 410 whose gain is adjusted and the plurality of non-flash images 320, 330, and 340.

  Here, a case where a region for calculating a motion vector is specified based on the difference regions 901 and 902 between non-flash images adjacent in time series will be described as an example. The system control unit 101 calculates a difference area (the above-described difference value) between frames of the non-flash images 320 and 330, and sets an area where the absolute value of the difference value exceeds a threshold as the difference area 901. Similarly, the system control unit 101 calculates a difference area (the above-described difference value) between the frames of the non-flash images 330 and 340, and sets an area where the absolute value of the difference value exceeds the threshold as the difference area 902. The difference areas 901 and 902 are areas for specifying motion vectors.

  Then, the system control unit 101 calculates motion vectors for the difference areas 901 and 902 (motion areas). For example, the system control unit 101 determines, among the areas included in the difference area 901, a vector directed to the center of gravity of the area corresponding to the image captured earlier in time from the center of gravity of the area corresponding to the image captured late in time. As a motion vector for the difference area 901. For example, in the difference area 901, the area corresponding to the image captured earlier in time is an area where the brightness of the first non-flash image 320 is significantly different from that of the second non-flash image 330. The area corresponding to the image captured late in time is an area where the brightness of the second non-flash image 330 is significantly different from that of the first non-flash image 320. The motion vector can be calculated for the difference area 902 as well as the difference area 901.

  At this time, the system control unit 101 may change the range for calculating the motion vector according to the size of the difference areas 901 and 902. For example, the larger the size of the difference regions 901 and 902, the larger the range for calculating the motion vector can be. For example, the system control unit 101 calculates a motion vector in a range including the difference areas 901 and 902. The system control unit 101 can obtain motion vectors 903 and 904 between the non-flash images 320 and 330 and between the non-flash images 330 and 340, respectively, by calculating the motion vector. In this embodiment, the motion vector indicating the movement of the subject in the two non-light emitting images is detected according to the size of the region where the pixel information of the two non-light emitting images has a difference exceeding the threshold. An example of changing the range is realized.

  Returning to the description of FIG. 8, in step S <b> 801, the system control unit 101 predicts a motion area between the first non-flash image 320 and the flash image 410. An example of a method of predicting a motion area will be described in detail with reference to FIGS. 9 and 10. FIG. 10 is a diagram for explaining an example of a method of predicting a motion area.

  In FIG. 10, motion vectors 1002 and 1003 correspond to the motion vectors 903 and 904 shown in FIG. 9, respectively, and are plotted on one plane. The moving direction 1001 indicates the direction of the motion vectors 1002 and 1003. The predicted vector 1004 corresponds to the predicted vector 905 shown in FIG. The system control unit 101 obtains a predicted vector 1004 based on, for example, the moving directions, the sizes, and the positions of the motion vectors 1002 and 1003.

  First, the system control unit 101 calculates the movement direction 1001 from the motion vectors 1002 and 1003. Thereafter, the system control unit 101 sets the start point of the motion vector 1002 as an end point, calculates the vector having the half size of the size of the motion vector 1002, and the orientation along the movement direction 1001 as the prediction vector 1004. In the present embodiment, the photographing interval between the flash image 310 and the non-flash images 320, 330, and 340 is very short, and the movement of the moving subject can be predicted as constant velocity movement. Further, in the present embodiment, the flash image 310 has an exposure time shorter by one stage than the non-flash images 320, 330, and 340. Therefore, it is estimated that the movement amount between the flash image 310 and the non-flash image 320 is 1/2 times the movement amount between the non-flash images 320, 330, and 340.

However, when the exposure times of the flash image 310 and the non-flash images 320, 330, and 340 are extremely short, the difference between the exposure times becomes small. For this reason, the magnitudes of the prediction vectors do not differ from the magnitudes of motion vectors between non-flash images. Therefore, the system control unit 101 can set a vector having the same size as the size of the motion vector between non-flash images as the prediction vector. The size of the predicted vector depends on the difference between the exposure time at the time of shooting of the flash image and the time of shooting of the non-flash image. Therefore, the system control unit 101 may switch the size of the predicted vector according to the difference in exposure time. For example, if the exposure time is within a predetermined range, the prediction vector can be estimated by constant velocity motion. Therefore, for example, when there is a difference in exposure between the flash image and the non-flash image of two steps, the system control unit 101 sets a vector having a size that is 1/4 times the size of the motion vector between the non-flash images. Make it a prediction vector.
In the present embodiment, as described above, a predicted vector for predicting the motion of the subject between the light emitting image and the non-light emitting image of one of the plurality of non-light emitting images based on the motion vector is used. An example of deriving is realized.

  Subsequently, the system control unit 101 sets the difference area 906 (white area) at the position closest to the prediction vector 905 in the difference area (white area) included in the difference information 914 at the position corresponding to the prediction vector 905. The motion difference area prediction result 907 is set. The position corresponding to the prediction vector 905 is determined based on, for example, the size, the orientation, and the position of the prediction vector 905. Derivation of the difference information 914 and the difference area 906 is realized by the same method as the method described above. The motion difference area prediction result 907 is a prediction result of difference information.

  In the present embodiment, for example, a first difference image which is an image indicating the difference between the pixel information of the light emission image and the pixel information of the non-light emission image of one of the plurality of non-light emission images by the difference information 914. An example is realized. Also, for example, in FIG. 9, the difference information between the non-flash images 320 and 330 (difference information including the difference area 901) indicates the difference between the pixel information of two non-emission images among the plurality of non-emission images. An example of a second difference image that is an image is realized. Similarly, in FIG. 9, an image showing differences in pixel information of two non-emission images among the plurality of non-emission images by difference information between the non-flash images 330 and 340 (difference information including the difference area 902). An example of the second difference image, which is

  Returning to the description of FIG. 8, in step S802, the system control unit 101 corresponds to the motion difference region prediction result 907 among the difference regions (white regions) included in the difference information 914 between the flash image 410 and the non-flash image 320. The difference area 906 is removed. As described above, in the present embodiment, for example, the difference area 906 realizes an example of the area of the subject with motion. Further, the difference area (white area) other than the difference area 906 in the difference information 914 realizes an example of the area of the subject without motion. Thus, in the present embodiment, for example, the motion difference region prediction result 907 is derived based on the difference information 914 and the prediction vector 905, and the difference region 906 is detected and removed based on the motion difference region prediction result 907. Therefore, based on the first difference image indicating the difference between the pixel information of the light emission image and the pixel information of the non-light emission image of one of the plurality of non-light emission images, the light emission image based on the prediction vector The area of the subject with motion is detected at.

  As described above, by using the motion difference area prediction result 907, even if the calculation accuracy of the prediction vector 905 is low, the difference area 906 can be identified and removed. However, when the calculation accuracy of the prediction vector 905 is high and the difference area 906 can be accurately derived from only the prediction vector 905, the system control unit 101 directly adds the difference area 906 included in the difference information 914 from the prediction vector 905. It may be identified and removed.

  In this manner, the system control unit 101 removes the difference area 906 from the difference information 914 between the flash image 410 and the non-flash image 320, and then the difference area (white area) included in the difference information 914 Detect as 908. The difference information 914 includes, as difference areas, the main subject that is not moving and the moving subject. Therefore, by removing the difference area that is the motion difference area prediction result 907, the area including the moving subject can be removed. Therefore, the system control unit 101 can detect the area of the main subject illuminated by the strobe light (main subject replacement area 908).

  As described above, in the present embodiment, the digital camera 100 uses the difference areas 901 and 902 of the non-flash images 320, 330, and 340 to calculate the prediction vector 905 for predicting the motion of the moving subject. In the difference area between the flash image 410 and the non-flash image 320, the digital camera 100 sets an area excluding the difference area 906 corresponding to the predicted vector 905 as a main subject replacement area 908. The digital camera 100 replaces the image of the area corresponding to the main subject replacement area 908 in the area of the background image 420 with the flash image 310 as the final image. Therefore, even if there is a moving subject, it is possible to appropriately determine and extract the area where the strobe light is applied among the areas of the subject included in the flash image 310. Therefore, for example, it is possible to preferably generate an image in which the brightness of both the night scene and the main subject is appropriate.

  In the present embodiment, the difference information (difference information obtained in S602 of FIG. 6) used in the background image generation process and the difference information (difference information obtained in S800 of FIG. 8) used in the main subject replacement area detection process The case of separately obtaining is described as an example. However, common difference information may be used in these processes. For example, in S800 of FIG. 8 in the main subject replacement area detection processing, the system control unit 101 obtains a motion vector between two images based on the non-flash image 320 as in S602 of FIG. Specifically, the system control unit 101 obtains a motion vector between the non-flash images 320 and 330 and a motion vector between the non-flash images 320 and 340. By using common difference information in this manner, it is possible to reduce the amount of computation and the processing time.

  Further, in the present embodiment, the configuration for performing motion area prediction using a prediction vector in the main subject replacement area detection process (S801 in FIG. 8) has been described, but the present invention is not limited thereto. For example, the motion difference region prediction result 907 shown in FIG. 9 may be specified using the shape, size, and position of the difference information between the frames obtained by the plurality of non-flash images 320, 330, and 340. A modification of the method of performing motion area prediction will be described in detail with reference to FIG. 11A and 11B show difference information 1110 and 1120 (difference information between non-flash images 320 and 330, non-flash images 330 and 340, respectively) including difference areas 901 and 902 shown in FIG. Indicates difference information).

  Since the sampling period for acquiring the flash image 310 and the non-flash images 320, 330, and 340 is short, the system control unit 101 assumes that the moving subject (difference regions 901 and 902) performs linear motion at constant velocity. Do. Under this assumption, the system control unit 101 causes the difference area 1103 in which the difference area 901 is linearly moved at the same speed by the sampling period for acquiring the flash image 310 and the non-flash images 320, 330, and 340, as shown in FIG. The motion difference area prediction result 1130 shown in FIG. Determining the motion difference region prediction result 1130 using the region information of the difference regions 901 and 902 eliminates the need for calculation processing for obtaining the motion vectors 903 and 904 and the prediction vector 905, and makes it possible to shorten the processing time.

  Further, in the present embodiment, the configuration is described in which imaging of one flash image and three non-flash images is performed a total of four times (however, gain adjustment of the flash image is not included). There is no. The noise reduction effect can be improved by performing the processing of the present embodiment by increasing the number of captured images (by setting the number of non-flash images to 3 or more). Further, since the number of images for calculating the direction, position, and size of the vector is increased, it is possible to improve the prediction accuracy of the motion area prediction (S801 in FIG. 8).

  Further, when the position of the digital camera 100 is fixed by a tripod or the like, the alignment process (S201 in FIG. 2) may not be performed. In this way, the processing time can be shortened.

Second Embodiment
Next, a second embodiment will be described. In the first embodiment, the case of using the three non-flash images 320, 330, and 340 acquired by the main photographing accompanying the photographing instruction by the operation of the input operation unit 115 by the user or the timer photographing is described as an example. . On the other hand, in the present embodiment, using the live view image (preview image), brightness difference information is obtained to obtain a motion vector, and a predicted vector is obtained from the obtained motion vector. Then, the area of the main subject is determined by removing the area of the moving subject using the predicted vector and the difference information of the two images (flash image and non-flash image) acquired by the main photographing. Then, the image in the area of the main subject included in the flash image and the non-flash image (an area other than the area of the main subject) are synthesized.

  As described above, in the present embodiment and the second embodiment, a part of the process for identifying the moving subject is mainly different. Accordingly, in the description of the present embodiment, the same parts as those of the first embodiment will be denoted by the same reference numerals as those of the reference numerals in FIGS. In the following description, the live view image is referred to as a screen display image or a screen display non-flash image as needed. Further, a flash image obtained by the main shooting is referred to as a main shooting flash image as needed, and a non-flash image obtained by the main shooting is referred to as a main shooting non-flash image as required, and these are as required Collectively referred to as an image.

In the present embodiment, a process of creating a background image from the flash image whose gain has been adjusted and a plurality of non-flash images is not performed. Therefore, based on the motion vector obtained from the live view image acquired before the main shooting, a predicted vector of the motion area between the flash image and the non-flash image is calculated.
The configuration of the digital camera 100 according to the present embodiment is the same as the configuration of the digital camera 100 according to the first embodiment shown in FIG.

  In the present embodiment, a screen display image is used. Therefore, the memory 107 illustrated in FIG. 1 stores an image for acquiring an evaluation value of camera processing and an image for screen display in addition to the image data acquired by the main photographing. In the present embodiment, an image for screen display is used in addition to a flash image and a non-flash image which are image data acquired by main imaging.

  An example of the entire process of the digital camera 100 of the present embodiment will be described with reference to the flowchart of FIG. 12. The detailed description of the same processing as that of the first embodiment is omitted. The flowchart shown in FIG. 2 is started when an instruction for photographing is given in the digital camera 100. On the other hand, the flowchart shown in FIG. 12 is started before the photographing instruction is given in the digital camera 100.

  In step S1200, the system control unit 101 performs an imaging process. Specifically, the system control unit 101 continues the imaging process of the screen display image before an imaging instruction is issued by the user or timer imaging of the digital camera 100 or the like. Thereafter, after a shooting instruction is issued, the system control unit 101 performs a main shooting and continuously captures a flash image and a non-flash image one by one. FIG. 13 is a diagram for explaining an example of a screen display image and an actual photographed image.

  In FIG. 13, the non-flash images for screen display 1301 to 1303 are non-flash images as the image for screen display taken before the imaging instruction. It is assumed that the first screen display non-flash image 1301, the second screen display non-flash image 1302, and the second screen display non-flash image 1303 are sequentially captured in order. Thereafter, after a shooting instruction is issued, the digital camera 100 continuously captures the main shooting flash image 1304 and the main shooting non-flash image 1305, which are main shooting images, in this order. The main non-flash image 1305 may be exposed for a long time unlike the first embodiment, and may have a configuration different from the exposure time of the non-flash images 1301 to 1303 for screen display. In addition, a plurality of images for screen display acquired before the imaging instruction is issued are held in the memory 107 in a range that fits in the capacity of the memory 107. In the present embodiment, for example, the non-flash images 1301 to 1303 for screen display realize an example of an image to be displayed on the screen. Also, for example, the main photographing flash image 1304 and the main photographing non-flash image 1305 realize an example of the main photographing image based on the photographing instruction.

Next, in step S1201, the system control unit 101 performs main subject replacement area detection processing for combining the main shooting flash image 1304 and the main shooting non-flash image 1305. This will be described in detail using an example of a method of detecting the main subject replacement area according to the present embodiment with reference to FIG.
In FIG. 14, the real shooting flash image 1410 is an image whose gain is adjusted so that the brightness of the background area matches the brightness of the real shooting non-flash image 1305 with respect to the real shooting flash image 1304.

  In the first embodiment, motion vectors 903 and 904 are obtained from the captured non-flash images 320, 330 and 340, and motion vectors 903 and 904 are used to obtain a prediction vector 905 to predict the motion of a moving subject. I do. On the other hand, in the present embodiment, the motion vector and the prediction vector are obtained using the non-flash image for screen display. The system control unit 101 calculates, for example, a difference area 1401 between the screen display non-flash images 1301 and 1302. Furthermore, the system control unit 101 calculates a motion vector 1403 in the difference area 1401. Similarly, the system control unit 101 calculates a difference area 1402 between the non-flash image for screen display 1302 and 1303, and calculates a motion vector 1404 in the difference area 1402. The method itself for obtaining the difference regions 1401 and 1402 and the motion vectors 1403 and 1404 is the same as the method described in the first embodiment.

  Next, the system control unit 101 calculates a prediction vector 1405 using the motion vectors 1403 and 1404 in the same manner as described in the first embodiment. After that, the system control unit 101 uses the difference information 1414 between the main shooting flash image 1410 after the gain adjustment and the main shooting non-flash image 1305 and the prediction vector 1405 in the same manner as described in the first embodiment. Then, the motion difference area prediction result 1407 is calculated. Finally, the system control unit 101 removes the difference area 1406 included in the difference information 1414 based on the difference information 1414 and the motion difference area prediction result 1407 in the same manner as described in the first embodiment. The main subject replacement area 1408 is extracted.

  Referring back to FIG. 12, in step S1202, the system control unit 101 combines the main shooting flash image 1304 and the main shooting non-flash image 1305 based on the detection result of the main subject replacement area 1408 obtained in the processing of S1201. Do. The method of combining these images can be realized by replacing the background image 420 with the real shooting non-flash image 1315 in the description of the process of combining the flash image 310 and the background image 420 in the first embodiment. Therefore, the detailed description is omitted.

  As described above, in the present embodiment, the digital camera 100 calculates the motion difference region prediction result 1407 using the non-flash images 1301 to 1303 for screen display. Therefore, even if the image captured in the main shooting is two images of a flash image and a non-flash image, it is possible to extract the main subject replacement area excluding the moving area. It can improve the harmful effect.

  In the present embodiment, a configuration is used in which a plurality of non-flash images for screen display 1301 to 1303 until the imaging instruction is output is held, and the main imaging flash image 1304 and the main imaging non-flash image 1305 after the imaging instruction is output. An example has been described. However, when it is known that the processing of the present embodiment is performed by the setting of the mode or the like in the digital camera 100, the motion vector is previously made before the photographing instruction using the non-flash image for screen display before the photographing instruction is issued. It is also possible to request 1403 and 1404. With this configuration, there is no need to always hold a non-flash image for screen display, and motion vectors are already calculated when main shooting is performed, which is effective in improving the processing speed. Furthermore, the predicted vector 1405 may be obtained before the imaging instruction.

Further, in the present embodiment, a plurality of non-flash images 1301 to 1303 for screen display until the imaging instruction is issued are held, and the main imaging flash image 1304 and the main imaging non-flash image 1305 after the imaging instruction is output are used. The configuration has been described by way of example. However, after the photographing instruction is issued, the main photographing flash image 1304 and the main photographing non-flash image 1305 may be photographed, and thereafter, a plurality of screen display non-flash images may be held. By adopting this configuration, it is not necessary to hold a plurality of screen display images when processing of the present embodiment is not performed, which is effective in saving memory.
Also in the present embodiment, the modification described in the first embodiment can be applied.

Third Embodiment
Next, a third embodiment will be described. In the first and second embodiments, the movement of the difference area is obtained in the entire difference information between the flash image and the non-flash image, and the difference area without movement is calculated as the main subject replacement area. . On the other hand, in the present embodiment, the movement of the difference area is not obtained for the mask area, with the area other than the area of the subject at the proper emission distance of the strobe light as the mask area. As described above, in the present embodiment and the first and second embodiments, a part of the process for identifying the moving subject is mainly different. Therefore, in the description of the present embodiment, the same parts as those in the first and second embodiments are denoted by the same reference numerals as those in FIGS. 1 to 14, and the detailed description will be omitted.

The configuration of the digital camera 100 according to the present embodiment is the same as the configuration of the digital camera 100 according to the first embodiment shown in FIG.
An example of the overall processing of the digital camera 100 of the present embodiment will be described with reference to the flowchart of FIG. The detailed description about the same processing as the first and second embodiments is omitted.

In step S1501, the system control unit 101 performs main subject replacement area detection processing for combining the main shooting flash image 1304 and the main shooting non-flash image 1305.
In the present embodiment, the system control unit 101 sets an area of the appropriate light emission distance information. For example, the system control unit 101 calculates the appropriate light emitting distance information from the aperture value (F value) of the photographing lens 121 and the guide number, and calculates the depth information. The proper light emitting distance information is information indicating the distance from the digital camera 100 when the strobe light properly reaches. The depth information of the scene is information indicating the distance from the digital camera 100 to the subject. Then, based on the proper light emitting distance information and the depth information, the system control unit 101 sets only the range of the subject of the proper light emitting distance of the strobe light as the area for obtaining the motion vector.

An example of how to obtain the main subject replacement area will be described with reference to FIG. Here, the detailed description of the same processing as the processing described with reference to FIG. 14 of the second embodiment will be omitted.
In FIG. 16, the system control unit 101 acquires the proper light emitting distance information 1601 from, for example, the aperture value (F value) of the photographing lens 121 and the guide number at the time of photographing of the main photographing flash image 1410. At the same time, the system control unit 101 calculates depth information 1602. As a method of calculating the depth information 1602, the following method may be mentioned.

  If the imaging device 123 is a pupil division imaging device, there is one method of calculating the depth information 1602 using the defocus amount calculated from the correlation operation of the parallax image (data for distance acquisition) acquired from the imaging device 123 It is mentioned as an eye method. This method can be realized using the method described in Patent Document 3. As a second method, there is a method of calculating depth information 1602 using a defocus amount calculated from correlation calculation of a plurality of images (data for distance acquisition) captured by changing the focal plane. This method can be realized using the method described in Patent Document 4. If the imaging unit 120 has a compound eye configuration, a third method is a method of calculating the depth information 1602 using a defocus amount calculated from correlation calculation of compound eye images (data for distance acquisition). This method can be realized using the method described in Patent Document 5. Note that each method is a means for acquiring distance information, and does not limit the configuration of the present embodiment.

  In the present embodiment, for example, an example of information indicating the emission distance of the strobe light from the imaging device is realized by the appropriate emission distance information 1601. Also, for example, the depth information 1602 realizes an example of information indicating the distance from the imaging device of the subject in the non-emission image.

  The system control unit 101 calculates an area corresponding to the proper light emitting distance information 1601 and the depth information 1602 obtained as described above as a proper flash area mask 1603. This masked area (appropriate flash area mask 1603) becomes an area 1610 in which the strobe light travels as appropriate light on the image.

  The system control unit 101 determines whether the difference area 1401 between the non-flash image for screen display 1301 and 1302 and the difference area 1402 between the non-flash image for screen display 1302 and 1303 are included in the area 1610. In the example illustrated in FIGS. 14 and 16, the difference area 1401 between the non-flash image for screen display 1301 and 1302 and the difference area 1402 between the non-flash image for screen display 1302 and 1303 are not included in the area 1610. In this case, the system control unit 101 does not calculate the motion vectors 1403 and 1404, the prediction vector 1405, and the motion difference region prediction result 1407. Then, the system control unit 101 extracts an area corresponding to the area 1610 in the difference information 1414 as a main subject replacement area 1408. As described above, it is possible to eliminate the need to perform the motion prediction of the subject with respect to the motion outside the area 1610, so that the calculation processing load can be reduced.

  On the other hand, when the difference area between the non-flash images for screen display is included in the area 1610, the system control unit 101 causes the motion vector and the prediction vector to be the difference area as described in the second embodiment. , And calculation of the motion difference area prediction result. Then, as described in the first and second embodiments, the system control unit 101 identifies the motion area in the area 1610 based on the motion difference area prediction result, and mainly determines the area excluding the motion area from the area 1610. A subject replacement area 1408 is extracted.

  According to the above-described processing, the image processing area for calculating the motion vector is narrowed, so that the calculation processing load can be reduced. Furthermore, it is possible to prevent the area outside the proper light emitting distance of the flash image from being synthesized, and the flash image can be replaced by focusing on the effective area of the strobe light.

Also, not the depth information 1602 but only the area near the distance from the light emission center position of the strobe light as the position information of the image is set as the area where the flash image is synthesized as the comparison processing area of the flash image and the non-flash image. Also good. This becomes an area that becomes less noticeable as it moves away from the flash light emission center position, so the area to be replaced with the flash image can be limited, and the processing load can be further reduced.
In the present embodiment, the form (the form using the non-flash image for screen display) has been described as an example based on the second embodiment, but the form (the non-flash image according to the main photographing) according to the first embodiment In the form of using. Also in the present embodiment, the modification can be applied to the first and second embodiments.

  The embodiments described above are merely examples of implementation for carrying out the present invention, and the technical scope of the present invention should not be interpreted in a limited manner by these. That is, the present invention can be implemented in various forms without departing from the technical concept or the main features thereof.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the aforementioned embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

  100: digital camera, 101: system control unit, 113: strobe

Claims (19)

An acquisition unit configured to acquire a light emission image captured in a state in which the strobe light is emitted and a non-emission image captured in a state in which the strobe light is not emitted;
An image processing apparatus, comprising: detection means for detecting an area of a subject having a motion in the light emission image based on a change in pixel information of the plurality of non-light emission images.   The image processing apparatus according to claim 1, further comprising a combination unit configured to combine the light emission image and an image based on the non-light emission image.   The image processing apparatus according to claim 2, further comprising a generation unit configured to generate an image obtained by combining the light emission image and the plurality of non-light emission images as an image based on the non-light emission image.   The generation means combines an image serving as a reference among the light emission image and the plurality of light emission images, and an image obtained by combining the light emission image and an image different from the image serving as the reference among the plurality of light emission images The method according to claim 3, wherein combining the reference image and an image not synthesized among the different images is repeated until the light emission image and the plurality of light emission images are combined. Image processing device.   The generation means is configured to generate an image which has already been synthesized according to a difference between pixel information of the reference image and pixel information of an image different from the reference image, and an image different from the reference image. The image processing apparatus according to claim 4, wherein a ratio of combining is determined.   The detection means derives a first difference image which is an image showing a difference between pixel information of the light emission image and pixel information of the non-light emission image of one of the plurality of non-light emission images; Deriving at least one second difference image, which is an image indicating a difference between pixel information of two non-emission images among the plurality of non-emission images, and, based on the second difference image, The image processing apparatus according to any one of claims 1 to 5, wherein at least detecting an area of a subject having a motion is performed from the difference image of. The light emission image further includes an adjustment unit that adjusts the brightness of the light emission image so that the brightness of the region of the background of the light emission image approaches the brightness of the non-light emission image.
The detection means is an image showing the difference between the pixel information of the light emission image whose brightness has been adjusted by the adjustment means and the pixel information of the non-light emission image of one of the plurality of non-light emission images. 7. The image processing apparatus according to claim 6, wherein the image processing apparatus is derived as a difference image of.   The detection means derives a motion vector indicating a motion of an object in the two non-emission images based on a difference between pixel information of the two non-emission images, and the emission image is generated in the emission image based on the derived motion vector The image processing apparatus according to any one of claims 1 to 7, wherein an area of a moving subject is detected.   The detection means is an area of the two non-emission images, and the subject in the two non-emission images according to the size of the area where the pixel information of the two non-emission images has a difference exceeding the threshold. The image processing apparatus according to claim 8, wherein a range in which a motion vector indicating a motion is detected is changed.   The detection means derives, based on the motion vector, a prediction vector for predicting the movement of the subject between the light emission image and the non-light emission image of one of the plurality of non-light emission images, and the prediction vector The image processing apparatus according to claim 8, wherein an area of a subject having a motion is detected in the light emission image based on the image.   The detection means is based on a first difference image indicating a difference between pixel information of the light emission image and pixel information of the non-light emission image of one of the plurality of non-light emission images, and the prediction vector. 11. The image processing apparatus according to claim 10, wherein an area of a subject having a motion in the light emission image is detected.   The image processing apparatus according to any one of claims 1 to 11, wherein the number of the plurality of non-light emitting images is three or more.   The image processing apparatus according to any one of claims 1 to 12, further comprising: alignment means for aligning the positions of the plurality of non-emission images. The non-emission image is an image to be displayed on a screen,
The image processing apparatus according to any one of claims 1 to 13, wherein the light emission image is an image that is actually photographed based on a photographing instruction.   The image processing apparatus according to any one of claims 1 to 12, wherein the non-light emitting image and the light emitting image are images that have been captured based on a shooting instruction.   The detection means is an area of a subject having a motion in the light emission image based on information indicating a light emission distance of the strobe light from the imaging device and information indicating a distance from the imaging device of the subject in the non-light emission image. The image processing apparatus according to any one of claims 1 to 15, wherein a range in which to detect is determined.   The image processing apparatus according to any one of claims 1 to 16, wherein the pixel information is information indicating brightness or a color difference. Acquiring an emission image captured in a state in which the strobe light is emitted and a non-emission image captured in a state in which the strobe light is not emitted;
A detection step of detecting an area of a subject having a motion in the light emission image based on a change in pixel information of the plurality of non-light emission images.   The program for functioning a computer as each means of the image processing apparatus in any one of Claims 1-17.