JP6115815B2 - Composite image generation apparatus and composite image generation method - Google Patents

Description

  The present invention relates to a composite image generation apparatus and a composite image generation method capable of generating a strobe image in which a movement trajectory of a subject is stored in a single image by combining a plurality of images.

  Technology that generates a strobe image that captures the movement trajectory of a subject in a single image by continuously shooting a moving subject with a fixed imaging device and combining multiple images obtained by continuous shooting It has been known. This type of technology is disclosed in Patent Document 1, for example.

  In Patent Document 1, a candidate for a video object (subject) is extracted from a frame image, and the position of a subject to be tracked is extracted based on a preset extraction condition from the extracted subject candidate images. In addition, a configuration is described in which a strobe image is obtained by obtaining a trajectory image in which only the obtained movement trajectory of the subject is created, and a trajectory image generated by combining the generated trajectory image and a frame image. Examples of extraction conditions include area, luminance, color, aspect ratio, and circularity.

JP 2005-123824 A

  In Patent Document 1, for example, in order to extract an arbitrary subject, it is necessary to appropriately set extraction conditions. Specifically, the user needs to select an item (for example, luminance) suitable for extraction of an arbitrary subject and set a value (for example, luminance value) suitable for extraction of the subject for the selected item. However, selecting an item suitable for extraction of an arbitrary subject and setting a value are not easy operations for the user because subject extraction accuracy varies depending on the type of item to be selected and the set value. For example, as the number of extraction condition items decreases, the subject extraction accuracy deteriorates, the probability that a tracking target subject is extracted decreases, and the probability that a subject that is not a tracking target is extracted increases. On the other hand, when the number of extraction condition items is increased, the accuracy of subject extraction is improved, the probability that a subject to be tracked will be increased, and the probability that a subject that is not a target will be extracted will decrease. However, in this case, if the combination of items is appropriately selected and the setting value of each item is not appropriately set in consideration of the combination of items, the subject to be tracked may be excluded from the extraction target. In addition, another problem arises that the processing operation becomes more complicated as the number of items increases. Therefore, the number of items cannot be easily increased even when it is desired to improve the extraction accuracy of the subject. As described above, in Patent Document 1, it is difficult to set an appropriate extraction condition for an arbitrary subject. Therefore, it is unsuitable for generating a strobe image with an arbitrary subject as a tracking target.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a composite image generation apparatus and an image generation method suitable for generating a strobe image with an arbitrary subject as a tracking target. That is.

  A composite image generating apparatus according to an aspect of the present invention is specified by a display unit that displays a captured image on a predetermined display screen, and a range specifying unit that allows a user to specify a range in the captured image displayed on the display screen. A subject registration unit for registering a subject included in the range, a subject detection unit for detecting a subject that can be regarded as the same as the registered subject registered by the subject registration unit from each of a plurality of captured images, and a subject detection Image synthesizing means for synthesizing the plurality of captured images.

  The range designating unit displays a predetermined frame in a superimposed manner on the captured image displayed on the display screen, changes the range in the captured image surrounded by the frame according to the user's operation, and It is good also as a structure which makes a user designate the range of this. For example, the shape and size of the frame and the position in the captured image are changed according to the user's operation.

  Further, the composite image generating apparatus includes a composite ratio setting unit that sets different composite ratios for a subject region in which a subject is detected by the subject detection unit and a non-subject region other than the subject region, in the captured image. It is good also as a structure. In this case, the image synthesizing unit weights each of the captured images to be synthesized based on the synthesis ratio set by the synthesis ratio setting unit, and synthesizes the plurality of weighted captured images.

  For example, when the composition ratio set in the subject area is defined as the first composition ratio and the composition ratio set in the non-subject area is defined as the second composition ratio, the composition ratio setting unit The composition ratio is set so as to continuously change from the first composition ratio to the second composition ratio in the vicinity of the boundary with the non-subject region.

  Further, the composition ratio setting means may be configured to set a composition ratio lower than that of the subject area for the non-subject area.

  Further, the composition ratio setting means may be configured to set a composition ratio of 1 / N times the subject area with respect to the non-subject area when the number of photographed images synthesized by the image composition means is defined as N. .

  In addition, when there are two or more photographed images whose subject areas overlap each other when they are synthesized by the image synthesizing unit, the synthesis ratio setting unit 2 prevents the subject from being overlapped and displayed in the synthesized image. A composition ratio of subject areas in two or more captured images may be set.

  The composition ratio setting means sets the composition ratio of the subject area in one captured image among two or more captured images to a positive value, and the composition ratio of the subject areas in the captured image other than the one captured image. It is good also as a structure which sets to zero.

  According to another aspect of the present invention, there is provided a composite image generating apparatus that specifies a captured image selection unit that allows a user to select one captured image from a plurality of captured images, and a range within the selected captured image. Range registration means, subject registration means for registering subjects included in the designated range, and photographed images not selected by the photographed image selection means are regarded as the same as the registered subject registered by the subject registration means. A subject detection unit that detects a subject that can be detected, and an image synthesis unit that combines the captured image selected by the captured image selection unit and the captured image from which the subject is detected by the subject detection unit.

  Moreover, the composite image generation method according to an aspect of the present invention includes a display step of displaying a captured image on a predetermined display screen, a range specifying step of allowing a user to specify a range within the captured image displayed on the display screen, A subject registration step for registering a subject included in a specified range; a subject detection step for detecting a subject that can be regarded as the same as the registered subject registered in the subject registration step from each of a plurality of captured images; And an image synthesis step of synthesizing a plurality of captured images in which the subject is detected.

  According to another aspect of the present invention, there is provided a method for generating a composite image in which a user selects a captured image selection step from which a user selects one captured image from a plurality of captured images, and specifies a range within the selected captured image. The same as the registered subject registered in the subject registration step from each of the range designation step to be performed, the subject registration step for registering subjects included in the designated range, and the photographed image not selected in the photographed image selection step A subject detection step for detecting a subject that can be regarded as a subject, and an image synthesis step for synthesizing the captured image selected in the captured image selection step and the captured image in which the subject is detected in the subject detection step. .

  According to the composite image generation apparatus and the composite image generation method according to the present invention, the user can directly specify an arbitrary subject from the captured image, and the subject that is regarded as the same as the subject specified by the user from other captured images Is detected. Therefore, a strobe image tracking an arbitrary subject can be obtained with high accuracy.

It is a block diagram which shows the structure of the imaging device of embodiment of this invention. It is a figure which shows the imaging | photography and production | generation flow of the flash image of embodiment of this invention. 3 is a flowchart showing a process step S1 (subject registration) of FIG. It is a figure which shows the example of a display screen displayed on LCD (Liquid Crystal Display) of an imaging device at the time of execution of subject registration mode. It is a figure for demonstrating process step S4 (object detection) of FIG. It is a flowchart which shows process step S5 (setting of a synthetic | combination ratio) of FIG. It is a figure which shows notionally an initial composition ratio arrangement | sequence and a region label. It is a figure which shows notionally the synthetic | combination ratio arrangement | sequence and area | region label with respect to the picked-up image of the 1st frame. It is a figure which shows notionally the synthetic | combination ratio arrangement | sequence and area | region label with respect to the picked-up image of the 2nd frame. It is a figure which shows notionally the synthetic | combination ratio arrangement | sequence and area | region label with respect to the picked-up image of the 2nd frame. It is a figure for demonstrating the production | generation of a flash image. It is a figure for demonstrating the production | generation of a flash image. It is a figure which shows notionally the aspect that a synthetic | combination ratio changes continuously in the vicinity of the boundary of the to-be-tracked object and its surrounding area.

  Hereinafter, a composite image generation apparatus according to an embodiment of the present invention will be described with reference to the drawings. In the present embodiment, the composite image generation device is mounted on a photographing device capable of photographing a digital image. The photographing device is, for example, a digital single lens reflex camera (with a quick return mirror) or a mirrorless single lens reflex camera with a replaceable photographing lens, a compact digital camera, a camcorder, a mobile phone terminal, a PHS (Personal Handy phone System), a smartphone, Feature phones, portable game machines, etc.

[Configuration of the photographing apparatus 1]
FIG. 1 is a block diagram showing a configuration of the photographing apparatus 1 of the present embodiment. As shown in FIG. 1, a photographing apparatus 1 includes a central processing unit (CPU) 100, a digital signal processor (DSP) 102, an operation unit 104, a photographing lens 106, a diaphragm 108, a shutter 110, an image sensor 112, and a diaphragm / shutter. A drive circuit 114, a ROM (Read Only Memory) 116, a memory card adapter 118, and an LCD 120 are provided.

  The operation unit 104 includes various switches necessary for the user to operate the photographing apparatus 1, such as a power switch, a release switch, and a mode setting switch. When the user presses the power switch, power is supplied from the battery (not shown) to the various circuits of the photographing apparatus 1 through the power line. After supplying power, the CPU 100 accesses the ROM 116 to read out a control program, loads it into a built-in memory (not shown), and executes the loaded control program to control the entire photographing apparatus 1. In FIG. 1, for the sake of clarity, the connection between the CPU 100 and other circuits is omitted.

  The DSP 102 includes an aperture 108 and an aperture 108 via an aperture / shutter drive circuit 114 so that an appropriate exposure can be obtained based on a photometric value measured by a TTL (Through The Lens) exposure meter (not shown) built in the photographing apparatus 1. The shutter 110 is driven and controlled. More specifically, drive control of the aperture 108 and the shutter 110 is performed based on an AE function designated by a mode setting switch, such as a program AE (Automatic Exposure), a shutter speed priority AE, and an aperture priority AE. The DSP 102 performs AF (Autofocus) control together with AE control. An active method, a phase difference detection method, a contrast detection method, and the like are applied to the AF control. Since the configuration and control of this type of AE and AF are well known, detailed description thereof is omitted here.

  The light flux from the subject passes through the photographing lens 106, the diaphragm 108, and the shutter 110 and is received by the image sensor 112. The image sensor 112 is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and accumulates an optical image formed by each pixel on the imaging surface as a charge corresponding to the amount of light, Convert to electrical signal. The DSP 102 performs predetermined signal processing such as color interpolation, matrix calculation, and Y / C separation on the electrical signal (photographing data) input from the image sensor 112 to generate a luminance signal Y and color difference signals Cb and Cr. , And compressed in a predetermined format such as JPEG (Joint Photographic Experts Group). The compressed image signal (photographed image data) is stored in the memory card 200 inserted into the memory card adapter 118. Further, the DSP 102 buffers the signal after Y / C separation in a frame memory (not shown) in units of frames. The DSP 102 sweeps the buffered signal from each frame memory at a predetermined timing, converts it into an image signal, generates an image, and displays it on the LCD 120. The user can view through the LCD 120 a real-time through image captured with appropriate brightness and focus based on AE control and AF control.

[Shooting and Generation of Strobe Image by Shooting Apparatus 1]
Next, shooting of a strobe image by the shooting apparatus 1 will be described. FIG. 2 is a diagram showing a strobe image capturing and generating flow. Shooting of a strobe image is normally performed in a state where the photographing apparatus 1 is fixed to a tripod or the like in order to prevent blurring of a background image or the like.

[S1 in FIG. 2 (Subject Registration)]
The DSP 102 includes a subject registration unit 102a. The subject registration unit 102a can register a subject to be tracked. FIG. 3 shows a flowchart for explaining this processing step S1 (subject registration) in more detail. As shown in FIG. 3, the subject registration unit 102a transitions to the subject registration mode in accordance with the operation of the operation unit 104 by the user (S101 in FIG. 3). 4A to 4C show examples of display screens displayed on the LCD 120 when the subject registration mode is executed.

  When the subject registration unit 102a transitions to the subject registration mode, the frame A is superimposed on the through image displayed on the LCD 120 as shown in FIG. 4A (S102 in FIG. 3). In the example of FIG. 4A, the frame A is displayed at the center of the screen. For example, the user can fit the subject B to be tracked in the frame A displayed in the center of the screen by adjusting the orientation of the photographing apparatus 1. The user can also move the frame A to an arbitrary position in the screen by operating the operation unit 104. The user can also place the subject B to be tracked in the frame A by moving the frame A within the screen. There are various sizes and shapes of the subject to be tracked. Therefore, when the subject is large, the subject may not fit within the frame A. If the subject is too small with respect to the frame A, a subject other than the tracking target enters the area between the subject to be tracked and the frame A. Even when the outer contour of the subject and the shape of the frame A are greatly different, subjects other than the tracking target tend to enter the area between the subject to be tracked and the frame A. Therefore, as illustrated in FIG. 4A, the size of the frame A can be enlarged or reduced by operating the operation unit 104. 4B, the shape of the frame A can be changed to a shape other than a rectangle (such as a circle, an ellipse, or a polygon) by operating the operation unit 104. The aspect ratio can be changed. The shape of the frame A may be determined by free form.

  When the registration button of the operation unit 104 is pressed by the user, the subject registration unit 102a cuts out the range in the photographed image surrounded by the frame A as shown in FIG. (S103 in FIG. 3). The predetermined message is “Register this subject”. The options are “OK” and “Cancel”. The user can confirm whether or not the subject B to be tracked can be selected by viewing the display on the LCD 120. The user selects “OK” when the subject B to be tracked can be selected, and selects “Cancel” when the subject B to be tracked cannot be selected. When “OK” is selected (S104: YES in FIG. 3), the subject registration unit 102a registers the subject B surrounded by the frame A (stores image information and coordinate information) (S105 in FIG. 3). Hereinafter, the subject registered in the subject registration unit 102a is referred to as “registered subject”. As a result, the subject registration mode ends, and the mode of the photographing apparatus 1 returns to the mode immediately before the transition to the subject registration mode. When “cancel” is selected (S104 in FIG. 3: NO), the subject registration unit 102a returns the process to the processing step S102 in FIG.

[S2 in FIG. 2 (Standby for Shooting Operation)]
The DSP 102 waits for an input of a strobe image photographing operation by the user. When the release switch of the operation unit 104 is pressed (S2: YES in FIG. 2), the DSP 102 proceeds to the processing step S3 (image shooting) in FIG.

[S3 in FIG. 2 (Image Capture)]
The DSP 102 controls the aperture 108, the shutter 110, and the image sensor 112 to photograph the subject, and processes the photographing data input from the image sensor 112 to generate an image. The photographing process is performed using a well-known continuous shooting function. With the continuous shooting function, for example, a plurality of images are taken per second while the release switch is being pressed. The continuous shooting speed can be appropriately changed through the operation of the operation unit 104. The continuous shooting period may be, for example, until a predetermined time elapses after the release switch is pressed, or until a predetermined condition is satisfied after the release switch is pressed (for example, described below). Processing step S4 (until no subject is detected from the photographed image in subject detection) may be used.

[S4 in FIG. 2 (Subject Detection)]
The DSP 102 includes a subject detection unit 102b. Each time an image is captured and generated by the continuous shooting function in the processing step S3 (image capturing) in FIG. 2, the captured image is transferred to the subject detection unit 102b. The subject detection unit 102b detects a subject that can be regarded as the same as the registered subject registered in the subject registration unit 102a from the transferred captured image. Hereinafter, a subject that can be regarded as the same as the registered subject is referred to as a “tracking subject”. The subject detection unit 102b stores the coordinates of the detected tracking target subject. More precisely, the subject detection unit 102b stores the coordinates of a range that is the smallest rectangular range that includes the entire tracking target subject and that substantially includes only the tracking target subject. The “minimum rectangular range” is replaced with “minimum circular range”, “minimum elliptical range”, “minimum polygonal range”, etc. according to the setting of the frame A. Hereinafter, such a range is referred to as a “tracking target range”. The tracking target subject is detected using a known method such as a template matching method, a condensation method, or a particle filter method. In this embodiment, the registered subject is a bird as shown in FIG. Because birds fold or spread their wings, their shape changes on the captured image. Further, the size of the captured image also changes depending on the distance. However, when the Condensation method, particle filter method, etc. are used, even when the shape or size of the captured image changes due to the bird being tracked folding or spreading its wings or flying away, these birds are regarded as the same. be able to. Therefore, a change in the shape and size of the subject in each captured image does not cause a substantial problem in detecting the tracking target subject. In addition, an exemplary description of the Condensation method can be referred to in the following document.
(Literature explaining the Condensation method by way of example)
Contour tracking by stochastic propagation of conditional density
Michael Isard and Andrew Blake
Proc. European Conference on Computer Vision, vol. 1, pp. 343--356, Cambridge UK, (1996).

FIG. 5 is a diagram for explaining the processing step S4 (subject detection). 4 in the left column of FIG. 5 shows captured images I _{1 to} I ₄ , and FIG. 4 in the right column of FIG. 5 shows tracking target ranges I ₁ ′ to I ₄ ′ in each of the captured images I _{1 to} I _4. Show. In the example of FIG. 5, the tracking target subject (bird) is detected from each of the captured images I _{1 to} I ₄ by executing this processing step S4 (subject detection), and the detected tracking target subject (bird) is selected. The coordinates of the tracking target range I ₁ ′ to I ₄ ′ are stored. The captured images I ₃ and I ₄ include a car as a dynamic subject. However, since the car is not a registered subject, it is not detected in this processing step S4 (subject detection).

  As described above, in this embodiment, the subject detection unit 102b automatically detects the tracking target subject. However, in another embodiment, the user individually selects the tracking target subject from each captured image by manual operation. You may do it.

[S5 in FIG. 2 (Setting of Composite Ratio)]
When the DSP 102 completes the shooting and generation of images by the continuous shooting function and the detection processing of the tracking target subject for each generated captured image, the DSP 102 sets the composition ratio of the captured images from which the tracking target subject is detected. The composition ratio is a weighting coefficient when composing each captured image. FIG. 6 shows a flowchart of the processing step S5 (setting of the composition ratio).

  Here, the number of frames of the captured image in which the tracking target subject is detected is defined as N, and a composite ratio array indicating the composite ratio for each pixel of the captured image of the n (= 1 to N) frame is represented by Wn (x, y ), And the region label used to avoid the overlapping of the tracking target subjects is defined as L (x, y), and within the composition ratio array Wn (x, y) and the region label corresponding to the tracking target range A region in L (x, y) is defined as Rn (x, y). “Overlapping tracking target subjects” means that when a plurality of frames of captured images are combined, each other's tracking target subjects (more precisely, the tracking target range) overlap on the strobe image. The composite ratio array Wn (x, y) and the area label L (x, y) have areas corresponding to the respective pixels of the captured image, and have the same size (area corresponding to the same pixel array) as the captured image. Have. One composite ratio array Wn (x, y) is prepared for each frame of captured images. On the other hand, the region label L (x, y) is used in common for the captured images of all frames, and there is only one.

  The DSP 102 includes a composition ratio setting unit 102c. The composition ratio setting unit 102c sets the initial value (= 1 / N) of the composition ratio (weighting factor) for all the regions of the composition ratio array Wn (x, y) corresponding to the captured image of each frame ( S201 in FIG. 6). The composition ratio setting unit 102c also sets a value of zero for all regions of the region label L (x, y). FIG. 7A shows a conceptual image of the composition ratio array Wn (x, y) in which the composition ratio (initial value 1 / N) is set for all regions. FIG. 7B shows a conceptual image of a region label L (x, y) in which values (initial values of zero) are set for all regions.

  When each of the N frames of the captured images is multiplied by the composite ratio 1 / N set in the composite ratio array Wn (x, y) and the captured images of all the frames (N frames) after the multiplication are combined, The total value of the composition ratio is 1 in each pixel of the image (flash image). For this reason, with respect to a subject in which all N frames overlap (a subject with a fixed background or the like), a luminance reproducibility equivalent to that of a normal image is realized in the strobe image. On the other hand, since the tracking target subject moves between frames, it basically does not overlap during image synthesis. For example, when the composition ratio is set to a low value such as 1 / N for the pixel corresponding to the tracking target subject of the first frame, the luminance of the tracking target subject of the first frame appearing in the strobe image is higher than that of the normal image. It becomes quite low.

Therefore, the composition ratio setting unit 102c sets the composition ratio set in the region R ₁ (x, y) of the composition ratio array W ₁ (x, y) corresponding to the captured image I ₁ of the first frame to an initial value of 1. The value is changed from / N to the value val (S202 in FIG. 6). The value val is a value set for convenience, for example, 10000, for example, a synthesis ratio (here, 1 / N) set in a region other than the region R ₁ (x, y) is so large that it can be substantially ignored. It is a positive value. Note that the value val is reset to 1, for example, when the captured image is synthesized. In this manner, the composition ratio setting unit 102c ensures the luminance of the tracking target subject in the first frame on the strobe image by setting the composition ratio of the region R ₁ (x, y) to a high value. Note that the composition ratio set in the region other than the region R ₁ (x, y) remains the initial value 1 / N. FIG. 8A shows the captured image I ₁ and tracking target range I ₁ ′ of the first frame. FIG. 8B shows a conceptual image of the composition ratio array W ₁ (x, y) in which the composition ratio of the region R ₁ (x, y) is changed to the value val.

The composition ratio setting unit 102c updates the value set in the region R ₁ (x, y) of the region label L (x, y) from the initial value zero to 1 (S203 in FIG. 6). A region where the value is set to 1 in the region label L (x, y) indicates a region where the tracking target subject (more precisely, the tracking target range) is planned to be arranged in the strobe image. Hereinafter, an area whose value is set to 1 in the area label L (x, y) is referred to as a “tracking target arrangement area”. Note that values set in regions other than the region R ₁ (x, y) remain at the initial value zero. FIG. 8C shows a conceptual image of the region label L (x, y) in which the value of the region R ₁ (x, y) is updated.

  The composition ratio setting unit 102c determines whether or not “overlapping subject to be tracked” occurs between the captured image of the next frame and the captured image of the previous frame (S204 in FIG. 6). Specifically, the composition ratio setting unit 102c determines that the tracking target subject area and the area R (x, y) of the next frame overlap at least partially in the area label L (x, y). It is determined that “overlap” occurs (S204 in FIG. 6: YES). The composition ratio setting unit 102c determines that the “tracking target subject overlap” does not occur when the tracking target arrangement region and the region R (x, y) of the next frame do not overlap at all (S204 of FIG. 6: NO). ).

Here, taking the second frame as an example, the “overlapping subject to be tracked” determination process will be described. In this example, the area R ₁ (x, y) of the first frame is the tracking target arrangement area. The composition ratio setting unit 102c determines whether or not the tracking target arrangement region and the region R ₂ (x, y) of the second frame overlap even partly in the region label L (x, y).

FIG. 9A shows an example of the captured image I ₂ and tracking target range I ₂ ′ of the second frame. In the example of FIG. 9A, the tracking target range I ₁ ′ and the tracking target range I ₂ ′ partially overlap each other. In this case, the tracking target arrangement region and the region R ₂ (x, y) of the second frame partially overlap even within the region label L (x, y). Therefore, the inclusion of the tracking target subject included in the captured image I ₂ on the stroboscopic image "overlapping tracked subject" occurs. The composition ratio setting unit 102c determines that the tracking target arrangement region and the region R ₂ (x, y) of the second frame overlap (S204 of FIG. 6: YES), and proceeds to the processing step S205 of FIG.

In processing step S205 of FIG. 6, the composition ratio setting unit 102c performs region R on the composition ratio array W ₂ (x, y) corresponding to the captured image I ₂ of the second frame, as shown in FIG. 9B. ₂ The composition ratio of (x, y) is changed. Specifically, the composition ratio setting unit 102c changes the composition ratio set in the region R ₂ (x, y) of the composition ratio array W ₂ (x, y) from the initial value 1 / N to zero. In this way, the composition ratio setting unit 102c sets the composition ratio of the region R ₂ (x, y) to zero, thereby causing the “tracking target object” to cause “tracking target object overlap”. Is not displayed on the strobe image.

The composition ratio setting unit 102c does not update the region label L (x, y) as shown in FIG. 9C because the tracking target subject of the second frame is not arranged in the strobe image (the second frame). Region R ₂ (x, y) is not added as a tracking target placement region).

FIG. 10A shows an example of the captured image I ₂ and tracking target range I ₂ ′ of the second frame. In the example of FIG. 10A, the tracking target range I ₁ ′ and the tracking target range I ₂ ′ do not overlap. In this case, the tracking target arrangement area does not overlap with the area R ₂ (x, y) in the second frame even in the area label L (x, y). Therefore, "overlapping tracked subject" does not occur even including tracked object included in the captured image I ₂ on the stroboscopic image. The composition ratio setting unit 102c determines that the tracking target arrangement region and the region R ₂ (x, y) in the second frame do not overlap (S204 of FIG. 6: NO), and proceeds to the processing step S206 of FIG. .

In the processing step S206 of FIG. 6, similarly to the first frame, the composition ratio setting unit 102c performs the region R ₂ (x, y) on the composition ratio array W ₂ (x, y) corresponding to the captured image I ₂ of the second frame. The composition ratio set in y) is changed from the initial value 1 / N to the value val. In this way, the composition ratio setting unit 102c ensures the luminance of the tracking target subject in the second frame on the strobe image by setting the composition ratio of the region R ₂ (x, y) to a high value. FIG. 10B shows a conceptual image of the composition ratio array W ₂ (x, y) in which the composition ratio of the region R ₂ (x, y) is changed.

Synthesis rate setting unit 102c also, as shown in FIG. 10 (c), the region _R 1 (x, y) region labels tracked disposed region L (x, y) for the region _R 2 (x, y ) Is updated from the initial value zero to 1. The composition ratio setting unit 102c refers to the updated region label L (x, y), and thereby sets the region on the strobe image corresponding to the region R ₁ (x, y) and the region R ₂ (x, y). It is possible to grasp that the tracking target subject (more precisely, the tracking target range) is arranged.

  The composition ratio setting unit 102c determines whether or not the processing step S205 or S206 of FIG. 6 has been executed for all captured images (N frames) in which the tracking target subject is detected (S207 of FIG. 6). When the composition ratio setting unit 102c determines that the processing step S205 or S206 of FIG. 6 has been executed for all captured images (N frames) (S207: YES in FIG. 6), the processing step S6 of FIG. The process proceeds to (Composition). If the composition ratio setting unit 102c determines that there are still frames for which the processing step S205 or S206 in FIG. 6 has not been executed (S207: NO in FIG. 6), the process returns to the processing step S204 in FIG. Processes subsequent to step S204 in FIG. 6 are executed for the captured image of the frame.

  In the process shown in the flowchart of FIG. 6, when “overlapping subject to be tracked” occurs, the subject to be tracked of the previously captured image is arranged with priority over the strobe image. In another embodiment, the tracking target subject of an image captured later may be arranged with priority over the strobe image, and the user may arbitrarily select the tracking target subject arranged in the strobe image. You may be able to do it.

[S6 in FIG. 2 (Composition of Captured Image)]
The DSP 102 includes an image composition unit 102d. The image composition unit 102d performs RGB luminance value i _R (N) of the captured image of N frames corresponding to the composition ratio array Wn (x, y) for which the composition ratio is set in the processing step S5 (composition ratio setting) in FIG. x, y), i _G (x, y), i _B (x, y) are synthesized by a weighted average with respect to the synthesis ratio array W _n (x, y). That is, when the strobe image is defined as I _R (x, y), I _G (x, y), I _B (x, y), the image composition unit 102d generates a strobe image according to the following equation.

  The image composition unit 102d displays the generated strobe image on the LCD 120 so that the user can visually recognize the generated strobe image. After displaying the strobe image on the LCD 120, the image composition unit 102d stores the strobe image in the memory card 200 in accordance with the operation of the operation unit 104 by the user. The user can visually recognize the strobe image displayed on the LCD 120 and adjust the strobe image when, for example, the subject cannot be tracked or combined as desired. As an example, the user can finely adjust the position and size of the tracking target range in the captured image of each frame automatically detected in the processing step S4 (subject detection) in FIG. The composition ratio setting unit 102c and the image composition unit 102d perform setting of the composition ratio and composition of the captured image using the captured image whose tracking target range is finely adjusted. As a result, the user can obtain a strobe image that further meets his / her desire.

11 and 12 are diagrams for explaining the generation of the strobe image according to the present embodiment. FIG. 5 in the left column of FIG. 11 shows captured images I _{1 to} I ₅ to be combined. FIG. 5 in the center column of FIG. 11 shows the composition ratio arrays W ₁ (x, y) to W ₅ (x, y) corresponding to the captured images I _{1 to} I ₅ . FIG. 5 in the right column of FIG. 11 shows the region label L (x, y) that is updated each time the processing step S205 or S206 of FIG. 6 is executed for the captured images I _{1 to} I ₅ . In FIG. 11, for convenience of explanation, the captured image to be combined is assumed to be 5 frames. However, the number of frames of the captured image to be synthesized is usually much larger than 5 frames.

In the example of FIG. 11, for the first frame, the composition ratio set in the region R ₁ (x, y) of the composition ratio array W ₁ (x, y) is changed from the initial value 1 / N to the value val. At the same time, the region R ₁ (x, y) is set as the tracking target placement region in the region label L (x, y). In the second frame, it is determined that the region R ₂ (x, y) overlaps the tracking target placement region. Therefore, the composition ratio set in the region R ₂ (x, y) of the composition ratio array W ₂ (x, y) is changed from the initial value 1 / N to zero. However, the area R ₂ (x, y) is not added as the tracking target arrangement area in the area label L (x, y) (that is, the area label L (x, y) is not updated). In the third frame, the composition ratio set in the region R ₃ (x, y) of the composition ratio array W ₃ (x, y) is changed from the initial value 1 / N to the value val, and the region label L ( A region R ₃ (x, y) is added as a tracking target arrangement region in x, y). In the fourth frame, it is determined that the region R ₄ (x, y) overlaps the tracking target placement region. Therefore, the composition ratio set in the region R ₄ (x, y) of the composition ratio array W ₄ (x, y) is changed from the initial value 1 / N to zero. However, the region R ₄ (x, y) is not added as the tracking target arrangement region in the region label L (x, y) (that is, the region label L (x, y) is not updated). In the fifth frame, the composition ratio set in the region R ₅ (x, y) of the composition ratio array W ₅ (x, y) is changed from the initial value 1 / N to the value val, and the region label L ( A region R ₅ (x, y) is added as a tracking target arrangement region in x, y). As a result, in the example of FIG. 11, the captured images of the next five frames are synthesized.
・ Photographed image I ₁
The composition ratio of the tracking target range I ₁ ′ is the value val, and the composition ratio 1 / N of the other image area
・ Image I ₂
The composition ratio of the tracking target range I ₂ ′ is zero, and the composition ratio 1 / N of other image areas
・ Photograph I ₃
The composition ratio of the tracking target range I ₃ ′ is the value val, and the composition ratio 1 / N of the other image area
・ Photograph I ₄
The composition ratio of the tracking target range I ₄ ′ is zero, and the composition ratio 1 / N of other image areas
・ Photograph I ₅
The composition ratio of the tracking target range I ₅ ′ is the value val, and the composition ratio 1 / N of the other image area

  In this way, in the example of FIG. 11, the tracking target subject (bird) in the first frame and the tracking target subject (bird) in the second frame overlap, so the composition ratio is set for the tracking target subject (bird) in the second frame. By setting it to zero, the tracking target subject (bird) in the second frame is prevented from appearing in the strobe image. Further, since the tracking target subject (bird) in the third frame and the tracking target subject (bird) in the fourth frame overlap each other, the synthesis ratio is set to zero for the tracking target subject (bird) in the fourth frame. The eye tracking subject (bird) is prevented from appearing in the strobe image.

FIG. 12 shows a stroboscopic image obtained by synthesizing the five frames of the captured image. The composition ratio is zero for the tracking subject (bird) in the second frame and the fourth frame. Therefore, as shown in FIG. 12, the subject to be tracked (birds) for three frames (1, 3, 5 frames) that do not overlap each other appears in the strobe image. The captured images I ₃ and I ₄ include a car as a dynamic subject. However, the composition ratio is set to 1 / N for the pixels in which the automobile is shown. The composition ratio is set to a low value for automobiles (generally, N is much larger than 5 and 1 / N is much lower than 1). Does not appear substantially (or does not stand out if it appears). As described above, according to the present embodiment, “overlapping of tracking target subjects” can be avoided, and even when there are a plurality of moving subjects, the movement trajectory of the tracking target subjects can be stored in the strobe image while The movement trajectory of the subject can be excluded from the strobe image (that is, the arbitrary subject selected by the user can be traced reliably).

In some cases, it is more convenient for the tracking target subjects to overlap each other in the strobe image (for example, when it is desired to visually recognize the pitcher's ball). In this case, the user can allow “overlapping of tracking target subjects” through the operation of the operation unit 104. Specifically, the composition ratio setting unit 102c does not use the region label L (x, y) when executing the processing step S5 (setting of the composition ratio) in FIG. 2, and the composition ratio array W _n (x , Y) for the region R _n (x, y), the synthesis ratio is uniformly set to the value val. As a result, the strobe image generated by the image composition unit 102d is allowed to overlap the tracking subject.

  The above is the description of the exemplary embodiments of the present invention. Embodiments of the present invention are not limited to those described above, and various modifications are possible within the scope of the technical idea of the present invention. For example, the embodiment of the present application also includes an embodiment that is exemplarily specified in the specification or a combination of obvious embodiments and the like as appropriate.

For example, in the present embodiment, in the composition ratio array W _n (x, y), the composition ratio changes in a stepless manner from the value val to the value 1 / N in the region R _n (x, y) and its peripheral region. (For example, see FIG. 8B). In another embodiment, the composition ratio continuously changes from the value val to the value 1 / N near the boundary between the region R _n (x, y) and the surrounding region in the composition ratio array W _n (x, y). You may do it. FIG. 13 conceptually shows an aspect in which the composition ratio continuously changes between the value val and the value 1 / N near the boundary between the region R _n (x, y) and the surrounding region. The composition ratio setting unit 102c defines a four neighborhood distance (City block distance) from the boundary between the region R _n (x, y) and the surrounding region to the center of the region R _n (x, y) as d, When the width of the region where the ratio is continuously changed is defined as wd, the composition ratio in the width wd is set using the following equation.
wd _n (x, y) = {val * d + (1 / N) * (wd−d)} / wd
Instead of the 4-neighbor distance, for example, an 8-neighbor distance (Chess board distance), an octagonal distance (Ectlideal distance), or an Euclidean distance may be used. According to this embodiment, when the tracking target subject is synthesized, the luminance change near the boundary between the tracking target subject and the background becomes smooth, so there is no possibility that the vicinity of the boundary looks unnatural.

In the present embodiment, the strobe image is generated by the processing steps (A) to (E) schematically shown below.
(A) Registration of subject through visual recognition of through image (B) Continuous shooting and storage of captured image (C) Detection of tracking target subject from each of a plurality of captured images taken continuously (or tracking target by manual operation) Select subjects individually)
(D) Setting of composition ratio (E) Generation and storage of strobe image based on set composition ratio

On the other hand, in another embodiment, a strobe image may be generated by processing steps (a) to (f) schematically shown below.
(A) Continuous shooting and storage of captured images (b) The user selects one captured image (hereinafter referred to as “representative image”) from a plurality of continuously captured images. (C) A subject is selected from the representative image. Registration (d) Detecting the tracking target subject from each of the captured images other than the representative image (or manually selecting the tracking target subject by manual operation)
(E) Composition ratio setting (f) Strobe image generation and storage based on the set composition ratio

  In the processing step (b), the user can select a representative image from all the captured images obtained by continuous shooting by operating the operation unit 104, for example. Alternatively, a representative image may be selected from a predetermined number of captured images (for example, 10 frames from a frame designated by operation of the operation unit 104).

1 photographing apparatus 100 CPU
102 DSP
102a Subject registration unit 102b Subject detection unit 102c Composition ratio setting unit 102d Image composition unit 104 Operation unit 106 Shooting lens 108 Aperture 110 Shutter 112 Image sensor 114 Aperture / shutter drive circuit 116 ROM
118 Memory card adapter 120 LCD
200 memory card

Claims (14)

Display means for displaying a captured image on a predetermined display screen;
A range specifying means for allowing a user to specify a range in a captured image displayed on the display screen;
Subject registration means for registering subjects included in the specified range;
Subject detection means for detecting a subject that can be regarded as the same as the registered subject registered by the subject registration means from each of a plurality of photographed images;
A region setting means for setting a subject region, which is a region of the detected subject, to region data corresponding to all pixels of the captured image;
A first composition ratio is set for a subject area set in the area data among all areas in the captured image, and a second composition lower than the first composition ratio is set for a non-area other than the subject area. A composition ratio setting means for setting the ratio;
Image combining means for weighting each of the plurality of photographed images based on the composition ratio set by the composition ratio setting means, and combining the plurality of weighted photographed images;
With
The area setting means includes
Compare the subject area of the frame before the current frame set in the area data with the subject area of the current frame,
If both do not overlap within the area defined by the area data, set both subject areas in the area data,
When both overlap at least partially within the area defined by the area data, only one subject area is set as the area data.
Composite image generation device. The range specifying means includes
Displaying a predetermined frame superimposed on the captured image displayed on the display screen;
The range in the captured image surrounded by the frame is changed according to a user operation,
Allowing the user to specify a range within the captured image enclosed by the frame ;
The composite image generation apparatus according to claim 1. The frame is
The shape and size and the position in the captured image are changed according to the user's operation .
The composite image generation apparatus according to claim 2. The synthesis ratio setting means includes
The composition ratio is continuously changed from the first composition ratio to the second composition ratio in the vicinity of the boundary between the subject area in which the first composition ratio is set and the non-region in which the second composition ratio is set . It is set so as to change to,
The composite image generation apparatus according to any one of claims 1 to 3 . The synthesis ratio setting means includes
When the number of photographed images to be synthesized by the image synthesizing unit is defined as N, a synthesis ratio of 1 / N times the subject area is set for the non-area .
Synthetic image generating apparatus according to any one of claims 1 to 4. The synthesis ratio setting means includes
Wherein one of the two object region determined to overlap at least a portion in a region domain data defining sets the mixing ratio of the object area set to the area data to a positive value in the area setting unit, the Set the composition ratio of the subject area that was not set in the area data to zero ,
The composite image generation apparatus according to claim 5 . Photographed image selection means for allowing the user to select one photographed image from a plurality of photographed images;
A range specifying means for allowing a user to specify a range in the selected captured image;
Subject registration means for registering subjects included in the specified range;
Subject detection means for detecting a subject that can be regarded as the same as the registered subject registered by the subject registration means from each of the photographed images not selected by the photographed image selection means;
A region setting means for setting a subject region, which is a region of the detected subject, to region data corresponding to all pixels of the captured image;
A first composition ratio is set for a subject area set in the area data among all areas in the captured image, and a second composition lower than the first composition ratio is set for a non-area other than the subject area. A composition ratio setting means for setting the ratio;
Image combining means for weighting each of the plurality of photographed images based on the composition ratio set by the composition ratio setting means, and combining the plurality of weighted photographed images;
With
The area setting means includes
Compare the subject area of the frame before the current frame set in the area data with the subject area of the current frame,
If both do not overlap within the area defined by the area data, set both subject areas in the area data,
When both overlap at least partially within the area defined by the area data, only one subject area is set as the area data.
Composite image generation device. A display step for displaying the photographed image on a predetermined display screen;
A range specifying step for allowing a user to specify a range in a captured image displayed on the display screen;
A subject registration step for registering a subject included in the designated range;
A subject detection step for detecting a subject that can be regarded as the same as the registered subject registered in the subject registration step from each of a plurality of captured images;
A region setting step for setting a subject region, which is a region of the detected subject, in region data corresponding to all pixels of the captured image;
A first composition ratio is set for a subject area set in the area data among all areas in the captured image, and a second composition lower than the first composition ratio is set for a non-area other than the subject area. A composition ratio setting step for setting the ratio;
Weighting based on the composition ratio set in the composition ratio setting step for each of the plurality of photographed images, and an image composition step of compositing the weighted photographed images;
Including
In the region setting step,
Compare the subject area of the frame before the current frame set in the area data with the subject area of the current frame,
If both do not overlap within the area defined by the area data, set both subject areas in the area data,
When both overlap at least partially within the area defined by the area data, only one subject area is set as the area data.
Composite image generation method. In the range specifying step,
Displaying a predetermined frame superimposed on the captured image displayed on the display screen;
The range in the captured image surrounded by the frame is changed according to a user operation,
Allowing the user to specify a range within the captured image enclosed by the frame;
The synthetic image generation method according to claim 8 . The frame is
The shape and size and the position in the captured image are changed according to the user's operation.
The synthetic image generation method according to claim 9 . In the composition ratio setting step,
The composition ratio is continuously changed from the first composition ratio to the second composition ratio in the vicinity of the boundary between the subject area in which the first composition ratio is set and the non-region in which the second composition ratio is set . Set to change to
The synthetic | combination image generation method as described in any one of Claims 8-10 . In the composition ratio setting step,
When the number of captured images to be combined in the image combining step is defined as N, a combination ratio of 1 / N times the subject area is set for the non-area .
Synthetic image generating method according to any one of claims 11 to claim 8. In the composition ratio setting step,
Of the two subject regions determined to overlap at least partially within the region defined by the region data in the region setting step, the composition ratio of the subject region set in the region data is set to a positive value, Set the composition ratio of the subject area not set in the area data to zero,
The synthetic image generation method according to claim 12 . A captured image selection step for allowing the user to select one captured image from a plurality of captured images;
A range specifying step for allowing a user to specify a range in the selected captured image;
A subject registration step for registering a subject included in the designated range;
A subject detection step for detecting a subject that can be regarded as the same as the registered subject registered in the subject registration step, from each of the captured images not selected in the captured image selection step;
A region setting step for setting a subject region, which is a region of the detected subject, in region data corresponding to all pixels of the captured image;
A first composition ratio is set for a subject area set in the area data among all areas in the captured image, and a second composition lower than the first composition ratio is set for a non-area other than the subject area. A composition ratio setting step for setting the ratio;
Weighting based on the composition ratio set in the composition ratio setting step for each of the plurality of photographed images, and an image composition step of compositing the weighted photographed images;
Including
In the region setting step,
Compare the subject area of the frame before the current frame set in the area data with the subject area of the current frame,
If both do not overlap within the area defined by the area data, set both subject areas in the area data,
When both overlap at least partially within the area defined by the area data, only one subject area is set as the area data.
Composite image generation method.

Images