JP2016134060A - Image processor, control method thereof, control program, and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To extract a subject as a moving object even though movement of the subject is very small, and to form one composite image from a plurality of images.SOLUTION: A reference image selection part 105 selects at least two images as reference images from a plurality of images obtained by consecutively photographing a subject. A motion vector detection part 106 detects a motion vector with a motion vector detection object and a selection reference image with each of the plurality of images as the motion vector detection object and with one of the two reference images as the selection reference image. A moving subject extraction part 107 extracts an area in which a movement amount is equal to or more than a prescribed threshold about each of the plurality of images as a moving object image, and a composition processing part 108 combines the moving object image with a background image with one of the plurality of images as the background image to acquire a composite image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, a control method thereof, a control program, and an imaging apparatus, and in particular, an image processing apparatus that extracts a subject from a plurality of images obtained by continuous shooting and combines the images. About.

  In general, there is an imaging apparatus such as a digital camera that continuously captures a plurality of images, extracts moving subjects from the plurality of images, and combines them into a single image. For example, a plurality of moving subjects (called moving subjects) are detected from a plurality of frames captured at a predetermined timing, and an image located on the locus of the moving subject selected from the plurality of moving subjects and a still image are combined. There is an image pickup apparatus configured to do so (see Patent Document 1).

JP 2013-162333 A

  However, in the imaging device described in Patent Document 1 described above, it is determined whether or not the subject is a moving subject between frames adjacent in time series. As a result, when the movement of the subject is small (small), it may be determined that there is no movement. In some cases, it is not possible to extract a moving subject from a plurality of images and combine them into a single image without extracting a region determined to have no motion as a moving subject.

  Accordingly, an object of the present invention is to provide an image processing apparatus, a control method thereof, and a control program that can extract a moving subject even if the subject's movement is minute and synthesize one image from a plurality of images. And providing an imaging apparatus.

  In order to achieve the above object, an image processing apparatus according to the present invention is an image processing apparatus that obtains a composite image by combining a plurality of images obtained by continuously photographing a subject, and at least from the plurality of images. Selecting means for selecting two images as reference images; each of the plurality of images as a motion vector detection target; and any one of the at least two reference images as a selection reference image; Detecting means for detecting a motion vector indicating the motion of the subject from the selection reference image; and extracting a region where the amount of motion indicated by the motion vector is a predetermined threshold or more for each of the plurality of images as a moving object image; Synthesizing means for synthesizing the moving object image with the background image by using one of the plurality of images as a background image. The features.

  An imaging apparatus according to the present invention includes an imaging unit that continuously captures a subject to obtain a plurality of images, the above-described image processing device, and a recording unit that records a composite image obtained by the image processing device. It is characterized by that.

  The control method according to the present invention is a control method for an image processing apparatus that obtains a composite image by combining a plurality of images obtained by continuously photographing a subject, and at least two images from the plurality of images are used as reference images. Selecting each of the plurality of images as a motion vector detection target, and selecting any one of the at least two reference images as a selection reference image, according to the motion vector detection target and the selection reference image. A detection step of detecting a motion vector indicating a motion of a subject; and a region where a motion amount indicated by the motion vector is greater than or equal to a predetermined threshold for each of the plurality of images is extracted as a moving body image; And combining the moving object image with the background image as a background image to obtain the synthesized image. To.

  The control program according to the present invention is a control program used in an image processing apparatus that obtains a composite image by combining a plurality of images obtained by continuously photographing a subject, the computer included in the image processing apparatus, A selection step of selecting at least two images as a reference image from a plurality of images, each of the plurality of images as a motion vector detection target, and any one of the at least two reference images as a selection reference image; A detection step of detecting a motion vector indicating the motion of the subject based on a vector detection target and the selection reference image; and a region in which a motion amount indicated by the motion vector is greater than or equal to a predetermined threshold for each of the plurality of images Extracted as an image, and using one of the plurality of images as a background image to the background image A synthesizing step of obtaining the composite image synthesized and characterized in that it is running.

  According to the present invention, at least two images are selected as reference images, a region in which a motion amount indicated by a motion vector is greater than or equal to a predetermined threshold is extracted as a moving image for each of a plurality of images, and a moving object is selected as a background image. Composite the images. Thereby, even if the movement of the subject is minute, it is possible to extract a moving body image (moving subject) and synthesize one image from a plurality of images.

1 is a block diagram illustrating a configuration of an example of an imaging apparatus including an image processing apparatus according to a first embodiment of the present invention. 3 is a flowchart for explaining an example of image processing performed by the camera shown in FIG. 1. It is a figure which shows an example of the image obtained as a result of the continuous imaging | photography performed with the camera shown in FIG. It is a figure which shows an example of the image obtained as a result of the continuous imaging | photography performed with the camera which concerns on the 2nd Embodiment of this invention. It is a flowchart for demonstrating an example of the image processing performed with the camera which concerns on the 2nd Embodiment of this invention. 6 is a flowchart for explaining intermediate reference image determination shown in FIG. 5.

  Hereinafter, an example of an image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 is a block diagram showing a configuration of an example of an imaging apparatus including an image processing apparatus according to the first embodiment of the present invention.

  The illustrated imaging apparatus 100 is, for example, a digital camera (hereinafter simply referred to as a camera), and includes a lens unit 101, an imaging element 102, an AFE (analog front end) 103, an image signal processing unit 104, a reference image selection unit 105, a motion. A vector detection unit 106, a moving subject extraction unit 107, a composition processing unit 108, an external recording memory 109, a memory 110, a nonvolatile memory 111, and a system control unit 112 are included. These blocks are connected to each other by an internal bus 150.

  The lens unit 101 has a plurality of lenses including a focus lens, drives the focus lens along the optical axis during AF, and acquires lens information including a focal length. An optical image (subject image) is formed on the image sensor 102 via the lens unit 101. The image sensor 102 is, for example, a CCD or CMOS image sensor, and outputs an electrical signal (analog signal) corresponding to the optical image.

  The AFE 103 converts an analog signal that is an output of the image sensor 102 into a digital signal (image data).

  The image signal processing unit 104 performs various types of image processing on the image data output from the AFE 103 or the image data read from the external recording memory 109 or the memory 110 by the system control unit 112. Image processing performed by the image signal processing unit 104 includes, for example, encoding processing, compression processing, decoding processing, enlargement / reduction processing (resizing), noise reduction processing, and color conversion processing.

  The reference image selection unit 105 selects a reference image to be used by the motion vector detection unit 106 from a plurality of image data (hereinafter also simply referred to as images) output from the image signal processing unit 104. Note that the reference image selection unit 105 may select a plurality of images as the reference image.

  The motion vector detection unit 106 detects a motion vector in the two images and outputs motion vector information indicating the motion vector. At the time of motion vector detection, the motion vector detection unit 106 divides each of the two images into a plurality of small regions, and performs template matching on the attention region that is one of the small regions and the surrounding region. , Detecting a motion vector.

  In accordance with the motion vector detected by the motion vector detection unit 106, the moving subject extraction unit 107 extracts a small region whose motion amount is equal to or greater than a predetermined threshold as a moving subject region. The composition processing unit 108 uses one of the plurality of images output from the image signal processing unit 104 as a background image, and extracts a moving subject region (also referred to as a moving body image) extracted by the moving subject extraction unit 107. Composite to the background image. The background image is an image serving as a reference for the composition, and the composition processing unit 108 sequentially composes the moving object image with the background image.

  The external recording memory 109 is a recording medium such as a memory card, for example. The non-volatile memory 111 is, for example, a hard disk (HDD) or ROM. The non-volatile memory 111 includes, for example, image data, audio data, other data, various programs for operating the system control unit 112, and the like. Stored. The memory 110 is a volatile memory such as a RAM, for example.

  Based on a program stored in the nonvolatile memory 111, the system control unit 112 controls the entire camera using the memory 110 as a work memory. The system control unit 112 reads image data from the external recording memory 109 via the internal bus 150 and writes the image data to the external recording memory 109.

  FIG. 2 is a flowchart for explaining an example of image processing performed by the camera shown in FIG. The process according to the flowchart shown in the figure is performed under the control of the system control unit 112, and the system control unit 112 develops the program recorded in the nonvolatile memory 111 into the memory 110 and executes the process according to the flowchart.

  When image processing is started, the system control unit 112 captures N (N is an integer of 2 or more) images obtained by continuous shooting into the memory 110 (step S101). In this case, the system control unit 112 controls the lens unit 101, the image sensor 102, the AFE 103, and the image signal processing unit 104 to perform continuous shooting, and stores an image obtained as a result of shooting in the memory 110.

  FIG. 3 is a diagram showing an example of an image obtained as a result of continuous shooting performed by the camera shown in FIG. Here, it is assumed that N images 401 to 402 are taken continuously in time.

  2 and 3, subsequently, the system control unit 112 controls the reference image selection unit 105 to select the first image and the Nth image from the N images stored in the memory 110. Each image is selected as a reference image (step S102). Then, the system control unit 112 sets the first image as a background image serving as a composition reference (step S103). Then, as will be described later, the moving body images extracted with respect to the background image are sequentially synthesized.

  Here, the first image is set as the background image, but the present invention is not limited to this, and may be an image selected from N photographed images. Further, for example, a plurality of images such as N photographed images may be combined and a single background image may be generated and used.

  Next, the system control unit 112 sets the second image as the initial image among the N images (step S104). Then, the system control unit 112 selects the nth image (motion vector detection target) among the N images from the memory 110 (step S105). Thereafter, the system control unit 112 detects a motion vector and extracts a moving subject for the nth image. Here, 2 ≦ n ≦ N−1.

  Subsequently, the system control unit 112 determines which reference image is used to detect the motion vector for the nth image selected in step S105 (step S106). In the illustrated example, as described above, the first image and the Nth image are selected as the reference images. In such a case, the system control unit 112 switches the reference image from the first image (first image) to the Nth image (last image) with the N / 2th image as a boundary. A reference image used when detecting a motion vector is called a selection reference image.

  In the process of step S106, the system control unit 112 determines whether or not the boundary for switching the reference image has been reached. That is, the system control unit 112 determines whether n ≦ N / 2. If n ≦ N / 2 (YES in step S106), system control unit 112 selects the Nth image as a reference image (step S107). On the other hand, if n> N / 2 (NO in step S106), system control unit 112 selects the first image as a reference image (step S108).

  After the process of step S107 or S108, the system control unit 112 sends the reference image and the nth image from the memory 110 to the motion vector detection unit 106. Then, the motion vector detection unit 106 detects the motion vector of the subject based on the reference image and the nth image, and obtains motion vector information (step S109).

  The system control 112 sends the motion vector information obtained by the motion vector detection unit 106 to the moving subject extraction unit 107. The moving subject extraction unit 107 detects a small area whose movement amount is equal to or greater than a predetermined threshold according to the motion vector information, and extracts a moving subject portion (moving body image) in the nth image (step S110).

  Subsequently, the system control 112 controls the synthesis processing unit 108 to synthesize the moving body image extracted by the moving subject extraction unit 107 and the background image stored in the memory 110 to form a synthesized image (step S111). Then, the system control unit 112 stores the composite image in the memory 110.

  Next, the system control unit 112 selects the next image with n = n + 1 (step S112). Then, the system control unit 112 determines whether or not the synthesis process has been performed for all N images. That is, the system control unit 112 determines whether or not n ≧ N (step S113).

  If it is determined that n ≧ N (YES in step S113), system control unit 112 ends the image processing. On the other hand, if it is determined that n <N (NO in step S113), system control unit 112 returns to the process in step S105.

  As described above, in the first embodiment, a plurality of reference images used for motion vector detection are set. Then, when detecting a motion vector, a reference image that is most distant from the detection target image in terms of time is used. As a result, even when the movement of the subject is small, it is possible to perform motion determination and extract it as a moving subject.

  In this embodiment, it is assumed that the shooting interval is substantially constant, and N / 2 images are used as boundaries for switching the selected images. However, the present invention is not limited to this. It is also possible to change the image serving as the boundary according to the movement of the subject during continuous shooting. For example, when the motion of the subject immediately after the start of continuous shooting is small and gradually increases, the motion vector of the subject is accurately obtained by making the boundary image an image after the N / 2nd image. be able to.

  Further, in the present embodiment, after selecting one selection reference image from two reference images, a motion vector is detected based on the selection reference image. A motion vector may be calculated as an image, and a moving subject portion (moving body image) in the nth image may be extracted by selectively using a motion vector having a motion amount equal to or greater than a predetermined threshold.

[Second Embodiment]
Next, an example of a camera provided with an image processing apparatus according to the second embodiment of the present invention will be described. The configuration of the camera according to the second embodiment is the same as that of the camera shown in FIG.

  In the second embodiment, processing (that is, composition processing) in the case where the moving direction of the subject changes in a predetermined period will be described.

  FIG. 4 is a diagram illustrating an example of an image obtained as a result of continuous shooting performed by the camera according to the second embodiment of the present invention.

  Here, it is assumed that N images 501 to 503 are taken continuously in time, and the subject moves in the direction indicated by the arrow. In the example shown in FIG. 4, the moving direction of the subject is changed, so that the last image 503 overlaps with other images. For this reason, when the moving direction of the subject changes, it is determined that the movement is small, and the moving body image may not be appropriately extracted.

  FIG. 5 is a flowchart for explaining an example of image processing performed by the camera according to the second embodiment of the present invention.

  When image processing is started, the system control unit 112 performs steps S201 to S204. The processing in steps S201 to S204 is the same as the processing in steps S101 to S104 described in FIG.

  In step S204, after setting the second image as the initial image in the N images, the system control unit 112 sets the number of intermediate reference images for frames (that is, images) in which the movement of the subject has changed. Initialization is performed (i = 0: step S205). Then, the system control unit 112 performs intermediate reference image determination as described later (step S206), and stores the total number Imax of intermediate reference images obtained by the intermediate reference image determination in the memory 110.

  Subsequently, the system control unit 112 selects the next image with n = n + 1 (step S207). Then, the system control unit 112 determines whether or not intermediate reference image determination has been performed for all images (step S208). That is, the system control unit 112 determines whether n ≧ N.

  If it is determined that n <N (NO in step S208), system control unit 112 returns to the process in step S206. On the other hand, if it is determined that n ≧ N (YES in step S208), the system control unit 112 sets the M [0] th sheet as the initial intermediate reference image (step S209). Then, the system control unit 112 selects the second image as an initial image (step S210).

  Next, the system control unit 112 determines whether or not the nth image has reached a switching boundary for switching the reference image (step S211). That is, the system control unit 112 determines whether n <M [i]. If it is determined that N ≧ M [i] (NO in step S211), the system control unit 112 controls the reference image selection unit 105 to control the M [i−1] -th and M [i] from the intermediate reference image. ] Is selected as the reference image (step S212).

  Subsequently, the system control unit 112 determines whether or not the nth image has reached a switching boundary for switching the reference image (step S213). Here, the system control unit 112 determines which of the reference images M [i−1] and M [i] selected in step S212 is most distant from the nth image in terms of time. That is, the system control unit 112 determines whether or not n ≦ (M [i] + M [i−1]) / 2.

  If it is determined that n ≦ (M [i] + M [i−1]) / 2 (YES in step S213), the system control unit 112 controls the reference image selection unit 105 to perform the M [i] th image. Are selected as reference images (step S214). On the other hand, when it is determined that n> (M [i] + M [i−1]) / 2 (NO in step S213), system control unit 112 controls reference image selection unit 105 to control M [i−. 1] The first image is selected as a reference image (step S215).

  If it is determined that N <M [i] (YES in step S211), the system control unit 112 determines whether the system control unit 112 has selected all intermediate reference images (step S216). That is, the system control unit 112 determines whether i = imax.

  If it is determined that i = imax (YES in step S216), the system control unit 112 determines whether or not the nth image has reached a switching boundary for switching the reference image (step S217). Here, the system control unit 112 determines which of the reference images M [i] and N is most distant from the n-th image in terms of time. That is, the system control unit 112 determines whether or not n ≦ (N + M [i]) / 2.

  If it is determined that n ≦ (N + M [i]) / 2 (YES in step S217), system control unit 112 controls reference image selection unit 105 to select the Nth image as a reference image ( Step S218). On the other hand, if it is determined that n> (N + M [i]) / 2 (NO in step S217), system control unit 112 controls reference image selection unit 105 to set the M [i] -th image as a reference. It selects as an image (step S219).

  If it is determined that i = imax is not satisfied (NO in step S216), system control unit 112 selects the next intermediate reference image as i = i + 1 (step S220). Then, the system control unit 112 returns to the process of step S211.

  After the process of step S214, S215, S218, or S219, the system control unit 112 performs the process of steps S221 to S225. Since the processing of steps S221 to S225 is the same as the processing of steps S109 to S113 described in FIG. 2, the description thereof is omitted here. If it is determined in step S225 that n <N, the system control unit 112 returns to the process of step S211.

  FIG. 6 is a flowchart for explaining the intermediate reference image determination shown in FIG.

  When the intermediate reference image determination process is started, the system control unit 112 controls the motion vector detection unit 106 to detect the subject between the nth image and the previous (n−1) th image. A motion vector is detected (step S301). Subsequently, the system control unit 112 controls the motion vector detection unit 106 to detect the motion vector of the subject in the nth image and the next (n + 1) th image (step S302). .

  Next, the system control unit 112 compares the directions of the motion vectors detected in steps S301 and S302 (step S303). Then, the system control unit 112 determines whether or not the amount of change in the direction of the motion vector has changed beyond a preset change threshold (step S304).

  If it is determined that the amount of change in the direction of the motion vector is greater than or equal to the change threshold (YES in step S304), the system control unit 112 determines that the amount of change in the direction of the motion vector is greater than or equal to the change threshold. Is added to the reference image as an intermediate reference image M [i]. That is, the system control unit 112 sets M [i] = n, i = i + 1, and Imax = i. Then, the system control unit 112 ends the intermediate reference image determination process, and proceeds to the process of step S207 shown in FIG.

  On the other hand, if it is determined that the amount of change in the direction of the motion vector is less than the change threshold value (NO in step S304), the system control unit 112 ends the intermediate reference image determination process and proceeds to the process of step S207 shown in FIG. move on.

  As described above, in the second embodiment, since the image in which the direction of the motion vector has changed is added as the intermediate reference image, the motion determination can be performed even when the motion of the subject has changed. It is possible to extract a moving subject and generate a composite image.

  As is clear from the above description, in the example shown in FIG. 1, the system control unit 112 and the reference image selection unit 105 function as a selection unit, and the motion vector detection unit 106 and the system control unit 112 function as a detection unit. In addition, the system control unit 112, the moving subject extraction unit 107, and the synthesis processing unit 108 function as a synthesis unit.

  Further, the lens unit 101, the image sensor 102, the AFE 103, and the image signal processing unit 104 function as an imaging unit, and the system control unit 112 and the external recording memory 109 function as a recording unit.

  In the example illustrated in FIG. 1, at least the reference image selection unit 105, the motion vector detection unit 106, the moving subject extraction unit 107, the synthesis processing unit 108, the memory 110, the nonvolatile memory 111, and the system control unit 112 are included in the image processing apparatus. Configure.

  As mentioned above, although this invention was demonstrated based on embodiment, this invention is not limited to these embodiment, Various forms of the range which does not deviate from the summary of this invention are also contained in this invention. .

  For example, the function of the above embodiment may be used as a control method, and this control method may be executed by the image processing apparatus. In addition, a program having the functions of the above-described embodiments may be used as a control program, and the control program may be executed by a computer included in the image processing apparatus. The control program is recorded on a computer-readable recording medium, for example.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101 Lens unit 102 Image sensor 103 AFE
104 image signal processing unit 105 reference image selection unit 106 motion vector detection unit 107 moving subject extraction unit 108 synthesis processing unit 110 memory 112 system control unit

Claims (8)

An image processing apparatus that combines a plurality of images obtained by continuously photographing a subject to obtain a composite image,
Selecting means for selecting at least two images as reference images from the plurality of images;
Each of the plurality of images is a motion vector detection target, one of the at least two reference images is a selection reference image, and a motion vector that indicates the motion of the subject by the motion vector detection target and the selection reference image Detecting means for detecting
For each of the plurality of images, a region in which the amount of motion indicated by the motion vector is equal to or greater than a predetermined threshold is extracted as a moving image, and one of the plurality of images is used as a background image and the moving image is added to the background image. Synthesizing means for obtaining the synthesized image by synthesizing
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the detection unit sets a reference image that is most distant in time from the motion vector detection target as the selection reference image.   The image processing apparatus according to claim 1, wherein the selection unit selects an image captured first and an image captured last among the plurality of images as the reference image.   The image processing apparatus according to claim 3, wherein the selection unit selects, as a reference image, an image in which the movement direction of the subject has changed in the plurality of images.   The image processing apparatus according to claim 4, wherein the selection unit selects, as the reference image, an image in which the amount of change in the movement direction of the subject is equal to or greater than a preset change threshold. Imaging means for continuously capturing a subject to obtain a plurality of images;
The image processing apparatus according to any one of claims 1 to 5,
Recording means for recording a composite image obtained by the image processing apparatus;
An imaging device comprising: A control method of an image processing apparatus that combines a plurality of images obtained by continuously photographing a subject to obtain a composite image,
A selection step of selecting at least two images from the plurality of images as reference images;
Each of the plurality of images is a motion vector detection target, one of the at least two reference images is a selection reference image, and a motion vector that indicates the motion of the subject by the motion vector detection target and the selection reference image A detecting step for detecting
For each of the plurality of images, a region in which the amount of motion indicated by the motion vector is equal to or greater than a predetermined threshold is extracted as a moving image, and one of the plurality of images is used as a background image and the moving image is added to the background image. Synthesizing to obtain the synthesized image;
A control method characterized by comprising: A control program used in an image processing apparatus that combines a plurality of images obtained by continuously photographing a subject to obtain a composite image,
In the computer provided in the image processing apparatus,
A selection step of selecting at least two images from the plurality of images as reference images;
Each of the plurality of images is a motion vector detection target, one of the at least two reference images is a selection reference image, and a motion vector that indicates the motion of the subject by the motion vector detection target and the selection reference image A detecting step for detecting
For each of the plurality of images, an area in which a motion amount indicated by the motion vector is equal to or greater than a predetermined threshold is extracted as a moving image, and one of the plurality of images is used as a background image and the moving image is added to the background image Synthesizing to obtain the synthesized image;
A control program characterized by causing

Images