JP2015008428A - Image processing apparatus, image processing method, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus, an image processing method, and an image processing program capable of simply generating an image giving no discomfort relating to the movement of a subject and background even in photographing with the apparatus held with the hands.SOLUTION: The image processing apparatus comprises: an imaging unit 102 which images subject consecutively and acquires plural images; a subject determination unit 114 which determines the area of a main subject and the area of a background subject in each of the acquired images; an object image determination unit 116 which determines a target image out of the acquired images; a motion vector estimation unit 118 which estimates the motion vector of at least the area of the background subject between the target image and an image captured at a time before or after the time when the target image was captured; a filter constitution unit 120 which constitutes a filter to be applied to the target image on the basis of the estimated motion vector; and a filter application unit 122 which applies the constituted filter to the target image.

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program that perform image processing that emphasizes the motion of a subject using a plurality of captured images.

  One of the photography techniques is “panning”. Panning is a technique in which a photographer moves a camera in accordance with the movement of a subject so that the position of the subject is fixed with respect to the angle of view. In panning, an effect is produced in which the background flows without blurring the subject, so that an image in which the sense of speed of the subject is emphasized can be obtained. Here, panning is often performed with an exposure time longer than that of normal shooting. As the exposure time becomes longer, camera shake tends to occur. For this reason, the subject as well as the background is often blurred during panning.

  As one of the techniques for suppressing subject shake during panning, a technique for suppressing subject blur by mechanically moving an image sensor or a lens in accordance with camera shake occurring during panning is conceivable. In this case, camera shake correction for panning is controlled separately from normal shooting, and the system is likely to be complicated. In addition, subject blur occurs unless mechanical camera shake correction is performed, and thus a time lag occurs between when the shutter button is pressed and when shooting is performed. For this reason, there is a possibility that an image having a composition that the photographer originally wants to shoot cannot be obtained.

  As another method for suppressing subject shake during panning, the method of Patent Document 1 has been proposed. The technique of Patent Document 1 obtains an image in which the blur of the subject is suppressed by shortening the exposure time, and separates the image into a subject region and a background region, so that the image of the background region flows. This is a method of performing blur image processing on the background area in accordance with the movement.

JP 2011-114823 A

  The technique of Patent Document 1 functions effectively only when the movement of the subject is parallel to the shooting screen and the relationship between the depth between the background object and the subject is unchanged. Actual panning is often performed in a situation involving movement of the subject in the depth direction (such as movement in which the subject approaches the photographer). At this time, the movement of the background area on the shooting screen is not necessarily uniquely determined from the movement of the subject. For this reason, in the method of applying uniform “motion blur” to the background as in Patent Document 1, there is a possibility that unnatural blur that cannot actually occur may occur.

  The present invention has been made in view of the above circumstances, and provides an image processing apparatus, an image processing method, and an image processing program that can easily generate an image that does not feel uncomfortable with the movement of the subject and the background even with hand-held shooting. Objective.

  In order to achieve the above object, an image processing apparatus according to a first aspect of the present invention includes an imaging unit that continuously images a subject to acquire a plurality of images, and each of the acquired plurality of images. A subject determination unit that determines a main subject region and a background subject region, a target image determination unit that determines a target image from the plurality of acquired images, and the target image and the target image are captured A motion vector estimation unit that estimates a motion vector of at least the background subject area between a time and an image captured at a time before and after, and a filter that is applied to the target image based on the estimated motion vector And a filter application unit that applies the configured filter to the target image.

  In order to achieve the above object, an image processing method according to a second aspect of the present invention includes acquiring a plurality of images by continuously capturing an image of a subject and performing a main process in each of the acquired plurality of images. Determining a region of the subject and a region of the background subject, determining a target image from the plurality of acquired images, and before and after the time when the target image and the target image were captured Estimating at least a motion vector of the area of the background subject between the image captured at the time to be performed, configuring a filter to be applied to the target image based on the estimated motion vector, and Applying the filter to the target image.

  In order to achieve the above object, an image processing program according to a third aspect of the present invention includes a function of continuously capturing an image of a subject to acquire a plurality of images, and a main function in each of the acquired plurality of images. A function for determining a subject area and a background subject area, a function for determining a target image from the plurality of acquired images, and a time sequence of the target image and the time when the target image was captured A function of estimating at least a motion vector of the area of the background subject between the image captured at the time to perform, a function of configuring a filter to be applied to the target image based on the estimated motion vector, and And a function of applying the filtered filter to the target image.

  According to the present invention, it is possible to provide an image processing apparatus, an image processing method, and an image processing program that can easily generate an image that does not give a sense of incongruity to movements of a subject and a background even with hand-held shooting.

It is a figure which shows the structure of an example of an imaging device as an application example of the image processing apparatus which performs an image process using the image processing method which concerns on the 1st Embodiment of this invention. 3 is a flowchart illustrating a shooting operation of the imaging apparatus according to the first embodiment of the present invention. It is a figure which shows the example of the image obtained by a panning. It is a figure which shows the example of the continuous shooting operation | movement in 1st Embodiment. It is a figure which shows the outline | summary of a to-be-photographed object determination. It is a flowchart which shows the process of target image determination. It is a figure showing the ideal photographic subject gravity center position by a three division method. It is a figure which shows the relationship between the gravity center position calculated from the area | region of the actual main subject calculated with respect to 3 images obtained by continuous shooting operation | movement, and an ideal subject gravity center position. It is a figure for demonstrating motion vector estimation. It is a figure which shows the example of the estimated motion vector. It is a figure which shows the setting of the size of a filter kernel. It is a figure which shows the example of the comprised filter. It is a figure shown about a boundary region. It is a figure which shows the example of the filter applied with respect to a boundary region. It is a flowchart which shows the process for setting a boundary area | region. It is a figure which shows the structure of an example of the imaging device as an application example of the image processing apparatus which performs an image process using the image processing method which concerns on the 2nd Embodiment of this invention. It is a flowchart which shows the imaging | photography operation | movement of the imaging device in the 2nd Embodiment of this invention. FIG. 18A is a diagram showing a region for estimating a motion as a background subject in the screen, and FIG. 18B is a diagram showing a configuration of a block for estimating a motion vector. It is a figure which shows the example of the scene in which subject determination is performed. FIG. 20A is a diagram showing motion vectors estimated when the shooting mode is the panning mode, and FIG. 20B is an integration of motion vectors estimated in the case of the panning mode in units of small areas. It is a figure which shows the motion vector obtained by this. FIG. 21A shows a motion vector estimated when the shooting mode is the handheld long-time shooting mode, and FIG. 21B shows a motion vector estimated when the shooting mode is the handheld long-time shooting mode. It is a figure which shows the motion vector obtained by integrating in a unit. It is a figure which shows the structure of an example of the imaging device as an application example of the image processing apparatus which performs image processing using the image processing method which concerns on the 3rd Embodiment of this invention. It is a figure which shows the specific example of the content displayed on a display part in the case of selection of a target image.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
(First embodiment)
[Constitution]
FIG. 1 is a diagram illustrating a configuration of an example of an imaging apparatus as an application example of an image processing apparatus that performs image processing using the image processing method according to the first embodiment. Here, the solid line arrow in FIG. 1 represents the flow of an image. Moreover, the broken line arrow of FIG. 1 represents the flow of control information other than an image. An imaging apparatus 100 illustrated in FIG. 1 includes an imaging unit 102, an operation unit 104, a shooting condition setting unit 106, a processing control unit 108, an information recording unit 110, an image storage unit 112, a subject determination unit 114, The target image determination unit 116, the motion vector estimation unit 118, the filter configuration unit 120, and the filter application unit 122 are included.

  The imaging unit 102 includes a lens 1021, an imaging element 1022, and an imaging control unit 1023. The lens 1021 is a single lens or a plurality of lenses that form a light flux from the subject on the image plane of the image sensor 1022. The image sensor 1022 converts the light beam incident on the image plane through the lens 1021 into an electric signal (image signal) by photoelectric conversion. Hereinafter, the image signal is simply referred to as an image. The imaging control unit 1023 controls the imaging operation of the imaging element 1022. For example, the imaging control unit 1023 controls the exposure amount of the imaging element 1022 by controlling the exposure time, sensitivity, and the like of the imaging element 1022. The imaging control unit 1023 controls the continuous shooting operation of the image sensor 1022. Furthermore, the imaging control unit 1023 performs color restoration processing (such as demosaicing processing when the imaging device 1022 has a single plate color filter), noise reduction processing, edge enhancement on the image obtained by the imaging device 1022. Image processing such as processing is performed.

  The operation unit 104 is an operation member that is operated by the photographer to operate the imaging apparatus 100. The operation unit 104 may be an operation member such as a button or a switch, or may be an operation member such as a touch panel. By the operation of the operation unit 104, for example, the shooting mode of the imaging device 100 is selected. The shooting modes in this embodiment include, for example, a “panning” mode and a “handheld long-time shooting” mode. The panning mode is a shooting mode for panning. The hand-held long-time shooting mode is a shooting mode for taking a long-time shooting by a photographer.

  The shooting condition setting unit 106 sets shooting conditions at the time of shooting. The shooting conditions include, for example, conditions related to the exposure amount of the image sensor 1022 at the time of shooting (for example, exposure time and sensitivity), conditions related to image processing (for example, noise reduction level in noise reduction processing), and the number of shots when continuous shooting is performed. Including. These photographing conditions are set according to a manual operation by the photographer operating the operation unit 104, for example. Further, the shooting condition may be set based on the brightness of the subject in the image obtained during the live view before the main shooting, for example. The live view is an operation for displaying an image obtained by continuously driving the image sensor 1022 on the display unit in real time before the main photographing.

  The process control unit 108 controls each block of the imaging apparatus 100 according to a program recorded in the information recording unit 110. For example, the processing control unit 108 controls the imaging control unit 1023 according to the imaging conditions set by the imaging condition setting unit 106. Further, the process control unit 108 instructs the start of the operation of the subject determination unit 114.

  The information recording unit 110 is, for example, a flash ROM, and records a program executed by the processing control unit 108, parameters necessary for image processing, and the like. The image storage unit 112 is, for example, a RAM, and stores an image obtained by the imaging unit 102. The image stored in the image storage unit 112 is read according to an instruction from the subject determination unit 114.

  The subject determination unit 114 determines the main subject region and the background subject region in the image stored in the image storage unit 112, respectively, and subject / background region information (for example, main subject information) indicating the main subject and background subject region information. , And a binary image obtained by dividing the background subject area). The target image determination unit 116 determines a target image to which the filter is applied by the filter application unit 122 from among a plurality of images obtained by the continuous shooting operation, according to the subject / background region information from the subject determination unit 114. To do. The motion vector estimation unit 118 estimates the motion vector of the subject between a plurality of images obtained by the continuous shooting operation. The filter configuration unit 120 configures a filter for adding motion blur to an image based on the motion vector estimated by the motion vector estimation unit 118. The filter application unit 122 applies the filter configured by the filter configuration unit 120 to the target image.

[Operation]
Next, the operation of the imaging apparatus 100 according to the present embodiment will be described. FIG. 2 is a flowchart illustrating a shooting operation of the imaging apparatus 100 according to the first embodiment. In FIG. 2, the shooting condition setting unit 106 sets shooting conditions at the time of shooting (step S101). The shooting conditions are set according to the shooting mode, for example. The shooting mode is selected by, for example, operating the operation unit 104 by the photographer. In the following, it is assumed that either the panning mode or the long-time shooting mode is selected. When the shooting mode is selected, information on the selected shooting mode is input to the shooting condition setting unit 106. The shooting condition setting unit 106 sets shooting conditions according to the input shooting mode information. In step S101, the number of shots, exposure time, sensitivity, noise reduction level, etc. are set as imaging conditions. Details of setting the shooting conditions will be described later.

  When the shooting condition is set, the processing control unit 108 controls the imaging control unit 1023 to execute a shooting operation (step S102). If the shooting mode is panning, the photographer performs shooting while moving the imaging apparatus 100 so as to follow a moving subject. In addition, if the shooting mode is handheld long-time shooting, the photographer performs shooting while fixing the imaging device 100 by hand. After the start of shooting, the processing control unit 108 controls the imaging control unit 1023 so that each shooting is performed under the shooting conditions set in step S101. The imaging control unit 1023 executes the imaging operation by the imaging element 1022 for the number of shots with the set exposure time and sensitivity. Then, the imaging control unit 1023 performs image processing on the image obtained by each imaging operation, and causes the image storage unit 112 to store the image subjected to the image processing. Here, the noise reduction level in the image processing is set as a shooting condition. After storing the set number of captured images in the image storage unit 112, the imaging control unit 1023 inputs information on completion of imaging to the processing control unit 108.

  When shooting completion information is input, the processing control unit 108 instructs the subject determination unit 114 to start processing and inputs shooting mode information. In response to the instruction to start processing, the subject determination unit 114 performs subject determination for each image stored in the image storage unit 112 (step S103). The subject determination unit 114 instructs the image storage unit 112 to extract an image. When receiving the instruction, the image storage unit 112 takes out a plurality of images stored according to the continuous shooting operation one by one and inputs them to the subject determination unit 114. Upon receiving the image input, the subject determination unit 114 determines the main subject region and the background subject region in the input image, and generates subject / background region information representing the main subject and background subject region information. To do. The subject determination unit 114 also generates motion estimation target region information based on the shooting mode information, and adds the generated motion estimation target region information to the subject / background region information. The motion estimation target region information is information indicating which of the main subject region and the background subject region is used when estimating a motion vector and applying a filter. As information on the motion estimation target area, for example, information on the shooting mode can be used.

  When the subject determination for all the images stored in the image storage unit 112 is completed, the subject determination unit 114 sends the input images and subject / background region information associated with the images to the target image determination unit 116. To the motion vector estimation unit 118. The target image determination unit 116 analyzes subject / background region information of each image, determines a target image to be applied with the filter from the input images, and determines the determined target image as the motion vector estimation unit 118. The data is sent to the filter application unit 122 (step S104). Details of the target image determination method will be described later.

  The motion vector estimation unit 118, based on the information of the motion estimation target region, the subject between the target image input from the target image determination unit 116 and an image having a shooting time that is temporally around the shooting time of the target image. And the estimated motion vector is input to the filter configuration unit 120 (step S105). For example, in the panning mode, the photographer moves the imaging device 100 to follow the moving main subject. Therefore, when viewed on the image, it is considered that the main subject is almost stationary and the background subject is moving. There is no need to add motion blur to a stationary subject. Therefore, in the panning mode, it is only necessary to estimate the motion vector of the background subject area. Of course, the motion vector of the area of the main subject may be estimated. On the other hand, in the hand-held long-time shooting mode, the photographer fixes the imaging device 100 by hand. In this case, when viewed on the image, the main subject may move, and the background subject is usually considered to be fixed. Therefore, in the handheld long-time shooting mode, it is only necessary to estimate the motion vector of the main subject area. Of course, the motion vector of the background subject area may also be estimated. The motion vector estimation unit is a pixel unit or a block unit composed of a plurality of adjacent pixels (for example, 5 × 5 pixels). Details of such motion vector estimation will be described later.

  When the motion vector is input, the filter configuration unit 120 configures a filter according to the motion vector, and inputs information about the configured filter to the filter application unit 122 (step S106). The filter configured by the filter configuration unit 120 is a smoothing filter that smoothes an image in accordance with a motion vector (direction and size). The filter is provided with a motion vector estimation unit. Therefore, no filter is formed in the region where the motion vector is not estimated. Details of the processing that constitutes the filter will be described in detail later.

  When the filter information is input, the filter application unit 122 performs filter processing (motion blur addition) on the target image determined by the target image determination unit 116 (step S107). Then, the filter application unit 122 outputs the filtered image to the output device (step S108). The output device is, for example, a display that is a device for displaying an image or a memory card that is a device for recording an image.

[Continuous shooting]
The setting of shooting conditions and the continuous shooting operation will be further described. FIG. 3 is a diagram illustrating an example of an image obtained by panning. FIG. 3A shows an image obtained by ideal panning. The main subject is the car in front, and the background subject is a tree or building in the back. In panning, shooting is performed while the imaging apparatus 100 is moved so as to follow a moving main subject. Here, in ideal panning, the main subject moving during exposure is stationary on the image. For this reason, the background subject other than the main subject is blurred by the movement on the image. However, ideal panning is difficult for photographers without skilled skills. In particular, camera shake tends to occur in hand-held panning shots by photographers. When camera shake occurs, as shown in FIG. 3B, the main subject that is not desired to be shaken is also shaken.

  In the present embodiment, as shown in FIG. 4, images are obtained by repeating short-time exposure continuously with Scene 1, Scene 2, and Scene 3. As a result, even in hand-held shooting, it is possible to greatly suppress blurring of a moving main subject. On the other hand, when the exposure time is shortened, the movement of the background subject cannot be felt. The background subject is blurred by applying smoothing according to the motion vector. This process is the process of step S107.

  Further, when the exposure time is shortened, the image becomes dark. As one of the techniques for making the brightness of the image appropriate, a technique for increasing the sensitivity of the image sensor 1022 (amplification factor of a signal obtained by the image sensor 1022) is conceivable. However, when the sensitivity of the image sensor 1022 is increased, noise in the image becomes conspicuous. Therefore, noise in the image is reduced by increasing the noise reduction level in the noise reduction processing according to the sensitivity.

  When the shooting conditions for the continuous shooting operation are automatically set, in order to suppress blurring, the single exposure time t in the continuous shooting operation is set as short as possible. Preferably, the exposure time is set shorter than the time referred to as a camera shake limit. The camera shake limit is a measure of the exposure time that is the limit at which camera shake does not occur, and is 1 / (lens focal length (mm)) second. In this case, the focal length of the lens is defined in terms of 35 mm film. Here, the exposure time T necessary for setting the brightness of the main subject to an appropriate brightness set in the imaging apparatus 100 is calculated by a known automatic exposure (AE) calculation when the lens aperture is fixed. The Therefore, the number of shots N is (T / t). However, if the exposure time for one exposure is t, the brightness is insufficient. This lack of brightness is compensated by increasing the sensitivity. That is, the sensitivity is set to (T / t) times = N times. The noise reduction level is set according to the increase in sensitivity. Thus, ideally, N images having the appropriate brightness of the main subject can be obtained. Note that the shooting conditions may be set by a manual operation by the photographer.

[Subject judgment method]
The subject determination method will be further described. FIG. 5 is a diagram showing an outline of subject determination in step S103. In the present embodiment, the main subject and other than the main subject (background subject) are determined for each of a series of images stored in the image storage unit 112 by the continuous shooting operation of the imaging unit 102. For the determination, for example, a known image recognition method is used. For this purpose, image feature amounts of objects (vehicles, people, animals, etc.) that accompany movement are registered in the information recording unit 110 in advance. In the determination, an image feature amount is calculated from each image obtained by the continuous shooting operation, and the calculated feature amount is compared with the image feature amount of each object based on a predetermined determination rule. Then, it is determined that the object having the most similar image feature amount is a subject in the image, and a binary image indicating the region of each subject becomes subject / background region information. Here, the image feature amount registered in the information recording unit 110 may be sequentially updated by learning.

  FIG. 5 shows the area of the car (Car) that is the main subject, the tree (Tree) that is the background subject, and the house (House) for each of a plurality of images (Scene 1 to Scene 3) obtained by the continuous shooting operation. ) Area is recognized. FIG. 5 shows whether the recognized subject is a main subject (main) or a background subject (back). Here, the main subject that matches the condition such as being located in front (not hidden by other subjects) or being an object with movement in image recognition is adopted. In the present embodiment, it is only necessary to distinguish the “main subject” from the “background subject”. Therefore, the tree (Tree) and the house (House) in FIG. 5 may be regarded as one region as the “background subject”. In this case, for example, if the subject determination is simple such that the foreground subject in the image is the main subject and the other subject is the background subject, a detailed image recognition method may not be used.

  The image shown in FIG. 5 is an image obtained by reducing noise with a short exposure time and high sensitivity. Therefore, the influence of subject blurring due to the movement of the imaging device 100 and the main subject is minimized. Therefore, highly accurate subject determination is performed without being affected by subject shake.

[Target image determination method]
The method for determining the target image will be further described. FIG. 6 is a flowchart showing the target image determination process in step S104. The example of FIG. 6 is an example in which the target image is an image in which the main subject is present at a position closest to the ideal position on the screen based on the three-division method used when determining the composition of the screen.

  In FIG. 6, the target image determination unit 116 initializes variables necessary for processing (step S201). Specifically, the target image determination unit 116 counts the reference order of images by N indicating the number of images to be processed as the number of images stored in the image storage unit 112 (matches the number of captured images). The image reference counter C is set to 1, the minimum center-of-gravity distance serving as a reference at the time of determining the target image is set to the image size (as a large value that is not calculated in pixel units), and the target image number P representing the order of the target image is set to Initialize to 0 respectively.

  After the initialization of the variable, the target image determination unit 116 executes a loop process from step S202. As the start of the loop process, the target image determination unit 116 reads out the C-th image in the order and the subject / background area information associated with the C-th image (step S202). Subsequently, the target image determination unit 116 calculates the center-of-gravity position in the main subject region from the subject / background region information (step S203). The barycentric position is calculated, for example, in pixel units of the image.

  Thereafter, the target image determination unit 116 calculates the distance between the calculated center of gravity position and a plurality of preset ideal subject center of gravity positions. Then, the target image determination unit 116 sets the smallest calculated distance as the center-of-gravity distance corresponding to the C-th image (step S204). Subsequently, the target image determination unit 116 determines whether or not the centroid distance corresponding to the Cth image is shorter than the minimum centroid distance (step S205). When the centroid distance corresponding to the C-th image is equal to or greater than the minimum centroid distance, the target image determination unit 116 does not replace the minimum centroid distance. On the other hand, when the centroid distance corresponding to the Cth image is shorter than the minimum centroid distance, the target image determination unit 116 replaces the minimum centroid distance with the centroid distance corresponding to the Cth image. Further, the target image determination unit 116 sets the current value of the image reference counter C to the target image number P (step S206). Then, the target image determination unit 116 increments the image reference counter C in preparation for reading out the next sequence of images (step S207).

  As the end determination of the loop processing, the target image determination unit 116 determines whether or not the image reference counter C is larger than the number N of processing target images (step S208). When the image reference counter C is larger than the number N of images to be processed, the target image determination unit 116 ends the loop process. When the image reference counter C is equal to or smaller than the number of images to be processed N, the target image determination unit 116 shifts the processing to step S202, which is the start of the loop processing.

  After completing the loop process, the target image determination unit 116 determines the P-th image corresponding to the minimum center-of-gravity distance as the target image for processing (step 209).

  FIG. 7 is a diagram showing the ideal subject gravity center position by the three-division method. In the case of the three-division method, grid lines that equally divide the screen into 3 × 3 are set. The ideal subject center-of-gravity position G0 is the four points where the grid lines intersect. Accordingly, the distance from the center of gravity of the area of the main subject is calculated for the four ideal subject center of gravity positions, and the smallest of these is the C-th center of gravity distance in step S204.

  FIG. 8 shows the relationship between the gravity center position G1 calculated from the actual main subject area calculated for the three images (Scene1, Scene2, Scene3) obtained by the continuous shooting operation and the ideal subject gravity center position G0. FIG. In the example of FIG. 8, when the gravity center distances of the images are compared, the distance between G0 and G1 is the smallest for the image of Scene2. Therefore, the image of Scene2 is determined as the target image.

[Motion vector estimation]
The motion vector estimation will be further described. FIG. 9A is a diagram illustrating a motion estimation target block set by the motion vector estimation unit 118. The motion estimation target block is set according to the motion estimation target area information (for example, information on the shooting mode) added to the subject / background area information. For example, when the shooting mode is the panning mode, the motion estimation target is the background subject area. Therefore, as shown in FIG. 9A, the motion vector estimation unit (square size block; for example, 5) for the background subject area other than the main subject area (main in FIG. 9A) in the Scene 2 image; X5 pixels) is defined. FIG. 9B shows the motion vector estimation unit block superimposed on the actual image (Scene2). A motion vector is estimated for each block other than the main subject area shown in FIG. 9B (main in FIG. 9B). The motion vector is a moving direction and a moving amount on the background subject image with respect to the image (Scene1) captured at the photographing time one time before the photographing time of the image Scene2 as the target image. is there. That is, it is possible to estimate the motion vector of the background subject area between the two images by searching in which area of the Scene2 image the background subject exists in the Scene1 image using a method such as pattern matching. it can.

  For estimation of the motion vector, an image taken at the time one time before the time when the target image was taken is at least required. On the other hand, since a part of the background subject is hidden by the main subject, there is a possibility that the corresponding background subject cannot be searched and the motion vector cannot be estimated. Therefore, in order to be able to estimate the motion vector for the entire background, an image taken at another time, for example, an image taken at a time immediately after the shooting time of the target image (Scene 3), before the shooting time of the target image. Alternatively, an image taken at the shooting time and an image taken at a shooting time after the shooting time of the target image may be used. By using a large number of images, it is possible to remove motion vectors with low reliability due to the effects of the above-described hiding and noise.

[Filter construction method from motion vectors]
A filter construction method from motion vectors will be further described. As an example, assume that a motion vector is estimated as shown in FIG. At this time, attention is paid to a part (block) of the house (back1) and a part of the tree (back2) which are background subjects. In the example of FIG. 10, the distance from the photographer is assumed to be farther away from the house than from the tree. Here, it is assumed that the photographer pans the image capturing apparatus 100 in an attempt to pan the car (main subject) that moves from right to left on the screen. In general, it is known that the movement of the background subject on the image with respect to the movement of the imaging apparatus 100 is larger as the subject is closer and is smaller as the subject is farther away. Therefore, in the example of FIG. 10, the magnitude of the motion vector of the house block is smaller than the magnitude of the motion vector of the tree block. The direction of the motion vector of the house block at the top of the screen is estimated as the lower right direction, and the direction of the motion vector of the tree block at the bottom of the screen is estimated as the upper right direction. As a result, in FIG. 10, the motion vector of the house block BL1 is V1, and the motion vector of the tree block BL2 is V2.

  A smoothing filter representing motion blur is configured using the motion vector obtained as shown in FIG. Specifically, the kernel size of the filter is determined so as to match the magnitude of the motion vector with the start point of the motion vector as the center. For example, as shown in FIG. 11A, if the motion vector of the house block BL1 is a vector of 3 blocks in the right direction and 1 block in the downward direction, the kernel size of the filter applied to the house is a block. 7 × 7 (= 3 + 3 + 1) is determined based on the number of blocks (3) in the larger direction (right direction). Further, as shown in FIG. 11B, if the motion vector of the tree block BL2 is a vector of 5 blocks in the right direction and 1 block in the upward direction, the kernel size of the filter applied to the tree is the block size. The number is determined to be 11 × 11 (= 5 + 5 + 1) based on the number of blocks (5) in the larger direction (right direction).

  When the kernel size of the filter is determined, the motion vectors are combined, and a smoothing filter is configured so as to have a filter element for the pixels to which the motion vector is applied. As shown in FIG. 12A, the filter applied to the house block BL1 is provided with filter elements in the four pixels in the lower right direction including the center. Also, as shown in FIG. 12B, the filter applied to the block BL2 of the tree is provided with filter elements in the six pixels in the upper right direction including the center. Since it is a smoothing filter, the filter elements are configured so that the total number is 1. That is, the filter element of the filter applied to the house block BL1 is 1/4, and the filter element of the filter applied to the tree block BL2 is 1/6. The filter configured in this way is a filter related to the pixels in the block. Using a similar method, filters are constructed for all blocks for which motion vectors have been estimated. As a result, the smoothing has an effect on the direction of center symmetry with the arrangement direction of the filter elements. That is, when the filter is applied to the house block BL1, motion blur that moves to the upper left is added. Further, when the filter is applied to the block BL2 of the tree, motion blur that moves to the lower left is added.

  Here, in the example of FIG. 12, the filter elements are determined so that smoothing is uniformly applied to a plurality of pixels. On the other hand, the filter element may be determined so as to apply the weighting to the target pixel. In this case, the value of the filter element is decreased as the distance from the center increases. When such a filter is applied, motion blur is added so that the original pattern moves to a certain degree. Such motion blur enhances the motion of the subject.

[Filter application method]
A method for applying the filter will be further described. Here, as an example, a method of applying a filter at the boundary between a car that is a main subject and a tree that is a background subject will be described. The subject determination unit 114 determines the main subject region and the background subject region in the image. However, the boundary between the main subject region and the background subject region is not always determined clearly. Further, if the smoothing filter is suddenly applied in the vicinity of the boundary between the main subject region and the background subject region, it is possible that the sense of discomfort in the processed image increases. Therefore, in this embodiment, the discomfort of the image is suppressed by changing the smoothing filter effect (filter strength) gradually in accordance with the distance from the main subject near the boundary. For example, for the boundary between the car and the tree shown in FIG. 13A, a plurality (four in the figure) of boundary areas (margin areas) corresponding to the filter kernel in the corresponding area as shown in FIG. 13B. Is provided. In the plurality of boundary regions, smoothing filters having different kernel sizes and different filter element values are applied. And the smoothing filter shown in FIG.12 (b) is applied with respect to the area | region of the tree beyond the boundary part.

  FIG. 14 is a diagram illustrating a configuration of a smoothing filter kernel applied to each boundary region. In the first boundary region, the kernel size of the smoothing filter is set to 3 × 3 as shown in FIG. Thereafter, the second boundary region is 5 × 5 as shown in FIG. 14B, the third boundary region is 7 × 7 as shown in FIG. 14C, and the fourth boundary region is shown in FIG. 14D. Thus, the kernel size of the filter increases as the distance from the main subject increases, such as 9 × 9. Also, the filter element value is changed as the filter kernel size increases. As a result, the strength of the filter increases as the distance from the subject increases, and the smoothing effect increases accordingly.

  FIG. 15 is a flowchart showing a process for setting a boundary region. Here, each boundary region is set using a morphological operation on the region of the main subject. The filter application unit 122 initializes variables necessary for processing (step S301). Specifically, the filter application unit 122 sets Fs, which indicates the maximum size of the smoothing filter, to the size of the smoothing filter generated by the filter configuration unit 120, and C, which indicates the number of times of expanding the main subject region, to 1. initialize. Further, the filter application unit 122 sets a binary image representing the region information in the region information Img of the main subject.

  After the variable initialization, the filter application unit 122 executes a loop process from step S302. As the start of the loop processing, the filter application unit 122 determines whether or not the number of expansions C is smaller than a value Fs / 2-1 based on the maximum filter size (step S302). When C is smaller than Fs / 2-1, the filter application unit 122 executes the processing after step S303. When C is Fs / 2-1 or more, the filter application unit 122 ends the loop process. In the example of FIG. 15, the loop process is executed when Fs is larger than 3 × 3.

  The filter application unit 122 executes a binary dilation operation on the binary image Img, and stores the binary image Img-Prc as a result in the image buffer of the filter application unit 122 (step S303). The structure element of Dilation is 3 × 3 and 8 neighborhoods. Subsequently, the filter application unit 122 calculates a difference between the binary image Img-prc and the binary image Img stored in the image buffer (step S304). This difference is calculated in order to take out the pixel expanded by the previous Dilation operation.

  Subsequently, the filter application unit 122 stores the binary difference image extracted by calculating the difference as information indicating the C-th boundary region (step S305). Thereafter, the filter application unit 122 replaces the binary image Img with the binary image Img-Prc generated by the previous Dilation operation (step S306). At this time, the binary image Img is a C-expansion operation image for the area of the main subject. Subsequently, the filter application unit 122 increments the number of expansions C (step S307), and returns the process in step S308.

  After the end of the loop processing, the filter application unit 122 determines that necessary boundary area information has been obtained, and integrates information indicating all boundary areas (step S309). Here, in the case of the processing of FIG. 15, the information indicating the boundary region is set by the number depending on the maximum size of the smoothing filter. In actual application, not all boundary area information may be used. For example, in the case of the boundary between a house and a car, the size of the smoothing filter related to the house is 7 × 7, so the third boundary area and the fourth boundary area shown in FIG. 14B are not used.

  As described above, in the first embodiment, the degree of smoothing is changed in accordance with the magnitude of movement for each of the main subject area and the background subject area. As a result, it is possible to generate an image that is not affected by camera shake and the like, and the main subject as shown in FIG. Can do. A smoothing filter may be applied only to the image to be processed, and no image synthesis processing or the like is required to add motion blur.

  In this embodiment, the motion of the main subject area and the background subject is detected by estimating the motion vector. In general, since the magnitude of the motion vector between images changes according to the depth of the background (subject distance), if the motion vector is estimated, there is no need to separately calculate depth information. By appropriately setting the motion vector estimation target region according to the shooting mode, natural motion blur can be added with low calculation cost.

  Furthermore, in this embodiment, an image having the center of gravity of the main subject at the ideal center of gravity determined by the three-division method is determined as the target image. As a result, it is possible to obtain a panning shot of a composition desired by the photographer and an image taken for a long time. In the example of FIG. 8, the target image is determined from the actual gravity center position of the main subject. In addition, the target image may be determined by comparing the size of the area of the main subject among a plurality of images obtained during the continuous shooting operation. For example, among images obtained during the continuous shooting operation, an image having a composition in which the main subject is enlarged in the screen and stays at a predetermined position on the screen may be set as the target image. By determining the target image in this manner, the effect of enhancing the movement of the main subject can be increased.

(Second Embodiment)
[Constitution]
FIG. 16 is a diagram illustrating a configuration of an example of an imaging apparatus that performs image processing using the image processing method according to the second embodiment. FIG. 16 will be described only for parts different from FIG. The motion vector estimation unit 118 in the second embodiment divides an image into blocks without distinguishing the subject and estimates a motion vector for each block. Then, the motion vector estimated by the motion vector estimation unit 118 is input not only to the filter configuration unit 120 but also to the subject determination unit 124. The subject determination unit 124 determines a subject using the motion vector input from the motion vector estimation unit. Other configurations in FIG. 16 are the same as those in FIG.

[Operation]
Next, the operation of the imaging apparatus 100 according to the present embodiment will be described. FIG. 17 is a flowchart illustrating a shooting operation of the imaging apparatus 100 according to the second embodiment. In FIG. 17, different points from the first embodiment will be described. That is, the operations in steps S403 and S404 in FIG. 17 are different from those in the first embodiment. Steps S401, S402, S405 to S408 are the same as steps S101, S102, S104, and S106 to S108 in FIG.

  In FIG. 17, when an image is stored in the image storage unit 112, the motion vector estimation unit 118 estimates a motion vector (step S403). The motion vector estimation unit 118 in the second embodiment estimates the motion vector for each block after simply dividing the image into blocks without distinguishing between the main subject and the background subject. The subject determination unit 124 performs subject determination using the motion vector estimated by the motion vector estimation unit 118 (step S404). The main subject does not necessarily exist in the center of the image, but does not exist in the entire peripheral portion of the image. Therefore, the subject determination unit 124 estimates the motion of the background subject using the motion vectors of the blocks located in the peripheral portion of the image. Then, the subject determination unit 124 determines a location that is clearly different from the movement of the background subject for each block as the main subject. For example, when the shooting mode is the panning mode, the background subject moves to some extent, but the main subject hardly moves or represents a motion different from the background subject. When the shooting mode is the handheld long-time shooting mode, the background hardly moves while the main subject moves. Based on this situation, the subject is determined from the motion vector. That is, in the panning mode, a block having a motion around the periphery of the image is determined as a background subject. On the other hand, in the hand-held long-time shooting mode, a block that moves around the periphery of the image is determined as the main subject.

  The subject determination will be described more specifically with reference to FIGS. FIG. 18A shows an area in which a motion as a background subject in the screen is estimated. Here, the entire screen is divided into small areas of 7 × 7 pixels, and the main subject and the background subject are determined in units of small areas. The central 3 × 3 pixel region represents a portion where the main subject may exist. FIG. 18B shows a configuration of a block for estimating a motion vector. The block size is, for example, 5 × 5 pixels, as in the first embodiment.

  When subject determination is performed in a scene as shown in FIG. 19 under this condition, the subject determination method is changed depending on the shooting mode. For example, when the shooting mode is the panning mode, the estimated motion vector is as shown in FIG. At this time, the movement of the background subject becomes dominant. At this time, when the motion vectors are integrated in units of small areas in FIG. 18A (for example, by averaging the motion vectors in the small areas), the state shown in FIG. 20B is obtained. Here, motion vectors exist in more than half of the small areas of the background subject, and these motion vectors have no significant variation in direction or size (there is no such thing as the opposite direction or extremely different size). It is recognized as corresponding to the movement of the background subject. That is, a portion different from the movement of the background subject is determined as the main subject. Here, a small area group (including a downward-sloping diagonal pattern portion in FIG. 20B) connected including a small area where the main subject may exist is set as a main subject area. When the shooting mode is the handheld long-time shooting mode, the estimated motion vector is as shown in FIG. 21A, and the background subject does not move. At this time, when motion vectors are integrated in units of small areas in FIG. 18A, the state shown in FIG. Here, since the motion vector does not exist in more than half of the small area of the background subject, the background subject does not move, that is, the portion where the motion is detected is determined as the main subject. Here, a small area group including a small area in which a main subject may exist (a right-downward diagonal pattern portion in FIG. 21B) is a main subject area.

  In this embodiment, the motion estimation area diagram 18 (a) and the motion vector estimation block diagram 18 (b) are set separately, but both may be shared.

  As described above, in the second embodiment, the main subject and the background subject are determined from the motion vector. Thereby, it is not necessary to perform image recognition processing in the subject determination unit 124. Note that subject determination may be performed using image recognition processing together.

(Third embodiment)
[Constitution]
FIG. 22 is a diagram illustrating a configuration of an example of an imaging apparatus that performs image processing using the image processing method according to the third embodiment. FIG. 22 will be described only for parts different from FIG. In the third embodiment, a photographer can select a target image. For this reason, the operation unit 104 of the third embodiment further includes an operation member for the photographer to set a target image. The imaging apparatus 100 according to the third embodiment also includes a display unit 126 for displaying candidate target images.

[Operation]
Next, the operation of the imaging apparatus 100 according to the present embodiment will be described. In determining the target image, the target image determination unit 116 according to the present embodiment causes the display unit 126 to display, for example, thumbnails of the images obtained by the continuous shooting operation. The photographer looks at the image displayed on the display unit 126 and selects which image is the output target by operating the operation unit 104. In response to the operation of the operation unit 104, the target image determination unit 116 acquires information on the selected image (for example, the index number of the selected image in the case of thumbnail display), and determines the target image.

  A specific example of the contents displayed on the display unit 126 is shown in FIG. Four thumbnail images are displayed on one screen, and index numbers are overlapped on each image. In addition to this, the operation unit 104 can display temporally previous and subsequent images and change the number of thumbnails displayed. When the target image is determined from the thumbnail display, the actual target image is determined based on the index number of the selected image. Note that the operation of the operation unit 104 may be realized by incorporating a touch panel in the display unit 126 and touching the screen.

  As described above, according to the present embodiment, motion blur can be added to an image having a scene and composition desired by the photographer.

  Although the present invention has been described above based on the embodiments, the present invention is not limited to the above-described embodiments, and various modifications and applications are naturally possible within the scope of the gist of the present invention. Further, in the description of each operation flowchart described above, the operation is described using “first”, “next”, and the like for convenience, but this does not mean that it is essential to perform the operations in this order. Absent.

  In addition, each processing method performed by the imaging apparatus in the above-described embodiment, that is, the processing illustrated in each flowchart is stored as a program that can be executed by the processing control unit 108. This program is stored and distributed in a storage medium of an external storage device such as a memory card (ROM card, RAM card, etc.), magnetic disk (floppy disk, hard disk, etc.), optical disk (CD-ROM, DVD, etc.), semiconductor memory, etc. can do. Then, the process control unit 108 reads the program stored in the storage medium of the external storage device, and the operation is controlled by the read program, so that the process described above can be executed.

  Further, the above-described embodiments include various stages of the invention, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some configuration requirements are deleted from all the configuration requirements shown in the embodiment, the above-described problem can be solved, and this configuration requirement is deleted when the above-described effects can be obtained. The configuration can also be extracted as an invention.

  DESCRIPTION OF SYMBOLS 100 ... Imaging device, 102 ... Imaging part, 104 ... Operation part, 106 ... Shooting condition setting part, 108 ... Processing control part, 110 ... Information recording part, 112 ... Image storage part, 114 ... Subject determination part, 116 ... Target image Decision unit 118 118 Motion vector estimation unit 120 Filter configuration unit 122 Filter application unit 124 Subject determination unit 126 Display unit 1021 Lens 1022 Imaging element 1023 Imaging control unit

Claims (9)

An imaging unit that continuously captures a subject and obtains a plurality of images;
A subject determination unit that determines a region of a main subject and a region of a background subject in each of the acquired plurality of images;
A target image determination unit that determines a target image from the plurality of acquired images;
A motion vector estimation unit that estimates a motion vector of at least the background subject area between the target image and an image captured at the time when the target image is captured and a time before and after the target image;
A filter constituting unit constituting a filter to be applied to the target image based on the estimated motion vector;
A filter application unit that applies the configured filter to the target image;
An image processing apparatus comprising: The motion vector estimation unit divides each of the acquired plurality of images into a plurality of blocks, estimates a motion vector between the images for each block,
The subject determination unit determines a main subject region and a background subject region in each of the plurality of acquired images based on the estimated motion vector;
The image processing apparatus according to claim 1.   The imaging unit shortens the exposure time to such an extent that blurring of the subject does not occur, increases the sensitivity by shortening the exposure time, and increases the noise reduction level of the noise removal processing by increasing the sensitivity. The image processing apparatus according to claim 1 or 2, wherein   The filter application unit provides a margin region at the boundary between the main subject region and the background subject region determined by the subject determination unit, and determines the filter strength in the margin region from the main subject region. The image processing apparatus according to claim 1, wherein the filter is configured to increase as the distance increases.   The image processing apparatus according to claim 1, wherein the subject determination unit determines the main subject region and the background subject region by image recognition.   The image processing apparatus according to claim 1, wherein the target image determination unit determines the target image based at least on the position of the region of the main subject determined by the subject determination unit.   The subject determination unit divides the acquired image into a central region and a peripheral region, respectively, and based on a motion vector of the peripheral region, the background subject region and the main subject region The image processing device according to claim 2, wherein the image processing device determines whether or not. Capturing a plurality of images by continuously capturing a subject;
Determining a main subject region and a background subject region in each of the plurality of acquired images;
Determining a target image from the plurality of acquired images;
Estimating a motion vector of at least the area of the background subject between the target image and an image captured at a time before and after the time when the target image was captured;
Configuring a filter to be applied to the target image based on the estimated motion vector;
Applying the configured filter to the target image;
An image processing method comprising: The ability to continuously capture a subject and acquire multiple images;
A function of determining a main subject region and a background subject region in each of the plurality of acquired images;
A function of determining a target image from the plurality of acquired images;
A function for estimating a motion vector of at least the background subject area between the target image and the image captured at the time when the target image was captured and the time before and after the target image;
A function of configuring a filter to be applied to the target image based on the estimated motion vector;
A function of applying the configured filter to the target image;
An image processing program for realizing a computer.

Images