JP5703788B2 - Imaging apparatus, image processing apparatus, and image processing program - Google Patents

Description

  The present invention relates to an imaging device, an image processing device, and an image processing program.

  A technique for estimating a moving area as a main subject area from a captured image has been proposed (see Patent Document 1).

JP 2002-232777 A

  However, when the main subject is moving, the photographer may take an image (panning, tilting, etc.) while moving the imaging device in accordance with the movement of the main subject. When imaging is performed while moving the imaging device, in the captured image, a moving main subject is an image with little movement, and a background region other than the main subject region is an image with much movement. For this reason, there is a problem in that the main subject region cannot be estimated appropriately in the method of detecting a moving image region as the main subject.

  The present invention has been made in view of such circumstances, and an object thereof is to provide an imaging device, an image processing device, and an image processing program that appropriately estimate a main subject region from a captured image.

The present invention has been made to solve the above-described problems. The present invention provides an imaging unit that images a subject, a shake detection unit that detects a shake amount, and a plurality of images that are captured in time series by the imaging unit. In the frame image, a frame image dividing unit that divides the frame image into a plurality of image regions, a motion vector detection unit that detects a motion vector of the image region for each image region, and the plurality of frame images are captured. When the shake amount of the imaging unit detected by the shake detection unit is greater than or equal to a predetermined amount, based on the motion vector of the image region detected by the motion vector detection unit, among the plurality of image regions the motion vector is smaller image region, and a main object estimation section for estimating that the region including the main subject, is detected by the motion vector detecting section The direction of the motion vector of the predetermined image area in the plurality of frame images and the shake direction of the imaging unit detected by the shake detection unit when the plurality of frame images are captured are a predetermined frame. When the same direction is continued for a number of times, the main subject estimation unit is an image having a small motion vector among the plurality of image regions based on the motion vector of the image region detected by the motion vector detection unit. The imaging apparatus is characterized in that the area is estimated to be a main subject area .

The present invention also provides a shake information input unit to which shake information of the imaging device is input, a frame image input unit to which a plurality of frame images captured in time series by the imaging device are input, and a plurality of the frame images. A plurality of frame images input from the shake information input unit; a frame image dividing unit that divides the image region; a motion vector detection unit that detects a motion vector of the image region for each image region; When the shake amount of the imaging device at the time is a predetermined amount or more, based on the motion vector of the image region detected by the motion vector detection unit, an image region having a small motion vector among the plurality of image regions, comprising a main subject estimator for estimating that the region including the main subject, wherein the plurality of frame images is detected by the motion vector detecting section The direction of the motion vector of the predetermined image area in the camera and the shake direction of the imaging device when the plurality of frame images input from the shake information input unit are captured are continuously the same for a predetermined number of frames or more. When the direction is a direction, the main subject estimation unit determines an image region having a small motion vector among the plurality of image regions based on the motion vector of the image region detected by the motion vector detection unit. It is an image processing apparatus characterized by presuming that it is.

The present invention also provides a shake information input procedure for inputting shake information of the imaging device to a computer as an image processing device, and a frame image input procedure for inputting a plurality of frame images taken in time series by the imaging device. A frame image dividing procedure for dividing the frame image into a plurality of image regions; a motion vector detecting procedure for detecting a motion vector of the image region for each image region; and When the shake amount of the imaging device is a predetermined amount or more, based on the motion vector of the image region detected by the motion vector detection procedure, an image region having a small motion vector among the plurality of image regions is selected as a main subject. anda main subject estimation procedure for estimating that the region included, the plurality of detected by the motion vector detection procedure The direction of the motion vector of a predetermined image area in the frame image and the shake direction of the imaging device when the plurality of frame images are captured by the frame image input procedure input from the shake information input procedure. When the same direction continues for a predetermined number of frames or more, the motion vector of the plurality of image regions is based on the motion vector of the image region detected by the motion vector detection procedure by the main subject estimation procedure. This is an image processing program for executing processing for estimating an image region having a small size as a main subject region .

  According to the present invention, the imaging device can appropriately estimate the main subject region from the captured image even when the main subject has a motion.

It is a schematic block diagram which shows the structure of the imaging device by 1st Embodiment. It is explanatory drawing which shows an example of the motion vector in a frame image. It is a flowchart which shows the main photographic subject estimation process in 1st Embodiment. It is a flowchart which shows the main photographic subject estimation process in 2nd Embodiment. It is a flowchart which shows the main to-be-photographed object estimation process in 3rd Embodiment. It is a flowchart which shows the main to-be-photographed object estimation process in 4th Embodiment. It is a flowchart which shows the position correction process in 5th Embodiment.

Embodiments of the present invention will be described below with reference to the drawings.
<First Embodiment>
FIG. 1 is a schematic block diagram showing the configuration of the imaging apparatus 100 according to the first embodiment of the present invention. 1 includes an imaging unit 2, a control unit 3, an image processing unit (image processing device) 4, a storage unit 5, a buffer memory unit 6, a display unit 7, a shake detection sensor 8 (a shake detection unit), A shake correction unit 9, an operation unit 11, a card connector 12, and a bus 15 are included.

The imaging unit 2 includes a lens unit 21, an imaging element 22, and an A / D conversion unit 23. The imaging unit 2 captures a subject and generates image data. The imaging unit 2 is controlled by the control unit 3 based on the set imaging conditions (for example, aperture value, exposure, etc.), and the subject luminous flux incident through the lens unit 21 forms a subject image on the imaging surface of the imaging element 22. Make an image. In addition, the imaging unit 2 converts the analog signal output from the imaging element 22 into a digital signal in the A / D conversion unit 23 to generate image data.
Note that the lens unit 21 described above may be attached to and integrated with the imaging apparatus 100, or may be detachably attached to the imaging apparatus 100.

  For example, the imaging unit 2 outputs moving image data in response to a moving image imaging operation performed by the operation unit 11. The imaging unit 2 outputs still image data in response to a still image imaging operation by the operation unit 11. The moving image image data and still image data captured by the imaging unit 2 are recorded in the memory card 200 by the control unit 3 via the buffer memory unit 6 and the image processing unit 4. The imaging unit 2 outputs image data of a through image during a standby state for performing still image imaging.

  The control unit 3 performs control of each unit included in the imaging apparatus 100 and various arithmetic processes. The image processing unit 4 performs image processing on the image data stored in the buffer memory unit 6 based on the image processing conditions stored in the storage unit 5. Here, the image data stored in the buffer memory unit 6 is, for example, moving image data captured by the imaging unit 2, still image data, or through image data, or read from the memory card 200. This is the output image data. The image processing unit 4 divides the captured image into a plurality of predetermined image areas, and detects a motion vector of the image area for each of the divided image areas. Then, the image processing unit 4 estimates the main subject from the captured image based on the detected motion vector. The configuration of the image processing unit 4 and the process of estimating the main subject will be described later.

  The storage unit 5 stores predetermined imaging control conditions, image processing conditions, reproduction control conditions, display control conditions, recording control conditions, output control conditions, and the like for controlling the imaging apparatus 100. For example, the storage unit 5 is a ROM (Read Only Memory). The storage unit 5 may be provided in the image processing unit 4. The storage unit 5 included in the image processing unit 4 may store predetermined image processing conditions used for image processing in the image processing unit 4. The storage unit 5 may record captured moving image data and still image data. In this case, for example, the storage unit 5 may be a flash memory or the like.

  The buffer memory unit 6 is used as a work area when the control unit 3 controls the imaging apparatus 100. The moving image image data, the live view image data, the still image data, and the image data read from the memory card 200 are buffered in the image processing process under the control of the control unit 3. It is temporarily stored in the memory unit 6. The buffer memory unit 6 is, for example, a RAM (Random Access Memory). The buffer memory unit 6 may be provided in the image processing unit 4 and used as a work area in the process of image processing under the control of the image processing unit 4.

  The display unit 7 is, for example, a liquid crystal display, and is an image based on image data captured by the imaging unit 2 or an image based on image data read from the memory card 200, a menu screen, or an operation of the imaging device 100. Displays information about status and settings.

  The shake detection sensor 8 detects the shake amount and the shake direction of the imaging device 100 as the movement of the housing of the imaging device 100. The shake detection sensor 8 is an angular velocity sensor that detects a biaxial angle and a biaxial angular velocity, for example. And the shake direction). Then, based on the detection result of the shake detection sensor 8, the control unit 3 controls the image processing unit 4 and the shake correction unit 9.

  The shake correction unit 9 moves the shake correction lens included in the lens unit 21 to change the optical axis. The shake correction unit 9 moves the shake correction lens of the lens unit 21 in a direction that cancels out the shake amount and the shake direction of the imaging device 100 detected by the shake detection sensor 8. Correct the relative position to reduce the influence of shake. As a result, the shake correction unit 9 detects the relative movement between the optical axis and the imaging surface based on the result detected by the shake detection sensor 8 with respect to, for example, the camera shake that occurs when the operator holds the image pickup apparatus 100 and takes an image. Correct the position.

  The operation unit 11 has an operation switch for an operator to input an operation to the imaging apparatus 100. For example, the operation unit 11 includes a power switch, a release switch, a mode switch, a menu switch, an up / down / left / right selection switch, a confirmation switch, a cancel switch, and other operation switches.

  A removable memory card (storage medium) 200 such as a card memory is inserted into the card connector 12. The image data is written to, read from, or deleted from the memory card 200 via the card connector 12. The memory card 200 is a storage unit that is detachably connected to the imaging apparatus 100, and stores, for example, image data captured by the imaging unit 2.

  The bus 15 is connected to the imaging unit 2, the control unit 3, the image processing unit 4, the storage unit 5, the buffer memory unit 6, the display unit 7, the shake detection sensor 8, the shake correction unit 9, the operation unit 11, and the card connector 12. Then, image data, control signals and the like output from each unit are transferred.

Next, the configuration of the image processing unit 4 will be described in detail. The image processing unit 4 includes a frame image division unit 40, a motion vector detection unit 41, a main subject estimation unit 42, and a position correction control unit 43. The frame image dividing unit 40 divides each of image data (hereinafter also simply referred to as frame images) of a plurality of frame images captured in time series by the imaging device 100 into a plurality of predetermined image areas. The motion vector detection unit 41 detects the motion vector of the image region for each image region divided by the frame image division unit 40.
In the following embodiment, a case where a plurality of frame images captured in time series are moving images will be described as an example. For example, the motion vector detection unit 41 detects a motion vector for each of the plurality of frame images of the moving image captured by the imaging device 100 for each image region divided by the frame image division unit 40.

  The main subject estimation unit 42 detects the image detected by the motion vector detection unit 41 when the shake amount of the imaging apparatus 100 detected by the shake detection sensor 8 during a period in which a plurality of frame images are captured is equal to or greater than a predetermined shake amount. Based on the motion vector of the region, an image region having a small motion vector among a plurality of image regions of the frame image is estimated as the main subject region. For example, the main subject estimation unit 42, for example, when the motion shake amount of the imaging device 100 during a period in which a moving image is captured is equal to or greater than a predetermined shake amount, the motion detected by the motion vector detection unit 41 in the frame image of the moving image. Based on the vector, an image area having a small motion vector among a plurality of image areas in the frame image is estimated as a main subject area.

  Here, the case where the shake amount of the imaging device 100 is equal to or greater than the predetermined shake amount is a case where the photographer intentionally moves the imaging device, such as panning the imaging device 100. On the other hand, the shake amount due to camera shake that occurs when the image pickup apparatus 100 is held and picked up is when the shake amount of the image pickup apparatus 100 is smaller than a predetermined shake amount. The predetermined shake amount or more may be a case where the shake amount is greater than or equal to a predetermined size, and the shake direction of the imaging apparatus 100 and the direction of the motion vector in the predetermined image area are predetermined. It may be the case that the number of frames continues to match.

FIG. 2 is an explanatory diagram illustrating an example of a motion vector in a frame image.
A process in which the main subject estimation unit 42 estimates the main subject region based on the motion vector detected by the motion vector detection unit 41 will be described with reference to FIG. FIG. 2A shows a case where the photographer is capturing a moving image while panning the imaging device 100 in accordance with the movement of the moving main subject A. In FIG. 2A, the photographer changes the imaging direction of the imaging apparatus 100 to the position B in accordance with the movement of the main subject A from the position A (1) to the position A (2) horizontally from right to left. Moving horizontally from right to left from (1) to position B (2). Here, a case where the photographer pans the imaging device 100 so that the main subject A is in the center of the angle of view will be described. In this case, the motion vector in the frame image of the captured moving image is small near the center of the angle of view, that is, in the region where the main subject A is imaged, and the background other than the end of the angle of view, that is, the background other than the main subject A is captured. It becomes larger in the background area.

  2B, one of the frame images of the moving image captured in FIG. 2A is divided into a plurality of image areas determined in advance by the frame image dividing unit 40, and the motion vector detecting unit 41 An example in which a motion vector in a frame image is detected as an image motion for each image region is shown. In FIG. 2B, the frame image dividing unit 40 divides the captured frame image of the moving image into predetermined “5 (horizontal direction) × 4 (vertical direction)” blocks (image areas). The motion vector detection unit 41 detects a motion vector in the frame image for each block. For example, when the motion vector detection unit 41 detects a motion vector in the block (1) of the “Nth” frame image, the image data of the block (1) of the “N−1” th frame image and the “Nth” ”Is compared with the image data of the block (1) of the frame image of“ ”(N is an integer).

  In FIG. 2B, since the photographer pans the imaging device 100 in accordance with the movement of the main subject A, the motion vector of the image area in which the main subject A is imaged, that is, near the center of the angle of view. The motion vectors of the blocks (15) to (20) are small motion vectors. On the other hand, the motion vector of the background area in which the background other than the main subject A is imaged, that is, the motion vectors of the blocks (1) to (14) near the edge of the angle of view, move in the direction opposite to the panning direction. The motion vector is larger than the nearby motion vector.

Then, the main subject estimation unit 42 estimates that the image region that is a small motion vector, that is, the blocks (15) to (20) near the center of the angle of view in FIG. For example, the main subject estimation unit 42 compares the magnitude of the motion vector detected for each block by the motion vector detection unit 41 with a predetermined threshold MV _th1, and a block having a motion vector larger than the threshold MV _th1 and the threshold MV Clustering into blocks of motion vectors equal to or less than _th1 . Then, the main subject estimation unit 42 estimates that the image region formed by the motion vector block equal to or less than the threshold MV _th1 is the main subject region. Here, the predetermined threshold value MV _th1 is a threshold value of the size of a predetermined motion vector in order to estimate a small motion vector region as the main subject region. The threshold value MV _th1 may be determined by the shake detection sensor 8. For example, the threshold value MV _th1 may be set to a larger value as the detection value output from the shake detection sensor 8 is larger.

  Although the case where the photographer pans the imaging apparatus 100 in accordance with the movement of the main subject has been described with reference to FIG. 2, the present invention is not limited to this. For example, the photographer may tilt the imaging device 100 in accordance with the movement of the main subject, or may move the imaging device 100 in an oblique direction in accordance with the movement of the main subject. Also good.

  In addition, an example has been described in which the frame image dividing unit 40 divides each of a plurality of captured frame images into predetermined “5 (horizontal direction) × 4 (vertical direction)” blocks (image regions). The blocks to be divided are not limited to “5 (horizontal direction) × 4 (vertical direction)”. The frame image dividing unit 40 may divide the number of blocks in the horizontal direction and the vertical direction other than “5 (horizontal direction) × 4 (vertical direction)”.

Further, the example in which the motion vectors detected by the motion vector detection unit 41 are clustered using one predetermined threshold MV _th1 has been described. However, the clustering threshold is not limited to one threshold and is clustered using a plurality of thresholds. Also good.

Next, the operation of the main subject estimation process in the first embodiment will be described.
FIG. 3 is a flowchart showing main subject estimation processing in the first embodiment.

  The frame image dividing unit 40 of the image processing unit 4 divides the captured frame image of the moving image into predetermined “5 (horizontal direction) × 4 (vertical direction)” blocks (image areas) (step S1). ). Next, the motion vector detection unit 41 detects a motion vector in the frame image for each divided block (step S2).

  The shake detection sensor 8 detects the movement (the shake amount and the shake direction) of the housing of the imaging device 100 (step S3). Then, the control unit 3 determines whether or not the shake amount of the imaging device 100 when the moving image is captured (period) is greater than or equal to a predetermined shake amount based on the detection result by the shake detection sensor 8 (step) S5).

In step S <b> 5, if the control unit 3 determines that the shake amount of the imaging apparatus 100 during the period in which the moving image is captured is equal to or greater than the predetermined shake amount, the main subject estimation unit 42 is detected by the motion vector detection unit 41. A block whose motion vector is equal to or smaller than a predetermined threshold MV _th1 is estimated to be a main subject region (step S6).

  On the other hand, when the control unit 3 determines in step S5 that the shake amount of the imaging device 100 during the period in which the moving image is captured is not equal to or greater than the predetermined shake amount, the image processing unit 4 performs the main subject estimation unit 42 to The process ends without executing the process of estimating the subject area.

  As described above, according to the first embodiment, when the shake amount of the imaging apparatus 100 is equal to or greater than the predetermined shake amount during the period in which the moving image is captured, the image processing unit 4 detects in the frame image of the moving image. Based on the motion vector in the frame image, an image region having a small motion vector among the image regions in the frame image is estimated as the main subject region. Therefore, even when the photographer captures an image while moving the imaging apparatus 100 in accordance with the movement of the main subject, the imaging apparatus 100 can appropriately estimate the main subject area from the captured image.

  3, the control unit 3 uses the shake amount and shake direction of the image pickup apparatus 100 detected by the shake detection sensor 8 to determine whether the shake amount of the image pickup apparatus 100 during a period during which a moving image is captured is a predetermined amount. The example in which it is determined whether or not the shake amount is greater than or equal to the shake amount has been described. However, the shake amount of the imaging apparatus 100 is determined based on the motion vector of the image area in the frame image detected by the motion vector detection unit 41 of the image processing unit 4. It may be determined whether or not is greater than a predetermined shake amount.

  For example, the control unit 3 determines whether or not the shake amount of the imaging apparatus 100 is equal to or greater than the predetermined shake amount based on the motion vectors of a predetermined image area in the plurality of frame images detected by the motion vector detection unit 41. Determine. Here, the predetermined image area is a predetermined image area near the edge of the angle of view (screen) that is likely to be a background area where a background other than the main subject is captured. That is, when the motion vector in the region near the edge of the angle of view is a motion vector having a predetermined size or more, the control unit 3 has a shake amount of the imaging apparatus 100 during a period in which a moving image is captured being a predetermined shake amount or more. Judge that there is. Note that when the control unit 3 determines that the motion vector of a predetermined image area in a plurality of frame images is a motion vector in the same direction continuously for a predetermined number of frames or more, the control unit 3 captures a period during which a moving image is captured. You may determine with the shake amount of the apparatus 100 being more than predetermined shake amount. Thereby, the movement of the imaging device 100 can be detected without using the shake detection sensor 8.

Second Embodiment
In the second embodiment, the control unit 3 detects both the shake amount and shake direction of the imaging apparatus 100 detected by the shake detection sensor 8 and the motion vector of the image detected by the motion vector detection unit 41. Based on the above, it is determined whether or not the shake amount of the imaging apparatus 100 during the period in which the moving image is taken is equal to or greater than a predetermined shake amount. The other configuration of the imaging apparatus 100 of the second embodiment is the same as the configuration of the first embodiment described with reference to FIG.

  The control unit 3 continues the motion vector direction of a predetermined image area in the plurality of frame images and the shake direction of the imaging device 100 during the period in which the plurality of frame images are captured for a predetermined number of frames or more. It is determined whether or not they match. Then, when it is determined that the number of frames continuously matches the predetermined number of frames, the main subject estimation unit 42 performs a plurality of operations based on the motion vector of the image area in the frame image detected by the motion vector detection unit 41. The image region having a small motion vector is estimated as the main subject region.

  For example, the predetermined image region in the plurality of frame images described above is an image region formed of a predetermined image region group among the plurality of image regions in the frame image. The control unit 3 sets the direction of the motion vector of the predetermined image region group to the image region having the maximum motion vector (image region having the maximum motion vector) among the plurality of image regions in the predetermined image region group. Determine the direction of the motion vector.

  The motion vector detection unit 41 detects the motion vectors of the blocks (1) to (14) near the edge of the angle of view shown in FIG. 2 as the direction of the motion vector of the predetermined image region group. The predetermined image region group is a preset image region group that is highly likely to be a background region in which a background other than the main subject is captured. The motion vector detection unit 41 detects the motion vectors of the blocks (1) to (14) near the edge of the angle of view as the motion vector of the background area. Then, the direction of the motion vector in the block having the maximum motion vector among the blocks (1) to (14) is set as the direction of the motion vector in the predetermined image region group, that is, the background region.

  For example, in the blocks (1) to (14) near the edge of the angle of view shown in FIG. 2, it is assumed that the person is located in the blocks (6) to (7) near the edge of the angle of view. If the person is chasing the main subject located in blocks (15)-(20), the motion vectors in blocks (6)-(7) are similar to the motion vectors in blocks (15)-(20) Become a vector. Also, assuming that the main subject, for example, a train, is located in the area where the blocks (15) to (20) and the blocks (6) to (7) are combined, the motion vector in the blocks (6) to (7) is the block ( The motion vector approximates the motion vector in 15) to (20). Therefore, the motion vectors in all the blocks (1) to (14) near the edge of the angle of view are not motion vectors approximate to the motion vectors in the blocks (15) to (20). Therefore, the motion vector detection unit 41 detects the direction of the motion vector in the block having the maximum motion vector size among the blocks (1) to (14) as the direction of the motion vector in the background region.

  Further, the control unit 3 determines, based on the result detected by the shake detection sensor 8, whether or not the shake direction and the shake amount of the imaging device 100 are equal to or greater than a predetermined shake amount. When the control unit 3 determines that the direction of the motion vector in the background area and the shake direction of the image pickup apparatus 100 detected by the shake detection sensor 8 continuously match a predetermined number of frames or more, the image pickup apparatus It is recognized that the shake amount of 100 is equal to or greater than the predetermined shake amount, that is, the photographer is taking an image while moving the imaging device 100 by panning or the like. In this case, as in the first embodiment, the main subject estimation unit 42 uses a plurality of image regions in the frame image based on the motion vector in the frame image detected by the motion vector detection unit 41 in the frame image of the moving image. Among them, an image area having a small motion vector is estimated as a main subject area.

  Here, the control part 3 determines by comparing the direction of a motion vector with a predetermined threshold value. For example, the control unit 3 compares the four angle thresholds (45 degrees, 135 degrees, 225 degrees, and 315 degrees) with the angle of the motion vector, so that the direction of the motion vector is the right or left in the horizontal direction. Or which of the four directions above or below the vertical direction is determined.

Specifically, the control unit 3 determines which of the four directions the direction of the motion vector is, for example, by comparing the thresholds of the four angles and the angles of the motion vector as follows. To do.
Horizontal right: When the angle of the motion vector is 0 degree or more and less than 45 degree, or 315 degree or more and less than 360 degree.
Horizontal left: The angle of the motion vector is 45 degrees or more and less than 135 degrees.
Vertical top: The motion vector angle is 135 degrees or more and less than 225 degrees.
Down in the vertical direction: The angle of the motion vector is 225 degrees or more and less than 315 degrees.

  Similarly, the control unit 3 determines which of the four directions the shake direction of the imaging apparatus 100 detected by the shake detection sensor 8 is. Then, based on the detected direction, the control unit 3 continuously matches the direction of the motion vector in the background region with the shake direction of the imaging device 100 detected by the shake detection sensor 8 for a predetermined number of frames or more. Whether or not it is detected is determined.

  Note that the direction of the motion vector in the background area detected by the motion vector detection unit 41 by the control unit 3 and the direction of the imaging device 100 detected by the shake detection sensor 8 are any of the four directions. The method for determining whether or not one of the four directions (up, down, left, and right) is detected by comparing with the threshold values of four angles. For example, it may be a method of detecting which of the eight directions including the diagonal directions in the up, down, left, and right directions by comparing with threshold values of eight angles. Further, other directions may be detected based on other threshold values.

FIG. 4 is a flowchart showing main subject estimation processing in the second embodiment.
Hereinafter, the main subject estimation process of the imaging apparatus 100 according to the second embodiment will be described with reference to the flowchart shown in FIG. Here, step S11 to step S12 in FIG. 4 correspond to step S1 to step S2 in FIG. 3, and description thereof will be omitted.

  The motion vector detection unit 41 detects motion vectors of a plurality of blocks (a predetermined image region group, that is, a plurality of blocks in the background region) near the edge of the angle of view. Then, the direction of the motion vector in the block having the maximum motion vector among the plurality of blocks is detected as the direction of the motion vector in the background area (step S13).

  The shake detection sensor 8 detects the shake direction of the imaging device 100 during the period in which the moving image is captured (step S14). Then, the control unit 3 continuously detects the shake direction of the imaging apparatus 100 detected by the shake detection sensor 8 and the direction of the motion vector in the background area detected by the motion vector detection unit 41 for a predetermined number of frames or more. It is determined whether or not it has been done (step S15).

In step S15, when it is determined that the shake direction of the imaging apparatus 100 and the direction of the motion vector in the background region continuously match the predetermined number of frames or more, the main subject estimation unit 42 determines the motion vector detection unit 41. A block whose motion vector is smaller than a predetermined threshold MV _th1 is estimated as a main subject region (step S16).

  On the other hand, in step S15, when it is determined that the shake direction of the imaging apparatus 100 and the direction of the motion vector in the background region do not continuously match the predetermined number of frames or more, the image processing unit 4 The process ends without executing the process of estimating the main subject area by 42.

  As described above, according to the second embodiment, the control unit 3 detects the shake direction of the imaging device 100 detected by the shake detection sensor 8 and the predetermined image region group (detected by the motion vector detection unit 41). When the direction of the motion vector in the background area is continuously consistent with the predetermined number of frames or more, it is determined that the shake amount of the imaging device 100 during the period in which the moving image is captured is equal to or greater than the predetermined shake amount. Then, based on the motion vector in the frame image detected in the frame image of the moving image, the image processing unit 4 determines that the image region having a small motion vector among the image regions of the frame image of the moving image is the main subject region. presume. In the case of camera shake by the photographer, the direction of the shake is unlikely to be the same direction for a long time. According to the above configuration, it is possible to reduce a possibility that the camera shake is erroneously determined as, for example, panning by the process of step S15. Therefore, even when the photographer captures an image while moving the imaging region of the imaging device 100 in accordance with the movement of the main subject, the imaging device 100 can appropriately estimate the main subject region from the captured image. .

  Further, the image processing unit 4 sets the direction of the motion vector in the background area as the direction of the motion vector in the image area having the maximum motion vector among the plurality of image areas in the background area. Therefore, even when a part of the background region includes a part of the main subject or a subject that moves close to the motion of the main subject, the image processing unit 4 appropriately sets the direction of the motion vector of the background region. Can be detected.

<Third Embodiment>
In the third embodiment, similarly to the second embodiment, the control unit 3 detects the shake amount and shake direction of the imaging apparatus 100 detected by the shake detection sensor 8 and the motion vector detection unit 41. Based on both detection results of the image motion vector, it is determined whether or not the shake amount of the imaging device 100 during the period in which the time-series image is taken is equal to or greater than a predetermined shake amount. In the third embodiment, the detection method of the direction of the motion vector of the image in the background region (predetermined image region group) is different from the second embodiment. Other configurations of the imaging apparatus 100 of the third embodiment are the same as the configurations of the first embodiment described with reference to FIG.

  In the second embodiment, the direction of the motion vector of the imaging device 100 is determined based on the direction of the motion vector of the image region having the maximum motion vector in the predetermined image region group. In the third embodiment, the image processing unit 4 sets the direction of the largest motion vector among the directions of the motion vectors for each of the plurality of image areas as the direction of the motion vector of the imaging apparatus 100. For example, the motion vector detection unit 41 detects the motion vectors of the blocks (1) to (14) near the end of the imaging region shown in FIG. 2 as the motion vector of the background region. Then, the direction of the most motion vector among the motion vectors in the blocks (1) to (14) is set as the direction of the imaging apparatus 100.

  In the blocks (1) to (14) near the edge of the angle of view shown in FIG. 2, a person is located in the blocks (6) to (7) near the edge of the angle of view, and the person is in the blocks (15) to (20 ) Is chasing the main subject located at). Alternatively, it is assumed that a main subject, for example, a train is located in a region where the blocks (15) to (20) and the blocks (6) to (7) are combined. In this case, the direction of the motion vector in the blocks (6) to (7) is the same as the direction of the motion vector in the blocks (15) to (20). Therefore, the direction of the motion vector in all the blocks (1) to (14) near the edge of the angle of view is not the same as the direction of the motion vector in the blocks (15) to (20). Therefore, the motion vector detection unit 41 uses the most motion vector direction among the blocks (1) to (14) as the motion vector direction in the background region, so that the background region other than the main subject is imaged. The motion vector is detected. Note that the method by which the motion vector detection unit 41 detects the direction of the motion vector is the same as the example described in the second embodiment, and the description thereof is omitted.

  Then, the control unit 3 continues the motion vector direction in the background area detected by the motion vector detection unit 41 and the shake direction of the imaging device 100 detected by the shake detection sensor 8 for a predetermined number of frames or more. If it is determined that the image has been captured, it is determined that the shake amount of the imaging device 100 is equal to or greater than the predetermined shake amount, that is, the photographer is imaging while moving the imaging device 100 by panning or the like. In this case, as in the second embodiment, the main subject estimation unit 42 uses the motion vector in the frame image of the moving image based on the motion vector in the frame image detected by the motion vector detection unit 41 in the frame image of the moving image. The image region having a small motion vector is estimated as the main subject region.

FIG. 5 is a flowchart showing main subject estimation processing in the third embodiment.
Hereinafter, the main subject estimation process of the imaging apparatus 100 according to the third embodiment will be described with reference to the flowchart illustrated in FIG. Here, Steps S21 to S22 and Steps S24 to S26 in FIG. 5 correspond to Steps S11 to S12 and Steps S14 to S16 in FIG. 4 and will not be described. Only Step S23 will be described.

  The motion vector detection unit 41 detects motion vectors of a plurality of blocks (a predetermined image region group, that is, a plurality of blocks in the background region) near the edge of the angle of view. Then, the direction of the most motion vector among the motion vectors of the plurality of blocks is detected as the direction of the motion vector in the predetermined image region group (step S23).

  As described above, according to the third embodiment, similarly to the second embodiment, the control unit 3 detects the shake direction of the imaging device 100 detected by the shake detection sensor 8 and the motion vector detection unit 41. If the direction of the motion vector of the predetermined image area (background area) that has been continuously matched is equal to or greater than the predetermined number of frames, the shake amount of the imaging apparatus 100 during the period in which the moving image is captured is equal to the predetermined shake amount It determines with it being above. Then, based on the motion vector in the frame image detected in the frame image of the moving image, the image processing unit 4 estimates an image region having a small motion vector among the image regions of the frame image of the moving image as the main subject region. To do. Therefore, even when the photographer captures an image while moving the imaging region of the imaging device 100 in accordance with the movement of the main subject, the imaging device 100 can appropriately estimate the main subject region from the captured image. it can.

  Further, the motion vector detection unit 41 sets the direction of the motion vector in the background area as the direction of the most motion vector among the directions of the motion vectors in the plurality of image areas in the background area. Therefore, even when a part of the background region includes a part of the main subject, a subject that moves close to the motion of the main subject, or a subject that moves differently from the motion of the background region, the motion vector detection unit 41 Can appropriately detect the direction of the motion vector of the background region.

<Fourth embodiment>
In the fourth embodiment, an example in which the process of estimating the main subject is switched depending on whether or not the shake amount of the imaging apparatus 100 is equal to or greater than a predetermined shake amount during a period in which a plurality of frame images are taken is shown. The other configuration of the imaging apparatus 100 of the fourth embodiment is the same as the configuration of the first embodiment described with reference to FIG.

  When the shake amount of the imaging apparatus 100 is equal to or greater than a predetermined shake amount during a period in which a plurality of frame images are captured, the main subject estimation unit 42 uses the motion vector in the frame image detected by the motion vector detection unit 41. Based on this, an image area having a small motion vector among a plurality of predetermined image areas in the frame image is estimated as the main subject area. For example, if the shake amount of the imaging apparatus 100 is equal to or greater than a predetermined shake amount during the period in which the moving image is captured, the main subject estimation unit 42 detects the motion vector detected by the motion vector detection unit 41 in the frame image of the moving image. Based on the above, it is estimated that an image area having a small image motion vector among a plurality of image areas in the frame image of the moving image is a main subject area.

  On the other hand, when the shake amount of the imaging apparatus 100 is not equal to or greater than a predetermined shake amount during a period in which a plurality of frame images are captured, the main subject estimation unit 42 detects a motion vector in the frame image detected by the motion vector detection unit 41. Based on the above, an image area having a large motion vector among a plurality of predetermined image areas in the frame image is estimated as the main subject area. For example, if the shake amount of the imaging apparatus 100 is not equal to or greater than a predetermined shake amount during the period in which the moving image is captured, the main subject estimation unit 42 uses the motion vector detected by the motion vector detection unit 41 in the frame image of the moving image. Based on this, it is estimated that an image region having a large image motion vector among a plurality of image regions in the frame image of the moving image is a main subject region.

As described above, the main subject estimation unit 42 switches the process of estimating the main subject based on the movement of the imaging device 100. For example, when the shake amount of the imaging apparatus 100 is equal to or greater than a predetermined shake amount, the main subject estimation unit 42 clusters the motion vectors detected by the motion vector detection unit 41 using a predetermined threshold MV _th1 . It is estimated that an image area formed by a block having a motion vector equal to or less than the threshold MV _th1 is a main subject area. On the other hand, when the shake amount of the imaging apparatus 100 is not equal to or greater than the predetermined shake amount, the main subject estimation unit 42 clusters the motion vectors detected by the motion vector detection unit 41 with a predetermined threshold MV _th2 to obtain a threshold MV. It is estimated that an image area formed by a block having a motion vector larger than _th2 is a main subject area. Here, the predetermined threshold MV _th2 is a predetermined motion vector threshold for estimating a large motion vector region as a main subject region. The threshold values MV _{th1 and} MV _th2 may be the same value.

FIG. 6 is a flowchart showing main subject estimation processing in the fourth embodiment.
Hereinafter, the main subject estimation process of the imaging apparatus 100 according to the fourth embodiment will be described with reference to the flowchart shown in FIG. Here, steps S1 to S6 in FIG. 6 are the same processes as steps S1 to S6 in FIG.

As in FIG. 3, in step S <b> 5, the control unit 3 determines that the shake amount of the imaging device 100 during the period during which the moving image is captured is equal to or greater than the predetermined shake amount based on the detection result of the motion vector detection unit 41. In this case, the main subject estimation unit 42 estimates that a block whose motion vector is equal to or less than a predetermined threshold MV _th1 detected by the motion vector detection unit 41 is a main subject region (step) S6).

On the other hand, when the control unit 3 determines in step S5 that the shake amount of the imaging apparatus 100 during the period in which the moving image is captured is not equal to or greater than the predetermined shake amount, the main subject estimation unit 42 is detected by the motion vector detection unit 41. A block whose motion vector is larger than a predetermined threshold MV _th2 is estimated to be a main subject region (step S7). Then, the image processing unit 4 ends the process.

  As described above, according to the fourth embodiment, when the shake amount of the imaging apparatus 100 is equal to or greater than the predetermined shake amount during the period in which the moving image is captured, the image processing unit 4 includes the captured frame image. An image area with a small motion vector is estimated as the main subject area. On the other hand, if the shake amount of the imaging apparatus 100 is not equal to or greater than the predetermined shake amount during the period in which the moving image is picked up, the image processing unit 4 uses an image region having a large motion vector as a main subject region in the captured frame image. Estimated. Therefore, in the case where the main subject is moving, either when the photographer takes an image while moving the imaging device 100 according to the movement of the main subject, or when the image is taken without moving the imaging device 100. In addition, the imaging apparatus 100 can appropriately estimate the main subject area from the captured image. In addition, the process of step S5 in the above configuration is performed in the direction of the motion vector of the shake detection sensor 8 and the screen edge region group (in the direction of the motion vector having the maximum size or the direction of the motion vector of the screen edge region group). It may be configured to determine whether or not the most common direction) continuously matches for a predetermined number of frames.

<Fifth Embodiment>
In the fifth embodiment, a process of correcting the imaged position of the main subject area in the imaging apparatus 100 based on the main subject area estimated by the image processing unit 4 will be described. The other configuration of the imaging apparatus 100 according to the fifth embodiment is the same as the configuration of the first embodiment described with reference to FIG.

  The control unit 3 uses the position correction control unit 43 of the image processing unit 4 so that the main subject region estimated by the main subject estimation unit 42 is located at a predetermined position in the frame image. The position of the main subject area is corrected. Here, the case where the predetermined position is the center position in the frame image will be described.

  For example, the position correction control unit 43 calculates the difference between the center position of the main subject area estimated by the main subject estimation unit 42 and the center position in the frame image. Then, the position correction control unit 43 calculates the shake correction value of the imaging optical system so that the difference becomes 0 or decreases. The control unit 3 corrects the optical axis by moving the shake correction lens included in the lens unit 21 based on the shake correction value input from the position correction control unit 43 via the shake correction unit 9. Thereby, the position of the main subject area to be imaged is corrected to the center in the frame image.

FIG. 7 is a flowchart showing position correction processing in the fifth embodiment.
Hereinafter, the main subject estimation process of the imaging apparatus 100 according to the fifth embodiment will be described with reference to the flowchart shown in FIG.

  The image processing unit 4 executes the process of estimating the main subject area shown in FIG. 3, FIG. 4, FIG. 5, or FIG. 6 (step S31). The position correction control unit 43 of the image processing unit 4 calculates the position of the center of gravity of the main subject area estimated by the main subject estimation unit 42 (step S32).

  Next, the position correction control unit 43 calculates the difference between the calculated center position of the main subject region and the center position in the frame image (step S33). Then, the control unit 3 determines whether there is a difference between the position of the center of gravity of the main subject region and the center position in the frame image based on the calculation result of the position correction control unit 43 (step S34).

  In step S34, when it is determined that there is a difference between the center of gravity position of the main subject region and the center position in the frame image, the position correction control unit 43 takes the imaging optical so that the difference becomes 0 or decreases. The system shake correction value is calculated. (Step S35). On the other hand, if it is determined in step S34 that there is no difference between the center of gravity position of the main subject area and the center position in the frame image, the image processing unit 4 ends the process.

  Note that the control unit 3 moves the shake correction lens included in the lens unit 21 via the shake correction unit 9 based on the shake correction value input from the position correction control unit 43 in step S35 to change the optical axis. to correct. Thereby, the position of the main subject area to be imaged is corrected to the center in the frame image.

  As described above, according to the fifth embodiment, based on the output result of the position correction control unit 43, the main subject is estimated so that the main subject region estimated by the main subject estimation unit 42 is positioned at the center in the frame image. Correct the position of. For example, even if the photographer attempts to capture an image while moving the imaging apparatus 100 in accordance with the movement of the main subject, it is difficult to continuously match the moving main subject to the center position of the frame image. According to the fifth embodiment, the imaging apparatus 100 can correct the position of the main subject area to the center position of the frame image from the difference between the estimated position of the main subject area and the center position of the frame image. it can.

  In the fifth embodiment, the case where the predetermined position is the center position in the frame image captured by the imaging apparatus 100 has been described. However, the present invention is not limited to this. The predetermined position may be predetermined at a position other than the center position in the captured frame image. The predetermined position may be set by the photographer to an arbitrary position in the captured frame image.

  Note that the position correction control unit 43 may correct the position of the main subject region in the frame image in a direction in which the movement of the image between the frame images in the main subject region estimated by the main subject estimation unit 42 decreases. . That is, the position correction control unit 43 may correct the position of the main subject region in the frame image in a direction in which the motion vector in the frame image in the main subject region estimated by the main subject estimation unit 42 decreases.

  For example, the position correction control unit 43 positions the main subject region in the “Nth” frame image with respect to the position of the main subject region in the “N−1” th frame image estimated by the main subject estimation unit 42. The motion vector of is calculated. Then, the position correction control unit 43 calculates a shake correction value of the imaging optical system so that the motion vector becomes small. Based on the shake correction value of the position correction control unit 43, the control unit 3 moves the shake correction lens of the lens unit 21 via the shake correction unit 9 to correct the relative position between the optical axis and the imaging surface. To do. Thereby, the position of the main subject area imaged in the frame image is corrected.

  According to this, the position correction control unit 43 corrects the position of the main subject so that the motion vector of the main subject region estimated by the main subject estimation unit 42 becomes small. For example, even if the photographer tries to capture an image while moving the imaging apparatus 100 according to the movement of the main subject, it is difficult to continuously position the moving main subject at a certain position in the frame image. As in the case of camera shake that occurs when imaging is performed, shake occurs at the position of the main subject area. However, when the imaging apparatus 100 is intentionally moved, the shake correction control of the imaging apparatus 100 is stopped because the shake amount of the imaging apparatus 100 exceeds the shake amount due to camera shake. In this case, the imaging apparatus 100 corrects the position of the main subject so that the estimated motion vector of the main subject area becomes small. Accordingly, when the photographer captures an image while moving the imaging apparatus 100 in accordance with the movement of the main subject, the imaging apparatus 100 can correct the estimated shake of the position of the main subject area.

In addition, although the description has been given of the example of the optical shake correction that corrects the optical axis by moving the shake correction lens included in the lens unit 21 via the shake correction unit 9, the imaging element 22 is provided via the shake correction unit 9. May be performed to correct the shake of the relative position of the imaging surface of the imaging element 22 with respect to the optical axis of the lens unit 21.

  The optical shake correction for correcting the optical axis via the shake correction unit 9 has been described as an example, but electronic shake correction may be performed. Here, the electronic shake correction means that an area smaller than the imageable area of the image sensor 22 is an imaging area, and the imaging area is moved within the imageable area so as to cancel out the shake according to the shake. This is shake correction for correcting shake.

In the fifth embodiment, the process of correcting the imaged position of the main subject area based on the main subject area estimated by the image processing unit 4 has been described. However, the estimated main subject area is used for the following control. May be used.
(1) The imaging apparatus 100 may estimate the type of the main subject (for example, a person, an animal, a car, or a building) from the estimated feature in the main subject area. In addition, the imaging apparatus 100 may record the estimated main subject type information in association with each frame image. Further, in the case where the information on the type of main subject is recorded in association with each other, the imaging apparatus 100 may record the information in an image data file, or may record in another data file associated with the image data file.
(2) The imaging apparatus 100 may execute AUTO control such as AF (Auto Focus), AE (Auto Exposure), or AWB (Auto White Balance) focusing on the estimated main subject area. Good. For example, the imaging apparatus 100 may control AF so that the estimated main subject area is focused on. The imaging apparatus 100 also performs AE so that the estimated brightness of the main subject region is appropriate in a scene where the background region is bright and the main subject region may be captured darkly, such as a backlight scene. You may control. In addition, the imaging apparatus 100 is used in a scene where there are a plurality of white balance standards in the imaging area, such as a scene where there are a plurality of light sources having different colors (for example, a light bulb, a tungsten light, and a fluorescent lamp). AWB may be controlled in preference to the estimated main subject area.
(3) The imaging apparatus 100 allows “capturing a still image or starting capturing a moving image” in accordance with the estimated position of the main subject region being a position in a preset frame image. Alternatively, control such as performing the operation automatically may be executed.
(4) The imaging apparatus 100 may extract a frame image in which the estimated main subject region does not exist from the captured frame images. Then, control may be performed such as deleting the extracted image or replacing it with a frame image in which the main subject region exists.
(5) When the estimated size of the main subject area is small, the imaging apparatus 100 may control the zoom to the telephoto side so that the main subject area has a predetermined size (suitable for imaging). . These are examples of control using the estimated main subject area in the imaging apparatus 100, and the estimated main subject area may be used for other controls.

  As described above, the imaging apparatus 100 in the above-described embodiment can appropriately estimate the main subject region from the captured image. In addition, the imaging apparatus 100 can perform position correction, shake correction, and various other controls in the imaging apparatus 100 based on the estimated main subject region.

  The main subject estimation unit 42 may use AF information in the frame image in addition to the motion vector information when estimating the main subject. Alternatively, the closest subject region may be detected based on the defocus information of the phase difference AF, and the main subject region may be estimated using the detection result. For example, when imaging a situation in which a plurality of subjects are moving in the same direction at approximately the same speed, such as two running cars, the main subject estimation unit 42 adds the AF information and estimates the main subject. By doing so, it can be estimated accurately.

  As a method for detecting a motion vector, a method for detecting using a pattern matching of images between frame images may be used. Alternatively, a feature point of the image may be extracted and detected from the movement amount and the movement direction of the feature point of the image between the frame images.

In addition, you may make it the structure which combines at least 2 of this Embodiment 1-5.

  In addition, although the example which estimates the main subject from the moving image imaged in this embodiment was demonstrated, the image imaged in time series is not restricted to a moving image. For example, the images captured in time series may be a through image or a plurality of still images in which the same main subject having different captured times is captured.

  In all of the embodiments, the imaging apparatus 100 illustrated in FIG. 1 has been described as an example. However, the image processing unit 4 is not limited to the configuration provided in the imaging apparatus 100. For example, the image processing unit 4 may be an image processing device that receives a plurality of frame images captured in time series, detects a motion vector from the input frame image, and estimates a main subject. In this case, the image processing apparatus may include an image input unit (frame image input unit) through which an image is input, and a plurality of frame images captured in time series may be input from the image input unit. In addition, the image processing apparatus may receive a plurality of frame images captured in time series from a storage medium such as the memory card 200. The image processing apparatus may be configured to include a card connector 12 into which a removable storage medium is inserted when reading image data from a storage medium, or a storage medium reading apparatus as an external apparatus. It may be configured to connect. Further, the image processing apparatus may be configured to include the display unit 7 or may be configured to be connectable to an external display device. Accordingly, the image processing apparatus can similarly execute the main subject estimation process on an image captured by an imaging apparatus other than the imaging apparatus 100 of the present embodiment.

  In addition, an image in which information related to shake of the imaging apparatus during the imaging period is recorded in association with a plurality of frame images captured in time series may be input to the image processing apparatus. Thereby, the image processing apparatus can detect the movement of the imaging apparatus detected during the period of imaging by the imaging apparatus from the information associated with the input image. In addition, the image processing apparatus includes a shake information input unit to which shake information of the imaging device is input, and information regarding shake of the imaging device during a period in which a plurality of frame images taken in time series is captured is input. Also good.

  Note that the motion vector detection unit 41, the main subject estimation unit 42, and the position correction control unit 43 in the image processing unit 4 in FIG. 1 may be realized by dedicated hardware, and may include a memory and a CPU ( It is also possible to realize a function by loading a program for realizing the function of each unit described above into a memory and executing the program.

  1 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed. You may perform the process of. Here, the “computer system” includes an OS and hardware such as peripheral devices.

Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes design and the like within a scope not departing from the gist of the present invention.

  2 imaging unit, 4 image processing unit (image processing apparatus), 8 shake detection sensor (shake detection unit), 40 frame image division unit, 41 motion vector detection unit, 42 main subject estimation unit, 43 position correction control unit, 100 imaging apparatus

Claims (9)

An imaging unit for imaging a subject;
A shake detection unit for detecting the shake amount;
In a plurality of frame images captured in time series by the imaging unit, a frame image dividing unit that divides the frame image into a plurality of image regions;
A motion vector detection unit that detects a motion vector of the image region for each image region;
Based on the motion vector of the image area detected by the motion vector detection unit when the shake amount of the imaging unit detected by the shake detection unit is greater than or equal to a predetermined amount when the plurality of frame images are captured. A main subject estimation unit that estimates an image region having a small motion vector among the plurality of image regions as a region including a main subject;
Equipped with a,
The direction of the motion vector of a predetermined image area in the plurality of frame images detected by the motion vector detection unit, and the imaging unit detected by the shake detection unit when the plurality of frame images are captured. If the shake direction is the same direction for a predetermined number of frames or more,
The main subject estimation unit includes:
Based on a motion vector of the image area detected by the motion vector detection unit, an image area having a small motion vector among the plurality of image areas is estimated as a main subject area.
An imaging apparatus characterized by that. The predetermined image area is a predetermined image area group of the plurality of image areas,
The motion vector detection unit
The direction of the motion vector in the predetermined image region group is the direction of the motion vector in the image region having the largest motion vector among the plurality of image regions in the predetermined image region group.
The imaging apparatus according to claim 1 . The predetermined image area is a predetermined image area group of the plurality of image areas,
The motion vector detection unit
The direction of the motion vector in the predetermined image region group is set as the direction of the motion vector having the largest direction of the motion vector for each image region among the plurality of image regions in the predetermined image region group.
The imaging apparatus according to claim 1 . A position correction control unit that corrects the position of the main subject region in the frame image captured by the imaging unit in a direction in which a motion vector in the main subject region estimated by the main subject estimation unit decreases.
The imaging apparatus according to any one of claims 1 to 3 , further comprising: The position correction control unit
The position of the main subject region in the frame image captured by the imaging unit is set so that the main subject region estimated by the main subject estimation unit is located at a predetermined position in the frame image. to correct,
The imaging apparatus according to claim 4 . A shake information input unit to which shake information of the imaging apparatus is input;
A frame image input unit for inputting a plurality of frame images captured in time series by the imaging device;
A frame image dividing unit for dividing the frame image into a plurality of image regions;
A motion vector detection unit that detects a motion vector of the image region for each image region;
Based on a motion vector of the image area detected by the motion vector detection unit when a shake amount of the imaging apparatus when the plurality of frame images input from the shake information input unit is captured is a predetermined amount or more. A main subject estimation unit that estimates an image region having a small motion vector among the plurality of image regions as a region including a main subject;
Equipped with a,
Direction of motion vector of a predetermined image area in the plurality of frame images detected by the motion vector detection unit and the imaging device when the plurality of frame images input from the shake information input unit are imaged If the shake direction is the same direction for a predetermined number of frames or more,
The main subject estimation unit includes:
Based on a motion vector of the image area detected by the motion vector detection unit, an image area having a small motion vector among the plurality of image areas is estimated as a main subject area.
An image processing apparatus. The predetermined image area is a predetermined image area group of the plurality of image areas,
The motion vector detection unit
The direction of the motion vector in the predetermined image region group is the direction of the motion vector in the image region having the largest motion vector among the plurality of image regions in the predetermined image region group.
The image processing apparatus according to claim 6 . The predetermined image area is a predetermined image area group of the plurality of image areas,
The motion vector detection unit
The direction of the motion vector in the predetermined image region group is set as the direction of the motion vector having the largest direction of the motion vector for each image region among the plurality of image regions in the predetermined image region group.
The image processing apparatus according to claim 6 . In a computer as an image processing device,
A shake information input procedure for inputting shake information of the imaging device,
A frame image entering procedure for inputting a plurality of frame images captured in time series by the imaging device;
A frame image dividing procedure for dividing the frame image into a plurality of image regions;
A motion vector detection procedure for detecting a motion vector of the image region for each image region;
When the shake amount of the imaging device when the plurality of frame images are captured is a predetermined amount or more, based on the motion vector of the image region detected by the motion vector detection procedure, A main subject estimation procedure for estimating an image region having a small motion vector as a region including a main subject;
Including
The direction of the motion vector of a predetermined image area in the plurality of frame images detected by the motion vector detection procedure and the plurality of frame images are captured by the frame image input procedure input from the shake information input procedure. When the shake direction of the imaging device is the same direction continuously for a predetermined number of frames,
By the main subject estimation procedure,
A process of estimating an image area having a small motion vector among the plurality of image areas as a main subject area based on a motion vector of the image area detected by the motion vector detection procedure;
An image processing program for executing

Images