JP2009260671A - Image processing apparatus and imaging device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To determine a motion vector affected by a moving subject highly precisely in performing vibration proof processing that eliminates an effect of the moving subject. <P>SOLUTION: An image processing apparatus has: a motion vector calculating means 109 for calculating the motion vector at a plurality of places in an image obtained by taking a moving image; a determining means 113 for determining the motion vector corresponding to the moving subject among motion vectors at the plurality of positions; and processing means 114 to 116 for performing vibration proof processing to reduce image vibration on the basis of the motion vector other than motion vectors corresponding to the moving subject among the motion vectors at the plurality of positions. The determining means determines a motion vector corresponding to the moving subject on the basis of the determination of whether a motion vector at a position in correspondence among the plurality of positions has a specific relationship. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image stabilization processing technique for reducing image blur due to camera shake.

  An image pickup apparatus such as a video camera may be equipped with an electronic image stabilization function for reducing image shake due to the shake of the image pickup apparatus.

  In electronic image stabilization, the area that is actually output for recording or display in the frame image generated by imaging is used as a cutout area, and coordinate conversion that shifts or deforms the cutout area for each frame image so that the image shake is canceled out. Process.

  As a shake detection method of an imaging apparatus using an image, there is a method in which a brightness projection histogram is created for each frame image, and shake (direction and amount) is calculated from a position difference of the histogram between the frame images. There is also a method of calculating a motion vector between frame images and obtaining a shake from the motion vector.

  These shake detection methods are effective when the entire image obtained by imaging moves due to the shake of the imaging apparatus. However, when a moving object (moving subject) such as a pedestrian or a vehicle exists in the image, the positional difference and motion vector of the histogram affected by the moving subject are calculated. The shake cannot be detected.

  For this reason, it is necessary to be able to perform favorable shake detection of the imaging apparatus even when a moving subject is present in the image.

  In Patent Document 1, an image obtained by imaging is divided into a plurality of areas, the reliability of motion vectors calculated in each area is determined, and the motion vectors of areas determined to be unreliable are captured. A shake detection method that is not used for shake detection of an apparatus is disclosed. Thereby, it is possible to detect the shake of the imaging apparatus excluding the influence of the moving subject.

In the moving subject detection method disclosed in Patent Document 2, the motion vector of the entire image is calculated from the motion vector calculated for each region in the image. Then, the motion vector of the entire image is accumulated to extract a statistical property regarding the change in the motion of the entire image, and based on this statistical property, the motion vector of the entire image in the subsequent frame image is predicted. At this time, if the value of the motion vector calculated in each region and the predicted value of the motion vector of the entire image are greatly different, it is determined that a moving subject exists in that region.
Japanese Patent No. 2512207 Japanese Patent Laid-Open No. 11-266459

  However, in the shake detection method disclosed in Patent Document 1, the presence of a moving subject is determined using a correlation value between two frame images. For this reason, when camera work such as panning or tilting is performed on the imaging apparatus, the correlation value changes greatly even in an area where there is no moving subject, making it impossible to determine the presence of the moving subject. Become.

  In addition, the moving object detection method disclosed in Patent Document 2 is based on the premise that the motion vector of the entire image changes linearly. Prediction accuracy decreases for an image obtained by imaging. For this reason, the accuracy of determining whether or not there is a moving subject also decreases.

  The present invention provides an image processing apparatus, an imaging apparatus, and an image processing method capable of determining a motion vector affected by a moving subject with high accuracy when performing an image stabilization process excluding the influence of the moving subject. .

  An image processing apparatus according to one aspect of the present invention corresponds to a motion vector calculation unit that calculates motion vectors at a plurality of positions in an image obtained by moving image capturing, and corresponds to a moving subject among the motion vectors at the plurality of positions. Determination means for determining a motion vector to be performed, and processing means for performing image stabilization processing for reducing image shake based on a motion vector excluding a motion vector corresponding to a moving subject from among the motion vectors at the plurality of positions. Have. The determining means determines a motion vector corresponding to the moving subject based on determining whether or not the motion vector at a symmetrical position among the plurality of positions has a specific relationship. To do.

  Note that an imaging apparatus having an imaging system for capturing a moving image and the image processing apparatus also constitutes another aspect of the present invention.

  An image processing method according to another aspect of the present invention includes a step of calculating motion vectors at a plurality of positions in an image obtained by moving image capturing, and a moving subject among the motion vectors at the plurality of positions. A determination step of determining a corresponding motion vector, and a step of performing an image stabilization process for reducing image blur based on a motion vector excluding the motion vector corresponding to the moving subject among the motion vectors at the plurality of positions. Have. In the determination step, the motion vector corresponding to the moving subject is determined based on the determination as to whether or not the motion vector at the symmetrical position among the plurality of positions has a specific relationship. To do.

  According to the present invention, a motion vector affected by a moving subject is determined with high accuracy based on whether or not the motion vector at a symmetrical position in an image including the moving subject has a specific relationship. can do. Therefore, it is possible to perform a good image stabilization process excluding the influence of the moving subject even on an image obtained by a shake of a high-frequency imaging device or an image taken under complicated camera work.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows a configuration of an imaging apparatus capable of capturing a moving image, such as a video camera or a digital still camera, including an image processing apparatus that is Embodiment 1 of the present invention.

  In FIG. 1, reference numeral 101 denotes an optical system that forms a subject image, and 102 denotes an image sensor such as a CCD sensor or a CMOS sensor that photoelectrically converts a subject image formed by the optical system 101.

  An image forming circuit 103 forms a video signal from an electrical signal output from the image sensor 102. The image forming circuit 103 includes an A / D conversion circuit 104, an auto gain control circuit (AGC) 105, and an auto white balance circuit (AWB) 106, and generates a digital video signal. The A / D conversion circuit 104 converts an analog signal into a digital signal. The AGC 105 corrects the level of the digital signal. The AWB 106 corrects the white level of the video. The imaging device 102 and the image forming circuit 103 constitute an imaging system that performs moving image imaging.

  A frame memory 107 temporarily stores and holds one frame image or a plurality of frame images of the video signal formed by the image forming circuit 103. Reference numeral 108 denotes a memory control circuit that controls video signals input to and output from the frame memory 107.

  Reference numeral 109 denotes a motion vector calculation circuit that detects a motion vector at each of a plurality of feature points extracted between two consecutive (adjacent) frame images, further divides the frame image into a plurality of regions, A representative motion vector in the region (position) is calculated. The motion vector calculation circuit 109 includes a motion vector detection circuit 110, an area division circuit 111, and a representative motion vector calculation circuit 112. In the following description, the frame image is also simply referred to as an image.

  The motion vector detection circuit 110 extracts a plurality of characteristic points (feature points) in the image, and calculates a motion vector indicating the direction and amount of movement (displacement) of the subject image between two consecutive frame images. The motion vector is motion information representing the apparent motion of the subject image existing in the image.

  The area dividing circuit 111 divides the frame image into a plurality of areas (for example, 4 × 4 16 areas).

  The representative motion vector calculation circuit 112 calculates, for each of a plurality of regions divided by the region dividing circuit 111, a representative motion vector that is one motion vector representing the region.

  Reference numeral 113 denotes a motion vector analysis circuit as determination means, which analyzes representative motion vectors in a plurality of regions obtained from the motion vector calculation circuit 109. The motion vector analysis circuit 113 determines a representative motion vector corresponding to the moving object (moving subject) from the analysis result of the representative motion vector.

  Reference numeral 114 denotes a moving object removal circuit. The moving object removal circuit 114 is a divided area (hereinafter, referred to as a divided motion area) in which a representative motion vector corresponding to the moving object is determined by the motion vector analysis circuit 113 among the divided areas (hereinafter referred to as divided areas). The moving object area is excluded from the subsequent processing. Note that excluding a moving object region is synonymous with excluding a representative motion vector corresponding to a moving object.

  Reference numeral 115 denotes a correction parameter calculation circuit, which calculates an image deformation amount from a motion vector group in the remaining divided areas from which the moving object area is excluded by the moving object removal circuit 114 among the plurality of divided areas. Further, the correction parameter calculation circuit 115 calculates a correction parameter for correcting (reducing) image blur from the image deformation amount.

  Reference numeral 116 denotes a shake correction circuit, which performs geometric transformation processing (ie, image stabilization processing) for image shake correction on the frame image using the correction parameter calculated by the correction parameter calculation circuit 115.

  The moving object removal circuit 114, the correction parameter calculation circuit 115, and the shake correction circuit 116 constitute processing means. The motion vector calculation circuit 109, the motion vector analysis circuit 113, the moving object removal circuit 114, the correction parameter calculation circuit 115, and the shake correction circuit 116 constitute an image processing apparatus.

  Reference numeral 117 denotes a video output circuit, which sequentially displays frame images that have undergone image blur correction processing (geometric transformation processing) on a display (not shown) or records them on a recording medium such as a semiconductor memory, an optical disk, or a magnetic tape. To do.

  Reference numeral 100 denotes a main that controls the operations of the image sensor 102, the image forming circuit 103, the motion vector calculation circuit 109, the motion vector analysis circuit 113, the moving object removal circuit 114, the correction parameter calculation circuit 115, the shake correction circuit 116, and the video output circuit 117. It is a controller. The main controller 100 is constituted by a CPU or the like. The main controller 100 also controls the operation of the memory control circuit 108.

  The operation of the imaging apparatus configured as described above will be described with reference to the flowchart shown in FIG. The operation described here is executed according to a computer program stored in a memory (not shown) of the main controller 100. The same applies to the following embodiments.

  In FIG. 2, in step S <b> 201, the main controller 100 causes the image sensor 102 to photoelectrically convert the subject image formed by the optical system 101. Further, the main controller 100 causes the image forming circuit 103 to process an analog signal output from the image sensor 102 in accordance with the subject brightness, thereby generating a video signal.

  The image forming circuit 103 converts the analog signal into, for example, a 14-bit digital signal by the A / D conversion circuit 104. Further, the digital video signal that has been subjected to signal level correction and white level correction by the AGC 105 and AWB 106 is recorded and held in the frame memory 107.

  In the imaging apparatus of this embodiment, frame images are sequentially generated at a predetermined frame rate, and the frame images stored and held in the frame memory 107 are sequentially input to the motion vector calculation circuit 109. In addition, the frame images stored in the frame memory 107 are also updated sequentially. The above operation is controlled by the memory control circuit 108 in accordance with an instruction from the main controller 100. In step S202 and subsequent steps, image blur correction is performed.

  In step S202, the main controller 100 causes the motion vector detection circuit 110 to detect a motion vector between two consecutive frame images. For the detection of the motion vector, a general detection method such as a template matching method or a gradient method may be used, but any method may be used.

  In addition, the detection of the motion vector is generally calculated based on the correlation between the pixel of interest between two consecutive frame images and the texture information of the neighboring area. For this reason, when there is no texture in the target pixel and its neighboring region and there is almost no change in luminance value, it is not possible to find a correlation between frame images.

  Therefore, an accurate motion vector can be detected by previously extracting corners and lines of an object (subject) as feature points in the image and using them as a target pixel in motion vector detection. The feature point extraction method includes a method using an intersection of edges in an image as a feature point, a method using color information, and the like, and any method may be used.

  In step S203, the main controller 100 causes the area dividing circuit 111 to divide the image into a plurality of areas. Further, the main controller 100 causes the representative motion vector calculation circuit 112 to calculate a representative motion vector in the divided area using the plurality of motion vectors detected in the divided area.

  Here, an example of the region division of the frame image by the region dividing circuit 111 is shown in FIG. Reference numeral 301 denotes the feature points detected in step S202 and motion vectors at the feature points. Reference numeral 302 denotes one divided area. Note that FIG. 3 shows a case where the region is divided into a 4 × 4 grid, but any number of divisions and division shapes may be used.

  FIG. 4 shows representative motion vectors calculated for each divided region by the representative motion vector calculation circuit 112. Reference numeral 401 denotes a representative motion vector in the divided region 302 shown in FIG. The representative motion vector is obtained by integrating a plurality of motion vectors at feature points extracted in one divided region. The representative motion vector can be calculated by obtaining an average value or an intermediate value of a plurality of motion vectors in the divided area, but any calculation method may be used.

  In step S204, the main controller 100 causes the motion vector analysis circuit 113 to analyze the representative motion vector obtained in step S203 and determine a divided region (moving object region) where the moving object exists. In other words, the representative motion vector corresponding to the moving object is determined. Then, the main controller 100 causes the motion vector analysis circuit 113 to remove the moving object region from the target region for subsequent processing.

  Here, a motion vector analysis method by the motion vector analysis circuit 113 in this embodiment will be described. Here, a representative motion vector is analyzed with two regions having a point-symmetrical positional relationship (symmetrical relationship) with respect to the center of the image (a position corresponding to the optical axis position of the optical system 101 in the image) as one set. In the following description, a set of divided areas having a symmetric positional relationship with respect to the center of the image is referred to as a point-symmetric area, and a representative motion vector in the set of divided areas is referred to as a symmetric position representative motion vector.

  When a shake occurs in the image pickup apparatus (that is, an image), the point-symmetric position representative motion vector corresponds to a specific relationship, that is, the type of motion of the entire image caused by the shake if there is no moving object. Has a unique relationship.

  When translational shake occurs, the point-symmetric position representative motion vectors have a specific relationship that they both face the same direction. Further, when a rotational shake occurs, the point-symmetric position representative motion vectors have a specific relationship that they are opposite to each other in the tangential direction of the rotation. Furthermore, when a forward / backward (optical axis direction) shake, that is, an enlargement / reduction shake, occurs, the point-symmetric position representative motion vectors have a specific relationship that they are opposite to each other in the radial direction. FIG. 13 shows the relationship of motion vectors at a plurality of positions in an image when translation, rotation, enlargement, and reduction shakes occur.

  Therefore, if the relationship of the point-symmetric position representative motion vector is different from the relationship peculiar to the motion of the entire image, there is a moving object in which one of the point-symmetric regions has a motion different from the motion of the entire image. It can be determined that it is a moving object area. In other words, when the relationship of the point symmetric representative motion vector is different from the above unique relationship, one of the point symmetric representative motion vectors can be determined as the representative motion vector corresponding to the moving object.

  FIG. 5 shows a method for determining a moving object region when translational shake occurs. Since the representative motion vectors in the point symmetry region 501 are both in the same direction, it can be determined that the representative motion vector has a specific relationship according to the motion of the entire image due to translational shake and does not include the moving object region. On the other hand, the representative motion vector in the point symmetry region 502 does not have the above specific relationship (more specifically, not only the translational shake but also the unique relationship with respect to the rotation and scaling shake). One of the areas 502 is determined as a moving object area.

  FIG. 6 shows how the representative motion vector appears 603 when translational shake 601 and reduced shake 602 occur simultaneously. In FIG. 6, only representative motion vectors in the divided areas at the four corners of the image are shown.

  When a plurality of types of shakes occur in this way, it is difficult to find a specific relationship (specific relationship) caused by the shakes only with representative motion vectors that are in a point-symmetric positional relationship. Therefore, in such a case, in addition to the point-symmetrical positional relationship, the movement is also determined using whether or not the representative motion vectors that are vertically and horizontally symmetrical relative to the center of the image have a specific relationship. Determine the object area.

  In FIG. 6, the horizontal size is the same between the representative motion vectors 604 and 606 that are in a vertically symmetrical positional relationship, and the vertical direction is between the representative motion vectors 604 and 607 that are in a symmetrical positional relationship. It is possible to find a peculiar relationship that the sizes of are the same. Such a relationship is unique when the motion of the entire image includes a motion due to translational shake and a motion due to reduced shake. By using this, it is possible to discriminate between the movement of the entire image and the movement of the moving object.

  Therefore, even when a plurality of types of shake affect the representative motion vectors of a plurality of divided regions, it is possible to determine a favorable moving object region by using a plurality of types of representative motion vectors in a symmetrical positional relationship. Can do. In addition, by determining the moving object region using the representative motion vectors having a plurality of types of symmetrical positional relationships, which one of the set of divided regions determined to be any moving object region is which is the true moving object. It is also possible to determine whether the area.

  FIG. 7 shows a moving object region determination method using a plurality of types of symmetrical positional relationships. Here, it is assumed that the divided area 702 is a moving object area.

  First, the moving object region is one of the divided regions 701 and 702 by determining whether or not the representative motion vector of the divided region 701 that is point-symmetric with the divided region 702 has the above-described specific relationship. Can be determined. Furthermore, the moving object region is one of the divided regions 702 and 703 by determining whether or not the representative motion vector of the divided region 703 that has a symmetrical positional relationship with the divided region 702 has the above-described specific relationship. Can be determined.

  Then, by determining whether or not the representative motion vectors of the divided areas 701 and 703 having a vertically symmetrical positional relationship have the above-described specific relationship, the divided areas 701 and 703 can be determined not to be moving object areas. . As a result, the moving object area can be specified as the divided area 702.

  In this way, by specifying a divided region that is a moving object region and excluding the divided region from the subsequent processing targets, the entire image with high accuracy can be obtained without being affected by the representative motion vector corresponding to the moving object. Can be calculated.

  In step S205 of FIG. 2, the main controller 100 causes the correction parameter calculation circuit 115 to use the representative motion vector group in the remaining divided areas from which the moving object area specified in step S204 is excluded, Causes correction parameters to be calculated. Thereafter, the main controller 100 causes the shake correction circuit 116 to perform a geometric transformation process using the correction parameter.

  In this embodiment, an affine transformation matrix is used for calculating the image deformation amount.

  A point a on the image,

(Formula 1)
Is the point a ′ in the next frame,

(Formula 2)
Suppose you move to. In this case, the correspondence between the points a and a ′ is obtained by using the affine transformation matrix Ha,

(Formula 3)
It can be expressed as.

  The affine transformation matrix Ha is a determinant indicating the amount of deformation due to translation, rotation, scaling, and shear between images, and can be expressed by the following formula.

(Formula 4)
Note that the points a, a ′ and the affine transformation matrix Ha are expressed using homogeneous coordinates. The same applies to the following.

  Each element of the affine transformation matrix Ha can be calculated by applying a statistical process such as a least square method using the representative motion vector group obtained in step S204, that is, the correspondence relationship of the divided areas between the frame images. it can. At this time, by removing each motion vector corresponding to the moving object in advance and calculating each element of the affine transformation matrix Ha, it is possible to obtain a good overall motion of the image that is not affected by the presence of the moving object.

The affine transformation matrix Ha obtained in this way represents the amount of image deformation due to the shake of the imaging device. Therefore, in order to correct the image blur, it is necessary to convert the affine transformation matrix Ha so as to indicate the amount of image deformation that cancels this. For this reason, the affine transformation matrix Ha is converted to its inverse matrix.

Convert to

  This

(Formula 5)
It can be expressed as.

This equation makes it possible to return the point a ′ after the shake has occurred to the same coordinates as the point a before the shake has occurred. In this example,

Is called a correction parameter.

  Then, image blurring is corrected (reduced) by performing geometric transformation processing on all pixels in the image using the correction parameters.

  In this embodiment, an affine transformation matrix is used as the amount of image deformation representing the shake amount. However, depending on the type of shake (motion) and the setting conditions of the camera at the time of imaging, other factors such as a Helmat matrix and a planar projection matrix The image deformation amount may be used.

  In step S206, the main controller 100 causes the video output circuit 117 to output an image with corrected image blur.

  As described above, the present embodiment utilizes the fact that the representative motion vector at the symmetrical position (region) has a specific relationship (specific relationship) according to the type of motion of the entire image. Then, it is determined whether or not there is a moving object in the image. Thereby, even when a moving object is present in the image, it is possible to perform a good image shake correction process (electronic image stabilization process).

  FIG. 8 shows a configuration of an imaging apparatus including an image processing apparatus that is Embodiment 2 of the present invention. In the present embodiment, the distribution of feature points in each divided region obtained by dividing the image is determined, and the divided region used for the subsequent processing is determined according to the result.

  In FIG. 8, the same components as those shown in FIG. In addition to the configuration illustrated in FIG. 1, the imaging apparatus according to the present exemplary embodiment includes a feature point distribution determination circuit 801 that determines the distribution of feature points in each divided region of an image divided by the region dividing circuit 111. The main controller 100 shown in FIG. 1 is omitted in FIG.

  Hereinafter, the operation of the imaging apparatus (main controller 100) of the present embodiment will be described with reference to the flowchart shown in FIG.

  Step S901 and step S902 are respectively the same as step S201 and step S202 shown in FIG.

  In step S903, the main controller 100 causes the feature point distribution determination circuit 801 to determine the feature point distribution in each divided region of the image divided by the region dividing circuit 111. Then, the main controller 100 determines a divided region (that is, a representative motion vector) to be used for subsequent processing according to the determination result.

  For example, a feature point cannot be extracted for a subject having a uniform luminance value such as a sky or a wall surface because there is no characteristic portion. In many cases, the representative motion vector calculated in such a region does not have sufficient accuracy. Therefore, in this embodiment, the distribution state of the feature points in the image is determined, and only a divided region (hereinafter referred to as an effective divided region) having a feature point distribution from which a representative motion vector can be obtained with good accuracy is used for the subsequent processing. Use it.

  FIG. 10 shows an outline of feature point distribution determination by the feature point distribution determination circuit 801. Here, a method of using the number of feature points in each divided region as the distribution state of feature points to be determined will be described.

  First, the feature point distribution determination circuit 801 measures how many feature points exist in each divided region. Next, whether or not each divided region is an effective divided region is determined using the minimum number of feature points that can obtain a representative motion vector with good accuracy as a threshold value.

  The feature point distribution determination circuit 801 determines that a divided region having a smaller number of feature points than a threshold value (predetermined value) is not an effective divided region, such as the divided region 1001 shown in FIG. Then, the divided area 1001 is excluded from the subsequent processing targets. On the other hand, the divided area 1002 in which the number of feature points is equal to or greater than the threshold is determined as an effective divided area.

  Here, the number of feature points in each region is used as the feature point distribution determination method, but the feature point distribution may be determined by other methods. For example, a variance value of feature point coordinates may be used.

  Step S904 in FIG. 9 is the same as step S203 shown in FIG.

  In step S905, the main controller 100 causes the moving object removal circuit 114 to exclude the moving object region from the subsequent processing targets, similarly to step S204 shown in FIG. Further, the main controller 100 causes the moving object removal circuit 114 to exclude a divided area (ineffective area) determined not to be an effective divided area by the above-described feature point distribution determination from subsequent processing targets.

  Steps S906 and S907 are the same as steps S205 and S206 shown in FIG. 2, respectively.

  As described above, in this embodiment, the representative motion vector group in the remaining divided areas excluding the divided areas determined as the moving object areas and the divided areas determined as the ineffective areas from the determination result of the feature point distribution is used. Then, image blur correction processing is performed.

  As a result, it is possible to perform better image blur correction processing (electronic image stabilization processing) that eliminates the influence of the representative motion vector corresponding to the moving object and the representative motion vector that is likely to be erroneously calculated.

  FIG. 11 shows a configuration of an imaging apparatus including an image processing apparatus that is Embodiment 3 of the present invention. In the present embodiment, instead of performing feature point extraction as in the first and second embodiments, motion vectors at a plurality of predetermined coordinate positions (coordinate points) are detected, and image blur correction processing is performed using them.

  In FIG. 11, the same reference numerals as those in FIG. 1 are given to components that have the same or similar functions as the components shown in FIG. 1. The image pickup apparatus according to the present embodiment has a configuration in which the region dividing circuit 111 and the representative motion vector calculation circuit 112 are omitted from the configuration illustrated in FIG. 1 and a motion vector reliability determination circuit 1101 is added. The main controller 100 shown in FIG. 1 is omitted in FIG.

  Hereinafter, the operation of the imaging apparatus (main controller 100) of the present embodiment will be described with reference to the flowchart shown in FIG.

  Step S1201 is the same as step S201 shown in FIG.

  In step S1202, the main controller 100 causes the motion vector detection circuit 110 to set a predetermined coordinate position as a detection point and detect a motion vector at each detection point. The arrangement of the detection points may be any arrangement as long as there are multiple sets of detection points that are symmetrical with respect to the center of the image, such as a lattice arrangement or a concentric arrangement. Good. In general, since it takes a lot of time to extract feature points, it is possible to speed up the processing by setting points for detecting motion vectors in advance as in this embodiment.

  In step S1203, the main controller 100 causes the motion vector reliability determination circuit 1101 to determine whether the motion vector detected at each detection point (each position) is reliable.

  The motion vector is not detected well in a region without a texture or a region where a repetitive pattern exists. Even if the image blur correction process is performed using a motion vector that is not correctly detected, it is not possible to obtain an image in which the image blur is favorably corrected.

  Therefore, in this embodiment, first, a reliability indicating the degree of correctness of the motion vector is determined. As the reliability, the similarity of the motion vector to be determined with respect to the motion vector detected in the vicinity of the motion vector to be determined, or the determination target detected in the current frame image with respect to the motion vector detected in the previous frame image A correlation value of a motion vector or the like can be used. A motion vector whose reliability is lower than a predetermined level is regarded as not being detected correctly and is not used in the subsequent processing.

  In step S1204, the main controller 100 causes the moving object removal circuit 114 to determine the motion vector corresponding to the moving object in the same manner as the moving object region determination process in step S204 of FIG. The moving object removal circuit 114 according to the present exemplary embodiment has a specific relationship between motion vectors at two detection points that are in a symmetrical relationship among a plurality of detection points (a unique relationship according to the type of motion of the entire image caused by shake). ) To determine a motion vector corresponding to the moving object. Then, the motion vector corresponding to the moving object is excluded from the subsequent processing targets.

  Steps S1205 and S1206 are the same as steps S205 and S206 shown in FIG.

  In this manner, in this embodiment, the image blur correction process is performed using the remaining motion vector group excluding the motion vector corresponding to the moving object.

  As described above, in this embodiment, since a motion vector is detected at a predetermined coordinate position, it is not necessary to perform feature point extraction processing and image division processing as in the first and second embodiments. For this reason, high-speed electronic image stabilization processing can be realized.

  In each of the above-described embodiments, the case where the image processing apparatus having the electronic image stabilization function is incorporated in the imaging apparatus has been described. However, the present invention is not limited to this.

  For example, as illustrated in FIG. 14, an image captured by the imaging device 1401 is transmitted to the personal computer 1402. The transmission method may be either a cable method or a wireless method, and may be transmitted via the Internet or a LAN.

  The personal computer 1402 performs processing from the motion vector calculation processing shown in the flowcharts of FIGS. 2, 9 and 12 to image shake correction processing and image output after shake correction according to a computer program stored in a hard disk or the like. In this case, the personal computer functions as the image processing apparatus according to the present invention.

  Each embodiment described above is only a representative example, and various modifications and changes can be made to each embodiment in carrying out the present invention.

  For example, in each of the embodiments described above, the case where the electronic image stabilization processing as the image processing is performed using the detected motion vector has been described. However, the present invention uses an optical element such as a lens using the detected motion vector. The present invention can also be applied to the case where an optical image stabilization process for moving the lens is performed.

1 is a block diagram illustrating a configuration of an imaging apparatus that is Embodiment 1 of the present invention. 6 is a flowchart illustrating the operation of the imaging apparatus according to the first embodiment. FIG. 5 is a diagram illustrating image area division processing according to the first exemplary embodiment. FIG. 6 is a diagram illustrating a representative motion vector calculation process for each divided region according to the first embodiment. FIG. 6 is a diagram illustrating a moving object region determination process according to the first embodiment. The figure which shows the representative motion vector in case the multiple types of shake in Example 1 is mixed. FIG. 4 is a diagram illustrating a moving object region determination method using a plurality of symmetrical positional relationships in the first embodiment. FIG. 3 is a block diagram illustrating a configuration of an imaging apparatus that is Embodiment 2 of the present invention. 9 is a flowchart illustrating the operation of the imaging apparatus according to the second embodiment. FIG. 10 is a diagram illustrating a feature point distribution determination process according to the second embodiment. FIG. 6 is a block diagram illustrating a configuration of an imaging apparatus that is Embodiment 3 of the present invention. 10 is a flowchart illustrating the operation of the imaging apparatus according to the third embodiment. The figure which shows the motion vector which arises by translation, rotation, and the shake of expansion / contraction. FIG. 10 is a diagram illustrating a usage pattern of an image processing apparatus that is Embodiment 4 of the present invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Main controller 101 Optical system 102 Image pick-up element 103 Image formation circuit 109 Motion vector calculation circuit 110 Motion vector detection circuit 111 Area division circuit 112 Representative motion vector calculation circuit 113 Motion vector analysis circuit 114 Moving object removal circuit 115 Correction parameter calculation circuit 116 Shaking Correction circuit 801 Feature point distribution determination circuit 1101 Motion vector reliability determination circuit

Claims (8)

Motion vector calculation means for calculating motion vectors at a plurality of positions in an image obtained by moving image capturing;
Determination means for determining a motion vector corresponding to a moving subject among the motion vectors at the plurality of positions;
Processing means for performing image stabilization processing for reducing image blur based on motion vectors excluding motion vectors corresponding to the moving subject among motion vectors at the plurality of positions;
The determination means determines a motion vector corresponding to the moving subject based on determination of whether or not a motion vector at a symmetrical position among the plurality of positions has a specific relationship. An image processing apparatus.   The image processing apparatus according to claim 1, wherein the symmetry relationship is at least one of point symmetry, vertical symmetry, and left-right symmetry with respect to the center of the image. The plurality of positions are a plurality of regions;
The motion vector calculation means calculates a motion vector at a feature point extracted from the image, calculates a representative motion vector for each region from the motion vector at the feature point,
The processing means performs the image stabilization processing based on a representative motion vector excluding a representative motion vector corresponding to the moving subject among the representative motion vectors in the plurality of regions,
The determination unit determines a representative motion vector corresponding to the moving subject based on determination of whether the representative motion vector in the symmetrical region of the plurality of regions has the specific relationship. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   The determination unit is configured to perform the image stabilization processing based on a representative motion vector obtained by excluding a representative motion vector of a region where the number of extracted feature points is less than a predetermined value among the representative motion vectors in the plurality of regions. The image processing apparatus according to claim 3, wherein:   The image processing apparatus according to claim 1, wherein the plurality of positions are a plurality of predetermined coordinate points. The processing means determines the reliability of the motion vector at each position,
The image stabilization processing is performed based on a motion vector excluding a motion vector having a reliability lower than a predetermined level and a motion vector corresponding to the moving subject among motion vectors at the plurality of positions. The image processing apparatus according to claim 5. An imaging system for capturing moving images;
An imaging apparatus comprising: the image processing apparatus according to claim 1. Calculating motion vectors at a plurality of positions in an image obtained by moving image capturing;
A determination step of determining a motion vector corresponding to a moving subject among the motion vectors at the plurality of positions;
Performing image stabilization processing for reducing image blur based on motion vectors excluding motion vectors corresponding to the moving subject among motion vectors at the plurality of positions,
In the determination step, a motion vector corresponding to the moving subject is determined based on determination of whether or not a motion vector at a symmetrical position among the plurality of positions has a specific relationship. Image processing method.