JP6525693B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus equipped with a technology for calculating the amount of geometric deformation between a plurality of temporally continuous images. The present invention relates to an imaging device such as a digital camera equipped with the image processing device.

  When watching sports or watching birds, generally, a telephoto lens is used to capture a large image of an object present at a distance.

  At this time, if the amount of geometric deformation between frame images is calculated using an image obtained from the telephoto lens in order to perform camera shake correction and alignment combination processing, the apparent movement amount appearing on the image is large. It becomes difficult to detect the amount of movement between them.

  In addition, since the angle of view of the obtained image is narrow in the telephoto lens, it is difficult to intentionally perform framing and camera work.

  Therefore, photographing is performed using a wide-angle lens, and a region corresponding to a desired telephoto angle is cut out from the obtained wide-angle image to perform display, thereby detecting the amount of deformation on the wide-angle image having a small apparent deformation. There is a method to do

  Furthermore, by using a wide-angle lens, it is possible to capture an image in a wider range than the desired telephoto angle of view, which makes it possible to grasp the situation around the telephoto angle of view and to easily perform framing and camera work. Is possible.

  The image processing apparatus described in Patent Document 1 estimates a motion vector of an arbitrary area in an image from a motion vector detected at the center of an image captured at a wide angle, and calculates a geometric deformation amount for the area. I am doing.

JP, 2008-160300, A

  However, the above-mentioned patent documents only calculate the amount of geometric deformation of the peripheral part of the image from the motion vector detected in the central part of the image, and take into consideration the movement of the whole image and the local movement due to the position of the peripheral part. I did not.

  Therefore, the present invention analyzes an amount of geometric deformation of the entire screen with respect to an arbitrary region to be subjected to calculation of the amount of geometric deformation to accurately calculate the amount of geometric deformation of the arbitrary region. Determine the motion vector inside.

  Thus, a method is proposed which makes it possible to calculate an appropriate geometric deformation amount according to the movement of the entire screen and the movement of an arbitrary area.

The image processing apparatus according to one aspect of the present invention includes a motion vector detecting means for detecting a motion vector between a plurality of images, a first determining means for determining a first angle region in the image, the first angle A first calculation unit that calculates a geometric deformation amount using a motion vector detected in the region, and a motion component generated in the first angle of view region from the geometrical deformation amount calculated by the first calculation unit a first analyzing means for analyzing and a second determining means for determining a narrower second angle region than and the first angle region encompassed by the first angle region in the image, the second Third determination means for determining the range of the angle of view of the motion vector used to calculate the amount of geometric deformation of the angle of view as the third angle of view, and the motion vector of the third angle of view A second calculating unit that calculates a geometric deformation amount of the second angle of view area; In the image processing apparatus, the third determination means may determine the third angle of view area from the result of the motion component analyzed by the first analysis means.

  According to the present invention, the motion vector necessary to calculate the amount of geometric deformation for a specific region in the input image is determined.

  This makes it possible to obtain better geometry as compared to using all of the motion vectors detected from the entire screen or calculating the amount of geometric deformation using only the motion vectors detected in a specific region. It becomes possible to calculate the amount of deformation.

A block diagram showing a configuration of an imaging apparatus that is Embodiment 1 of the present invention Flowchart showing the operation of the imaging apparatus of the first embodiment Template matching outline Schematic representation of geometric deformations that occur between frame images Alignment result by local geometric deformation A schematic view of the first angle of view area in the presence of a moving object Method of determining size of first angle of view area when moving object is present A schematic view of the first angle of view area when there is a low contrast area Method of determining size of first angle of view area in the presence of low contrast area A schematic view of a first angle of view area disposed about a second angle of view area Use vector decision outline chart Vector determination method when the second view angle area is not in focus The block diagram which shows the structure of the imaging device which is Example 2 of this invention Flowchart showing the operation of the imaging apparatus of the second embodiment A table showing the relationship between the size of the motion component and the size of the first angle of view area

Example 1
FIG. 1 shows the configuration of an image processing apparatus according to a first embodiment of the present invention.

  In the figure, reference numeral 101 denotes an imaging optical system for forming an object image, and reference numeral 102 denotes an imaging device such as a CCD sensor or a CMOS sensor for photoelectrically converting an object image formed by the optical system 101.

  A camera signal processing unit 103 forms a video signal from the electrical signal output from the imaging device 102. The camera signal processing unit 103 includes an A / D conversion unit (not shown), an auto gain control unit (AGC), and an auto white balance unit, and forms a digital signal.

  The imaging element 102 and the camera signal processing unit 103 constitute an imaging system for acquiring an image. A memory 104 temporarily stores a video signal, a detection result of a motion vector, and a calculation result of a geometric deformation amount.

  The motion vector detection unit 105 detects a motion vector between two images input from the development processing unit 103 and the memory 104. Reference numeral 106 denotes a first determination unit, which determines an area to be subjected to calculation of the overall geometric deformation amount of the screen.

  The first calculation unit 107 calculates the geometric deformation amount of the first angle of view area, and the first analysis unit 108 analyzes the motion component included in the geometric deformation amount.

  The second determining unit finally determines an area in the image for which the amount of geometric deformation is to be calculated.

  Reference numeral 110 denotes a use vector determination unit, which is based on the analysis result of the geometric deformation amount in the first analysis unit 108 and the view angle region determined in the second determination unit 109, the geometric deformation amount of the second view angle region Determine the motion vector needed to calculate.

  A step 111 calculates the amount of geometric deformation of the second angle of view area using the motion vector determined by the motion vector determination unit 110 from among the motion vectors detected by the motion vector detection unit 105.

  The output unit 112 stores the geometric deformation amount calculated by the second calculation unit 111 in the memory 104.

  Then, the main microcomputer 113 controls the operation of the optical system 101 and each arithmetic unit.

  The operation of the imaging apparatus configured as described above will be described using the flowchart shown in FIG.

  In FIG. 2, in step S201, the image of the subject formed by the optical system 101 is output as an analog signal according to the subject brightness in the imaging device 102, and the process of the camera signal processing unit 103 is performed to generate a video signal.

  The camera signal processing unit 103 converts an analog signal into, for example, a 12-bit digital signal by an A / D conversion unit (not shown).

  Further, a digital video signal subjected to signal level correction and white level correction by AGC and AWB (not shown) is stored and held in the memory 104.

  In the imaging apparatus of the present embodiment, frame images are sequentially generated at a predetermined frame rate, and the frame images stored and held in the memory 104 are input to the motion vector detection unit 105. Also, the frame images stored and held in the memory 104 are sequentially updated.

  In step S202, the motion vector detection unit 105 detects a motion vector between the two input frame images.

  In this embodiment, a method using template matching will be described as an example of a motion vector detection method. FIG. 3 shows a schematic view of template matching.

  In the figure, (a) is an original image, (b) is a reference image, and these images are image data stored and held in the memory 104.

  Then, as shown at 301 in FIG. 3, the template block is placed at an arbitrary position in the original image, and the correlation value between the template block 301 and each area of the reference image is calculated.

  At this time, since calculating the correlation value for the entire area of the reference image requires a large amount of calculation, as shown in 302, the rectangular area for calculating the correlation value on the reference image is actually searched for Set as.

  Here, the position and the size of the search range 302 are not particularly limited, but if the search range 302 does not include an area corresponding to the moving destination of the template block 301, it is possible to detect a correct motion vector. Absent.

  In this embodiment, a sum of absolute differences (hereinafter abbreviated as SAD) is used as an example of a method of calculating the correlation value. The calculation formula of SAD is shown in Formula 1.

  In Equation 1, f (i, j) represents a pixel value at coordinates (i, j) in the template block 301, and g (i, j) is a block 303 to be subjected to correlation value calculation in the search range 302. Represents each pixel value in

  In SAD, the absolute value of the difference is calculated for each pixel value f (i, j) and g (i, j) in both blocks, and the correlation value S_SAD can be obtained by calculating the sum.

  Therefore, the smaller the value of the correlation value S_SAD, the smaller the difference in pixel value between the two blocks, that is, the textures in the blocks of the template block 301 and the correlation value calculation area 303 are similar.

  Although SAD is used as an example of the correlation value in the present embodiment, the present invention is not limited to this method, and other correlation values such as differential sum of squares (SSD) and normalized cross correlation (NCC) may be used. good.

  However, when using a correlation value other than SAD, there are two cases where the similarity is higher as the correlation value is smaller according to the characteristics, and the case where the similarity is higher as the correlation value is larger. The process also needs to be changed.

  A correlation value is calculated between the template block 301 and the search range 302, and the position where the value becomes the smallest is determined.

  Then, it becomes possible to detect to which position in the reference image the template block on the original image has moved, that is, to detect the motion vector between the images.

  The motion vector detection process as described above is performed at a plurality of coordinate positions between input frame images.

  The motion vector detection positions may be arranged at regular intervals in a grid or may be arranged on feature points such as corners, and the range may be arranged so as to cover the entire image, or the first method described later. There is a method of arranging in the angle of view area. Then, the detected motion vector group is transmitted to the memory 104 and the first calculation unit 107.

  In step S202, a first angle of view area is set between two frame images, and the amount of geometric deformation for the area is calculated.

  Here, the first angle of view area in the present embodiment is a second angle of view area to be described later, which is determined as a second angle of view area. This is an area wider than the second angle of view area required for

  Here, first, a method of determining the first angle of view area in the present embodiment will be described.

  FIG. 4 schematically shows the motion generated in two frame images.

  It is assumed that a rotational motion occurs in the next frame image (b) due to the motion of the imaging device with respect to the frame image (a) captured at a certain time.

  In FIG. 6A, reference numeral 401 denotes a second angle-of-view area which is to be finally obtained as a geometric deformation amount.

  Reference numeral 402 denotes a second angle-of-view region in the frame image (b), and reference numeral 403 schematically denotes a motion vector group in which a rotational motion between frames is detected.

  Here, in order to calculate the amount of geometric deformation occurring between the second angle of view regions 401 and 402, it is assumed that the amount of geometric deformation is calculated using the motion vector 404 detected inside 402.

  In this case, since the motion vector 404 represents the motion in the lower left direction, the amount of geometric deformation calculated using it is also only the translation component in the lower left direction.

  At this time, FIG. 5 shows the state of the subject in the second angle of view area when the alignment of the second angle of view area between the frame images is performed using the calculated amount of geometric deformation.

  In FIG. 4A, reference numeral 501 denotes an object present inside the second angle of view area 401 in FIG. 4A, and reference numeral 502 denotes an object existing inside the second angle of view area 402 in FIG. 4B.

  As shown in FIG. 5, if alignment is performed using a geometric deformation amount calculated using only a local motion vector such as the motion vector 404, translational motion can be correctly aligned. .

  However, since the inclination of the main subject caused by the rotation of the entire frame image is not corrected but remains as it is, as a result, it is not possible to perform accurate alignment.

  The above is caused by calculating the amount of geometric deformation of the main subject using the motion vector detected from the local region without considering the movement of a wide range in the image.

  Further, the amount of geometric deformation is calculated using all motion vectors detected from the entire image indicated by 403 in FIG.

  In this case, the accuracy of the amount of geometric deformation may decrease due to the influence of the moving object present in the position away from the second angle of view area, the near and far conflict area, and the vector that fails to be detected.

  Therefore, in this step, in order to calculate the geometric deformation amount of the second angle of view region with high accuracy, it is determined as the first angle of view region with respect to what degree of angle of view region in the frame image.

  In order to accurately calculate the amount of geometric deformation for the first angle of view area, the first angle of view area is determined so as to exclude a motion vector indicating a motion different from the motion of the imaging device in the first angle of view area. There is a need to.

  As a method of determining the first angle of view area, as shown in FIG. 6, a method of preventing moving objects existing outside the second angle of view area detected using difference information between frame images from entering There is.

  In FIG. 6A, reference numeral 601 denotes a second angle of view area, and reference numeral 602 denotes a moving object present outside the second angle of view area.

  And (b) of the figure represents the state which the rotational motion has produced in the following flame | frame of (a), and 603 has shown the motion vector detected by the mobile body 602. FIG.

  The motion vector detected by the moving body 602 has a motion different from that of the vector group 604 representing the motion of the rotation of the entire image, which becomes an error factor in calculating the amount of geometric deformation.

  Therefore, in this case, it is possible to accurately calculate the amount of geometric deformation for the first angle of view area by setting the first angle of view area as 605 and excluding the motion vector of the moving object.

  At this time, the first angle of view area can be automatically set according to the position and size of the detected moving body.

  FIG. 7 is a graph showing a method of setting the first angle of view area. In the figure, the vertical axis of the graph is the distance between the moving object and the second angle of view area, and the horizontal axis shows the size of the first angle of view area.

  The size and the position of the first angle of view area are set as wide as possible so that the second angle of view area is included in the inside and does not protrude outside the image.

  Therefore, it is possible to calculate the amount of geometric deformation reflecting the movement of the entire screen.

  For that purpose, as shown in FIG. 7, the size of the first angle of view area may be set to be in direct proportion to the distance between the movable body and the second angle of view area.

  At this time, when a plurality of moving objects are detected, the setting of the first angle of view region is performed using the distance to the moving object closest to the second angle of view region.

  Here, in the case of a distance at which the moving body enters into the second angle of view area, the first angle of view area is used to calculate the amount of geometric deformation in a range wider than the second angle of view area. , Not smaller than the second angle of view area.

  By doing as described above, it is possible to set the first angle of view area as wide as possible while excluding the moving body area as shown by 605 in FIG. 6B.

  There is also a method of determining a low contrast area existing in a frame image based on the variance of pixel values, and determining the first angle of view area so that the area is not included.

  FIG. 8 (a) shows a frame image taken at a certain time, FIG. 8 (b) shows a case where a rotational movement occurs in the next frame, and 801 shows a second angle of view in FIG. It represents an area.

  A good geometric deformation amount is calculated by setting the first angle of view area 803 by excluding the motion vector that failed to be detected in the area without texture such as sky or ground as shown in 802 of FIG. You can do it.

  At this time, the first angle of view area can be automatically set in accordance with the position and size of the detected low contrast area, as in the case where the above-described moving body is present.

  FIG. 9 is a graph showing a method of setting the first angle of view area. In the figure, the vertical axis of the graph represents the distance between the low contrast area and the second angle of view area, and the horizontal axis represents the size of the first angle of view area.

  As shown in FIG. 9, the size of the first angle of view area may be set to be in direct proportion to the distance between the low contrast area and the second angle of view area.

  At this time, when a plurality of low contrast areas are detected, the first angle of view area is set using the distance to the low contrast area closest to the second angle of view area.

  Here, when the low contrast area is intruded into the second angle of view area, the first angle of view area is for calculating the amount of geometric deformation in a range wider than the second angle of view area. Do not make it smaller than the second angle of view area.

  By doing the above, it is possible to set the first angle of view area as wide as possible while excluding the low contrast area as shown in 803 of FIG. 8B.

  The above describes the method of automatically setting the first angle of view area according to the moving object and the low contrast area in the image, but as the other method, the photographer manually sets it while looking at the liquid crystal monitor of the imaging device It is good.

  As another method of determining the first view angle area, the first view angle area 1002 may be determined so that the second view angle area 1001 is always positioned at the center as shown in FIG.

  As a result, it is possible to calculate the amount of geometric deformation of the first angle-of-view region by emphasizing the geometric deformation of the second angle-of-view region.

  Further, with regard to the size of the first angle of view area, the size of the first angle of view area of the current frame can also be determined based on, for example, the analysis result of the geometric deformation of the first angle of view area in the previous frame.

  FIG. 15 shows the relationship between the size of each motion component of the geometric deformation amount in the previous frame and the size of the first angle of view area set in the current frame.

  According to FIG. 15, when the translational component is main in the geometric deformation amount in the previous frame, the first field angle area of the current frame is narrowed, thereby causing movement other than translational movement that causes an error in the geometric deformation amount calculation. It is possible to reduce the possibility of the motion vector detecting the entering.

  On the contrary, if the geometric deformation amount in the previous frame is one of the main motion components, that is, any one of the rotation component, the tilt component, and the motion component of the scaling component, the present Widen the first angle of view area at the frame.

  The motion component of the rotation component, the tilt component, and the scaling component is detected as a motion vector of magnitude and direction with a constant rule in each region in the image, so the motion vector of the widest possible range is used The geometric deformation can be calculated more accurately.

  When at least one of the absolute value of the rotation component, the absolute value of the tilt component, and the absolute value of the scaling component in the first angle of view area is larger than a predetermined value, using the motion vector detected in the first angle of view area A geometric deformation amount of the second angle of view area is calculated.

  In addition, when all motion components are small, they are easily affected by moving objects present in the image and motion vectors that fail to be detected, and therefore, they are not affected by narrowing the first angle of view area. Let's do it.

  When the translational component is large and the rotational component, the tilt component, and the scaling component are small, the degree of influence of the other motion components is smaller than that of the translational component in the calculation of the geometric deformation amount. The corner area can be widely set as compared with the case where all the motion components described above are small.

  For determination of which motion component is the main motion, for example, a determination value as the main motion is set for each motion component.

  If the magnitude of the motion component is zero, it is determined that the determination value is zero, that is, it is not the main motion, and the value of the determination value is increased as the magnitude of the motion component increases.

  The determination value of each motion component is compared, and for example, two components are determined to be main motion components in the order of the larger determination value, or the one having the largest determination value is determined as the main motion.

  Also, as another example of the main motion determination method, if the motion component is larger than a predetermined threshold value, it may be determined that the motion is the main motion.

  In the present embodiment, it has been described where in the frame image the first angle of view area is determined, but by making the first angle of view area the same as the shooting angle of view, the amount of geometric deformation of the entire frame image is calculated. You may do it.

  Further, in the present embodiment, the first view angle area is represented as a rectangular area, but the present invention is not limited to this, and any shape such as a circle or a trapezoid may be used without any problem.

  The second angle of view area is determined in step S205. The second angle-of-view area determining unit 109 determines an angle-of-view area to be finally calculated with respect to the amount of geometric deformation in the frame image as a second angle-of-view area.

  As a method of determining the second angle of view area, a photographer manually designates an arbitrary subject or area while looking at a live view image or the like, and determines an area in which the user can include it.

  As a result, it is possible to determine an area for which the geometric deformation amount is to be finally calculated as the second angle of view area.

  Using the motion vector detected as described above, the first view angle area geometric deformation amount calculation unit 107 calculates the geometric deformation amount of the first view angle area. The method of calculating the geometric deformation amount using the motion vector group will be described below.

  In this embodiment, a calculation method will be described in the case where the homography matrix is used as the image deformation amount as a model of geometric deformation. A point a on the current frame image,

Is point a 'in the next frame,

Suppose you move to Here, the subscript T indicates that it is a transposed matrix.

  The correspondence between the point a in the equation 2 and the point a ′ in the equation 3 can be obtained by using the homography matrix H

It can be expressed as.

  The homography matrix H is a determinant indicating the amount of deformation due to translation, rotation, magnification change, shear, or tilt between images, and can be expressed by the following equation.

  Each element of the homography matrix H can be calculated by applying statistical processing such as the least squares method using the motion vector group obtained in step S204, that is, the correspondence of representative points between frame images. .

  The homography matrix H determined in this manner represents the amount of deformation of the image due to the movement of the imaging device.

  In the present embodiment, although the homography matrix has been described as a model representing the geometric deformation amount of the first angle of view area, the present invention is not limited to this, and other models such as a Helmert matrix or affine matrix may be used.

  In step S204, using the geometric deformation amount acquired from the first calculation unit 107, analysis is performed as to what kind of motion component is included in the geometric deformation amount of the first angle of view area between the frame images.

  In this embodiment, a method of decomposing a homography matrix into each motion component will be described as an example of a method of extracting each motion component from a geometric deformation amount.

  The homography matrix H shown in Equation 5 can represent the movement of translation t, rotation R, scaling s, shear K, and tilt v between frame images, and can be deformed as follows.

  However, each variable is as follows.

  Here, θ represents a rotation angle around the optical axis, and b and φ represent shear coefficients.

  Also, the matrix A has the relationship shown in Equation 14.

  Next, a method of calculating R and K will be described. First, RK is obtained from Expression 15 obtained by transforming Expression 14.

  Then, by utilizing the property that K is an upper triangular matrix, the matrix A is finally calculated by performing separation of R and K by QR decomposition.

  As a result, it is possible to obtain motion components of translation t, rotation R, scaling s, shear K, and tilt v between frame images.

  Then, the determination process is performed on the size of each motion component to analyze the amount of geometric deformation of the first angle of view area.

  As the determination method, for example, when the amount of movement is larger than a predetermined threshold value for each movement component, it is determined that the movement component is the main movement component in the first angle of view area.

  In addition, the movement amount of each pixel is actually calculated individually for each movement component, and it is determined from the ratio of the movement amount, etc. which movement amount occupies the largest proportion in the entire geometric deformation amount. The main movement may be determined.

  In the present embodiment, the method of decomposing the homography matrix into the motion component has been described in order to analyze the geometric deformation amount, but the present invention is not limited thereto. For example, each component of the homography matrix is approximated as the motion component Alternatively, a method may be used in which the determination is made.

  The analysis result of the geometric deformation of the first angle of view area calculated as described above is transmitted to the use vector determination unit 110.

  In step S205, the area to be the target of which the amount of geometric deformation is finally calculated is determined as the second angle of view area by the method described above.

  Information on the position and range of the second angle of view area determined as described above is transmitted to the use motion vector determination unit 110.

  In step S206, using the analysis result of the geometric deformation amount of the first angle of view area obtained from the first analysis unit 108 and the information of the second angle of view area obtained from the second determination unit 109, a second angle of view area Calculate the geometric deformation of.

  In this step, first, the vector determination unit 110 determines a motion vector used to calculate the amount of geometric deformation of the second angle of view area.

  Here, in the analysis result obtained from the first analysis unit 108, for example, it is assumed that one or more motion components of rotation, tilt, and scaling are determined as the main motion in the first angle of view region. .

  As a method of determining whether each motion component is a main motion component, for example, a determination value as a main motion is set for each motion component.

  If the magnitude of the motion component is zero, it is determined that the determination value is zero, that is, it is not the main motion, and the value of the determination value is increased as the magnitude of the motion component increases.

  The determination value of each motion component is compared, and for example, two components are determined to be main motion components in the order of the larger determination value, or the one having the largest determination value is determined as the main motion.

  In addition, as another determination method, each motion component may be compared with a predetermined threshold value, and a motion component having a motion larger than the threshold value may be determined as the main motion component in the image. FIG. 11 shows a schematic diagram of use vector determination.

  In FIG. 11A, reference numeral 1101 denotes a first angle of view area, and reference numeral 1102 denotes a second angle of view area.

  When it is determined that any one or a plurality of motion components of rotation, tilt, and scaling is the main motion, a motion vector detected in a wider range than the second angle of view region 1102 as indicated by 1103. Try to use

  As a result, the motion of the second angle of view region is erroneously determined to be a motion of translation only, or the influence of the motion vector of the moving object at a position away from the second angle of angle region or the detection failure vector in the low contrast region It is possible to calculate a good amount of geometric deformation without receiving

  On the other hand, if it is determined that the main motion component of the geometric deformation in the first angle of view region is a component of translation, it is detected in the second angle of view region as indicated by 1104 in FIG. The geometric deformation amount of the second angle of view area is calculated using the motion vector.

  Thus, if the first angle of view area is performing only translational movement, the amount of geometric deformation of the second angle of view area is also only translational movement.

  Therefore, the translational movement of the second angle of view area can be accurately calculated by using only the motion vector detected in the second angle of view area for calculating the amount of geometric deformation.

  Also, as an example of the method of determining the range of the use vector shown in 1103 and 1104, in addition to the type of the main motion component, it can be determined according to the magnitude of the main motion component.

  For example, as the magnitude of each motion component increases, the range 1103 of motion vectors to be used is broadened.

  As a result, in the case of a large amount of motion, the amount of geometric deformation is calculated using more motion vectors, so that the amount of geometric deformation can be calculated with higher accuracy.

  Conversely, when the size of the main motion component is small, the calculation accuracy of the geometric deformation amount is not greatly affected even if the number of motion vectors used is small, and the calculation cost reduction effect can be expected.

  Here, as another example of the used vector determination method, it can be determined according to the state of the subject in the second angle of view area.

  An area including a specific subject area present in the image obtained by the image recognition process is set as the second angle of view area.

  For example, the difference between the pixel values in the second angle of view area between the frames is calculated, and the proportion of pixels in the second angle of view area in which the difference value is larger than the predetermined threshold is calculated. Do.

  Thus, it can be determined whether a non-rigid body with large motion exists in the second angle of view area.

  The threshold value at this time may be set arbitrarily. For example, when pixels having a difference value of 10 or more occupy 80% or more of the total pixels in the second angle of view area, most of the second angle of view area is non-rigid. It is sufficient to judge that it is occupied by

  A schematic diagram of the use vector determination in this case is shown in FIG. In FIG. 12, reference numeral 1201 denotes a second angle of view area including a non-rigid area, 1202 denotes an existing range of a used motion vector group, and 1203 denotes a motion vector detected within the second angle of view area.

  The motion vector detected on the non-rigid body is different from the motion of the whole image, and the motion is unstable between each frame, so using them may not be able to obtain a good amount of geometric deformation .

  Therefore, when there is a non-rigid body in the second angle of view area, motion vectors in the second angle of view area among the used vectors in 1202 are not used as shown in FIG. Then, the geometric deformation amount of the second angle of view area is calculated.

  As a situation in which the vector in the second angle of view area is not used, in addition to the presence of the non-rigid body, the subject in the second angle of view area may not be in focus.

  Even in such a case, the accuracy of the motion vector detected in the second angle of view area is low, which may cause an error in calculating the geometric deformation amount.

  Therefore, by not using motion vectors in the second angle of view area, it is possible to calculate the amount of geometric deformation of the second angle of view area with high accuracy.

  As to the determination as to whether or not the subject in the second angle of view area is in focus, it is determined that the focus is not in focus if the variance value of the pixel values in the second angle of view area is less than a predetermined threshold. Just do it.

  As another method, it can be determined from which AF region in the image the image is in focus from the AF parameter at the time of shooting.

  The second calculation unit 111 may set the first angle of view area so that an area smaller than a predetermined contrast value present in the image is not included.

  Information on the use vector determined as described above is transmitted to the second calculation unit 111.

  The second calculator 111 uses the motion vector determined by the use vector determiner 110 to calculate the amount of geometric deformation of the second angle of view area.

  The method of calculating the amount of geometric deformation of the second angle of view area may be the same as the process of calculating the amount of geometric deformation in the first calculation unit 107.

  In FIG. 2, in step S <b> 207, the geometric deformation amount calculated by the second calculation unit 111 is stored in the memory 104 or transmitted to an external storage device (not shown).

  As described above, when calculating the amount of geometric deformation for a specific angle of view area in a frame image, adaptively determine at which position the motion vector detected is to be used, including the area around it. .

  This makes it possible to calculate the amount of geometric deformation of a specific angle of view area in a frame image with high accuracy.

Example 2
FIG. 13 shows the configuration of an imaging apparatus according to a second embodiment of the present invention.

  In this embodiment, geometric deformation is calculated using images and motion vectors obtained from two different optical systems.

  In the figure, the same reference numerals as in FIG. 1 are given to the same components as those shown in FIG.

  The imaging apparatus of the present embodiment has a second optical system 1301, a second imaging element 1302, a second development processing unit 1303, and a second motion vector detection unit 1305 in the configuration shown in FIG.

  Furthermore, in the present embodiment, a second determination unit 1304 is provided instead of the second determination unit 109.

  Further, a flowchart in the present embodiment is shown in FIG. In the present embodiment, only a part performing processing different from that of the first embodiment in FIG. 14 will be described.

  Steps S201 to S204 are the same as S201 to S204 in FIG.

  In step S1301, the second determination unit 1304 determines which area in the image acquired from the first optical system 101 the frame image acquired from the second optical system 1301 corresponds to the second angle of view Determined as an area.

  Here, the first optical system 101 can acquire an image having a wider angle than the second optical system 1301, and an image obtained from the second optical system is an image obtained from the first optical system. It is assumed that a part of the angle of view in the middle is photographed.

  As a method of determining the second angle of view area, there is a method of first grasping a relative positional relationship from each other from position and orientation information of each optical system obtained from the imaging device.

  If the direction of the second optical system 1301 with respect to the direction of the first optical system 101 is known, it is known which region in the image obtained from the first optical system the second optical system is imaging It can.

  Then, the second optical system 1301 picks up an extent of the range in the image obtained from the first optical system 101 by using the focal length information of each optical system similarly obtained from the imaging device. You can calculate the

  There is also a method of using image information obtained from each optical system as a method of determining the second angle of view area.

  A feature point such as a corner of the texture is extracted from each image, and a feature amount is determined for the feature point.

  Then, by performing feature point matching processing, it is possible to determine where in the image obtained from the first optical system 101 there is a region that matches the image obtained from the second optical system 1301.

  In step S 1402, the second motion vector detection unit 1305 detects a motion vector on the image obtained from the second optical system 1301.

  The motion vector detection method may be similar to that of the motion vector detection unit 105 described in the first embodiment.

  In step S206, using the analysis result of the geometric deformation amount of the first angle of view area obtained from the first analysis unit 108 and the information of the second angle of view area obtained from the second determination unit 1304, the second optical system Calculate the geometric deformation of the image obtained from

  The calculation method may be the same as step S206 in the first embodiment, but the second optical system 1301 detected by the second motion vector detection unit 1305 as a motion vector in the second angle of view area. Use the motion vector for the image obtained from.

  At this time, since the first and second optical systems have different focal lengths, the apparent amount of movement appearing on the image is different even if the subject in the same movement is imaged in real space. .

  Therefore, when determining the use vector and calculating the geometric deformation amount of the second angle of view area, it is necessary to equalize the apparent motion amount by performing conversion processing according to the focal length on the motion vector.

  As a method of equalizing the motion amounts, for example, one motion vector may be matched to the other motion vector, assuming that the apparent motion amount is proportional to the focal length, and as another method, each motion vector is focal length A method such as normalization with.

  Step S207 in FIG. 14 is the same as S207 in FIG.

  Up to this point, the operation of the present proposal in the second embodiment has been described.

  The first embodiment has described a method of calculating the amount of geometric deformation of an arbitrary angle of view region for an image obtained from one optical system.

  On the other hand, in the present embodiment, in a plurality of optical systems having different focal lengths, there has been described a method for favorably calculating the amount of geometric deformation of a telephoto image by using motion vector information obtained from a wide-angle image.

  Thus, for example, it is possible to calculate an image of the second angle of view area having a higher resolution than the extraction of a part of the wide angle image as the second angle of view area as in the first embodiment and the amount of geometric deformation thereto. .

  Further, in the present embodiment, the case where a plurality of optical systems are provided in one imaging device has been described, but geometric deformation can be calculated by the same method as in the present embodiment between images captured by different imaging devices. It becomes.

  Here, as an application of the present invention, there is a case of shooting a long distance object such as watching sports or watching birds. Then, when a subject region is cut out from a wide-angle image and displayed as a telephoto image, it is possible to generate an image in which the blur occurring in the subject on the image is favorably corrected.

  Furthermore, as another application, for example, there is an application to a surveillance camera.

  In the case of photographing with a surveillance camera, when the region of interest such as a suspicious person is cut out and displayed from an image photographed at a wide angle, the method of the present invention is used. Therefore, it is possible to generate an image in which blurring is properly corrected for the subject in the attention area.

  As a result, the visibility of the subject in the region of interest is enhanced, and it is possible to accurately execute the face identification process and the tracking process performed as the process of the latter stage.

(Other embodiments)
The object of the present invention can also be achieved as follows. That is, a storage medium storing a program code of software in which a procedure for realizing the function of each embodiment described above is recorded is supplied to a system or an apparatus. Then, the computer (or CPU, MPU or the like) of the system or apparatus reads out and executes the program code stored in the storage medium.

  In this case, the program code itself read out from the storage medium implements the novel function of the present invention, and the storage medium storing the program code and the program constitute the present invention.

  Further, as a storage medium for supplying the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk and the like can be mentioned. Further, a CD-ROM, a CD-R, a CD-RW, a DVD-ROM, a DVD-RAM, a DVD-RW, a DVD-R, a magnetic tape, a non-volatile memory card, a ROM and the like can be used.

  In addition, by making the program code read out by the computer executable, the functions of the above-described embodiments are realized. Furthermore, when the OS (Operating System) or the like running on the computer performs part or all of the actual processing based on the instruction of the program code, and the functions of the above-described embodiments are realized by the processing. Also included.

  Furthermore, the following cases are also included. First, the program code read out from the storage medium is written to a memory provided in a function expansion board inserted in the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, a CPU or the like provided in the function expansion board or the function expansion unit performs part or all of the actual processing.

  Furthermore, the present invention is not limited to devices mainly for photographing such as digital cameras, and includes built-in or external connection with an imaging device such as a mobile phone, personal computer (laptop type, desktop type, tablet type, etc.) and game machine. It is applicable to any equipment that Therefore, the "imaging device" in the present specification is intended to encompass any electronic device having an imaging function.

107 first calculation unit 108 first analysis unit 110 motion vector determination unit 111 second analysis unit

Claims (15)

Geometric by using the motion vector detecting means for detecting a motion vector between a plurality of images, a first determining means for determining a first angle region in the image, the motion vector detected by said first angle region A first calculation unit that calculates a deformation amount; a first analysis unit that analyzes a motion component occurring in the first angle of view area from the geometric deformation amount calculated by the first calculation unit; and the image And second determining means for determining a second angle-of-view area included in the first angle-of-view area and narrower than the first angle- of- angle area, and for calculating the amount of geometric deformation of the second angle-of-view area. Geometric deformation of the second angle of view area is calculated using third determining means for determining the range of the angle of view area of the motion vector to be used as the third angle of view area and the motion vector of the third angle of view area. An image processing apparatus having a second calculation unit;
3. The image processing apparatus according to claim 1, wherein the third determination unit determines the third angle of view area from the result of the movement component analyzed by the first analysis unit. Said geometric deformation amount calculated by the first calculating means and the geometric deformation amount calculated by the second calculating means, translational component, rotational component, scaling component, at least one motion component of tilt component The image processing apparatus according to claim 1, characterized in that   The image processing apparatus according to claim 1, wherein the first determination unit sets the first angle of view area so that a moving object present in an image is not included.   4. The first determination unit according to any one of claims 1 to 3, wherein the first determination unit sets the first angle of view area so that an area smaller than a predetermined contrast value present in an image is not included. An image processing apparatus according to claim 1.   The second determination means sets an area including a specific subject area present in an image obtained by image recognition processing as the second angle of view area. An image processing apparatus according to any one of the preceding claims. The second calculation means is configured to set the first value when at least one of an absolute value of the rotation component, an absolute value of the tilt component, and an absolute value of the scaling component in the first angle of view region is larger than a predetermined value. The image processing apparatus according to claim 2, wherein a geometric deformation amount of the second angle of view area is calculated using a motion vector detected in the angle of view area. When the absolute value of the translational component in the first angle of view area is large, the second calculation means performs geometric deformation of the second angle of view area using a motion vector detected from within the second angle of view area. The image processing apparatus according to claim 2, wherein the amount is calculated.   The image processing apparatus according to any one of claims 1 to 7, wherein the third determination unit enlarges the third angle of view area as the movement component increases. The second calculation means is included in the second angle of view area among the motion vectors detected in the first angle of view area when the second angle of view area is occupied by a non-rigid body. The image processing apparatus according to any one of claims 1 to 7, wherein the geometric deformation amount of the second angle of view area is calculated using a non-motion vector. The second calculation means is included in a second angle of view area among motion vectors detected in the first angle of view area when the subject in the second angle of view area is not in focus. The image processing apparatus according to any one of claims 1 to 7, wherein the geometric deformation amount of the second angle of view area is calculated using a non-motion vector. The second calculation means is weighted at the time of calculation of the amount of geometric deformation as the detected position of the motion vector used to calculate the amount of geometric deformation of the second angle of view region moves away from the second angle of view region. The image processing apparatus according to any one of claims 1 to 10, characterized in that A motion vector detection unit that detects a motion vector between a plurality of images; and a calculation unit that calculates an amount of geometric deformation of a first region of the image used to detect the motion vector.
The calculation means uses the motion vector detected in a region other than the first region in a second region including the first region and wider than the first region, the first region And calculating an amount of geometric deformation with respect to the image processing apparatus.   An imaging device comprising the image processing device according to any one of claims 1 to 12 and an imaging element for outputting the motion vector. Geometric by using the motion vector detection step of detecting a motion vector between a plurality of images, a first determination step of determining a first angle region in the image, the motion vector detected by said first angle region A first calculation step of calculating a deformation amount; a first analysis step of analyzing a motion component occurring in the first angle of view area from the geometric deformation amount calculated in the first calculation step; a second determination step of determining said narrower than and the first angle region encompassed by the first angle region second angle region, in order to calculate the geometric deformation of the second angle region calculating a third determination step of determining the range of the angle of view region of the motion vector to be used as the third angle region, the geometric deformation of the second angle region by using the motion vector of the third angle region An image processing method having a second calculation step;
The image processing method characterized in that the third determination step determines the third angle of view area from the result of the motion component analyzed in the first analysis step . A motion vector detection step of detecting a motion vector between a plurality of images; and a calculation step of calculating a geometric deformation amount for a first region of the image used for detection of the motion vector,
The calculating step includes using the motion vector detected in a region other than the first region in a second region including the first region and wider than the first region. And calculating an amount of geometric deformation for the image processing method.

Images