JP2005157924A - Image movement detector - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately and easily detect the movement of images whose object is a target or background under consideration from the complicated change of picked-up images generated by the movement of an image pickup device or the like. <P>SOLUTION: An image movement detector comprises: a part-under-consideration information input means 1803 for inputting or setting the information of the part of the target or the background under consideration; a setting means 1802 for weighting or the like for collating the information from the part-under-consideration information input means 1803 and the contents of the image, performing weighting so as to increase weight at the part under consideration and reduce the weight of the part considered as the part other than the part under consideration and setting the initial value of movement amount calculation as needed; and a movement amount calculation means 1801 in which weighting or the like is taken into consideration for utilizing the weighting coefficient of weighting and the initial value of the movement amount calculation set in the setting means 1802 for weighting or the like and estimating the movement amount of the image paying attention to the part under consideration or excluding the part under consideration. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention is necessary in a wide range of fields such as image stabilization, image pasting, image collation, image fusion, image sharpening, moving target detection, target tracking, image guidance, image compression, stereo image processing, and three-dimensional image analysis. The present invention relates to an image motion detection apparatus that measures the deformation of an image and the amount of motion of the image.

  In general, image stabilization means detecting a motion amount of an image and correcting it to generate a stable image. FIG. 8 is an example of a block diagram of processing for stabilizing a moving image (image stabilization). A moving image 101 is input, and a stabilized moving image 102 is output as a processing result. The motion amount calculation means 103 compares the current input image 107 of the moving image 101 with the reference image 108 to estimate the motion amount. From the estimated motion amount, the motion amount filtering 104 estimates the estimated motion so as to stabilize the image output so that the small vibration of the image is absorbed and the motion that the entire image pans slowly remains unchanged. Based on the amount, an appropriate filtering process is performed to obtain a correction amount of the motion of the image. The image warp unit 105 performs geometric transformation on the input image 107 based on the obtained image motion correction amount to stabilize the image. The image warped and stabilized image becomes a reference image 108 which is one of the inputs of the motion amount calculation means 103 after passing through the image buffer 106.

  Here, the example of image stabilization has been described, but detection of the amount of motion between images as performed by the motion amount calculation means 103 is performed by image pasting, image matching, image fusion, image sharpening, moving target detection, target This function is necessary in a wide range of fields such as tracking, image guidance, image compression, stereo image processing, and three-dimensional image analysis.

  For example, when combining images, that is, when combining a plurality of images that have been divided and taken together to combine a single wide-field / high-resolution panoramic image, alignment between the images is performed. Therefore, it is necessary to accurately determine the amount of motion between images. As an example of image fusion, if it is possible to detect the amount of motion between an image focused at a distance and an image focused at a close distance and match the images, it is possible to create a composite image that is focused both at a distance and near it can.

  In the following, a conventional technique for estimating the amount of deformation and movement of an image necessary for various image processing such as image stabilization and image pasting will be described. There are (1) correspondence search method, (2) lightness gradient method, and (3) analysis method in frequency space, etc., for measuring the amount of deformation and motion between images. The present invention can be applied to a method for measuring the amount of deformation and the amount of motion between all these images. For simplicity, here is an example of the lightness gradient method, which basically does not require a search and can be processed faster than the corresponding search method (also called the matching method) that searches all possible movements one by one. It explains using. Note that the amount of motion of the image handled here is not limited to the amount of motion of a single point, but is intended for a region having a certain spread or the amount of deformation or motion of the entire image.

First, how the amount of motion of a certain point (pixel) can be obtained from two images by the brightness gradient method will be described. The lightness gradient method is a method that can calculate the amount of motion of an image from the change in lightness of pixels at high speed without the necessity of performing association processing, which is a problem in image processing. FIG. 9 shows the principle of the brightness gradient method. Here, for the sake of simplicity, it is considered in one dimension, the vertical axis is the image brightness I, the horizontal axis is the coordinate x on the image, and the gradient of the image brightness I (x) is constant and linearly changing. And Then, the image I ₁ at time t _{1 (x)} is changed at time t ₂ in the image I _{2 (x).} That is, the point at x ₁ in the image I ₁ (x) has moved to the position x _{2 in} the image I ₂ (x). Here, it is assumed that the brightness of the object does not change even after movement. The gradient of the triangle formed between two straight lines with parallel image brightness is Ix, the height is the brightness difference dI = I ₂ (x ₁ ) −I ₁ (x ₁ ), and the base is the movement amount dx = x ₂ −x since it is _1, the movement amount dx it can be seen that can be calculated from the slope Ix and lightness difference dI of the point _{x 1 (dx = -dI / Ix} ).

  The one-dimensional relationship shown here also holds between the two-dimensional motion amount and the brightness change (see the following formula).

Expression (1) is a relational expression (constraint expression of the lightness gradient method) showing the relationship between the two-dimensional motion amount (u, v) and the lightness change. However, Ix: gradient in the x-axis direction, Iy: gradient in the y-axis direction, It: brightness difference. The gradient Ix in the x-axis direction, the gradient Iy in the y-axis direction, and the brightness difference It between images are given by equation (2). The brightness difference It calculates the brightness difference between the images at that point. Further, the gradient Ix in the x-axis direction and the gradient Iy in the y-axis direction can be calculated by a difference operator as shown in FIG. 10, for example. Therefore, the gradient Ix in the x-axis direction, the gradient Iy in the y-axis direction, and the brightness difference It between the images can be obtained directly from the image, and the two-dimensional motion amount (u, v) can be obtained from the constraint equation of equation (1). Can be calculated. However, since there are two unknowns u and v in equation (1), the two-dimensional motion amount {u (x, y), v (x, y)} at a certain point (x, y) is For example, it can be solved simultaneously with a constraint equation for a point in the vicinity of the point (x, y) that can be regarded as having the same amount of movement. Thus, the lightness gradient method is a good method because the basic calculation is simple and does not require a search. However, as shown in the above description, the lightness change (lightness gradient) is constant and the lightness I ₁ and I ₂ are Equation (1) holds only when it is almost translated. Even if it can be considered that the brightness gradient of an actual image is locally constant in the brightness change, there is a limit to this, and Equation (1) can be applied only when the amount of motion is relatively small. Further, if the number of equations to be simultaneously set is small, that is, if the size of the image window for calculating the motion amount is small, the estimated value of the motion amount to be obtained becomes considerably unstable.

FIG. 11 shows whether or not the lightness gradient method can be successfully applied when the amount of motion is small and large. It can be seen that the amount of motion between the images I ₁ and I _{2 in} the case of FIG. 11A is small, and there are portions where the lightness curves are parallel to each other. In contrast, the amount of motion between the images I ₁ and I _{2 in} the case of FIG. 11B is large, and there are few portions where the lightness curves are parallel to each other. For this reason, in the case of FIG. 11B, even if the lightness gradient method is directly applied, the amount of motion between images cannot be obtained successfully.

  Therefore, in order to apply the lightness gradient method in the case of a large amount of motion as shown in FIG. 11 (b), the amount of motion is repeatedly calculated little by little using a pyramid image (multi-resolution image) or the like. Such a method is taken.

In the previous section, we explained how to determine the amount of movement of a point on an image from two images using the brightness gradient method. Next, how to obtain not only the movement amount of one point but also the deformation amount and the movement amount of an area having a certain spread or the entire image will be described. Basically, if you divide a wide area into individual pixels and small areas, you can calculate the amount of movement individually in the same way as explained in the previous section, but here the amount of deformation and movement of the entire area is compact. The story proceeds as what is required in a parametric form. The parametric form is obtained, for example, by modeling the amount of movement of each point with some motion model and obtaining the coefficient of the motion model (expressed as a function of some image position (x, y)). . Assume that the brightness of image 1 and image 2 is represented by I ₁ (x, y) and I ₂ (x, y), respectively. FIG. 12 shows an example in which the motion amount of the entire image from the image I ₁ (x, y) to the image I ₂ (x, y) is expressed parametrically. Here, the motion amount of each point of the image is a two-dimensional vector (called a motion vector), and is represented by affine transformation, which is one of geometric transformations. The calculation of the amount of motion between images is nothing other than finding the coefficients (a, b, c, d, e, f) of this affine transformation (see the following formula).

式(4)のアフィン変換のモデルを用いれば、画像の平行移動、拡大・縮小、回転及びくさび形変形の動きを表現することができる。簡単な平行移動だけの場合の時は平行移動のモデルの式(3)の形で良い。式(5)は３次元空間中にある平面が移動した場合( 図１３参照) に、撮像される画像上に引き起こされる移動量を近似したモデル(ここでは、移動３Ｄ平面モデルと表現する) である(より厳密な移動量は平面射影変換の変換式で計算することもできる)。対象とするシーンが遠景である場合や、近くでも建物や壁、黒板などの平面的でカメラの回転量が小さい場合は、この移動３Ｄ平面モデルで近似できる。

By using the affine transformation model of Equation (4), it is possible to represent the movement of the image in parallel movement, enlargement / reduction, rotation, and wedge-shaped deformation. In the case of only simple translation, the form of the translation model (3) may be used. Equation (5) is a model (here, expressed as a moving 3D plane model) that approximates the amount of movement caused on the captured image when a plane in the three-dimensional space moves (see FIG. 13). There is (more exact amount of movement can be calculated by the conversion formula of plane projective transformation). If the target scene is a distant view, or if it is a plane, such as a building, wall, or blackboard, and the amount of camera rotation is small, it can be approximated by this moving 3D plane model.

Next, a method for calculating the amount of deformation and the amount of motion of the image on the premise of these image motion models will be described. The translation model {Expression (3)}, the affine transformation model {Expression (4)}, and the moving 3D plane model {Expression (5)} can be expressed step by step more complicatedly. The calculation of the deformation amount and the motion amount of the image using the motion model is to calculate what the coefficients of a, b, c,. A method often used to calculate the coefficients a, b, c,..., H is a method of minimizing the square error of the constraint equation {Equation (1)} of the brightness gradient method in the target region R ( Least squares method). That is, finding a solution to the square error E ² represented by the following formula (6) to a minimum.

平行移動のモデルの場合は式(7)、アフィン変換のモデルの場合は式(8)、移動３Ｄ平面モデルの場合は下記式(10)に示す連立方程式を解けば、係数ａ，ｂ，ｃ，…，ｈを算出することができる。ここで式(7)から式(11)のΣは、対象とする領域Ｒでの加算を表している。

Coefficients a, b, and c are solved by solving the simultaneous equations shown in Equation (7) for a translation model, Equation (8) for an affine transformation model, and Equation (10) below for a moving 3D plane model. ,..., H can be calculated. Here, Σ in the equations (7) to (11) represents the addition in the target region R.

  Although not described in detail here, nonlinear optimization calculation may be necessary even if a model such as plane projective transformation that performs exact coordinate calculation or a model that assumes that the target is a curved surface may be required. It is possible to calculate the amount of movement. However, the scenes that are actually captured are complex and cannot be applied with simple planes or curved surfaces. In such a case, the amount of motion is calculated by dividing the image area within an approximate range. In the case of fine division, for example, even if it is extremely divided into areas of about one pixel, the amount of motion can be calculated in principle, but calculation is also necessary at each point, and a stable and accurate amount of motion can be calculated. Not necessarily. Usually, it is often calculated in an area of a certain size where the calculation of the movement amount is stable.

  As described in the previous section, the amount of movement of the entire image can be accurately obtained if the scene is a distant scene or a scene at the same distance or an object on the same plane. If the scene to be captured is complex, the image area can be divided into a plurality of small areas within a range that can be approximated, and the amount of motion can be calculated for each.

  However, how to divide actually becomes a problem. If the distance and shape to the target are known in advance, division that reflects this is possible, but usually it does not have such information. In addition, since there is a technique for restoring a three-dimensional shape from a moving image such as factorization method, it is conceivable to divide the image using such a technique. In order to restore the three-dimensional shape, it is necessary to measure the amount of motion of the image as preprocessing, and this is not an essential solution.

  For example, FIG. 14 shows a situation in which a distant object is captured together. This is a scene where trees are in the foreground and vehicles and mountains are in the distance. As shown in FIG. 14, when the camera moves to the left, the objects in the image as a whole move to the right. Certainly, the object in the image as a whole moves to the right, but the amount of movement differs depending on the content of the image, that is, the tree in front is large and the distant vehicle is small. When moving in this way, it is difficult to detect the exact amount of motion of the entire image without dividing the image into multiple regions (in other words, it is difficult to describe using a conventional motion amount model). ). For example, it is possible to determine the amount of motion using a moving 3D planar model {Equation (5)}, plane projective transformation, or the like, but it is not possible to detect an accurate amount of motion. However, in actual applications, the amount of motion is often calculated by forcibly processing to cause some problems. In real-time image stabilization and other applications, processing time becomes a problem, so complicated processing cannot be performed, and only a horizontal and vertical movement amount can be detected with a simple parallel movement model {Equation (3)}. Some devices can only correct the horizontal and vertical movement. In the case where objects in the distance as shown in FIG. 14 are mixed, a conventionally well-known method is to narrow down which side is of interest and stabilize the image of only the region of interest. .

  FIG. 15 shows a processing block for calculating a motion amount used for conventional real-time image stabilization. A motion amount calculation range setting means 1401 is provided before the motion amount calculation means 103 that compares the current input image 107 of the moving image with the reference image 108 to estimate the motion amount. A range (partial region) 1403 in which is calculated can be set. As a result, the processing range to be processed by the motion amount calculation means 103 is set by the motion amount calculation range setting means 1401, and only the motion amount in the range 1403 specified thereby is calculated. In this way, even if it is impossible to stabilize the entire image 1402, it is possible to stabilize the region of interest (need to be approximately modeled by a plane or the like).

  The above shows how to calculate the amount of deformation and the amount of motion of an image. Also, in the case of a complex background, a method has been shown in which a region of interest is designated and the amount of motion for only that portion is obtained. However, the target object does not exist at the same position in the actual image. If you want to process the recorded image once, you may be able to instruct which part to stabilize in advance, but depending on the change of the scene even if you think about stabilizing the image in real time etc. In fact, there are various problems such as it is impossible for a person to set the processing range of the motion amount calculation.

Therefore, the present invention intends to solve the problems listed below.
(1) It is difficult to appropriately input a region for calculating the amount of motion (region of interest) for an image. It is normal to focus on an area with a certain spread on the image, not on a point, and in the form of various areas. Even if a person specifies a simple rectangular window including the area, it is necessary to give the center position of the rectangular window and the vertical and horizontal sizes of the window. In addition, when a moving image is a target, the content of the image changes from moment to moment, and the position, size, and shape of the region to be noticed change accordingly. It is difficult for people to specify this one by one. If possible, it is ideal that the calculation range of the motion amount is automatically determined in such a way that detailed instructions are not necessary. For example, if you set a condition that you want to pay attention to a target at a certain distance, I would like you to automatically select and identify the area with the specified condition and perform processing such as image stabilization.
(2) Even if the region of interest is specified, unstable parts will enter.
In a moving image, a stationary object that has been included in the background until now may start to move. For example, even if the position and shape of the region of interest are set appropriately, if the object included in the target range suddenly moves, the correct amount of motion is calculated by calculating the amount of motion using a plane model that assumes a stationary background. It may not be possible. In addition, the object in the background of the image not only moves, but also the position of the shadow changes due to the deviation of the shooting time, or the instability that interferes with the motion detection of the whole image due to the difference in shooting environment and image sensor Some parts may get in.

  Other objects and novel features of the present invention will be clarified in embodiments described later.

In order to achieve the above object, the invention of claim 1 of the present application is an image motion detection device for detecting a target or background motion amount in an image.
A means for inputting information of a target portion for inputting or setting information of a target or background portion of interest;
Weighting refers to setting a weighting coefficient that changes according to an image position by associating the portion of interest with an image position. The information from the information input unit of the portion of interest is combined with the content of the image, A weighting setting unit that sets a weight for a part to be large and weights a part considered to be other than the part of interest to be small, and sets an initial value of motion amount calculation as necessary;
Using the weighting weighting coefficient set by the weighting setting means or the initial value of the motion amount calculation, weighting for estimating the motion amount of the image while paying attention to the portion of interest or excluding the portion of interest A motion amount calculation means that takes into account,
In the process of detecting the amount of motion of the image, the weighting setting unit and the motion amount calculation unit taking into account the weighting, etc., estimate the final amount of motion of the image by repeating at least one iteration. The weighting setting means has a function of adaptively adjusting the weighting by reflecting the motion amount estimation result of the image of the motion amount calculation means considering the weighting in the iterative processing, and taking the weighting into consideration The motion amount calculating means uses the weighting weighting coefficient adaptively adjusted by the weighting setting means in the iterative process.

  The image motion detection apparatus according to claim 2 of the present application is the image motion detection apparatus according to claim 1, wherein the information input means of the target portion is the image position, image range, distance, size of the target target or background portion. Deformation degree, two-dimensional shape, three-dimensional shape, texture, color, reflectance or transmittance, appearance, existence probability, vehicle type, model, category, number, speed, acceleration, angular velocity, movement information of image pickup device or It is characterized in that at least one or more pieces of information are input or set.

  An image motion detection apparatus according to a third aspect of the present invention provides a multi-resolution image from an input image for measuring a motion amount in order to more efficiently perform the iterative processing required in the first or second aspect of the invention. Is generated, a coarse / fine search is performed using the multi-resolution image, and the motion amount of the image is gradually estimated in the process of repetition.

  The image motion detection device according to the invention of claim 4 is the image motion detection device according to claim 1, 2 or 3, wherein the information input means of the part of interest uses information of the previous processing result in time series, It is characterized by setting the information of the part of interest.

  The image motion detection apparatus according to claim 5 of the present invention is the image motion detection device according to claim 1, 2, 3, or 4, wherein the weighting coefficient that changes the image position is related to the portion of interest in rectangular window units or arbitrary It is realized by processing in the form of handling with the same weighting factor in the area unit of the shape, or by processing equivalent to selecting the weighting factor as 0 or 1 in the rectangular window unit or the area unit of any shape It is characterized by that.

  An image motion detection apparatus according to claim 6 of the present application is the image motion detection device according to claim 1, wherein the weighting coefficient associates a portion of interest with the image position and changes according to the image position. An image change is a change that occurs due to differences in the shooting environment and the image pickup device between the images for which the amount of motion of the image is measured, and the change in the image that becomes an obstacle to measuring the amount of motion between images. The feature is that the weighting factor is adaptively adjusted so as not to pay attention to a large portion.

  The image motion detection apparatus according to claim 7 of the present application calculates a correspondence between images acquired by different types of image sensors such as a visible image and an infrared image, a near-infrared image, and a far-infrared image in claim 6. It is characterized by that.

  First, how the basic invention of claim 1 works will be described. Consider a case in which distant objects are mixed in the image, and for example, the amount of motion is extracted by paying attention to the distant object.

Such a situation is shown in FIG. It is assumed that the area A and the area B are shown next to each other on the image I ₁ captured at the time t ₁ , the object of the area A of interest is far from the camera, and the object of the area B is near the camera. Shows an example of a brightness change of the image I ₁ is a diagram 16 (a). In this figure, for the sake of simplicity, the change in brightness is shown in one dimension (x-axis). FIG. 16B below shows an example of brightness change of the image I ₂ captured at time t ₂ . In the image I ₂ , the area A and the area B are both shifted in the right hand direction due to the camera moving in the left hand direction. However, the amount of each movement is small because the object in the region A is far from the camera, and the object in the region B is large because the object is near the camera. The region generated between the regions A and B is a shaded area that has not been hidden in the image I ₁ in the region B point in time t _1.

The image motion detection apparatus according to the first aspect of the present invention is based on the premise that the motion amount is calculated stepwise by at least one or more iterations. For this reason, it is assumed here that a provisional value obtained by estimating the amount of motion in the region A is obtained by some clue, initial setting, or iterative processing. The provisional value of the amount of movement is a. In order to obtain a more accurate motion amount, it is necessary to calculate the remaining motion amount (motion amount correction value). FIG. 16C shows a state in which the image I ₂ is returned to the left by the provisional value a of the amount of motion and is superimposed on the image I ₁ . As can be seen from FIG. 16 (c), since the image is returned in accordance with the movement of the area A, there is little deviation in brightness in the area A, and I ₁ (x) and the image I ₂ are converted into a provisional value a for the amount of movement. As a result, the brightness of I ₂ (x + a) returned to the left is almost overlapped. In such a portion (represented here as a matching region), the remaining amount of motion can be calculated almost correctly by the lightness gradient method. In addition, the correct amount of motion cannot be obtained by the lightness gradient method in other regions B and hidden portions (herein expressed as non-matching regions). Of course, there is no problem if the positions and shapes of the areas A and B are designated from the beginning, but it is assumed here that the area information is not given clearly. Therefore, if the motion amount as a whole (region A and region B) is to be obtained, not only the correct motion amount cannot be calculated for the region B or the hidden portion, but also when the motion amount of the region A is obtained. Cause an error. However, when the amount of motion is calculated by the least square method or the like, the error in the matching region such as the region A is greatly evaluated, and the error is not much in the non-matching region such as the other region B or the hidden portion. If it is solved in such a way that it is not greatly evaluated, only the amount of movement focused on the area A can be calculated. Here, from the relationship of the coefficients in the error evaluation formula, it is important to emphasize the movement and evaluate the error greatly (so that the error is reduced as a result). The evaluation of the error being small is expressed by the expression “small weight coefficient or small weight”. This weighting factor (weighting) must be determined by some method. For example, it is determined by evaluating the lightness difference between the image I ₁ (x) and the image I ₂ (x + a) and the degree of coincidence of the gradients. Can do.

  The flowchart shown in FIG. 17 shows a process flow in which the weighting coefficient is adaptively changed by matching the images in the process of repeatedly processing the weighting coefficient. As described above, when position information or the like of a part of interest is first given, the weighting coefficient is automatically determined in accordance with the contents of the image. If the weighting factor is determined appropriately, the amount of movement of the region of interest can also be extracted appropriately. The first aspect of the present invention shows how to implement the processing flow specifically with each means required.

  In the image motion detection apparatus according to the first aspect of the present invention, first, information on a clue (image position, image range) that can be linked to position information on a first target region by means of inputting information on the target region. , Distance, size, degree of deformation (tolerance of variation with respect to the reference image), 2D shape, 3D shape, texture (pattern), color, reflectance or transmittance, appearance (how 3D objects look ), Existence probability (probability that a specific target exists), vehicle type, type (classification that is finer than the vehicle type), category (for example, whether the target is a person, vehicle, tree, etc.), number, speed, acceleration , Information on the angular velocity, movement of the image pickup device (a three-dimensional camera movement amount obtained from a gyro attached to the camera or the like) or prediction information thereof). In addition, the weighting setting means adds the information and the contents of the image from the information input means for the part of interest, increases the weight of the part of interest, and reduces the weight of the part considered to be other than the part of interest. The initial value of motion calculation can be set as necessary. Then, in the motion calculation means considering weighting or the like, the initial position value of the weighting or motion calculation set by the weighting etc. setting means is used to pay attention to the part of interest or to exclude the part of interest and change the position of the image. It is possible to estimate. Also, the weighting setting means and the motion calculation means considering weighting, etc. are configured to estimate the final image position change by repeating at least one iteration. The weighting setting means has a function of adaptively adjusting the weighting by reflecting the result of the motion calculation means considering the weighting in the iterative processing, and the iterative processing in the motion calculation means considering the weighting etc. The weighting adaptively adjusted by the weighting setting means can be used. Accordingly, it is possible to appropriately extract the amount of motion of only the region of interest in the form of the iterative process shown in FIG.

  According to a second aspect of the present invention, there is provided an image motion detection apparatus comprising: information on a clue that can be linked to position information of a first target region in the first aspect of the invention, the image position, the image range, the distance, the size, Deformation degree, two-dimensional shape, three-dimensional shape, texture, color, reflectance or transmittance, appearance, existence probability, vehicle type, model, category, number, speed, acceleration, angular velocity, movement information of image pickup device or those The prediction information is specifically limited.

  According to a third aspect of the present invention, in the iterative processing of the first aspect of the invention, the image motion detection apparatus performs processing in the form of a coarse / fine search using a multi-resolution image. If the processing is performed in the form of a coarse / fine search, it is possible to estimate the change in the position of the image gradually in the process of the iteration, and thus it is possible to estimate the amount of motion very efficiently.

  According to a fourth aspect of the present invention, there is provided an image motion detection apparatus that automatically sets information of a portion of interest by using information of a previous processing result in time series as an input means of information of a portion of interest. In this way, it is possible to save the trouble of inputting information of a portion of interest every time.

  The image motion detection apparatus according to the invention described in claim 5 is the same as the operation of setting the weighting coefficient that changes depending on the image position, in which the portion of interest is associated with the image position, in units of rectangular windows or regions of arbitrary shapes. Thus, the same effects as those of the inventions according to claims 1 to 3 can be obtained.

  An image motion detection apparatus according to a sixth aspect of the present invention is a shooting environment (weather, shooting) between images to be measured for the amount of motion of an image, which is an obstacle to measuring the amount of motion between images. Since the weight coefficient is adaptively adjusted so as not to pay attention to a portion where the change in the image resulting from the difference in the type of image sensor is large, the amount of motion between images can be measured correctly.

  According to a seventh aspect of the present invention, there is provided an image motion detection apparatus according to the sixth aspect of the invention, in particular, associating images obtained by different types of image sensors such as a visible image and an infrared image, and a near-infrared and far-infrared image sensor. The weighting factor is adaptively adjusted so that even if there is a change in the way the image is captured between images, the part where the change in the image is large is not noticed. It can be measured correctly.

  The present invention has the following effects.

(1) Easily set the area for calculating the amount of motion (the area of interest) in the image.
By using the image motion detection device according to the first, second, and third aspects of the invention, the range of the target region can be extracted by adaptive weighting from the information that is the first clue. This eliminates the need for detailed instructions on the region of interest and the object, and can reduce the labor of the operator when stabilizing a real-time moving image, which is very effective. Further, if the invention according to claim 4 is used, there is no need to always indicate where the region of interest in the moving image is, and the portion of interest of the image that automatically follows only with the information of the region of interest given first is Since the contents are updated somewhere, the operation of the apparatus becomes much easier.

(2) The error part is eliminated, and the stability of the motion estimation is improved.
Even if a moving object that deviates from the movement of the moving 3D plane assumed in the attention area is generated, an inconsistent area is excluded by weighting in the iterative processing, and as a result, the amount of movement of the attention area Can be estimated accurately and stably.

(3) Information such as three-dimensional information can be extracted from the image.
If there is a moving object that deviates from the movement of the moving 3D plane assumed in the region of interest, the weight of the moving object part becomes small. On the contrary, it is also possible to automatically detect a moving object that moves other than assumed by using the weighting information. In addition, by applying the present invention, it is possible to determine whether or not the designated portion can be approximated in a planar manner from the consistency of the image. Therefore, when restoring the three-dimensional information of the scene, which portion is planar approximated Important information such as whether it can be obtained.

(4) Images with different properties obtained from different types of image sensors can be aligned.
Visible and infrared are completely different in appearance, but parts with different appearance can be excluded by reducing the weighting factor, and attention can be paid to edges such as the contour of the object and the overall correspondence. (Adjustment) can be accurately performed (if the images of different types of image sensors can be correlated, it will be possible to create a single image containing various information by merging, or to classify objects by spectral analysis, etc. Become.).

  DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of an image motion detection device will be described with reference to the drawings as the best mode for carrying out the present invention.

  Corresponding to the description in the prior art, here also an explanation will be given focusing on an example realized by the brightness gradient method. Further, in the following description, in many cases, it is expressed in the form of weighting in each pixel, but it includes processing equivalent to the weighting. For example, the processing that uses a specific weight for the entire processing window of a certain rectangle, or an extreme story, the processing window is selected if the weight is controlled by 0 and 1, and the weighting at each pixel is a difference in expression. Even if there is, it is essentially the same thing. Where the processing is not affected, it is not necessary to calculate the weight for each pixel of the image. In addition, for simplification of processing, there is a case where an average consistency of an image is calculated over a certain rectangular window area without using a weighting coefficient for each pixel, and one weighting coefficient is used for the area. . For example, the motion amount may be calculated individually in individual window areas as shown in FIG. 18, and one weighting coefficient may be used. FIG. 18 shows an example in which one image composed of 320 horizontal pixels × 240 vertical pixels is divided into 25 window regions of 5 × 5. In the following description, the divided window area may be expressed as a block. Therefore, the invention described in claim 5 shows that weighting can be performed in units of rectangular windows or regions of arbitrary shapes, and the embodiment of the invention described in claim 5 is claimed in the claims. In the description of the embodiments of the invention described in 1-4, it will be described as appropriate.

(Embodiment 1)
A processing block diagram of the first embodiment corresponding to the first aspect of the present invention is shown in FIG. In Figure 1, is input to the current input image 107 and the reference motion amount calculation unit 1801 image 108 and is considering weighting for a moving image, the motion amount calculation unit 1801 in consideration of weighting like the reference image I ₁ and the input image Using I ₂ as an input, the amount of motion of the part of interest is estimated. The motion amount calculation means 1801 in consideration of the weighting is for obtaining the motion amount by at least one or more repeated processes, and in the repeated processes, the part of interest has a large weight and is considered to be other than the part of interest. Information 1804 such as a weighted weighting factor and weighting of an initial value of motion amount calculation, if necessary, is given from the weighting setting means 1802 so as to reduce the weight of the portion. The weighting setting means 1802 also performs at least one repetition process in conjunction with the motion amount calculation means 1801 taking weighting into account. In this weighting setting means 1802, information 1805 of the target or background to be noticed from the information input means 1803 of the part of interest is combined with information 1806 such as the intermediate processing result of the motion amount calculation means 1801 in consideration of weighting, The portion to be noticed (including the target position 1807 to be noticed) is weighted so that the weight is large, and the weight of the portion considered other than the portion to be noticed is small. Also, the weighting setting unit 1802 sets an initial value for the motion amount calculation as necessary. In the information input means 1803 for the part of interest, information 1805 such as the target or background to be given to the weighting etc. setting means 1802 can be input or set.

  FIG. 2 shows a detailed implementation example of the motion amount calculation means 1801 in consideration of weighting. Prior to the detailed description of FIG. 2, a method for calculating the amount of motion of an image in consideration of a weighting factor at each point will be described. The weighting coefficient of each point is represented by S (x, y). This weighting factor S (x, y) will be described later as to how it will be obtained, but here it is assumed that it has already been obtained.

The following equation (12) intended to correspond to formula (6) evaluation formula ordinary least squares method, and to consider when to aggregate squared error E ^2, the weighting factor S a (x, y) It is an evaluation formula.

例えば、重み係数Ｓ(x,y)を考慮し最小二乗法による平行移動モデルでの動き量を計算する連立方程式は以下の式(13) の形となる。

For example, simultaneous equations that calculate the amount of motion in the translation model based on the least square method in consideration of the weight coefficient S (x, y) have the form of the following equation (13).

他の動きのモデル(アフィンモデル、移動３Ｄ平面モデル)の場合でも、それぞれ対応する、式(8)〜式(11) のΣの後に重み係数Ｓ(x,y)を挿入した式になるので、その重みを考慮した連立方程式を解けば良い。

Even in the case of other motion models (affine model, moving 3D plane model), the weight coefficient S (x, y) is inserted after Σ in the corresponding equations (8) to (11). It is sufficient to solve the simultaneous equations in consideration of the weights.

With reference to FIG. 2, the overall motion of the motion amount calculation means 1801 considering weighting will be described. First, the two target images are the reference image 108 (I ₁ ) and the input image 107 (I ₂ ). Processing is started with these images as inputs. First, the initial value 1908 of the motion amount (estimated value) given from the weighting setting means 1802 (the initial value is set to zero if there is no appropriate predicted value), and the input image I ₂ is the image warp unit 105. And warped image I ₂ ^W 1906 is generated. The warp at this time is exactly the reverse warp, and the input image I ₂ is deformed and returned to the reference image I ₁ . The image gradient calculation unit 1902 calculates the gradient I _{x in} the x-axis direction and the gradient I _y in the y-axis direction of the reference image I ₁ using a difference operator as shown in FIG. Further, the brightness difference calculation section 1901 calculates a brightness difference _{I t} of the reference image _{I 1} and the input image _I ^{2 W} was warped. At this point, the image coordinates (x, y), the brightness gradient is uniform (I _x, I _y), brightness difference I _t. In a summation unit 1903 of flow feature quantities (here, values such as I _x ² , xI _x ² , yI _x ^2, etc.) are shown, a weighting factor S (x, 1907) given from the weighting setting means 1802. y), and the image coordinates (x, y), the brightness gradient (I _x, I _y), using the brightness difference I _t, the ΣS made to the coefficients of the simultaneous equations (x, y) flow weighted such I _x ² Accumulate feature values. Here, in the form such as a raster scan, the image coordinates of each point (x, y), the weighting factor S (x, y), the brightness gradient (I _x, I _y), only to calculate the brightness difference I _t For example, the flow feature amount can be calculated and accumulated in the same raster scanning form, which is efficient.

  Next, the motion amount calculation unit 1904 calculates the motion amount by solving the simultaneous equations with the flow feature amount accumulated in the entire image as an input. After several iterations, the image warp unit 105 warps the input image with the motion amount calculation value, and the motion amount calculation unit 1904 obtains the remaining motion amount (motion amount correction). Value). The motion amount addition unit 1905 adds the motion amount correction values obtained by the motion amount calculation unit 1904 (for example, (Δa,..., Δh) in the moving 3D plane model) to obtain a provisional value of a new motion amount. After several iterations, the provisional value of this amount of motion is output as an estimated amount of motion 1909 {eg (a, b, c, d, e, f, h)} for a moving 3D plane model}. The

  On the weighting setting means 1802 side, the information 1806 such as the intermediate processing result (reference image, warp image, etc.) of the motion amount calculation means 1801 in consideration of this weighting is reflected in the evaluation value calculation and repeated. In the process, it is used to adaptively modify the weighting factor S (x, y). The evaluation value referred to here is an evaluation value for evaluating whether or not it is consistent with the assumed motion model of the region of interest.

  The information input means 1803 for the part of interest only needs to be able to input information of a target to be noticed. For example, when specifying the position, the operator can specify the position with a pointing device such as a mouse or a cursor. Means for receiving such target position information from another target tracking device may be provided. Further, it may be calculated where the operator's line of sight is detected and where attention is paid. If the color of the object or background of interest is known, the position of the target or background can be specified simply by specifying the color. Furthermore, if the amount of movement of the image pickup device (camera) itself can be obtained from a gyro sensor or the like, it is possible to specify in which direction and position the part of interest in the image has shifted. As described above, the information related to the target target or background position information or usable as a clue is the distance, size, degree of deformation, two-dimensional shape, 3 Dimensional shape, texture, color, reflectance or transmittance, appearance, existence probability, vehicle type, model, category, number, speed, acceleration, angular velocity, movement information of the image pickup apparatus, prediction information thereof, and the like.

  In the setting means 1802 for weighting or the like, the information given from the information input means 1803 for the part of interest is combined with the contents of the image, the part of interest has a large weight, and the weight of the part considered other than the part of interest is Weighting is performed to make it smaller, and an initial value for motion amount calculation is set as necessary.

A specific method for realizing the setting unit 1802 for weighting and the like will be described with an example. There are various methods for associating information such as the target or background of interest with the weighting coefficient S (x, y), which is a function of the image position (x, y). here,
(1) When the image position that is the center of the target area is specified,
(2) A case where attention is paid to an area as far away as possible will be described as an example.

For example, in the case of (1), if the center of the target region is (x _p , y _p ), the degree of proximity from the center (x _p , y _p ) of the region is expressed by the following equation (16). Can be evaluated.

式(16)では、この近接度をＰ(x,y)で表している。

In the equation (16), this proximity is represented by P (x, y).

Also, as in the case of (2), if attention is paid to a far target, a block unit having a shape as shown in FIG. 18 (for example, the original image having 320 horizontal pixels and 240 vertical pixels is represented by 64 pixels. The motion amount as the translation model equation (3) is calculated in advance in ₂₅ blocks B _{1 to} B ₂₅ of pixels × 48 pixels), and the translation amount {a (x, y), d (x, y)}. Here, the translation amount {a (x, y), d (x, y)} is obtained by using the translation amount (a _i , d _i ) of a certain block B _i obtained in block units and the translation amount of a block in the vicinity thereof. This is assumed to be obtained by interpolation. Using this, it is possible to calculate the evaluation value of the smallness of motion (probability of being far away) Q (x, y) in the form as in equation (17) (the evaluation of being far away is the translation amount In fact, it is more reliable if the rotation component received by the camera rotation is corrected by the previous iteration.) As an evaluation value other than this, the degree of matching M (x, y) after warp is important. This can be obtained by evaluating the lightness difference before and after the warp and the similarity of the gradient. Equation (14) is to calculate the evaluation value of the matching degree from the brightness difference I _t ^w after warped and warped previous brightness difference I _t. Furthermore, with the evaluation value of only the expression (14), the reliability is low in a place where the image brightness with few image features is flat, so that the weight coefficient is increased only when a certain amount of image features exist, the expression ( An evaluation value J (x, y) for evaluating the image feature strength as shown in 15) is also required. These evaluation values are calculated at each point of the image position (x, y), but may be similarly calculated for each block.

  The following formulas (18) to (21) correspond to the above formulas (14) to (17) and show examples of calculation formulas for obtaining an evaluation value for each block.

なお、これらの式の中のｋ_１〜ｋ_４ は各評価値を適切に調整する係数である。ブロック毎にワープ後の整合度の評価値を計算する場合、勾配法の拘束式に明度の変動α(ブロック内で定数とする)を導入して下記の式(22)で示されるような２乗誤差Ｅ^２を最小化するようにして解（ａ,ｄ,α）を算出し、この求まったαを利用して評価値を計算することもできる。

Note that k ₁ to k ₄ in these equations are coefficients for appropriately adjusting each evaluation value. When calculating the evaluation value of the degree of consistency after warp for each block, introduce a variation α of brightness (constant in the block) into the gradient method constraint formula, and 2 as shown in the following formula (22) The solution (a, d, α) can be calculated so as to minimize the multiplication error E ² , and the evaluation value can be calculated using the obtained α.

明度の変動αを考慮したときの動き量を算出する連立方程式は、式(23)の形になる。また、検出した明度の変動αで整合度の評価値を算出する式の例を式(24)に示す。ここでσは、変動αのブレの大きさを調べ調整する係数である。連立方程式を変えたくなければ、ブロックの整合度は対象とするブロックで計算される動き量の大きさを利用しても良い。一般に、明度勾配法を適用できないところで明度勾配法で動き量を算出すると大きな動き量が算出される傾向がある。

The simultaneous equations for calculating the amount of motion when the brightness variation α is taken into account are in the form of equation (23). Further, an example of an expression for calculating the evaluation value of the degree of matching with the detected brightness variation α is shown in Expression (24). Here, σ is a coefficient for investigating and adjusting the magnitude of fluctuation α. If it is not desired to change the simultaneous equations, the degree of motion calculated by the target block may be used as the degree of matching of the block. In general, when a motion amount is calculated by the lightness gradient method where the lightness gradient method cannot be applied, a large amount of motion tends to be calculated.

Next, an example of calculating the weighting coefficient S (x, y) from these evaluation values is shown. Equations (25) and (26) below represent the weighting factor S () expressed by the image position (x, y) and the block B _i when the position of interest on the image (such as the center of the region) is designated. x, y), it is an example of a formula for S _i.

Ｊ^ｌ(x,y)等の上に付く添え字のｌは、各反復処理又は、ピラミッドレベルを識別する番号である。ここでのΣは、これまでの反復処理で計算された値を集積する意味で使用している。また、なるべく遠方の領域等に着目したい時には、式(27)及び式(28)のような式で重み係数を計算すればよい。ここで示すのは代表的な例に過ぎない。色々な方法でまた目的にあった評価値を導入できるし、評価式や重み係数も色々な形で表現できるし、計算しやすいように変形して利用することができる。

The subscript l attached on J ^l (x, y) or the like is a number for identifying each iteration or pyramid level. Here, Σ is used in the sense of accumulating the values calculated in the iterative processing so far. Further, when it is desired to pay attention to a distant area as much as possible, the weighting coefficient may be calculated using equations such as equations (27) and (28). This is only a representative example. Evaluation values suitable for various purposes can be introduced in various ways, evaluation formulas and weighting coefficients can be expressed in various forms, and can be used by being modified so that they can be easily calculated.

(Embodiment 2)
An embodiment for performing a coarse / fine search using a multi-resolution image (pyramid image) according to the invention described in claim 3 will be described. FIG. 3 shows an example in which the motion amount calculation means 2601 considering weighting using a pyramid image is taken into account. In FIG. 3, the feature is that a pyramid image generation means 2602 and a pyramid level selection means 2603 are provided in the preceding stage of the motion amount calculation means 1801 in consideration of weighting and the like, and the other parts are the same as in FIG. The same reference numerals are given to the same or corresponding parts.

  An example of the pyramid image 2604 generated by the pyramid image generation means 2602 will be described with reference to FIG. At the bottom of FIG. 4 is a normal image, here an image of 320 pixels in the horizontal direction × 240 pixels in the vertical direction. This original starting image level (pyramid level) is represented by L0. To put it simply, the image in which the vertical and horizontal directions of this image are successively reduced by ½ is an image of pyramid levels L1, L2, L3,. On the area, the area of the image becomes smaller by 1/4. In FIG. 4, the L0 original image is at the bottom, the L1 image (horizontal direction 160 pixels × vertical direction 120 pixels), the L2 image (horizontal direction 80 pixels × vertical direction 60 pixels), and the L3 image ( 40 pixels in the horizontal direction × 30 pixels in the vertical direction) and vertically above.

  Once the input image is converted into such a pyramid image, and the motion amount is estimated while gradually estimating the motion amount from the upper image (high image with high pyramid level), a large motion amount can be obtained. In addition to being able to catch, the total amount of calculation in the iterative process can be reduced.

  In the above explanation, when generating the pyramid image 2604, it is expressed that the vertical and horizontal directions are reduced by 1/2 each time. However, when thinning out simple pixel data, aliasing noise (for example, a jagged line appears) occurs. Therefore, pixel thinning is performed while removing noise with a Gaussian filter or the like. An implementation example of the pyramid image generation means 2602 is shown in FIG. The selection circuit 2801 operates so that the input from the input terminal B is selected after the normal image L0 first enters from the input terminal A. Therefore, the image L0 passes through the selection circuit 2801 and is output as an image of L0 at the same time. At the same time, the image L0 is input to the Gaussian filter 2802. Then, the image L0 passes through the Gaussian filter 2802 and the thinning-out processing unit 2803 and becomes an L1 image. After the output timing is adjusted by the image buffer 106, the image L0 is selected as the input of the input terminal B of the selection circuit 2801, and Output as an image. By repeating such an operation, it is possible to generate higher order smaller pyramid images sequentially as L0, L1,.

  The pyramid level selection 2603 in FIG. 3 serves to provide a low-order pyramid image as an input to the motion amount calculation means 1801 in consideration of weighting or the like in order from the high-order pyramid image in the motion amount calculation repetitive processing. You can repeat the same pyramid several times and skip the pyramid level if necessary. Depending on the content of the image and the purpose of application, there may be a case where the estimated value of the motion amount with sufficient accuracy can be obtained without processing with the lowest pyramid image.

  FIG. 6 shows a state in which the amount of motion is estimated using the pyramid image according to the second embodiment and sequentially using a coarse / fine search method in an iterative process, and is the same as or equivalent to FIG. A reference is attached.

Using this figure, it is the effect of the present invention,
(1) Easier to specify the area of interest
(2) Explain that it is possible to narrow down the estimation of the amount of motion by excluding those that move differently than expected.

Here, for the sake of simplicity, a case will be described where processing is performed once each from pyramid levels 2 to 0 to estimate the amount of motion of the image. It is assumed that the character A is shown as the reference image I ₁ and another circular object is covered in front of it, and that the character A is rotated and moved in the upper left direction as the input image I ₂ . Further, the region of interest here is the region of image A, and it is assumed that the image position of the + mark is designated in the form of a target that focuses on a part of character A as shown in the figure (the entire character A Note that no shape is specified). After the pyramid image of the reference image and the input image is generated, this processing flow is started. The pyramid level 2 input image I ₂ (L2) (here, (LN) represents the image of the pyramid level N of the image) is image-warped with the initial value given by the weighting setting means 1802, and 1906 Using the warped image I ₂ ^w (L2) and the reference image I ₁ (L2), the motion calculation unit 2901 of the pyramid level 2 calculates a temporary motion amount (motion amount correction value). Here, it is assumed that the motion calculation units 2901 to 2903 include portions for calculating the amount of motion by performing lightness gradient calculation, lightness difference calculation, flow feature accumulation, and the like of the motion calculation means in consideration of weighting and the like. . The initial weighting used here is based on the information on the part of interest (information from the information input unit on the part of interest) so that the weighted setting unit 1802 makes the + mark part high and the other parts low. Is set. Therefore, the motion correction value calculated here extracts the motion amount centered on the character A.

In the next pyramid level 1 processing, the image is changed to pyramid level 1 for processing. At this time, image warping is performed in a manner approaching the A image warping section more reference image I ₁ in 105. That is, this is because the motion amount centered on the character A is calculated in the level 2 processing in the previous stage. At this time, when the warp image I ₂ ^w (L1) and the reference image I ₁ (L1) are overlapped, the evaluation of the consistency of the brightness of the circular area portion becomes low. The circular area is given weighting so that the weighting factor is low. As a result, the motion amount calculation unit 2902 of the pyramid level 1 excludes the motion of the circular area portion, and estimates the remaining motion amount of only the portion that can be regarded as the character A portion.

  The pyramid level 0 continues to operate in the same manner as the pyramid level 1. Therefore, the circular area portion has a lower weight coefficient, and the movement amount calculation unit 2903 of the pyramid level 0 only performs the remaining movement of the letter A portion. The amount will be estimated more accurately. As a result, the finally estimated amount of motion is the amount of motion focusing on the character A portion.

  In this way, although the image position indicated by the + mark of a part of the first character A is specified, the weighting factor is adaptively adjusted with reference to the first clue, and finally It automatically finds where the area belonging to the letter A is. Thereby, it is not necessary to accurately trace and input the entire shape of the character A, and it becomes easy to specify a region to be markedly focused. In addition, as described above, due to the weight coefficient that is sequentially updated and adaptively changes, the portion of the circular region that is considered to perform another motion is substantially excluded from the motion amount estimation, and the motion amount estimation for the letter A is performed. Also accurate and stable.

(Embodiment 3)
An embodiment corresponding to the invention of claim 4 for efficiently setting information of a part of interest by using information of previous processing results in time series will be described. FIG. 7 shows the flow of processing when the invention of claim 4 is carried out. In FIG. 7, the information input means for the part of interest is represented by the expression 3001 or 3002 for setting the information of the part of interest. Also, the weighting setting means 1802 and the motion amount calculation means 1801 taking weighting into consideration are collectively shown in one processing block for the sake of simplicity, and the same reference numerals are assigned to the same or corresponding parts as in FIG. Here, a description will be given with respect to an image I ₁ , an image I ₂ , an image I ₃ ,... Sequentially input in time series, and a process for stabilizing the image.

First, the setting 3001 information of the target portion at the top of FIG. 7, once on the basis of the image I _1, the information region of interest is set. Based on it, in which an image I ₂ and subsequent steps are stabilized. The stabilized image is denoted by I _i ^* . And region of interest that is set to the image I ₁ on the reference information, the motion amount of the target site from the image I ₁ and the image I ₂ is calculated by the motion amount calculation unit 1801 in consideration of weighting like. The motion amount and the filtering process with the motion amount of filtering 104, the actual amount of correction is determined, stabilized image I ₂ ^* is generated from the image I ₂ by the processing of the image warping section 105. This image I ₂ ^* becomes a reference image for the next processing, and is used for calculating the amount of motion of the image I ₃ . Setting 3002 information of the portion of interest, the image I ₂ ^* and information of the target portion of the motion amount calculation image I ₃ but is set, there is where the focus portion of the image I ₂ ^* image I ₁ and the information used for calculating the amount of motion of image I ₂ (including information on the region of interest specified first) and the correction amount used when the stabilized image I ₂ ^* was generated Can do.

  In this way, information can be handed over in sequence where the target part is. Even if the shape of the part of interest is changed little by little, the weighting factor is automatically adjusted according to the image successively, so that the amount of motion can be calculated continuously. In addition, if the information of the previous frame is used, even if the target is temporarily hidden behind the obstacle in front, it is possible to proceed with the assumption that the target is hidden behind by setting the information of the part of interest It is. In this way, there is no need to always indicate where the part of interest is, and the contents of the part of interest in the image that automatically follows are automatically updated with only the information of the first part of interest, so that the operation of the device is greatly improved. Becomes easier.

  To implement the invention described in claim 6, the shooting environment (weather, the type of image sensor to be shot) between the images to be measured for the amount of motion of the image, which is an obstacle to measuring the amount of motion between the images. The weighting factor may be adjusted adaptively so as not to pay attention to a portion with a large change in the image resulting from the difference in the above.

  In general, it is difficult to calculate the amount of motion between images obtained by different types of image sensors such as a visible image and an infrared image. This is because even if the same object is photographed, the visible way is different between visible and infrared. However, it is possible to associate images from the edges such as the contour of the object and the overall correspondence. To implement the invention according to claim 7, for example, in the block diagram (FIG. 1) of the first embodiment, the image acquired by the visible image sensor can be used as the reference image, and the image acquired by the infrared image sensor can be used as the input image. (Here, which is the reference image and which is the input image is not an essential problem.) Then, even if there is a difference in how the images are captured between the images, the weighting factor is adaptively adjusted so as not to pay attention to the part where the difference between the images is large. It can be measured. At this time, since there is a problem that the brightness level is different between the visible image and the infrared image, it is necessary to perform some conversion to convert the brightness level between images and to make an image (edge image etc.) easy to collate. It is effective.

Even when the invention of this patent is implemented other than when the invention according to claim 7 is implemented, it is not necessarily the original image that is used as the input of the lightness gradient method. An image of the magnitude of the gradient calculated by a Sobel operator or LoG
A Laplacian image extracted by a filter such as a (Laplacian of Gaussian) filter may be used. When the lighting conditions etc. change significantly and the brightness of the object changes significantly, convert the relative brightness change into an image with information, and then calculate the amount of deformation or movement of the image Can be processed stably. Further, in the embodiment, the example of the monochrome image has been described, but naturally, it can be easily extended to a multidimensional image such as a color image or a multispectral image. Even when two or more images are used simultaneously to calculate the amount of motion of the image, the processing framework of the present invention is the same and can be easily applied. Although the lightness gradient method has been described as an example for calculating the amount of motion, the present invention can naturally be applied to the correspondence search method and other methods for analysis in the frequency domain. For example, in correspondence search processing for detecting corresponding points in stereo image processing or the like, a matching degree of a region that may be supported is examined rather than solved by simultaneous equations. The degree of matching is calculated by using cross-correlation between regions, sum of SAD (Sum of Squared Differences), SSAD (Sum of SADs), or the like. What is necessary is just to calculate in consideration of weighting when calculating this cross-correlation or SSAD. Further, the calculation is performed by adaptively changing the weighting so as to exclude a portion that is clearly inconsistent in the matching window (for example, a portion that has changed due to occlusion or the like in the stereo image). At this time, the ratio of the part to be excluded changes depending on the position to check the correspondence, but if there are many parts to be excluded, the reliability as a whole becomes low, so if the degree of matching is corrected considering that , More accurate processing.

  Although the embodiments of the present invention have been described above, it will be obvious to those skilled in the art that the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the claims.

It is a processing block diagram of Embodiment 1 corresponding to the invention of Claim 1. 6 is a processing block diagram illustrating an implementation example of motion calculation means in consideration of weighting and the like according to Embodiment 1. FIG. It is Embodiment 2 corresponding to invention of Claim 3, Comprising: It is a processing block diagram which calculates a motion amount using a pyramid image as a multi-resolution image. It is explanatory drawing which shows the example of a pyramid image. It is a processing block diagram which shows the implementation example of a pyramid image generation means. It is a flowchart which shows the procedure which calculates a motion amount hierarchically using a pyramid image. It is Embodiment 3 corresponding to invention of Claim 4, Comprising: It is a block diagram in the case of processing efficiently in time series using the processing result of the previous frame. It is a block diagram of the example of a process which stabilizes a moving image. It is explanatory drawing explaining the principle of the lightness gradient method in one dimension. It is explanatory drawing which shows the example of the calculation mask which calculates the gradient of an image. It is explanatory drawing which shows that the gradient method is applicable when there is little motion amount. It is explanatory drawing which shows the example which expresses the motion amount between two images with an affine coefficient. It is explanatory drawing which shows the plane in the three-dimensional space used for description. It is explanatory drawing which shows the case where a problem occurs in simple motion detection (when near objects are mixed). It is a processing block diagram of the motion calculation of the conventional system which designates the range to process beforehand. It is explanatory drawing which shows why it is necessary to attach a weight. It is explanatory drawing of adjusting a weighting coefficient adaptively. It is explanatory drawing of the example of a division | segmentation of an image when processing by an individual window area | region (block).

Explanation of symbols

101 Video (input)
DESCRIPTION OF SYMBOLS 102 Stabilized moving image 103 Motion amount calculation means 104 Motion amount filtering 105 Image warp unit 106 Image buffer 107 Input image 108 Reference image 1401 Motion amount calculation range setting means 1402 Whole image 1403 Range where motion amount is calculated 1801 Weighting 1802 Weighting setting unit 1803 Information input unit of interest 1804 Information of weighting 1805 Information of target or background 1806 Information of intermediate processing result 1807 Position of target 1901 Lightness difference calculation unit 1902 Image gradient calculation unit 1903 Flow feature amount integration unit 1904 Motion amount calculation unit (solves simultaneous equations)
1905 Motion amount addition unit 1906 Warp image 1907 Weighting coefficient 1908 Initial value of motion amount (estimated value) 1909 Estimated value of motion 2601 Motion amount calculating means 2602 considering weighting using a pyramid image 2602 Pyramid image generating means 2603 Pyramid level Selection means 2604 Pyramid image 2801 Selection circuit 2802 Gaussian filter 2803 Thinning process
2901 Pyramid level 2 motion amount calculation unit 2902 Pyramid level 1 motion amount calculation unit 2903 Pyramid level 0 motion amount calculation unit 3001 Setting of information of a portion of interest (start)
3002 Setting the information of the part of interest (using the previous frame)
3003 Previous frame result

Claims (7)

In an image motion detection device for detecting a target or background motion amount in an image,
A means for inputting information of a target portion for inputting or setting information of a target or background portion of interest;
Weighting refers to setting a weighting coefficient that changes according to an image position by associating the portion of interest with an image position. The information from the information input unit of the portion of interest is combined with the content of the image, A weighting setting unit that sets a weight for a part to be large and weights a part considered to be other than the part of interest to be small, and sets an initial value of motion amount calculation as necessary;
Using the weighting weighting coefficient set by the weighting setting means or the initial value of the motion amount calculation, weighting for estimating the motion amount of the image while paying attention to the portion of interest or excluding the portion of interest A motion amount calculation means that takes into account,
In the process of detecting the amount of motion of the image, the weighting setting unit and the motion amount calculation unit taking into account the weighting, etc., estimate the final amount of motion of the image by repeating at least one iteration. The weighting setting means has a function of adaptively adjusting the weighting by reflecting the motion amount estimation result of the image of the motion amount calculation means considering the weighting in the iterative processing, and taking the weighting into consideration The motion amount calculating means uses a weighting coefficient that is adaptively adjusted by the weighting setting means in the iterative process.   The information input means for the part of interest includes an image position, an image range, a distance, a size, a deformation degree, a two-dimensional shape, a three-dimensional shape, a texture, a color, a reflectance, or a transmission of the target target or background part. 2. At least one information among rate, appearance, existence probability, vehicle type, model, category, number, speed, acceleration, angular velocity, movement information of the image pickup apparatus or prediction information thereof is input or set. The image motion detection apparatus described.   3. A multi-resolution image is generated from an input image for measuring a motion amount, a coarse-fine search is performed using the multi-resolution image, and the motion amount of the image is gradually estimated in the process iteration. The image motion detection apparatus described.   4. The image motion detection apparatus according to claim 1, wherein the information input means for the part of interest sets information of the part of interest by using information of a previous processing result in time series.   The weighting coefficient that changes the image position by associating the part of interest with the image position is realized by processing in the form of handling the same weighting coefficient in units of rectangular windows or regions of arbitrary shapes, or in units of rectangular windows or arbitrary The image motion detection apparatus according to claim 1, 2, 3, or 4, embodied by a process equivalent to selecting a weighting factor of 0 or 1 for each area of the shape.   The weighting factor is a weighting factor that changes depending on the image position, associating the portion of interest with the image position, and the image change is the difference in the shooting environment and the image pickup device between the images to be measured for the amount of motion of the image 6. The weight coefficient is adaptively adjusted so as not to pay attention to a portion of a large image change that is an obstacle to measuring the amount of motion between images. Image motion detection device.   The image motion detection device according to claim 6, wherein correspondence between images acquired by different types of image sensors such as a visible image and an infrared image, a near-infrared image, and a far-infrared image is calculated.

Images