JP2007278844A - Image processing apparatus and its processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of accurately measuring the photographing location or attitude of a photographing device or coordinates of objects on the basis of time-serially changing photograph images. <P>SOLUTION: A series of photograph images are acquired by time-serial photographing. Characteristic points are extracted from the photograph images. The characteristic points are tracked and related to one another. Stereoscopic images are selected from the series of photograph images related to one anther, oriented, and three-dimensionally measured. Three-dimensional coordinates of corresponding characteristic points determined by the three-dimensional measurement are back-projected to the photograph images to compute the residual between coordinates of the corresponding characteristic points and coordinates of the back-projected corresponding characteristic points. On the basis of the computed residual, the appropriateness of the corresponding characteristic points to three-dimensional measurement is judged. Corresponding characteristic points judged as inappropriate are deleted. By performing orientation and three-dimensional measurements without the deleted corresponding characteristic points, it is possible to improve measurement accuracy. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and a processing method thereof. Specifically, the present invention relates to an image processing apparatus that tracks a dynamic image when an imaging apparatus moves relative to an object and measures coordinates of the imaging apparatus or the object, and a processing method thereof.

  There has been a technique for measuring the position of an imaging apparatus by continuously imaging an object while performing relative movement. However, when an imaging device is actually mounted on a moving body such as an automobile or when a person takes a picture with a hand, a stable image is not necessarily obtained due to the shaking of the automobile or person. It is necessary to correct vertical movement, tilt, etc., and other cars, people, Asuka, fallen leaves etc. enter between the imaging device and the object, and the feature points are hidden behind and disappear, It may be revived again. In addition, it may be difficult to see the movement in the three-dimensional space on the two-dimensional image. Therefore, it is necessary to perform processing for identifying and deleting feature points and corresponding points that are inappropriate for three-dimensional measurement, as well as such processing for swinging the photographing apparatus and processing for feature points that disappear and are reproduced. On the other hand, when a stationary object is imaged with a stationary imaging device, a three-dimensional measurement technique that automatically searches for and measures corresponding points with high accuracy is disclosed. (See Patent Document 1)

JP 2003-264852 A (paragraphs 0018 to 0073, FIGS. 1 to 11 etc.)

  In order to measure the three-dimensional coordinates of a photographing apparatus or an object from a moving image, feature points corresponding to each other on two or more moving images, that is, the same point on the object (hereinafter referred to as a corresponding feature point) are obtained. Need to keep track of this. However, when the corresponding feature points are obtained, for example, when they move in the depth direction three-dimensionally, they do not appear to move when viewed two-dimensionally, and there are two differences in the depth in three-dimensional space. The so-called pseudo feature points and pseudo reference points that are recognized as intersections on the image by the above lines cannot be removed correctly. In addition, the conventional moving image analysis is based on the evaluation of the movement of the two-dimensional point, and what is evaluated for the movement of the three-dimensional point is not found, but if the correct miscorrespondence removal cannot be performed, As a result, the measurement accuracy of the three-dimensional coordinates deteriorates and the reliability decreases.

  Therefore, the technology for photographing a stationary object with a stationary imaging device is developed and applied when one or both of them moves, and feature points and corresponding points inappropriate for 3D measurement are identified and deleted. It is possible to provide a technique that can accurately measure the shooting position, posture, or coordinates of an object of a shooting device even in the case of a shot image that changes in time series such as a moving image that requires processing to be performed. is needed.

  An object of the present invention is to provide an image processing apparatus and a processing method thereof that can accurately measure the shooting position, posture, or coordinates of an object of a shooting apparatus from a shot image that changes in time series such as a moving image. .

In order to solve the above problems, the image processing apparatus 100 according to claim 1, for example, as shown in FIG. 2, an object relatively moving, three or more images mutually Ri next to an overlapping area In this way, a captured image acquisition unit 2 that acquires a series of captured images captured in time series, a feature extraction unit 3 that extracts a feature point from any captured image captured in time series, and a series of captured images The feature point tracking unit 4 that tracks feature points and associates the feature points with each other, the stereo image selection unit 6 that selects a stereo image that is a pair of images from a series of captured images, and the stereo image selection unit 6 select An orientation processing / three-dimensional measurement unit 7 for performing orientation and three-dimensional measurement using corresponding feature points (representing the same point on the object) associated with each other of the stereo images thus obtained, and stereo From image to 3D measurement A back projection unit 81 that back-projects the obtained three-dimensional coordinates of the corresponding feature point onto the photographed image, and, for the photographed image, calculates a residual between the coordinates of the corresponding feature point and the back-projected corresponding feature point The residual calculation unit 82, the corresponding point suitability determination unit 83 that determines whether the corresponding feature point is suitable for three-dimensional measurement based on the residual calculated by the residual calculation unit 82, and the corresponding point suitability determination unit 83 A miscorresponding point deletion unit 84 that deletes the corresponding feature point determined to be inappropriate, and the orientation processing / three-dimensional measurement unit 7 determines the corresponding feature point deleted by the miscorresponding point deletion unit 84 from the corresponding feature point. Except for this, orientation and three-dimensional measurement are performed again.

  Here, the relatively moving object is typically shot with either the object or the imaging device moving and the other stationary, but is shot with both moving together. Also good. That is, the image may be taken in a state in which both move relatively. It is possible to track feature points by sharing an overlapping portion between three or more adjacent images. The larger the number of overlapping images, the better because the measurement coordinate accuracy can be improved. For example, 10 or more is preferable, and 50 or more is more preferable. A series of captured images taken in time series is captured images that are sequentially acquired over time, and may be images extracted from frames of moving images continuously captured by a video camera. It may be an image taken sequentially at an appropriate time interval by a camera. Further, it may be acquired from all the frames of the moving image or may be acquired every several frames. Also, the tracking of feature points is performed by searching corresponding points corresponding to the second image with respect to the feature points of the first image. Corresponding corresponding points in the third image are searched as points, and feature points are sequentially associated and tracked. Here, feature points associated with each other in a series of captured images are referred to as corresponding feature points. Also, feature points are newly generated, disappeared, and reproduced with the passage of time. Therefore, in a series of photographed images, the feature points associated with each other may be in at least three or more photographed images. In addition, it is preferable to select a stereo image from a corrected image obtained by correcting the position, magnification, and inclination of a series of captured images by reflecting corresponding feature points. In addition, since they are associated with each other by three-dimensional measurement, that is also acceptable. The orientation is a process for calculating the photographing position and inclination of the photographing apparatus, and the three-dimensional measurement is a process for calculating the three-dimensional coordinates of each feature point. Moreover, the captured image to be back-projected does not necessarily have to be an image selected as a stereo image. In addition, the suitability of the corresponding feature point for 3D measurement does not mean that a miscorresponding point that is a feature point that degrades the accuracy of 3D coordinate measurement (and therefore also a feature point that adversely affects the orientation). Determined with the intention of excluding it as appropriate. It should be noted that, in addition to feature points that are not originally associated but incorrectly associated with miscorresponding points, feature points, pseudo feature points, and pseudo reference points attached to objects that move differently from the object being imaged Etc. are also included.

  With this configuration, even in the case of captured images that change over time such as moving images that require processing to identify and delete feature points and corresponding points that are inappropriate for 3D measurement, they are inappropriate. Image processing that can accurately measure the shooting position, orientation, or coordinates of an object from a shooting image that changes in time series, such as a moving image. Equipment can be provided.

  The invention according to claim 2 is the image processing apparatus according to claim 1, wherein the orientation processing / three-dimensional measurement unit 7 has a substantially equal base length as a stereo image selected by the stereo image selection unit 6. The measured values from a plurality of stereo images are statistically processed to obtain the three-dimensional coordinates of the corresponding feature points. Here, the baseline length refers to the distance between the optical axes of the photographing apparatus that has photographed a stereo image. If comprised in this way, a three-dimensional coordinate can be made highly accurate using a statistical process.

  According to a third aspect of the present invention, in the image processing device according to the first or second aspect, the corresponding point suitability determination unit 83 sets the residual calculated by the residual calculation unit 82 to a predetermined threshold value. If it exceeds, the corresponding feature point is determined to be inappropriate. Here, the predetermined threshold is 3σ, for example. With this configuration, inappropriate feature points can be automatically removed using a predetermined threshold.

  According to a fourth aspect of the present invention, in the image processing apparatus according to the third aspect, the predetermined threshold value is variable for each number of determinations by the corresponding point suitability determination unit 83. With this configuration, for example, as the number of determinations is increased, the threshold value can be reduced to increase the accuracy.

  According to a fifth aspect of the present invention, in the image processing apparatus according to the first or second aspect, the corresponding point suitability determination unit 83 adds the residual time series data calculated by the residual calculation unit 82 to the time series data of the residual. Based on this, the suitability of the corresponding feature point is determined. When configured in this way, there is a high possibility that a miscorresponding point will have a large residual at any point in time-series data, which is effective in detecting these miscorresponding points.

According to a sixth aspect of the present invention, in the image processing apparatus according to the fifth aspect, the corresponding point suitability determination unit 83 includes corresponding characteristic points including data having an abnormally large residual in the time series data of the residual. Is found, the corresponding feature point is determined to be inappropriate.
With this configuration, the corresponding feature point including data having an abnormally large residual may be unstable. For example, the reliability of the measurement value can be increased by deleting the corresponding feature point.

  According to a seventh aspect of the present invention, in the image processing apparatus according to the fifth aspect, the corresponding point suitability determining unit 83 is a point where the corresponding feature point is moving based on the time series data of the residual. It is determined whether or not. If comprised in this way, it will be useful for the analysis in case a corresponding feature point is a moving point.

  Further, in the image processing method according to claim 8, for example, as shown in FIG. 3, a relatively moving object is photographed in time series so that three or more adjacent images share an overlapping portion. A captured image acquisition step (S100) for acquiring a series of captured images, a feature extraction step (S110) for extracting feature points from any of the captured images captured in time series, and feature points for a series of captured images. A feature point tracking step (S120) for tracking and associating feature points with each other, a stereo image selection step (S140) for selecting a stereo image from a series of captured images, and a stereo image selected in the stereo image selection step (S140) The first orientation step (S150) for performing orientation using the corresponding feature points and the first three-dimensional measurement step for performing three-dimensional measurement using the orientation results of the first orientation step (S150) ( 160), a back projection step (S170) for back projecting the three-dimensional coordinates of the corresponding feature points obtained from the stereo image in the three-dimensional measurement step (S160) to the photographed image, and the coordinates of the corresponding feature points for the photographed image A residual calculation step (S175) for calculating a residual with the back-projected coordinates of the corresponding feature point, and a three-dimensional measurement of the corresponding feature point based on the residual calculated in the residual calculation step (S175). Corresponding point suitability determining step (S185) for determining the suitability of the corresponding point, an erroneous corresponding point deleting step (S190) for deleting the corresponding feature point determined to be inappropriate in the corresponding point suitability determining step (S185), and the corresponding feature point Three-dimensional measurement using the orientation results of the second orientation step (S150) and the second orientation step (S150) in which the corresponding feature points deleted in the erroneous correspondence point deletion step (S190) are removed and the orientation is performed again. The second three to do And a source measurement step (S160).

  With this configuration, even in the case of captured images that change over time such as moving images that require processing to identify and delete feature points and corresponding points that are inappropriate for 3D measurement, they are inappropriate. Image processing that can accurately measure the shooting position, orientation, or coordinates of an object from a shooting image that changes in time series, such as a moving image. Can provide a method.

  The invention according to claim 9 is an invention of a computer-readable program for causing a computer to execute the image processing method according to claim 8.

  ADVANTAGE OF THE INVENTION According to this invention, the image processing apparatus and its processing method which can measure the imaging | photography position, attitude | position of an imaging device, or the coordinate of a target object from the captured image which changes in time series, such as a moving image, can be provided.

  Embodiments of the present invention will be described below with reference to the drawings.

[First Embodiment]
FIG. 1 is a diagram for explaining the concept in the present embodiment. This is an example in which a camera is attached to an automobile, time, that is, the position of the automobile is changed little by little, and an urban area as an object is photographed, and the position coordinates of the camera, that is, the locus of the automobile are obtained from the tracking results in these captured images. This makes it possible to continuously display the position of the car in the car navigation, but it is also significant to be used complementarily in a section where GPS radio waves cannot be received. In the following embodiments, in a captured image, miscorresponding points such as feature points that are incorrectly associated, feature points with movement, and feature points with large fluctuations are inappropriate for obtaining the position coordinates of the camera. Remove because there is. In the first embodiment, an example will be described in which a feature point is determined to be inappropriate based on the time series data of the residual calculated by the residual calculation unit.

  FIG. 2 shows a configuration example of the image processing apparatus 100 in the present embodiment. In the figure, reference numeral 1 denotes a control unit that controls each unit of the image processing apparatus 100 so as to function as an image processing apparatus. Specifically, the control unit 1 instructs the captured image acquisition unit 2 to acquire a captured image and the feature extraction unit 3. Instruction for executing feature point extraction, instruction for tracking execution to the feature point tracking unit 4, stereo image selection instruction to the arithmetic processing unit 5, instruction for execution of orientation / three-dimensional measurement, instruction for evaluation / deletion processing of corresponding feature points, etc. I do.

  Reference numeral 2 denotes a captured image acquisition unit that sequentially acquires captured images that change in time series, such as moving images. In addition to acquiring captured images, the captured image is output to the feature extraction unit 3, stored in the image memory 10, and the like. I do. Note that a captured image may be acquired by communication from another imaging device without performing imaging with the own imaging device. Reference numeral 3 denotes a feature extraction unit that extracts feature points from sequentially acquired captured images. Extraction of feature points from the captured image input from the captured image acquisition unit 2 and extraction of the extracted feature points to the feature point tracking unit 4 are performed. Perform output, etc. 4 is a corresponding point corresponding to the feature point input from the feature extraction unit 3 (strictly speaking, it is a candidate corresponding point until it is determined, but in this embodiment, it is a corresponding point including the candidate corresponding point). It is a feature point tracking unit that searches and tracks feature points. In addition to the tracking process, the tracking result is output to the corresponding point information memory 9, the arrangement of the corresponding points is determined, and the feature points to the feature extracting unit 3 are Instruct new establishment. A series of photographed images in which feature points are associated in the feature point tracking unit 4, that is, to which the corresponding feature points are attached, are corrected images with corrected magnification, fluctuation, and inclination, together with the acquired original image and the image memory 10. Is remembered.

  An arithmetic processing unit 5 includes a stereo image selection unit 6, an orientation processing / three-dimensional measurement unit 7, and a corresponding point evaluation removal unit 8. The stereo image selection unit 6 selects a stereo image from a series of images to which corresponding feature points are attached in the feature extraction unit 3 and the feature point tracking unit 4. The orientation processing / three-dimensional measurement unit 7 executes orientation calculation and three-dimensional measurement using the stereo image selected by the stereo image selection unit 6. To the display unit 11 of the orientation result and the three-dimensional measurement result, besides, , Output the orientation results and 3D measurement results to the outside. One or more stereo images may be used for orientation calculation and three-dimensional measurement, but accuracy can be improved by performing statistical processing such as averaging using a large number of stereo images. Corresponding point evaluation removal unit 8 uses the three-dimensional measurement result, and miscorresponding points such as feature points incorrectly associated, unstable feature points with movement and fluctuation, that is, feature points inappropriate for three-dimensional measurement And corresponding points are removed. The corresponding point evaluation removing unit 8 backprojects the three-dimensional coordinates of the corresponding feature points obtained by the three-dimensional measurement of the stereo image onto the photographed image, and is back-projected with the coordinates of the corresponding feature points in the photographed image. A residual calculation unit 82 for calculating a residual with the coordinates of the corresponding feature point, and a corresponding point determination for determining whether the corresponding feature point is suitable for three-dimensional measurement based on the residual calculated by the residual calculation unit 82 A determination unit 83; and an erroneous corresponding point deletion unit 84 that deletes the corresponding feature point determined to be inappropriate by the corresponding point suitability determination unit 83. A stereo image is selected again by the stereo image selection unit 6 from the series of images from which the inappropriate corresponding feature points have been removed by the corresponding point evaluation removing unit 8, and the orientation processing / three-dimensional measurement unit 7 again performs orientation. Calculation and three-dimensional measurement are executed.

  Reference numeral 11 denotes a display unit that displays an image of an object subjected to orientation processing or three-dimensional measurement by the arithmetic processing unit 5 and a locus of a captured position in a two-dimensional or three-dimensional manner, and 9 denotes a feature point and its corresponding point (candidate corresponding point Corresponding point information memory 10 for storing information relating to (including), 10 is an image memory for storing captured images, corrected images, and other images. The corresponding point information memory 9 and the image memory 10 are referred to and written whenever necessary, such as when tracking feature points, when evaluating / removing corresponding points, when selecting a stereo image, during orientation calculation, or when measuring three-dimensionally. .

  FIG. 3 shows a flow example of the image processing method according to the first embodiment. First, the photographic image acquisition unit 2 acquires a series of photographic images that change in a time series such as a moving image for an object that moves relatively (S100: photographic image acquisition step). A series of captured images are acquired so that three or more adjacent images share an overlapping portion. The image may be obtained by taking an image with its own photographing camera, or an image taken with another photographing apparatus may be obtained via a communication line. The control unit 1 sequentially supplies a photographic image that changes in time series such as a moving image from the photographic image acquisition unit 1 to the feature extraction unit 3. In the present embodiment, since a photographing camera is attached to a car and photographing is performed while moving, the photographed image is a photographed image that changes little by little in terms of time or space, and the object is common to most of the image. The feature extraction unit 3 extracts feature points from one of the captured images taken in time series (S110: feature extraction step), and the extracted feature point data is supplied to the feature point tracking unit 4 to correspond points. It is stored in the information memory 9. The feature point tracking unit 4 searches for a corresponding point corresponding to the feature point in the series of photographed images, and associates and tracks the feature point (S120: feature point tracking step). In the tracking of the feature point, the corresponding point of the second image is searched for the feature point of the first image, and when the corresponding point is matched, the corresponding point of the second image is set as a new feature point. Corresponding points of the image are searched, and the correspondence and feature points are sequentially tracked. Here, feature points associated with each other in a series of captured images are referred to as corresponding feature points. Corresponding feature point data is stored in the corresponding point information memory 9. The captured image with the corresponding feature point is stored in the image memory 10 as a corrected image in which the magnification, fluctuation, and inclination are corrected (S125). If the feature point data extracted by the feature extraction unit 3 and the corresponding feature point data correlated by the feature point tracking unit 4 are sequentially supplied to the arithmetic processing unit 5 in real time, a moving object (such as an automobile) ), It is highly possible that the orientation process and the three-dimensional measurement are performed at an early stage and reflected in the navigator.

  Next, a stereo image is selected from a series of photographed images to which corresponding feature points are attached (S140: stereo image selection step). It is preferable to use a corrected image in which the magnification, fluctuation, and inclination are corrected to reflect the corresponding feature points for the series of images. Since it is matched, it is still possible. Here, a corrected image is used.

  FIG. 4 shows an example of stereo image selection. In this example, an image several images away from the acquired image is selected as a stereo image. If the frame interval between two images that form a pair is made constant, the baseline length can be kept substantially constant, which is preferable. One or more stereo images may be selected. However, since accuracy can be improved by performing statistical processing such as averaging using a large number of stereo images, a large number of tracked images are used here. A stereo image is selected and statistical processing is performed.

  Returning to FIG. 3, using the corresponding feature points of the stereo image selected by the stereo image selection unit 6, the orientation processing and the three-dimensional measurement unit 7 perform orientation processing and three-dimensional measurement. In the orientation process, relative orientation is performed on the selected stereo image using the coordinates of the feature points and the corresponding points, and the photographing position and inclination of the photographing camera are calculated (S150: first orientation step). Connection orientation between stereo images is also performed. Note that, for a captured image for which a stereo image could not be set, the shooting position and tilt of the shooting camera can be calculated by performing single photo orientation. Subsequently, the three-dimensional coordinates of each feature point are calculated by three-dimensional measurement using the orientation result (S160: first three-dimensional measurement step). The three-dimensional measurement is obtained, for example, by performing bundle adjustment using the position and inclination (external orientation element) of the photographing camera obtained by the orientation process as initial values. As a result, not only the three-dimensional coordinates but also a more accurate position and inclination of the photographing camera can be obtained.

  Next, the corresponding point evaluation removing unit 8 uses the three-dimensional measurement result to remove erroneously corresponding points such as feature points that are incorrectly associated and unstable feature points that are moved or fluctuated. First, the back projection unit 81 back projects the 3D coordinates of the corresponding feature points obtained in the 3D measurement process for the stereo image onto the captured image (S170: back projection process). Back projection is basically performed based on the collinear conditional expression (5) described later. Next, the residual calculation unit 82 calculates the residual between the coordinates of the corresponding feature points and the coordinates of the corresponding feature points back-projected with respect to the series of captured images to which the corresponding feature points are attached (S175: residual). Calculation step). Here, it is determined whether or not the overall residual is smaller than the set threshold (S180). If this value is less than or equal to the set threshold value, the three-dimensional coordinates are determined (S195), and the process ends. Otherwise, the process proceeds to the next, and the corresponding point suitability determination unit 83 performs a residual evaluation based on the residual calculated by the residual calculation unit 82, that is, whether or not the corresponding feature point is suitable for three-dimensional measurement. Determination (S185: Corresponding point suitability determination step). Next, the corresponding feature point determined to be inappropriate by the corresponding point suitability determining unit 83 is deleted by the erroneous corresponding point deleting unit 84 (S190: erroneous corresponding point deleting step).

Next, the process returns to the stereo image selection step (S140) again, the corresponding feature points deleted from the corresponding feature points by the erroneous corresponding point removal unit 8 are removed, and the stereo image selection (S140) is performed again. : Second orientation step), and then, the 3D measurement is performed again using the orientation result (S160: second 3D measurement step). In the second orientation step and the second three-dimensional measurement step, since erroneous correspondence points, that is, detected inappropriate correspondence feature points are excluded, more precise orientation is performed and three-dimensional coordinates are calculated. The By repeating this loop processing (S140 to S190), the measurement accuracy can be improved. The loop process is repeated until the overall residual is below the threshold. Thereby, the three-dimensional coordinates are determined (S195), and the measurement is terminated.
Below, each process is demonstrated as needed.

[Feature point extraction]
The feature extraction unit 3 extracts feature points from each captured image (S110, see FIG. 5). Typically, the initial frame is extracted from the entire screen, and the next frame is extracted from a new screen area that does not overlap the initial frame. For extracting feature points in the initial frame, for example, MORAVEC operator (HP Moravec. Towers Automatic Visual Observed Aviation, Proc. 5th International Joint Conference, Art. An operator such as can be appropriately employed.

[Tracking process]
FIG. 5 shows an example of a processing flow for feature point tracking. The feature point tracking unit 4 tracks each feature point selected by the feature extraction process (S120). That is, a candidate corresponding point corresponding to a feature point is obtained, a feature point movement vector and a screen relative movement amount are obtained, and these are connected to obtain a movement locus. The screen relative movement amount is a relative movement amount on the screen between the photographing camera and the photographing object, and the movement vector is a relative movement vector of each feature point on the two-dimensional photographed image. In tracking feature points, first, template matching is performed on adjacent captured images (S13) to obtain candidate corresponding points corresponding to the feature points. Thereby, the movement vector of each feature point is obtained. Further, by performing projective transformation using the adjacent photographed image (S15), the screen relative movement amount with respect to the photographing camera is obtained. That is, since the overall movement between frames is very short in time, it is assumed that it can be approximated by projective transformation, and the screen relative movement amount is estimated by projective transformation. Next, the movement vector of each feature point is compared with the relative movement amount of the screen between frames, and the quality of the movement vector is determined (S14). Then, candidate corresponding points that are considered to be erroneous correspondences indicating abnormal movement are removed (S16). By repeating steps S15 and S16, the accuracy of projective transformation is improved.

  Next, the arrangement of candidate corresponding points is determined (S17). That is, the arrangement of feature points and corresponding points on the captured image is confirmed. If the arrangement of feature points is extremely biased and a blank portion is generated, the feature extraction unit 3 is instructed to newly set a point existing in the newly generated blank portion as a new feature point. Then, the process returns to the feature extraction (S110) again, and the feature extraction (S110) and the tracking process (S120) are sequentially repeated in real time for new adjacent images. If feature extraction has been completed for a series of captured images, the process returns to template matching (S13), and after that, tracking processing (S120) is sequentially performed for new adjacent images.

  For example, template matching is used for tracking feature points (S13). Neighboring images are sequentially selected from the acquired captured images to form a stereo pair, and for example, stereo matching is performed by a technique such as an SSDA (Sequential Similarity Detection Algorithm) method to obtain corresponding points (S13). In the SSDA method, the similarity is determined using the residual, and the position where the partial matrix residual is minimum is obtained as the corresponding point. SSDA template matching is considered to be relatively fast as template matching and easy to implement in hardware. Other methods such as a normalized correlation method can also be employed. For template matching, it is important to optimally select a template size and a search range, and the search range is set optimally based on the frame rate, moving speed, etc. of the video camera.

  In the shooting screen, when a feature point is given to a moving object such as a running car, a person, an asuka, a fallen leaf, or when the camera shakes heavily, an erroneous response point may occur. The camera swing can be corrected by projective transformation. On the other hand, an object that moves differently from the object to be photographed causes a miscorresponding point. Therefore, by removing the miscorresponding points caused by the movement of the object etc., the reliability of the feature points (including corresponding points and candidate corresponding points) is improved, the accuracy of the mismatching determination is improved, It is possible to cope with upsets.

[Relative orientation processing]
Next, the orientation calculation will be described.
FIG. 6 is a diagram for explaining relative orientation. The origin of the model coordinate system is taken as the left projection center, and the line connecting the right projection centers is taken as the X axis. For the scale, the base line length is taken as the unit length. The parameters to be obtained at this time are the rotation angle κ ₁ of the left camera, the rotation angle φ _{1 of} the Y axis, the rotation angle κ _{2 of the} right camera, the rotation angle φ ₂ of the Y axis, and the rotation of the X axis. the five of the rotation angle of the corner ω _2. In this case, the X-axis rotation angle ω ₁ of the left camera is 0, so there is no need to consider it.

First, parameters required to determine the positions of the left and right cameras are obtained by the following coplanar conditional expression (1).

Under the above conditions, the coplanar conditional expression (1) is transformed as shown in Expression (2), and each parameter can be obtained by solving Expression (2).
Here, between the model coordinate system XYZ and the camera coordinate system xyz, the following coordinate transformation relational expressions (3) and (4) hold.

Using these equations, unknown parameters are obtained by the following procedure.
(I) The initial approximate values of the parameters (κ ₁ , φ ₁ , κ ₂ , φ ₂ , ω ₂ ) are normally 0.
(Ii) The coplanar conditional expression (2) is Taylor-expanded around the approximate value, and the value of the differential coefficient when linearized is obtained from the expressions (3) and (4), and an observation equation is established.
(Iii) Applying the least square method, a correction amount for the approximate value is obtained.
(Iv) The approximate value is corrected.
(V) The operations from (ii) to (v) are repeated using the corrected approximate value until convergence.
If it converges, connection orientation is further performed. This is a process for unifying the inclination and scale between the models to make the same coordinate system.

When this processing is performed, a connection range expressed by the following equation is calculated.
ΔX _j = (X _jr −X _jl ) / (Z ₀ −Z _jl )
ΔY _j = (Y _jr −Y _jl ) / (Z ₀ −Z _jl )
ΔZ _j = (Z _jr −Z _jl ) / (Z ₀ −Z _jl )
ΔD _j = √ (ΔX _j ² + ΔY _j ² )
(ΔX _jl ΔY _jl ΔZ _jl) ): j-th left model of unified coordinate system
(ΔX _jr ΔY _jr ΔZ _jr ): If the j-th right model ΔZj and ΔDj in the unified coordinate system are 0.0005 (1/2000) or less, it is considered that the connection orientation is normally performed. If not successful, an error is output in the orientation result display to indicate which image is bad. In this case, if there is another orientation point on the image, it is changed and the above calculation from (ii) to (v) is repeated. If not, change the location of the orientation point.

[Bundle adjustment]
Next, bundle adjustment used for three-dimensional measurement will be described. In the bundle adjustment, the object space coordinates (X, Y, Z), the image coordinates (x, y), and the projection center (X ₀ , Y ₀ , Z ₀ ) are on the same straight line as shown in the equation (5). It is based on the collinear conditional expression that exists.

  Based on the equation (5), an observation equation is created for all target space points, and a simultaneous solution is obtained by the least square method for the external orientation elements and the target space coordinates, which are the position and inclination of the camera at the time of shooting. In this case, since the equation (5) is a nonlinear equation, the external orientation element, the target space point, and the initial value of the focal length of the camera are necessary. However, the initial value of the external orientation element and the target space point is obtained by the orientation processing. Use the default value. Also, an approximate value of the focal length at the time of shooting is input as the initial value of the focal length of the camera. Since the position and orientation of the camera are known from the 3D measurement, the 3D coordinates of the corresponding points are calculated by the forward intersection method, for example.

  FIG. 7 is a diagram illustrating the result of the three-dimensional measurement, showing the photographing position where the points arranged in a substantially straight line below the left side surface are obtained, and a little small plot scattered in the three-dimensional diagram is three-dimensional. The feature corresponding point to which the coordinate value was given is shown.

[Evaluation and removal of corresponding points]
By the tracking process in the corresponding point tracking unit 4 described above, feature points are associated with all frames. However, these associations are based on a two-dimensional analysis, and may actually include inappropriate feature points and corresponding points such as pseudo feature points and pseudo reference points.
Therefore, in this embodiment, such a pseudo feature point, a pseudo reference point, a feature point that is incorrectly associated, a feature point attached to an object that moves differently from the object to be photographed, and other feature points that are inappropriate for three-dimensional measurement And corresponding points (corresponding feature points), that is, erroneous corresponding points are removed. These inappropriate corresponding feature points are removed by using the results of the orientation process and the three-dimensional measurement process in the corresponding point evaluation removing unit 8 from the back projection step (S170) to the erroneous corresponding point deletion step ( S190).

[Back projection]
In the back projection unit 81, the 3D coordinates of the corresponding feature points obtained in the 3D measurement step (S160) are back projected onto the captured image (S170). Back projection is basically performed by substituting into the collinear conditional expression (5). The collinear conditional expression indicates that the target space coordinates (X, Y, Z), the image coordinates (x, y), and the projection center (X ₀ , Y ₀ , Z ₀ ) exist on the same straight line. Formula (5) is the basis. Further, the position of the imaging device obtained in the three-dimensional measurement step (S160) is used as the projection center (X ₀ , Y ₀ , Z ₀ ) of back projection.

[Residual calculation]
Next, the residual calculation unit 82 calculates the residual between the coordinates of the corresponding feature points and the back-projected coordinates of the corresponding feature points with respect to the series of captured images to which the corresponding feature points are attached (S175). For the residual, for example, an overall mean square error σ is obtained, and this value, for example, is 3 times (3σ) as a threshold value. In this case, the loop processing (S140 to S190) is repeated, and the process ends when all the threshold values are below 3σ. Note that the number of feature points that can use these statistical properties is necessary, but it is sufficiently possible to select and process a large number of stereo images from a series of captured images.

[Correspondence point suitability determination (residual evaluation)]
The result of the three-dimensional measurement may include erroneous corresponding points, that is, inappropriate feature points and corresponding points. If there are feature points and corresponding points that are inappropriate for three-dimensional measurement, it is highly probable that they will be reflected in any residual in the time-series data. Therefore, the erroneous corresponding points are removed by removing those having a large residual on the image from the result of the residual measurement (S185 to S190).

  In the present embodiment, an example will be described in which a feature point is determined to be inappropriate based on time series data of residuals calculated by a residual calculation unit. Corresponding feature points are removed by, for example, paying attention to a certain corresponding feature point and examining a time-series change in residual on the image at that point. In the present embodiment, since a series of photographed images changes in time series, if residuals are plotted for these series of photographed images, time series data of residuals can be obtained.

FIG. 8 shows a first example of the residual time-series change (example of point Pa). The first example shows a case where the corresponding feature point Pa has a larger residual at the center between the stereo pairs A and B in a frame sandwiched between the stereo pairs A and B.
In this case, the residual is small at the time of the left and right stereo pairs A and B, but it is actually considered to be a miscorresponding point such as a pseudo feature point. Accordingly, the stereo pairs A and B and the corresponding points on the frame between them are removed in the entire period between AB.

FIG. 9 shows a second example (example of point Pb) of the time series change of the residual. The second example shows a case in which the corresponding feature point Pb has a large residual in some places in the frame sandwiched between the stereo pairs A and B.
In this case, the corresponding points are removed only for the frames having a large residual among the frames sandwiched between the stereo pairs A and B, but the corresponding points are the same as those in the first example for the points where the number of frames having the large residual is extremely large. Is removed in the entire period between A and B. Specifically, when 60% or more between stereo pairs A and B is large and there is no continuity (for example, there is no continuous one or more seconds), the corresponding points are A, as in the first example. It is removed in the entire period between B.

FIG. 10 shows a third example of the time series change of the residual (example of the point Pc). The third example shows a case where the residual becomes larger as the corresponding feature point Pc approaches the stereo image of either A or B in the frame sandwiched between the stereo pairs A and B.
In this case, it is considered that the corresponding point has moved three-dimensionally or was a pseudo reference point, and is removed as an erroneous corresponding point. However, when frames with very small residuals continue (for example, when they continue for 1 second or longer), it can be determined that the frame has not moved during this period, so that portion is not removed.

FIG. 11 shows a fourth example of the time series change of the residual (example of the point Pd). The fourth example shows a case where the residual increases as the corresponding feature point Pd moves away from the stereo pair in a frame that is not sandwiched between the stereo pairs A and B.
In this case, there is no problem between the corresponding points on the stereo pairs A and B and the corresponding points on the frame sandwiched between the stereo pairs A and B. However, other points with large residuals on the frame are removed.

  In the present embodiment, it is possible to set and apply an appropriate residual removal algorithm corresponding to various residual time series patterns. In addition, if this algorithm is programmed, inappropriate corresponding feature points can be automatically removed.

[Another orientation process and 3D measurement]
Next, stereo image selection (S140), orientation (S150: second orientation process), and three-dimensional measurement are performed again (S160: first), except for the corresponding feature points deleted by the corresponding point evaluation removing unit 8. 2 three-dimensional measurement process). In these steps, processing is performed except for detected erroneous corresponding points, that is, inappropriate corresponding feature points, and the process is repeated until the overall residual is equal to or less than a threshold value, so that highly accurate three-dimensional coordinates are calculated. The
As described above, according to the present embodiment, in the case of a captured image that changes in time series, such as a moving image that requires processing for identifying and deleting feature points and corresponding points that are inappropriate for three-dimensional measurement. However, it is possible to perform three-dimensional measurement by removing inappropriate feature points and corresponding points, so it is possible to accurately determine the shooting position, orientation, or object coordinates of the shooting device from shot images that change over time, such as moving images. It is possible to provide an image processing apparatus and a processing method thereof that can measure well.

[Second Embodiment]
In the first embodiment, the feature point suitability determination unit has described an example in which the feature point is determined to be inappropriate based on the time series data of the residual calculated by the residual calculation unit. The form shows an example in which the feature point is determined to be inappropriate when the residual calculated by the residual calculation unit exceeds a predetermined threshold. For the residual, for example, an overall mean square error σ is obtained, and three times this value (3σ) is set as a threshold value. When obtaining a more accurate result, the threshold value may be lowered to 2σ. Further, as the number of determinations by the feature point suitability determination unit is increased, the threshold value can be reduced to increase the accuracy. In this way, when determining with the threshold value, it is not necessary to refer to the time series data, and corresponding feature points having a large residual can be automatically deleted uniformly. Typically, if a corresponding feature point exceeding the threshold is found at any point in time, the corresponding feature point is deleted from all captured images. For example, when the threshold is exceeded three times in succession It is also possible to determine the removal conditions as appropriate, such as deleting only the corresponding feature points near the threshold value. Other points are the same as those of the first embodiment.

[Third Embodiment]
The present embodiment is an example in which not only the corresponding feature points are deleted, but also the captured images having many corresponding feature points having a large residual are removed for each image. Therefore, when a large number of corresponding feature points having a large residual are found for any one of the photographed images, the image itself is deleted from the series of photographed images and includes stereo image selection, orientation, and three-dimensional measurement. Perform loop processing. This is to improve the reliability of three-dimensional measurement when an image in which the entire feature point becomes unstable due to fluctuation, instantaneous movement, or the like is removed. Other points are the same as those of the first embodiment.

[Fourth Embodiment]
The present embodiment shows an example of extracting corresponding feature points that are moving based on the time series data of the residuals calculated by the residual calculation unit. Corresponding feature points whose residuals are stably changing with a constant tendency in time series are determined to be moving feature points. For example, a feature point where the residual increases or decreases at a constant rate, a feature point where the residual changes at a gradual cycle, and the like. For these feature points, it is possible to estimate the trajectory of the three-dimensional coordinates of the moving object, for example, by assuming that the coordinate difference (distance) of the moving object between stereo images is a constant value. Other points are the same as those of the first embodiment.

[Fifth Embodiment]
In the first embodiment, in the calculation of the three-dimensional coordinates, the three-dimensional coordinates obtained from the plurality of stereo images are statistically processed to calculate the three-dimensional coordinates of each feature point, and this is back-projected on the photographed image. In this embodiment, the residual is obtained by back projecting the three-dimensional coordinates obtained from the individual stereo images onto the photographed image. In this case, the residual is less likely to appear as compared with the first embodiment, but for example, if a sufficient base line length is taken, an unstable feature point is likely to show a residual, so that an inappropriate feature point can be removed. It is. Other points are the same as those of the first embodiment.

[Sixth Embodiment]
In the present embodiment, the stereo image selection unit has a correspondence table that represents the relationship between the baseline length of the stereo image and the measurement accuracy of the three-dimensional coordinates, and the stereo image has a baseline length that satisfies the required accuracy from the correspondence table. An example of selecting is shown. Since the measurement accuracy depends on the baseline length, a three-dimensional measurement result that meets the requirements can be provided by determining the baseline length of the stereo image in accordance with the required accuracy. Other points are the same as those of the first embodiment.

[Seventh Embodiment]
In the first embodiment, an example in which the photographing apparatus moves while the photographing target is stationary is described. However, the present embodiment is an example in which the photographing target moves while the photographing apparatus is stationary. Even in this case, in addition to the original object, a moving object may be interrupted between the image capturing apparatus and the object, or the image capturing apparatus may be shaken. It is possible to obtain the three-dimensional coordinates of the feature points related to the object in the captured image that changes gradually. In addition, if the object itself rotates, the feature point repeatedly disappears and is restored, so that the present invention can be applied. In addition, a plurality of objects may move differently, and the present invention can be applied to each object even in such a case.

  The present invention can also be realized as a program for causing a computer to execute the image processing method described in the above embodiment. The program may be stored and used in a built-in memory of the control unit 1, may be stored in a storage device inside or outside the system, or may be downloaded from the Internet and used. Moreover, it is realizable also as a recording medium which recorded the said program.

  Although the embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and it is obvious that various modifications can be made to the embodiment.

  For example, in the above embodiment, an example has been described in which a captured image is acquired in a state where one of the object or the imaging apparatus is moving and the other is stationary. However, the present invention is applied when both are moving. May be. For example, the present invention is sufficiently applicable when one moving speed and direction are constant. Even if the removal of the abnormal value in the tracking process is omitted, it is possible to remove inappropriate feature points using the residual measurement in the present invention. In addition, the residual measurement according to the present invention may be applied after removing unique feature points from the difference of the movement vectors in advance in the tracking process. Further, although an example in which the MORAVEC operator is used for feature point extraction and the SSDA template matching is used for template matching has been described, other operators and template matching methods may be used. The number of stereo images to be used, the base line length, the number of feature points, and the like can be selected as appropriate.

  The present invention is used for a photographing apparatus using a moving image or measurement of position coordinates of a photographing target.

It is a figure for demonstrating the concept of 1st Embodiment. It is a block diagram which shows the structural example of the image processing apparatus in 1st Embodiment. It is a figure which shows the example of a flow of the image processing method in 1st Embodiment. It is a figure which shows the example of a stereo image selection. It is a figure which shows the example of a processing flow of feature point tracking. It is a figure for demonstrating mutual orientation. It is a figure which illustrates the result of three-dimensional measurement. It is a figure which shows the 1st example of the time series change of a residual. It is a figure which shows the 2nd example of the time series change of a residual. It is a figure which shows the 3rd example of the time series change of a residual. It is a figure which shows the 4th example of the time series change of a residual.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Control part 2 Captured image acquisition part 3 Feature extraction part 4 Feature point tracking part 5 Operation processing part 6 Stereo image selection part 7 Orientation process and three-dimensional measurement part 8 Corresponding point evaluation removal part 9 Corresponding point information memory 10 Image memory 11 Display Unit 81 back projection unit 82 residual calculation unit 83 corresponding point suitability determination unit 84 incorrect corresponding point deletion unit 100 image processing apparatus

Claims (9)

A captured image acquisition unit that acquires a series of captured images obtained by capturing a relatively moving object in time series so that three or more adjacent images share an overlapping portion;
A feature extraction unit that extracts feature points from any of the captured images captured in time series;
A feature point tracking unit that tracks the feature points of the series of captured images and associates the feature points with each other;
A stereo image selection unit that selects a stereo image that is a pair of images from the series of captured images;
An orientation processing / three-dimensional measurement unit that performs orientation and three-dimensional measurement using corresponding feature points that are the feature points associated with each other of the stereo image selected by the stereo image selection unit;
A back projection unit that back projects the three-dimensional coordinates of the corresponding feature points obtained from the stereo image by three-dimensional measurement onto the captured image;
A residual calculation unit for calculating a residual between the coordinates of the corresponding feature point and the coordinates of the back-projected corresponding feature point with respect to the captured image;
A corresponding point suitability determining unit that determines the suitability of the corresponding feature point for three-dimensional measurement based on the residual calculated by the residual calculating unit;
An erroneous corresponding point deletion unit that deletes the corresponding feature point determined to be inappropriate by the corresponding point suitability determination unit;
The orientation processing / three-dimensional measurement unit performs orientation and three-dimensional measurement again by removing the corresponding feature point deleted by the erroneous corresponding point deletion unit from the corresponding feature point;
Image processing device. The orientation processing / three-dimensional measurement unit statistically processes measurement values from a plurality of stereo images having substantially the same base line length as the stereo image selected by the stereo image selection unit, and calculates the three-dimensional coordinates of the corresponding feature points. Ask;
The image processing apparatus according to claim 1. The corresponding point suitability determining unit determines that the corresponding feature point is inappropriate when the residual calculated by the residual calculating unit exceeds a predetermined threshold;
The image processing apparatus according to claim 1. The predetermined threshold is variable for each number of determinations by the corresponding point suitability determination unit;
The image processing apparatus according to claim 3. The corresponding point suitability determining unit determines the suitability of the corresponding feature point based on the time series data of the residual calculated by the residual calculating unit;
The image processing apparatus according to claim 1. The corresponding point suitability determination unit determines that the corresponding feature point is inappropriate when a corresponding feature point including data having an abnormally large residual is found in the time series data of the residual;
The image processing apparatus according to claim 5. The corresponding point suitability determination unit determines whether the corresponding feature point is a moving point based on the time series data of the residual;
The image processing apparatus according to claim 5. A captured image acquisition step of acquiring a series of captured images obtained by capturing a relatively moving object in time series so that three or more adjacent images share an overlapping portion;
A feature extraction step of extracting feature points from any of the captured images taken in time series;
A feature point tracking step of tracking the feature points of the series of captured images and associating the feature points with each other;
A stereo image selection step of selecting a stereo image from the series of captured images;
A first orientation step of performing orientation using the corresponding feature points of the stereo image selected in the stereo image selection step;
A first three-dimensional measurement step of performing three-dimensional measurement using the orientation result of the first orientation step;
A back projection step of back projecting the three-dimensional coordinates of the corresponding feature points obtained from the stereo image in the three-dimensional measurement step onto the captured image;
A residual calculation step for calculating a residual between the coordinates of the corresponding feature point and the coordinates of the back-projected corresponding feature point for the captured image;
A corresponding point suitability determination step for determining the suitability of the corresponding feature point for three-dimensional measurement based on the residual calculated in the residual calculation step;
An erroneous corresponding point deleting step of deleting the corresponding feature point determined to be inappropriate in the corresponding point suitability determining step;
A second orientation step of performing orientation again by removing the corresponding feature points deleted in the erroneous corresponding point deletion step from the corresponding feature points;
A second three-dimensional measurement step of performing a three-dimensional measurement using the orientation result of the second orientation step;
Image processing method. A computer-readable program for causing a computer to execute the image processing method according to claim 8.