JP2005189055A - Image measuring method and system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a three-dimensional measuring method and a system which do not require experts' knowledge and are simple and highly accurate. <P>SOLUTION: The three-dimensional measuring system is provided with: a processing part 22 for estimating corresponding lines between frames by using a large number of continuous frame images made of video images and automatically tracking corresponding points; and an optimum frame selection processing part 24 for selecting appropriate representative frames from among the large number of continuous frame images. The three-dimensional measuring system is further provided with a processing part 23 for inputting data on the corresponding lines between frames capable of verifying and modifying the corresponding points of selected images in order to facilitate the work of verifying the corresponding points and making computation processing highly accurate. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a measurement method and apparatus using an image, and more particularly to a method for efficiently and accurately performing data processing for estimating distance data based on an image.

  As a method for constructing three-dimensional information based on two-dimensional image information, for example, as described in Patent Document 1, based on projective geometry based on information on a plurality of two-dimensional images and their corresponding points. A calculation method has been proposed. This method can be implemented with high accuracy and at a relatively low price due to the popularization and low price of digital cameras, and is widely used in the fields of civil engineering / architecture and measurement of plant equipment. However, this method has a problem that a large amount of labor is required for setting corresponding points and preparing markers in order to recognize the three-dimensional shape and distance of complicated parts such as the pipe length inside the plant.

  In order to solve this problem, a method of generating three-dimensional data using a moving image photographed using a video camera, which has been studied for computer graphics, has been proposed. In this method, as described in Patent Document 2, for example, the corresponding points in a plurality of two-dimensional image data are obtained by tracking the optical flow of feature points based on the two-dimensional image data of each frame data of the moving image. It automatically extracts and automates data processing for setting corresponding points.

  If it is considered that a plurality of two-dimensional image data extracted from a moving image is equivalent to an image obtained by photographing the same object using the data of the corresponding points, as shown in Patent Document 1, for example, epipolar constraint And a three-dimensional shape can be calculated by using projective geometry calculation. Furthermore, if the absolute value of the distance of the reference image is given, it can be given as absolute coordinates of a three-dimensional shape.

Special table 2002-517712 gazette

Japanese Patent Laid-Open No. 9-326029 Image comprehension-Mathematics of three-dimensional recognition Wavelet image analysis (Science & Technology Publishing, written by Koichi Niijima) Pattern recognition (Asakura Shoten, Otsu et al.)

  However, since there are a large number of frames in the moving image, the selection of an appropriate frame greatly affects both the convergence speed and accuracy of the three-dimensional shape calculation. Also, the corresponding points obtained by automatic tracking are not always 100% correct.

  An object of the present invention is to provide a simple and highly accurate three-dimensional measurement method by selecting a frame having a highly reliable corresponding point, including a reference distance, and suitable for three-dimensional shape calculation from a moving image. And providing a system.

  In order to achieve the above object, the present invention provides an image measurement method for measuring a target distance in an image by analyzing image data of a plurality of image frames, including a predetermined frame including a measurement target portion from the plurality of image frames. Selecting and setting the measurement target part and the reference distance part on the same screen and inputting the reference distance, and automatically estimating the measurement target part in another image frame to calculate the target distance Features.

  The automatic tracking is estimated by tracing a change in the position of a feature point in an image between image frames (optical flow analysis). Further, an optimal frame is selected based on the estimation result, and a three-dimensional distance calculation is performed on the optimal frame.

  If there is an error in the corresponding point as a result of the selection of the optimum frame, the corresponding point between the frames is estimated again by the corresponding point correction input.

  An image measurement apparatus according to the present invention analyzes image data of a plurality of image frames and measures a target distance in an image. The measurement target portion and a reference distance portion in a predetermined image frame are displayed on the same screen. An interface for setting and inputting a reference distance; an inter-frame corresponding line estimation processing unit for estimating a measurement target portion in another frame image by tracking a change in a position of a feature point between image frames; An optimal frame selection processing unit that selects a frame image having a large change in the center of gravity of the tracked measurement target portion and the reference distance portion by optimal calculation, and a three-dimensional distance calculation processing unit that performs distance calculation on the selected optimal frame. It is characterized by providing.

  And a confirmation interface for confirming a result of tracking the measurement target portion and the reference distance portion of another frame corresponding to the measurement target portion and the reference distance portion in the image frame initially set using the interface, To do.

  According to the present invention, a moving image frame suitable for three-dimensional shape calculation can be selected by inputting a corresponding point with high reliability and a reference distance, so that it is possible to realize a highly accurate and easy-to-use image measurement method.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the three-dimensional measurement apparatus according to the first embodiment of the present invention, and FIG. 2 is a diagram showing the system configuration of the three-dimensional measurement apparatus.

  The image data initial setting unit 1 includes a video image input processing unit 14, a video image storage unit 17, a measurement target line data input processing unit 15, a reference line data input processing unit 16, a measurement target line data storage unit 18, and a reference line data storage unit. 19.

  The video image data 11 is input as moving image data by the video image input device 30. The moving image format is two-dimensional data image data with a time stamp such as AVI, MPEG, or QuickTime. This is the document "Book to become a master of audio, video, and document file formats-MP3 / UMA / MPEG / QT / WAVE / AVI ... file mechanism and conversion method (I / O separate volume, edited by 1st IO Editor)" It is described in.

  In the video image input processing unit 14, a range used for three-dimensional measurement of the moving image is limited by moving frames of the moving image as necessary. The moving image thus input is physically stored in the porcelain disk device 31 and is stored in the video image storage unit 17 on the memory.

  FIG. 3 shows an example of the frame data of the moving image held. As shown in FIGS. 3A and 3B, the moving image is stored as two-dimensional image information for each time. Point A and point A ′, and B and B ′ in the figure are corresponding points at the same position between the two frames. The format of the two-dimensional image of each frame is an arbitrary format suitable for subsequent feature extraction, such as grayscale monochrome, 24-bit full color, or 16 color.

  Next, in the measurement target line data input processing unit 15, an appropriate frame of the moving image is selected on the display 34, and the measurement target line 12 is input and set using the keyboard 32, the mouse 33, and the like.

  FIG. 5 shows an example of an interface for inputting a measurement target line. As shown in the figure, a frame including the measurement target part is selected and displayed, and the measurement target part is set by overwriting the measurement target part in the same manner as normal drawing software. The set measurement target line (dotted line) is a measurement target as a set of the time stamp of the image frame stored in the video image storage unit 17 and the coordinate information of the start point and end point of the two-dimensional image data corresponding to the time stamp. It is stored in the line data storage unit 18.

  Similarly, the reference line data input processing unit 16 selects an appropriate frame of the moving image on the display 34 and inputs and sets the reference line data 13 using the keyboard 32, the mouse 33, and the like.

  FIG. 6 shows an example of an interface for inputting a reference line. As shown in the figure, a frame including the reference line part is selected and displayed, and the reference line target part is set by overwriting the reference line part in the same manner as normal drawing software. Further, a sub window is displayed by right-clicking with the mouse 33 or the like, and the length of the reference line is set as shown. The start point, the end point position coordinates on the frame, and the reference line that have been set are stored in the reference line data storage unit 19.

  The above processing and storage are functions realized in the computer program, are described in a program language such as C ++, and are normally stored in the magnetic disk device 31. When the program is activated, the program is allocated to the memory 35, and various calculations are executed using the CPU 36 (central processing unit), the memory 35, and the like. The program may be stored in advance in a dedicated memory, or the function of the program may be configured to be executed using a dedicated CPU for each function.

  The three-dimensional distance calculation unit 2 includes an inter-frame correspondence line estimation processing unit 22, an optimum frame selection processing unit 24, a frame correspondence line data input processing unit 23, a three-dimensional distance calculation processing unit 25, and a distance calculation result storage unit 26. Yes.

  The inter-frame correspondence line estimation processing unit 22 is based on the measurement target line data input from the measurement target line data storage unit 18 and the reference line data input from the reference line data storage unit 19. Estimate the correspondence. That is, how each point on the measurement target line corresponds to each point on the reference line in each image frame input from the video image storage unit 17 is estimated.

FIG. 7 schematically shows the content of the corresponding point estimation process. In the figure, reference numerals 71, 72, and 73 denote two-dimensional image data of time stamps t _k , t _{k + 1} , and t _{k + 2} of the video image data, respectively, and 71 denotes a reference line and a measurement target line. 2D image data of a defined frame. Reference numerals 72 and 73 are two-dimensional image data of frames continuing from 71 frames. The dotted line arrows in the figure indicate which point corresponds to the frame 72 and the frame 73 as a result of estimating the corresponding points of the end points of the reference line and the measurement target line by the frame corresponding line estimation processing 22 set in the 71 frame. Estimated results are shown.

  Such a corresponding line estimation process can be realized by a known method called optical flow analysis. For example, a method using wavelet analysis as described in Chapter 13: Moving object detection of Non-Patent Document 2 or a feature point as described in Japanese Patent Application Laid-Open No. 5-34608 is used. It is a method of analysis.

  In this embodiment, in the measurement target line data input processing unit 15 and the reference line data input processing unit 16, the feature extraction of the frame of the current image corresponding to the set line is executed by the inter-frame corresponding line estimation processing unit 22. . It is possible to find a corresponding point with high accuracy by tracking the line where the feature amount matches by the optical flow analysis of the above document. For example, track the length of the line, the color of the portion of the line, the brightness, and the edge corresponding to the end point of the line.

  As shown in Non-Patent Document 1, it is possible to restore three-dimensional absolute coordinates by specifying eight points or more in a two-dimensional image and specifying a distance between the two points. In this case, if the correspondence of the remaining four points in addition to the two endpoints of the reference line in the designated frame and the two points to be measured can be estimated, the inter-frame correspondence line estimation processing unit 22 will theoretically Can be converted into three-dimensional absolute coordinates. The remaining four points can be selected by adding two reference lines or measurement target lines in total, or by automatically extracting feature points in the inter-frame corresponding line estimation processing unit 22 to cope with the remaining four points. A method of recognizing the relationship can be considered. For example, the technique of creating a correspondence line between frames by automatically extracting feature points can be performed by the method described in Chapter 13 “Detection of moving object” of Non-Patent Document 2.

  However, it is inevitable that a certain amount of error occurs in the inter-frame correspondence line estimation processing unit 22. Further, the error varies depending on which frame is used among the frames having corresponding points. Furthermore, Non-Patent Document 1 shows that accuracy can be improved by using frame corresponding points that include one or more corresponding points redundantly.

  For this reason, in the optimal frame selection processing unit 24, a frame for accurately converting to a three-dimensional absolute coordinate among the frames in which the correspondence between the reference line and the measurement target line is taken in the inter-frame estimation processing 22. Select one or more. The selected frame is checked by the user in the inter-frame correspondence line data input processing unit 21 and correction input is performed as necessary. As a result, compared to the case where the corresponding line is confirmed and corrected for all frames, it is possible to greatly reduce the user's effort without degrading the accuracy.

  FIG. 8 shows an example of an interface for correction input. The frame selected as the optimal frame on the right side is displayed in a small size. Select the one that seems to be wrong from this, and operate the eraser, reference line, etc. in the same way as normal drawing software on the left side screen Make corrections while selecting the icon.

  Since the tracking of feature points and lines is normally performed sequentially according to the time stamp of the frame, if a corresponding line shifts in the middle, the blur propagates to subsequent time stamps. For this reason, after inputting the correction result from the interface of FIG. 8, pressing the save button, and then pressing the tracking button at the lower right in the figure, the inter-frame correspondence line estimation processing unit 22 reflects the correction result of the user. Corresponding line tracking is performed. As a result of the recalculation, the frame to be selected may change. Therefore, the optimum frame selection processing unit 24 reselects the optimum frame, and the interframe correspondence line data input processing unit 23 again executes the user correction input. To do.

  Such confirmation / correction work is carried out until the correction input is completed, and after the correction input is completed, the correction is completed by pressing a correction completion button in the interface of FIG.

  In the three-dimensional distance calculation processing unit 25, three-dimensional calculation is performed based on the image data of the selected frame and the data on the correspondence between the reference line and the measurement target line. Then, the three-dimensional absolute coordinates for the image included in the frame selected by the optimum frame selection processing unit 24 and the calculated distance of the measurement target line are stored in the distance calculation result storage unit 26.

  The optimum frame selection process and the three-dimensional distance calculation, which are the keys of the invention, will be described below using a mathematical expression based on a general homogeneous coordinate system. The point of the simultaneous coordinate system (m1, m2, m3) corresponds to the equation 1 when m3 ≠ 0. However, f is an arbitrary point other than 0.

  Physically, as shown in FIG. 9, the point (X, Y, Z) is projected by the plane Z = f perpendicular to the Z-axis direction from the origin O of the (XYZ) coordinate system. This corresponds to the coordinates on the projection plane with the origin O in the figure.

  The lines of the simultaneous coordinate system (m1, m2, m3) correspond to the number 2 straight line on the image plane.

  Further, the coordinate of the simultaneous coordinate system is considered as a vector, and a normalized vector is called an N vector. For example, the N vector m of the point (a, b) on the screen coordinates is expressed by Equation 2.

  First, when two or more corresponding points are determined by two frames and the absolute values of the lengths of the two points are known, a method for constructing the positions of the points in the three-dimensional absolute coordinate space will be described. To do. However, camera correction has already been implemented. The correction of the camera can be performed by, for example, the method described in the document “3D Vision; Chapter 6 (Kyoritsu Shuppan, Saburo Tsuji, Author Tsuyoshi Tsuyoshi)”.

  The N vectors of the N pairs of camera-corrected points are represented by {mα} and {m′α} (α = 0, ..., N (N is 8 or more)), and the distance D between m0 and m1 is already set. Suppose that

  FIG. 10 shows a calculation procedure of the three-dimensional absolute coordinates. In step 200, corresponding point data {mα}, {m′α}, measurement reference line data, and reference line data are read. In step 201, a camera rotation R and a translation vector h in a plurality of frames are calculated from these data. In step 202, the absolute distance r to the camera is calculated. In step 203, the three-dimensional absolute coordinates of the corresponding points are calculated from these calculation results.

  FIG. 11 shows the calculation procedure of the camera rotation matrix R and translation vector h. This procedure is described in accordance with the contents described in Chapter 4 of Non-Patent Document 1.

In step 300, first, in order to obtain the translation vector h and the rotation matrix R, a basic matrix G defined as G = (h × r _1, h × r _2, h × r ₃ ) is calculated. Here, r _1, r _2, r ₃ are column vectors of the first, second, and third columns of the rotation matrix R, and x is an outer product of the vectors. At this time, theoretically, for each α, the inner product of the original point and the corresponding point converted by the basic matrix is zero. That is, (mαGm′α) = 0.

Actually, since there is a measurement and calculation error, it is not possible to determine G such that this inner product becomes 0 at all points. Therefore, G that minimizes the sum of inner products at each corresponding point is obtained. Further, since only this will cause indefiniteness of a constant multiple of G, Tr (GG ^T ) = 2. Here, Tr represents a matrix trace, and is a condition for normalizing the size of the translation vector to 1. The G thus obtained is set as G ^* . Such G ^* can be numerically solved by a method that reduces to the problem of obtaining the minimum eigenvalue.

In step 301, using the basic matrix G ^* thus obtained, a translation vector h ^* normalized to a magnitude of 1 is obtained as follows. G ^* g ₁ ^* a column _vector, g _{2 ^*,} when the g ₃ ^*, the number 3, Σα = 1, ..., 3 (g i *,, h) 2 → Min, and || h || A unit vector h satisfying = 1 is obtained by the method of least squares, and this is defined as h ^* .

This can be obtained by singular value decomposition or the like, as shown in pp105 to 106 of Non-Patent Document 1. The code is determined according to the following conditions.
Σα = 1, ..., N [h ^* mαGm'α]> 0 (Left side symbol [] is a scalar triple product)
Next, in step 302, the rotation matrix R is obtained as follows using the standardized translation h ^* thus obtained.
Σα = 1,..., 3 || h ^* × r _i , −g _i ^* || ² → minimum R = (r _1, r _2, r ₃ ) is determined as R ^* . This can be solved numerically by, for example, the least square method.

Up to this point, the translation vector and the rotation matrix have been obtained in a form that leaves the indefiniteness of the scale, but since the scales are both standardized, their relative sizes are not fixed. In order to obtain this, in step 303, if the distance from each corresponding point to the camera center is r ^* α, r ′ ^* α, (α = 0,..., N), this can be obtained by the following equation. .
r ^* α mα = r ′ ^* αR ^* m′α−h (α = 0,..., N)
As a result, r ^* α and r ′ ^* α are obtained as in Expression 4.

  As a result, the rotation vector and translation vector to be obtained, and the distance from each corresponding point to the camera center are obtained as follows, leaving the indefinite value k.

R = R ^* , h = h ^*
rα = kr ^* α, r′α = kr ′ ^* α
Therefore, finally, using the points m ₀ and m ₁ whose distances are known, the absolute value of k is obtained as follows.

^{_{| Kr * 0 m 0, -kr}} * 1 m 1 | = D
Here, D is a specified distance between m ₀ and m ₁ . From this equation, k can be obtained by using already obtained rα, r′α.

  When there are a plurality of reference lengths, the optimum k may be obtained by the least square method or the like so that the distance values match.

  Thus, when the translation vector h and the rotation matrix R are obtained, the coordinate x in the three-dimensional absolute space can be calculated from the corresponding point on the image by Equation 5, and the distance between the end points of the measurement target line is also calculated. Can be sought.

  The above is the procedure for calculating the rotation matrix R and the translation vector h of the camera using the method described in Non-Patent Document 1. On the other hand, in the present embodiment, the optimum frame selection processing unit 24 (FIG. 1) automatically selects and presents the frames, thereby reducing the trouble of the user's corresponding line confirmation processing and improving the accuracy. Can do.

  What should be considered as a method for automatic frame selection is that the value of the translation vector h needs to be sufficiently large in the previous formulation. In order to improve the accuracy by using a plurality of frames, it is necessary to increase the amount of information used by using frames from various directions.

  When there is no translation only by rotation, the center of gravity on the two-dimensional screen does not change. Therefore, a method of selecting a frame having a large change in the center of gravity of the frame can be considered. On the other hand, if the viewing direction changes greatly, it is conceivable that the variation in the direction change becomes large when the change of the corresponding point of each point is vectorized.

  FIG. 12 shows a procedure for automatic frame selection based on the above concept. In step 400, the centroid of the corresponding points of each frame is calculated, and the corresponding points are regarded as a sequence of points ordered in the order of the time stamps of the frames.

  In step 401, approximation is performed with a preset number of polygonal lines by, for example, dynamic programming described in p62 of Non-Patent Document 3. A frame corresponding to the end point of the broken line is a candidate for the optimum frame.

  FIG. 13 shows an example of a point sequence of the center of gravity and an approximation of the point sequence by a broken line. A white circle is an end point of the broken line, and a frame corresponding to the white circle is a candidate for the optimum frame.

In step 402, two optimum frames used for three-dimensional calculation are selected from the optimum frame candidates. For this reason, the direction φ _m of the direction vector when N corresponding points are connected on the image coordinates is calculated for all the two combinations of optimal frame candidates, and the combination with the largest variance is determined as the optimal frame. And

FIG. 14 shows the calculation process of φ _m , and the positions of corresponding points in two optimal frame candidates (assumed to be frames F1 and F2) are shown on the same frame screen. The black circles (points A to H) in the figure are the coordinates of the corresponding points in the frame F1, the white circles are the coordinates in F2 corresponding thereto, A ′ corresponds to A, and B ′ corresponds to B. As shown in the figure, the set of these corresponding points is connected, and the inclination of the frame image in the horizontal axis direction is φ _m . In this case, the variance value of φ _m of 8 corresponding points is calculated. Such calculation of the variance value is calculated for all combinations of optimum candidate frames, and the combination of frames having the maximum variance value is determined as the optimum frame.

  In step 403, the optimal frame candidate selected in step 401 and the optimal frame selected in step 402 are output to the inter-frame correspondence line data input processing unit.

  FIG. 4 summarizes the process flow described above. In step 100, video image data to be used is read. This processing is performed by the video image input processing unit 14. Next, measurement target line data is set (101) and reference line data is set (102). Step 101 is performed by the measurement target line data input processing unit 15, and step 102 is performed by the reference line data input processing unit 16.

  Based on these inputs, an inter-frame corresponding point estimation process (103) is performed by the inter-frame corresponding line estimation processing unit 22. Based on the processing result, the optimum frame selection processing (104) is performed by the optimum frame selection processing unit 24. For the optimal frame candidate selected here, the inter-frame corresponding line data input processing unit 23 presents the corresponding line estimation processing result, and if there is an error in the inter-frame corresponding line (105), the inter-frame corresponding point correction input Processing is performed (106).

  The processing from step 103 to step 106 is repeated until there is no error. If there is no error, the three-dimensional distance calculation processing unit 25 performs the three-dimensional distance calculation processing using the optimum frame set by the optimum frame selection processing unit 24 (107), and the calculation result is registered (108). .

  INDUSTRIAL APPLICABILITY The present invention provides a simple measurement method in building work and measurement of product dimensional inspection, and can be widely used.

The functional block diagram by one Example of the image measurement system of this invention. The hardware block diagram of an image measurement system. Explanatory drawing which shows the example of image frame data. The flowchart of the process by one Example of the image measuring method of this invention. Explanatory drawing which shows the example of the input interface of a measurement object line. Explanatory drawing which shows the example of the input interface of a dimension reference line. Explanatory drawing which shows the example of the optical flow of a corresponding point. Explanatory drawing which shows the example of the corresponding line input interface between frames. The flowchart which shows the procedure of a definition of N vector. The flowchart which shows a three-dimensional data production | generation process. The flowchart which shows the process of a translation vector and a rotation matrix. The flowchart which shows the selection process of the optimal frame. Explanatory drawing of the selection process of the optimal frame candidate. Explanatory drawing of the optimal frame selection process.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Image data initial setting part, 2 ... Three-dimensional distance calculation part, 14 ... Video image input process part, 15 ... Measurement object line input process part, 16 ... Reference line data input process part, 22 ... Inter-frame corresponding line estimation process 23: Inter-frame correspondence line data input processing unit, 24 ... Optimal frame selection processing unit, 25 ... Three-dimensional distance calculation processing unit, 26 ... Distance calculation result storage unit, 30 ... Video image input device, 31 ... Magnetic disk device 34 ... Display, 35 ... Memory, 36 ... CPU.

Claims (7)

In an image measurement method for analyzing image data of a plurality of image frames and measuring a target distance in an image,
A predetermined frame including a measurement target portion is selected from the plurality of image frames, the measurement target portion and a reference distance portion are set on the same screen, a reference distance is input, and measurement target portions in other image frames are automatically set. An image measurement method characterized by calculating an object distance by estimating by tracking. In an image measurement method for analyzing image data of a plurality of image frames and measuring a target distance in an image,
Read moving image data consisting of time-series image frames, select a predetermined frame including the measurement target portion, set the coordinates of the measurement line target portion and the reference distance portion in the predetermined frame and input the reference line data, Corresponding points in other image frames with respect to the coordinates set in the measurement line target part are estimated by automatically tracking the change in the position of the feature points in the image between the image frames. And measuring the three-dimensional distance on the optimum frame.   3. The image measurement method according to claim 2, wherein an image frame having a large parallel movement or center of gravity movement is selected as the optimum frame.   4. The image measurement method according to claim 2, wherein if there is an error in the corresponding point as a result of the selection of the optimum frame, the corresponding point is estimated again between the frames by a corresponding point correction input. In an image measurement apparatus that analyzes image data of a plurality of image frames and measures a target distance in an image,
An interface for setting a measurement target portion and a reference distance portion in a predetermined image frame on the same screen and inputting the reference distance, and a position in the image of a feature point between image frames for the measurement target portion in another frame image An inter-frame correspondence line estimation processing unit that estimates by tracking a change in data, and an optimal frame selection processing unit that selects a frame with a large change in the center of gravity of the tracked measurement target portion and the reference distance portion by optimal calculation. An image measuring apparatus comprising a three-dimensional distance calculation processing unit for calculating a distance on the optimum frame.   6. The confirmation interface according to claim 5, wherein a confirmation interface for confirming a result of tracking the measurement target portion and the reference distance portion of another frame corresponding to the measurement target portion and the reference distance portion in the image frame that is initially set using the interface is provided. An image measuring apparatus characterized by that. The image measurement apparatus according to claim 6, wherein the confirmation interface displays and outputs a display frame for confirming the optimum frame.

Images