JP2005140623A - Image measuring method and instrument, program, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To calculate an actual dimension by setting only a measuring range on one image, when measuring the actual dimension of an object within the image, using stereoscopic view. <P>SOLUTION: This image measuring instrument has a photographing information calculating part 200 for calculating a projection matrix when photographing two images photographed with a measuring object and a reference object at the same time, and a matrix for imparting a restriction relation between the two images, an objective area determination part 300 for receiving a two-dimensional rectangular assignment surrounding an objective area to be stored, in one of the images, for calculating a coupling coefficient when sides of the rectangle are expressed by linear coupling of an image of three sides crossed in one apex in an image of the reference object, and for correcting a shape of the rectangle to store apex positions thereof, a corresponding objective area determination part 400 for calculating an area corresponding to the rectangle as a corresponding objective area, in the other image, to store apex positions thereof, and an objective area actual dimension calculating part 500 for calculating three-dimensional coordinates of the apexes of the objective area, using the projection matrix, the apex positions in the objective area and the apex positions in the corresponding objective area. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention uses a plurality of images obtained by simultaneously capturing an object whose actual size and shape are known (reference object) and an object whose actual size and shape are unknown (measurement target object). The present invention relates to an image measurement method and apparatus for calculating an actual size.

  Measuring the actual size of a subject using a photographed image is widely performed in the field of so-called photogrammetry. In recent years, the accuracy of digital image capturing apparatuses typified by digital cameras has improved, and the capacity of recording media for recording data (digital images) captured by digital image capturing apparatuses has increased. The number of vertical and horizontal pixels (pixels), that is, the resolution is increased, whereby digital images have an image quality comparable to analog images taken by a film camera (silver salt camera). Since digital images can be easily processed by computers, with the development of these digital image photographing devices, it has become mainstream to use digital image photographing devices as photographing means for photogrammetry.

  Actual-scale surveys using images are usually performed by the triangulation principle of restoring the distance from the viewpoint at the time of shooting by applying stereo vision. It is described in many references.

  Hereinafter, the digital image capturing device is simply referred to as a camera or the like, and an image captured by the digital image capturing device is referred to as a digital image or simply an image. Also, the 3D Cartesian coordinate system set in the real world to be surveyed is called the world coordinate system or the world coordinate system.

And so on. A three-dimensional Cartesian coordinate system having the origin at the photographing viewpoint position (camera center) is called a camera coordinate system. Also, a two-dimensional Cartesian coordinate system on a captured and recorded digital image is called an image coordinate system, and a two-dimensional point in the image coordinate system is called

And so on. In the following description, for simplification of description, various symbols are described in italic bold unless otherwise confused.

May be described simply in a typeface (for example, m, m ′).

FIG. 1 shows the principle of stereo vision. In the figure, two images used for stereo vision are represented by I and I ', respectively, and O and O' are the camera centers of the images I and I ', respectively. O _w is the origin of the world coordinate system,

Is three-dimensional in the world coordinate system, m and m 'are in I and I'

It is a statue of. Also,

Is an epipole.

  Here, as shown in FIG.

If the images are taken in different images, they appear in each image

Is called a corresponding point. That is, in FIG. 1, the corresponding point on the image I ′ of the point m on the image I is m ′, and conversely, the corresponding point on the image I of m ′ is m. Of course, if there are a large number of portions that are captured in duplicate in a plurality of images, the number of corresponding points increases accordingly.

  Now, the principle of surveying by stereo is as follows. If the corresponding point of the reference object on the two images taken by changing the center position of the camera, that is, the image coordinate value of the position where the same position in the world coordinate system is reflected in each image is known, A projection matrix of the camera that has taken the image, that is, a matrix for calculating to which point in the image a certain point in the world coordinates is projected is known. The number of corresponding points required varies depending on what camera model is assumed. At this time, if only the pixel coordinate values of the necessary corresponding points are known, the original three-dimensional coordinate values of the subject can be restored only on a relative scale.

  However, if the three-dimensional coordinate value in the world coordinate system of the actual three-dimensional world point where the necessary corresponding point is copied and the image coordinate value in each image of the pair of corresponding points are known, that is,

If is known, a projection matrix including a scale factor can be obtained. This set of the three-dimensional coordinate value in the world coordinate system and the image coordinate value in each image of the set of corresponding points is represented by the following symbol:

Alternatively, it is simply expressed as S.

  How many Ss should be known depends on what camera model is assumed. For example, assuming the affine camera described in Non-Patent Document 2 and Non-Patent Document 3, if four different Ss are known, the projection matrix of two images can be known. This situation will be described below.

Is a three-dimensional point in the world coordinate system, and m is a two-dimensional point on the actually photographed digital image, generally, the point m = [u, v] ^T on the photographed image and the world coordinates Point in space

The following relationship holds between Where A ^T is a symbol representing the transpose of a matrix or vector A:

Here, s is a scale factor, and P is a 3 × 4 projection matrix.

Assuming an affine camera, the unknown component of the matrix P is represented by p _ij (i = 1, 2; j = 1, 2, 3, 4), as shown in Non-Patent Document 2 and Non-Patent Document 3. In general, equation (1) can be simplified as follows:

Accordingly, the projection matrices of the affine camera that captured the two images I and I ′ are P and P ′, and the unknown components thereof are p _ij and p ′ _ij (i = 1, 2; j = 1, 2, 3, 4) and a point in world coordinates

Is projected to corresponding points m = [u, v] ^T , m ′ = [u ′, v ′] ^T on the images I and I ′, the following equation holds:

  Therefore, when obtaining a projection matrix assuming an affine camera, there are 16 unknowns in the matrix on the right side of equation (4), and there are four equations for each three-dimensional world coordinate and corresponding point pair S on the image. As it comes out, you only need to know 4 sets of information.

  Therefore, once the projection matrices P and P ′ are obtained, unknown points in world coordinates

When there was

By finding the points m and m 'that are projected onto each image,

Can be requested. If this is used, the actual size of the object being photographed can be obtained by obtaining the position of the point to be measured on the object photographed in the two images on the two images and determining the distance in the world coordinates. Will be understood.

  However, this method has a problem in that it is necessary to determine where on the image the point of the object to be measured in the world coordinate system is projected over at least two images. In addition, even if an image of a measurement target point is designated in one image I and a corresponding point is obtained by searching for a corresponding point on the other image I ′, an accurate result cannot be obtained if an incorrect correspondence occurs. There was a problem.

  In the conventional technique, a technique for improving the measurement accuracy by accurately obtaining the corresponding points is disclosed. For example, Patent Document 1 discloses a method of assigning corresponding points by parallelizing two images. Japanese Patent Application Laid-Open No. 2004-228561 discloses a technique of searching while updating the search window position in order to obtain the corresponding position of the marker position in images taken by a plurality of cameras.

However, both Patent Document 1 and Patent Document 2 have a problem that the range to be measured cannot be specified intuitively.
JP 2003-185434 A JP 2001-141418 A Toni Scenk, translated by Toshiharu Murai and Hirofumi Chitsutsu, “Digital Photogrammetry”, published by Japan Surveying Association, 2002 Gang Xu and Zhengyon Zhang, "Epipolar Geometry in Stereo, Motion and Object Recognition: A Unified Approach", Kluwer Academic Publishers, 1996 Sato, "Computer Vision-Geometry of Vision", published by Corona, 1999

  According to the present invention, when measuring the size of an object photographed in an image, if a measurement range is set as a rectangle (that is, a rectangle) on one image, a corresponding point on another image is set. An image measurement method and apparatus capable of comparing the actual size by obtaining the coordinate value in the world coordinate system of the measurement range without specifying and calculating the actual size and combining the measurement range with another image. It is to provide.

The image measurement method of the present invention includes:
A rectangular area surrounding the measurement object from at least two images of the measurement object and a reference object forming a cube, a rectangular parallelepiped, or a parallelepiped whose length of three sides forming at least one vertex is known. An image measurement method for calculating a three-dimensional coordinate value of a vertex forming a castle,
Inputting at least two images I and I ′ in which a measurement object and a reference object are simultaneously photographed;
A projection matrix assuming an affine camera at the time of photographing two images I and I ′ by acquiring images of four vertices of the reference object image in the input two images I and I ′, and Calculating and storing a constraint matrix that provides a constraint relationship between the two images I and I ′;
A region to be subjected to actual size measurement is set as a target region M, and a designation of a two-dimensional rectangle on the screen surrounding the target region M is received and stored in one image I, and the sides of the rectangle are stored in the image of the reference object. Calculating a coupling coefficient when represented by a linear combination of three side images intersecting at one vertex, correcting the shape of the rectangle, and storing the vertex position on the rectangular screen;
In the other image I ′, a corresponding target area M ′ is calculated using the previously designated rectangular corresponding area as the corresponding target area M ′, and the vertex position on the screen of the corresponding target area M ′ is stored. ,
Using the calculated projection matrix, the vertex position in the target area M, and the vertex position in the corresponding target area M ′ to calculate the three-dimensional world coordinates of the vertices of the target area M.

  In the present invention, when calculating a projection matrix or a constraint matrix, it is preferable to acquire information on the actual size and shape of the reference object.

  In the present invention, using the target region vertex position on the image I or I ′ and its three-dimensional world coordinates, it can be combined with another image or a three-dimensional scene and displayed.

  Further, in the present invention, the step of storing the vertex position on the screen of the rectangular area as the corresponding target area is performed by using, for example, the calculated coupling coefficient of three sides that intersect at one vertex of the reference object image. Using the step of calculating and evaluating the vertex position on the screen of the corresponding target area by linear combination of the images, and using the matrix that gives the constraint between the calculated two screens, the other image I ′ of the vertex of the target area A step of calculating and evaluating the corresponding position, a feature point obtained from the luminance information in the target region M in one image I, and a feature point obtained from the luminance information in the corresponding target region M ′ in the other image I ′ And calculating the degree of similarity between them, and determining the corresponding target region M ′ while changing the position and shape of the corresponding target region M ′ based on the evaluation values obtained in the respective evaluating steps. .

  Further, in the present invention, after determining the corresponding target area M ′ in the other image I ′ of the target area M specified in one image I, the corresponding target area M ′ is regarded as a new target area, and again A new corresponding target area in the image I may be obtained.

The image measuring apparatus according to the present invention includes at least two images of a measurement object and a reference object having a cube, a rectangular parallelepiped, or a parallelepiped whose length of three sides forming at least one vertex is known. From the image measuring device for calculating the three-dimensional coordinate value of the vertex forming the rectangular castle surrounding the measurement object,
Image input means for inputting, storing and / or displaying at least two images of the measurement object and the reference object taken simultaneously;
By acquiring the images of the four vertices of the reference object image in the two input images, a projection matrix assuming an affine camera and the constraint between the two images at the time of photographing the two images Photographing information calculation means for calculating and storing a matrix for giving a relationship;
With the target area of the actual size measurement as the target area, in one of the two images, the designation of a two-dimensional rectangle on the screen surrounding the target area is accepted and stored, and the sides of the rectangle are The target area for storing the vertex position on the rectangular screen by calculating the coupling coefficient when the image of the reference object is represented by the linear combination of the three side images that intersect at one vertex. A determination means;
Corresponding target area determining means for calculating the area corresponding to the rectangle in the target area determining means as the corresponding target area in the other of the two images and storing the vertex position on the screen of the corresponding target area; ,
A target area actual size calculating means for calculating the three-dimensional coordinates of the vertices of the target area using the projection matrix, the vertex position in the target area, and the vertex position in the corresponding target area;
Have

  The present invention has an effect that the actual size of the range can be obtained as long as the range of the measurement object captured in the image is specified by a rectangle only with one image.

  Next, a preferred embodiment of the present invention will be described with reference to the drawings. FIG. 2 is a block diagram showing the entire measuring apparatus according to the embodiment of the present invention. FIG. 3 is a block diagram showing details of the measuring apparatus.

  The measuring device is a measuring object based on a measuring object and at least two images of a cube, a rectangular parallelepiped, or a parallelepiped reference object whose lengths of three sides forming at least one vertex are known. 3D coordinate values of vertices forming a rectangular castle (rectangular region) surrounding the image, an image input unit 100, an imaging information calculation unit 200, a target region determination unit 300, and a corresponding target region determination unit 400, a target area actual size calculation unit 500, and a target area display unit 600.

  The image input unit 100 includes an image reading unit 101, an image storage unit 102, and an image display unit 103, and includes at least two images of a measurement object and a reference object that forms a cube, a rectangular parallelepiped, or a parallelepiped at the same time. Enter an image and store or display it. The imaging information calculation unit 200 includes a reference object information input unit 201, a reference object extraction unit 202, a corresponding point acquisition unit 203, a correspondence calculation / storage unit 204, and a projection matrix calculation / storage unit 205. By acquiring the actual size and shape information, and acquiring the images of the four vertices of the reference object image in the input two images (one image and the other image), Then, a projection matrix assuming an affine camera and a matrix giving a constraint relationship between two images are calculated and stored. The target region determining unit 300 includes a base vector deriving unit 301, a target region selecting unit 302, a coupling coefficient deriving unit 303, a region deforming unit 304, and a region position storage unit 305. Accepts and stores a specification of a two-dimensional rectangle on the screen that encloses a region (this is called a corresponding region), and the side of the rectangle is an image of three sides that intersect at one vertex of the reference object image. The coupling coefficient when expressed by linear coupling is calculated, the shape of the rectangle is corrected, and the vertex position on the rectangular screen is stored.

  The correspondence target region determination unit 400 includes a region shape calculation unit 401, a region position calculation unit 402, a feature amount calculation unit 403, and a corresponding region movement / evaluation unit 404. In the other image, the previously specified rectangle An area corresponding to (this is called a corresponding target area) is calculated, and a vertex position on the screen of the corresponding target area is stored. In particular, the region shape calculation unit 401 uses the calculated coupling coefficient to determine the vertex position on the screen of the corresponding target region by linear combination of three side images that intersect at one vertex of the image of the reference object. Calculate and evaluate. The region position calculation unit 402 calculates and evaluates the corresponding position of the vertex of the target region in another image using the calculated matrix that gives the constraint between the two screens. The feature amount calculation unit 403 calculates and evaluates the similarity between the feature point obtained from the luminance information in the target region in one image and the feature point obtained from the luminance information in the corresponding target region in the other image. To do. Then, the corresponding region movement / evaluation unit 404 determines the corresponding target region while changing the corresponding target region position and shape based on the evaluation values from the region shape calculating unit 401, the region position calculating unit 402, and the feature amount calculating unit 403. To do.

  The target area actual size calculation unit 500 includes a world coordinate calculation unit 501 and a relative coordinate calculation unit 502. The calculated projection matrix, the vertex position of the target area calculated and stored by the target area determination unit 300, and Using the vertex position of the corresponding target area calculated and stored by the corresponding target area determining unit 400, the three-dimensional coordinates of the vertex of the target area are calculated. The target area display unit 600 includes a texture object generation unit 601, a polygon generation unit 602, an image generation unit 603, and a display unit 604. The input image, the vertex position on the screen of the target area, Using three-dimensional coordinates, the image data in the target area is displayed by being combined with another image or a three-dimensional scene.

  Next, a process for calculating the three-dimensional coordinate values of the vertices forming a rectangular castle surrounding the measurement object from at least two images of the measurement object and the reference object simultaneously photographed using the measurement apparatus will be described. Here, the reference object is a cube, a rectangular parallelepiped or a parallelepiped whose lengths of three sides forming at least one vertex are known.

  First, the image reading unit 101 of the image input unit 100 inputs at least two images obtained by photographing the measurement target and the reference object. At this time, in at least two images, it is desirable that the camera center of the camera to be photographed is moved, and it is desirable that three sides passing through one vertex of the reference object appear on the image. Further, it is desirable that the images of these three sides have a length of several pixels or more on the image. Further, it is desirable that one of the three sides is in the vertical direction in the real world, and the other two sides are perpendicular to the side in the vertical direction. In the following description, it is assumed that three sides passing through one vertex of the reference object satisfy the above condition. The two non-vertical sides are preferably perpendicular to each other, but may not be perpendicular as long as they are not on the same straight line. Hereinafter, the number of images is two, and the images are I and I ′.

  Next, the image storage unit 102 of the image input unit 100 stores the image read by the image reading unit 101 and the image display unit 103 displays the image. The display of the image is for receiving a rectangular input by the user in a target area selection unit 302 of the target area determination unit 300 described later.

When the input and storage of the image in the image input unit 100 are completed as described above, the imaging information calculation unit 200 is activated next, and the reference object information input unit 201 inputs the reference object previously input. The actual size and shape of the above three sides are acquired. Hereinafter, the three-dimensional world coordinates of the vertex where the three sides of the reference object intersect are X ₀ , and the three-dimensional world coordinates of the vertex of the reference object forming the side corresponding to the vertically downward direction subtracted from X ₀ are X ₂ . In the world coordinate system, when X _o is regarded as the origin, X ₀ X ₂ is regarded as the downward Y axis, and the remaining two sides passing through X ₀ are regarded as the X axis and the Z axis, these X axis, Y The X and Z axes are taken so that the axis and the Z axis form a right-handed system. Of the remaining two sides of the reference object, the side forming the X axis is X ₀ X ₁ and the side forming the Z axis is X _0. and X _3. In the present invention, since it may be for where the origin of the world coordinate system, the X _o as the origin as described above, X ₀ X axis X _1, Y axis _{X 0 X 2, X 0 X} 3 and Z-axis Let's say.

The reference object information input unit 201 acquires the lengths X ₀ X ₁ , X ₀ X ₂ , and X ₀ X ₃ . This length may be input manually by the user, or may be set in advance as a known one. For example, a template model in a database server connected to a network is compared with a reference object, and the length is verified. You may make it acquire the length in a template model according to a result.

  Next, the reference object extraction unit 202 extracts images of four vertices forming three sides of the reference object in the images I and I ′. That is, the coordinate values of the four vertex images in the two images are obtained from the two-dimensional coordinate values set in the respective images. This may be specified by manual input while displaying these images on the screen by the user, for example, and a model that has been registered in advance by extracting edges by Hough transform assuming that the shape of the reference object is known. Template matching may be performed. When the images of the four vertices are extracted, the corresponding point acquisition unit 203 associates the images in the four vertex images I and I ′. The extraction of the vertex image in the reference object extraction unit 202 and the association in the corresponding point acquisition unit 203 may be simultaneously performed by manual input as described above. A known method, for example, block matching or optical flow may be used for association in the corresponding point acquisition unit 203.

Hereinafter, the images in the image I of the four vertices X ₀ , X ₁ , X ₂ , X ₃ of the reference object are respectively shown.

And the images in image I '

far. For example, as shown in FIG.

Is an example. FIG. 4 is a diagram illustrating a search for a corresponding area. 4A shows one image I, and FIG. 4B shows the other image I ′.

Are the images of the vertices of the reference objects in the images I and I ′, respectively, and m _i and m ′ _i (i = 0, 1, 2, 3) represent the vertices of the rectangular area. Also

Respectively show epipolar lines in the image I 'of the vertex m _i.

  Next, the correspondence calculation / storage unit 204

From the coordinate values, a constraint matrix is obtained using a constraint relationship established between the images I and I ′.

  In general, in two images having different camera centers at the time of shooting, the corresponding points, that is, the images on the respective images of the same three-dimensional world coordinate points are denoted by m and m ′, respectively, and the homogeneous coordinates of m and m ′. Each notation

Then,

The constraint equation holds. This is called the generalized epipolar equation,

Is a matrix of 3 rows and 3 columns, and is called a Fundamental Matrix or F matrix. As described in Non-Patent Documents 1 to 3, assuming an affine camera, if four sets of corresponding points are known, this F It is known that matrix components can be calculated.

  In addition, the corresponding point in the other image I ′ of the point m on one image I is a straight line on the image I ′ that satisfies the following equation:

It is known that the constraint condition of being above is also satisfied.

  Straight line like this

Is generally called an epipolar line.

  If internal parameters such as camera focal length are known,

Can be obtained instead of an essential matrix or an E matrix, but in the present invention, it is called a constraint matrix without distinction.

It will be expressed as That is, the correspondence calculation / storage unit 204

Using the constraint matrix

Will be remembered.

  Next, the projection matrix calculation / storage unit 205

Is used to calculate and store the affine projection matrix P at the time of shooting one image I and the affine projection matrix P ′ at the time of shooting the other image I ′.

  Next, details of the target area determination unit 300 will be described. The target area determination unit 300 determines rectangular vertex coordinate values surrounding a portion to be measured in the image I.

  First, the basis vector deriving unit 301

From the world coordinate system

The image of

Ask for.

On the other hand, the target area selection unit 302 sets an area to be measured. At this time, a rectangular area on one image I may be selected by an input from the user, or a rectangular area surrounding a certain target may be selected. The vertices of the area set in this way are hereinafter referred to as m _i (i = 0, 1, 2, 3). An example of the region surrounding the person image in FIG. In the present embodiment, corresponding points m ′ _i (i = 0, 1, 2, 3) in the other image I ′ of m _i (i = 0, 1, 2, 3) constituting this rectangular area are obtained. The main feature is that it can.

First, let M _i (i = 0, 1, 2, 3) be the coordinate values of m _i (i = 0, 1, 2, 3) in the three-dimensional world coordinate system. In the three-dimensional world coordinate system, the rectangle M _i (i = 0, 1, 2, 3) is a vertical side parallel to X ₀ X ₂ forming the Y axis of the reference object (m ₁ m _{2 in} FIG. 4A). and m ₃ m ₀ and sides) formed by the world coordinate system, ZX plane of the reference object, namely X ₀ X ₁ and X ₀ m of X ₃ and parallel to the plane spanned by the horizontal side (FIG. 4 (B) ₀ Assume that m ₁ and m ₂ m ₃ form an edge in the world coordinate system.

If the Y axis of the reference object is parallel to the vertical line of the 3D world coordinate system and the ZX plane of the reference object is parallel to the horizontal plane of the 3D world coordinate system, then the rectangle M _i (i = 0, 1, 2, 3) represents a plane represented by a rectangle perpendicular to the horizontal plane in the three-dimensional world coordinate system.

  In the affine camera, a set of straight lines parallel to the world coordinate system is projected onto a set of parallel straight lines on the image. Therefore, a straight line parallel to the horizontal plane in the world coordinate system is projected on the horizontal plane on the image. Parallel to So, for example,

Can be expressed as an image on one image I by the affine camera.

Similarly, the image to the other image I 'by the affine camera is also

It becomes.

When the derivation of the base vector in the base vector deriving unit 301 and the selection of the target region in the target region selecting unit 302 are completed, the coupling coefficient calculating unit 303 obtains α _x and α _z using the equation (8). At this stage

Are known, since they are known.

Next, since the region deforming unit 304 has the vertical side of the rectangle M _i (i = 0, 1, 2, 3) parallel to X ₀ X ₂ of the reference object, m _i (i = 0, 1, 2, 3),

DOO moves the m _{i (i} = 0,1,2,3) so as to be parallel. At that time, the two points m _i (i = 0, 1, 2, 3) may be moved while maintaining the distance of the vertical side.

After that, the region position storage unit 305 changes m _i (i = 0, 1, 2, 3) after deformation to m _i (i = 0, 1, 2, 3) and stores it in the memory. As described above, the vertex coordinate values m _i (i = 0, 1, 2, 3) of the rectangular area in I are determined, and the processing in the target area determining unit 300 ends.

Next, the corresponding target area determining unit 400 determines the corresponding points m ′ _i (i = 0, 1, 2, 3) of m _i (i = 0, 1, 2, 3) in the other image I ′. . Processing in the corresponding target area determination unit 400 will be described.

First, the region shape calculating unit 401 uses the coupling coefficients α _x and α _z obtained by the coupling constant deriving unit 303 of the target region determining unit 300 to calculate the horizontal m ′ _i (i = 0, 1, 2, 3). Calculate the direction vector of the side, and further calculate the direction vector of the vertical side.

And From this direction vector, the shape of m ′ _i (i = 0, 1, 2, 3) is obtained. However, it is impossible to determine at which position on the image m ′ _i exists only by the calculation by the region shape calculation unit 401. On the other hand, the region position calculation unit 402 is a constraint matrix related to epipolar lines.

Thus, the corresponding point in the other image I ′ of m _i (i = 0, 1, 2, 3) is

Using the above, the position of m ′ _i (i = 0, 1, 2, 3) is determined. However, it is impossible to determine which point on the epipolar line is m ′ _i only by the calculation in the region position calculation unit 402. If the region shape calculation unit 401 and the region position calculation unit 402 are used, the shape of m ′ _i and the side length can be determined, but a unique position cannot be determined. At this stage, since any corresponding point of one point may Motomare of m _i, for example, by optical flow discovery and block matching method, the corresponding point of any one point m _i on the other image I ' It can be considered that m ′ _i is uniquely determined. However, if all m _i are within an area where there is no change in luminance, for example, such a corresponding point cannot be searched.

Therefore, in the feature amount calculation unit 403, a certain amount in an area surrounded by m _i (i = 0, 1, 2, 3) of one image I and m ′ _i (i = 0) of the other image I ′. , 1, 2, 3) and a certain amount in the region surrounded by Specifically, for example, some special points in the area surrounded by m _i of the image I are obtained as corners based on luminance values, for example, and each characteristic point is obtained by, for example, a vertex m _n (n∈ with a rectangle m _i ). {0, 1, 2, 3}) is used as a starting point, and the horizontal and vertical sides of the rectangle are represented by a linear combination thereof as a basis vector, and the average position (centroid position) of the coordinates of the feature points is calculated. Similarly, a corresponding point in the region surrounded by m ′ _i of the other image I ′ of the feature point is obtained by, for example, a block matching method or an optical flow search, and m ′ _n corresponding to m _n as the origin is obtained. A method is used in which the average position (centroid position) of the coordinates of the corresponding feature point is calculated using the vertical and horizontal sides of m ′ _i as basis vectors, and the coupling coefficients of the two are compared.

The surrounded by the area in m _i and m _'i, since the image of the subject to be measured is captured, feature points such as corners described above greatly possibly present, corresponding points m _i mentioned above It is easier to process than the method of searching.

Next, the corresponding area movement / evaluation unit 404 selects the apex with m ′ _i as an epipolar line.

While moving above, the processing of the region shape calculation unit 401, the region position calculation unit 402, and the feature amount calculation unit 403 is repeated, and the region shape calculation unit 401, the region position calculation unit 402, and the feature amount calculation unit 403 If the calculated amount is smaller than a certain threshold value, the process is terminated. Otherwise, the vertex having m ′ _i is again connected to the epipolar line.

Move these up to calculate these quantities again.

As the amount to be calculated, for example, in the region shape calculation unit 401, a function for evaluating the difference between the coupling coefficients α ′ _x and α ′ _z calculated from the current m ′ _i and α _x and α _z.

May be used, or the positive square root thereof may be used.

  In addition, as an evaluation function in the region position calculation unit 402,

The [delta] _i Distant distance between, the function representing the degree of restraint to the epipolar line of the corresponding point

May be used.

  Further, as an evaluation function in the feature quantity calculation unit 403, the coupling coefficients of the centroid positions of the feature points in the images I and I ′ are set as (μ, ν) and (μ ′, ν ′),

May be used.

  Also, the weighted sum of these evaluation functions is

Alternatively, the determination may be made by providing a threshold value ε as a criterion for determining the end of processing.

Note that m ′ _i obtained in this way may be newly regarded as a target region, and the same processing may be repeated for one image I with m _i as an initial value.

Once m ′ _i is obtained by the corresponding target area determination unit 400, the target area actual size calculation unit 500 next obtains a three-dimensional world coordinate value of M ′ _i . Hereinafter, processing in the target area actual size calculation unit 500 will be described.

The world coordinate calculation unit 501 uses the vertices m _i , m ′ _i (i = 0, 1, 2, 3) and the projection matrices P, P ′ that have already been obtained to calculate the three-dimensional M _i by the method described in the background art. Find the world coordinate value. M _i represents a rectangle (rectangle) surrounding the target object in the three-dimensional world coordinates, and if the rectangle is the minimum size surrounding the target object, by obtaining the lengths of the vertical and horizontal sides of the rectangle, The vertical and horizontal sizes of the target object can be obtained.

When the three-dimensional world coordinate value of M _i is obtained, the relative coordinate calculation unit 502 sets M _i to an appropriate relative coordinate value. Since M _i is a coordinate value with X ₀ of the reference object as the origin, it is converted into a relative coordinate value by a method such as expressing M ₀ as a starting point and expressing other points as position vectors from M _0. Is done. Of course, the calculation of relative coordinate values may be omitted. With the above processing, only by setting a rectangular area on one image I, the world coordinate value of the vertex of the rectangular area can be calculated.

  Since the actual size of the rectangular area in the image was found in this way, next, the target area display unit 600 generates the rectangular area of the image as a three-dimensional polygon defined by the actual size, and Texture mapping of pixels in the rectangular area of the image with respect to the polygon is performed. Further, by simultaneously displaying the polygons texture-mapped for other target objects generated in the same manner in a scene having the same field of view information, it is possible to compare the size information of each other. Hereinafter, processing in the target area display unit 600 will be described.

The texture object generation unit 601 obtains pixel data of an area surrounded by m _i (i = 0, 1, 2, 3) of I from one image I and m _i (i = 0, 1, 2, 3). . Of course, data obtained from the other images I ′ and m ′ _i (i = 0, 1, 2, 3) may be used. Then, the polygon generation unit 602 generates a quadrilateral polygon having a vertex M _i (i = 0, 1, 2, 3). Here, the pixel data generated by the texture object generation unit 601 may be texture-mapped to the polygon generated by the polygon generation unit 602, or the shape vertex _ap ( _i ) of the pixel data generated by the texture object generation unit 601. Since the position M _i (i = 0, 1, 2, 3) in the world coordinate system of i = 0, 1, 2, 3) is known, the polygon may not be generated and projected. The image generation unit 603 generates one or a plurality of objects generated in this way as a plane object in a scene having certain visual field information. Thereafter, the display unit 604 displays the composite scene generated by the image generation unit 603 on the screen.

  The measurement apparatus according to the preferred embodiment of the present invention has been described above. The above-described measuring device can be configured as a dedicated hardware device, and is also realized by reading a computer program for realizing it into a computer such as a personal computer or a workstation and executing the program. it can. A program for performing image measurement is read into a computer by a recording medium such as a magnetic tape or a CD-ROM, or via a network. Such a computer generally has a CPU, an external storage device such as a hard disk device for storing programs and data, a main memory, an input device such as a keyboard, a mouse, and a digital camera, a CRT, a liquid crystal display, and the like. Display device and a reading device for reading a recording medium such as a magnetic tape or a CD-ROM. The external storage device, main memory, input device, display device, and reading device are all connected to the central processing unit. This computer loads a recording medium storing a program for performing the above-described measurement into a reading device, reads the program from the recording medium and stores it in an external storage device, or downloads such a program via a network. When the CPU executes the program stored in the external storage device and then stored in the external storage device, it functions as the above-described measurement device.

  Hereinafter, processing steps in the computer program when the measuring apparatus of the present embodiment is realized as a computer program will be described with reference to FIG. FIG. 5 shows the flow of processing in such a computer program, where

Is the F matrix between the images I and I ′,

Is a three-axis image of the reference object in the images I and I ′, and [α _x , α _z ] is a linear coupling coefficient of the sides of the target region rectangle expressed by the axis of the reference object,

Are the world coordinate values of the target area rectangle and the images in the images I and I ′, respectively.

  In the figure, SPxxx indicates steps implemented as a group of program blocks in such a computer program. Here, the numbers in the part xxx are associated with the reference numerals of the respective processing units in the measuring apparatus shown in FIG. For example, the SP 202 is a program block corresponding to the processing in the reference object extraction unit 202 in the above description.

  First, in SP101, an image in which the reference object and the target object are simultaneously captured is read. In the present embodiment, it is assumed that the reference object has a rectangular parallelepiped shape and is placed on a horizontal plane. Further, it is assumed that the reference object and the target object are stationary, and two images taken by changing the camera position are read. At this time, camera parameters, that is, zoom magnification and the like may not be fixed. This is because in the present invention, the camera does not need to be calibrated between two captured images. Next, in SP1, the read image is stored as digital data. At that time, it may be stored simultaneously with some identifier (for example, file name) for identifying the image. In this description, for the sake of simplicity, all steps related to storage are represented by SP1. Of course, all the steps related to storage in each stage may be independent steps. In parallel with image storage, the image is displayed in SP103.

  After displaying the image, in SP202, images of three sides of the reference object that are orthogonal to each other and intersect at one point are extracted from the two images. This can be done by selecting the position on the screen with a mouse or the like on the screen, or selecting on one screen by performing block matching or optical flow search processing between the two images. You may make it obtain | require as a corresponding point in the other screen. Thereafter, in SP203, the four vertices are associated. This may be associated by the user selecting points in order, or may be associated again after selecting the points.

After associating the four vertices, in SP204, F matrices of two images are calculated from the four associated vertices and stored in SP1. Further, the reference object information is acquired in SP 201 by, for example, a template, manual input by a user, initial setting, or the like. Here, the length of three orthogonal sides in the reference object is obtained. For example, as described above, the three-dimensional coordinate of the intersection of three orthogonal sides is X ₀ , the three-dimensional coordinate of the vertex of the reference object that forms the side corresponding to the vertically downward direction subtracted from X ₀ is X ₂ , and the remaining two sides Are set to X ₁ and X _3, and three sides are set so that X ₀ X ₁ , X ₀ X ₂ , and X ₀ X ₃ form a right-handed system. However, X ₀ X ₂ is the Y axis. Here, assuming that the length of each side is X ₀ X ₁ = d ₁ , X ₀ X ₂ = d ₂ , X ₀ X ₃ = d ₃ , X ₀ = (0,0,0), X ₁ = (d ₁ , 0, 0), X ₂ = (0, d ₂ , 0), and X ₃ = (0, 0, d ₃ ). After SP201, the SP205, X _i and these images

Affine projection matrices P and P ′ are obtained from the above and stored in SP1.

Further, after execution of SP203, as an independent process to the above, in the SP301, X _i image of

From the above, as the basis vector

Ask for.

  In addition, after execution of SP202, as a process independent of the above, in SP302, a minimum rectangular area surrounding a subject to be measured is set for I (one image) which is one of the images displayed in SP202. specify. This process may be performed by receiving input from the user, and the sides of the rectangle may be parallel to the vertical and horizontal sides of the image. An example of the region surrounding the person image in FIG.

  When both the processing of SP301 and SP302 is completed, in SP303, the horizontal side of the rectangular area set in SP302 is displayed.

1 is obtained. For example, in FIG.

The

And real numbers (α _x , α _z ) are obtained and stored in SP1.

  Similar to SP303, in SP304, in FIG.

But keep the length

The positions of m ₀ and m ₁ are corrected so that they are parallel to each other. Of course, the positions of m ₂ and m ₃ may be corrected. In addition, if the subject to be measured protrudes from the rectangle due to this correction, the processing may be repeated by returning to SP302.

Thus, to determine the m _i, is stored by SP1.

Next, the corresponding point m ′ _i (i = 0, 1, 2, 3) of m _i in the other image I ′ is determined using the information obtained by the processing described above.

First, in SP403d, it is determined whether or not the process for determining m ′ _i has ended. If not completed, it is determined in SP403e whether m ′ _i is found in the next image. For example, if only m _i on one image I is not yet determined, Yes is selected, and processing for obtaining m ′ _i on the other image I ′ (SP 401 and SP 402 below) is performed. In addition, m ′ _i has been obtained. However, m ′ _i on the other image I ′ may be set as a known area, and for example, m _i may be obtained on one image I. Alternatively, when there are more than two images, processing for obtaining a corresponding area in the third and subsequent images may be performed. The criteria for making these determinations may be set by the user except that the processing for obtaining m ′ _i on the other image I ′ is performed on m _i on one image I. The program may be set according to the number of input images.

In SP401, the direction vector of the side of m ′ _i is obtained based on the coupling constants α _x and α _z .

If this is the case, the

And α ′ _x = α _x and α ′ _z = _z as initial values. The direction vector of the vertical side is

And

In SP402a in parallel with SP401, epipolar lines corresponding to m _i

Calculate

Next, after the end of SP401 and SP402a, for example, the position of m ′ ₀ is set in SP402b.

Determined appropriately above, a straight line drawn from m ′ _{0 in} the direction of (17)

Let m ' _{1 be} the intersection with, and from m' ₀

A straight line drawn in the direction of

Let m ' _{3 be} the intersection with. The remaining m ′ ₂ may be determined so that m ′ _i (i = 0, 1, 2, 3) forms a parallelogram. After that, for m ′ _i (i = 0, 1, 2, 3) determined in this way,

Calculate

Next, in SP404, first, a feature point in a region surrounded by m _i is found on one image I, and then a corresponding point on the other image I ′ of the feature point is found. As a method for finding the corresponding points, any method such as block matching or optical flow search may be used. Considering only those where corresponding points are found, and _assuming that the number is N, all feature points q _n (n = 1, 2,..., N) on one image I are all

The average position q of all q _n is expressed as

Ask for. Next, similarly for the other image I ′

Asking,

Calculate

  After the feature amount is calculated in this way, the weighted sum of these evaluation functions is then calculated in SP403a.

Calculate as The weights w _i may be equal to w _i = 1/3 or may be arbitrarily ordered. This value is compared with a preset threshold value ε that is a criterion for determining the end of processing. If the value is smaller than the threshold value, the processing ends and the processing returns to SP403d. Otherwise, the processing proceeds to step SP403b.

In SP403b, it is determined whether or not to end the process depending on whether or not the number of iterations exceeds a preset upper limit value (MAX_ITERATE). If the upper limit value is exceeded, the process ends and the process returns to SP403d. If not, the process proceeds to SP403c. In SP403c, for example, the position of m ′ ₀ is

And the number of iterations is incremented by 1, and the process returns to SP402b. The magnitude of the movement may be one pixel, a smaller value, or may be reduced according to the number of repetitions. Further, the direction of movement may be the same direction if the value of E is decreased by the first movement, and may be moved in the opposite direction otherwise.

Since m ′ _i (i = 0, 1, 2, 3) has been calculated in this way, it is determined in SP403d that the corresponding calculation has been completed, and in SP1, m ′ _i (i = 0, 1, 2, 3). Remember.

Next, in SP501, the three-dimensional world coordinate value of M ′ _i is obtained from m _i , m ′ _i (i = 0, 1, 2, 3) and the projection matrices P and P ′. _{i is set} to an appropriate relative coordinate value and stored in SP1. These stored data may be output to an external storage device in SP503.

Then, in SP2, these information, i.e. the image of the identifier, and the rectangular region in the image vertex coordinate values m _i and (i = 0, 1, 2, 3) and M _i are stored memory or the external storage device Read from file. In SPS 601, a texture object of the image is generated, and in SP 602, a three-dimensional Borgon having a vertex M _i is generated. Thereafter, a scene for display, for example, a three-dimensional scene generated by three-dimensional computer graphics, is generated in SP603a, and the texture object generated in SP601 is texture-mapped to the three-dimensional polygon generated by SP602 in SP603b. . When there are polygons to which a plurality of texture objects are mapped, these are arranged at appropriate positions in the scene. Finally, in SP604, the above scene and the texture-mapped polygon are displayed, which makes it possible to compare and display, for example, the size of the subject shot at different shooting sites in the same scene.

It is a figure showing the principle of stereo vision. It is a figure showing the basic composition of the present invention. It is a block diagram showing embodiment of this invention. It is a figure showing search of a corresponding area. It is a figure showing the flow of the process in the Example of this invention.

Explanation of symbols

100 Image Input Unit 200 Imaging Information Calculation Unit 300 Target Area Determination Unit 400 Corresponding Target Area Determination Unit 500 Target Area Actual Size Calculation Unit 600 Target Area Display Unit

Claims (13)

A rectangle surrounding the measurement object from at least two images of a measurement object and a reference object having a cube, a rectangular parallelepiped, or a parallelepiped whose length of three sides forming at least one vertex are known. An image measurement method for calculating a three-dimensional coordinate value of a vertex forming a castle,
Inputting at least two images of the measurement object and the reference object taken simultaneously;
A projection matrix assuming an affine camera and the two images at the time of photographing the two images by acquiring four vertex images of the reference object image in the two input images. Calculating and storing a matrix that gives a constraint relationship between them;
An area to be measured in actual size is set as a target area, and in one of the two images, a two-dimensional rectangle designation on the screen surrounding the target area is received and stored, and the side of the rectangle Is calculated as a linear combination of images of three sides that intersect at one vertex of the image of the reference object, and the rectangular shape is corrected by correcting the shape of the rectangle. Remembering
Calculating an area corresponding to the rectangle as a corresponding target area in the other of the two images, and storing a vertex position on the screen of the corresponding target area;
Calculating the three-dimensional coordinates of the vertices of the target region using the projection matrix, the vertex positions in the target region, and the vertex positions in the corresponding target region;
An image measurement method comprising:   Using at least one of the two images, the vertex position on the screen of the target area, and the three-dimensional coordinates of the vertex position of the target area, the image data in the target area is converted into another image or 3 The image measurement method according to claim 1, further comprising a step of combining and displaying the three-dimensional scene. The step of storing the vertex position on the screen of the rectangular area as a corresponding target area,
Calculating and evaluating a vertex position on the screen of the corresponding target area by linear combination of images of three sides intersecting at one vertex of the image of the reference object using the calculated coupling coefficient; and
Calculating and evaluating the corresponding position in the other image of the vertices of the target area using a matrix that gives a constraint between the calculated two screens;
Calculating and evaluating a similarity between a feature point obtained from the luminance information in the target area in the one image and a feature point obtained from the luminance information in the corresponding target area in the other image; ,
Determining the corresponding target region while changing the position and shape of the corresponding target region based on the evaluation values obtained in the respective evaluating steps;
The image measurement method by the image of Claim 1 or 2 which has these.   After determining the corresponding target area in the other image of the target area specified in the one image, determining the corresponding target area as a new target area and obtaining a new corresponding target area in the one image again The image measurement method according to claim 1, comprising: A rectangle surrounding the measurement object from at least two images of a measurement object and a reference object having a cube, a rectangular parallelepiped, or a parallelepiped whose length of three sides forming at least one vertex are known. An image measuring device for calculating a three-dimensional coordinate value of a vertex forming a castle,
Image input means for inputting, storing and / or displaying at least two images of the measurement object and the reference object taken simultaneously;
A projection matrix assuming an affine camera and the two images at the time of photographing the two images by acquiring four vertex images of the reference object image in the two input images. Photographing information calculation means for calculating and storing a matrix that gives a constraint relationship between
An area to be measured in actual size is set as a target area, and in one of the two images, a two-dimensional rectangle designation on the screen surrounding the target area is received and stored, and the side of the rectangle Is calculated as a linear combination of images of three sides that intersect at one vertex of the image of the reference object, and the rectangular shape is corrected by correcting the shape of the rectangle. Target area determination means for storing
A corresponding target area determining means for calculating an area corresponding to the rectangle as a corresponding target area in the other of the two images, and storing a vertex position on the screen of the corresponding target area;
Using the projection matrix, the vertex position in the target area, and the vertex position in the corresponding target area, target area actual size calculating means for calculating the three-dimensional coordinates of the vertex of the target area;
An image measuring device having   Using at least one of the two images, the vertex position on the screen of the target area, and the three-dimensional coordinates of the vertex position of the target area, the image data in the target area is converted into another image or 3 The image measurement apparatus according to claim 5, further comprising target area display means for combining and displaying the three-dimensional scene. The corresponding target area determining means includes
Region shape calculation for calculating and evaluating a vertex position on the screen of the corresponding target region by linear combination of images of three sides intersecting at one vertex of the image of the reference object using the calculated coupling coefficient Means,
Area position calculating means for calculating and evaluating a corresponding position in the other image of the vertex of the target area using a matrix that gives a constraint between the calculated two screens;
A feature amount for calculating and evaluating a similarity between a feature point obtained from luminance information in the target area in the one image and a feature point obtained from luminance information in the corresponding target area in the other image A calculation means;
A corresponding region moving / evaluating unit that determines the corresponding target region while changing the position and shape of the corresponding target region based on the evaluation values obtained by the region shape calculating unit, the region position calculating unit, and the feature amount calculating unit; ,
An image measuring apparatus using an image according to claim 5 or 6.   Means for determining a corresponding target area in the other image after determining a corresponding target area in the other image of the target area specified in the one image, and obtaining a new corresponding target area in the one image again The image measuring device according to claim 5, comprising: Computer
Input at least two images of a measurement object and a reference object having a cube, a rectangular parallelepiped, or a parallelepiped having a known length of three sides forming at least one vertex, and storing and / or storing them Image input means for displaying,
A projection matrix assuming an affine camera and the two images at the time of photographing the two images by acquiring four vertex images of the reference object image in the two input images. Photographing information calculation means for calculating and storing a matrix that gives a constraint relationship between
The target area of the actual size measurement is set as the target area, and the designation of a two-dimensional rectangle on the screen surrounding the target area in one of the two images is received and stored, and the sides of the rectangle are stored. , Calculating a coupling coefficient when the image of the reference object is represented by a linear combination of images of three sides intersecting at one vertex, and correcting the shape of the rectangle to obtain the position of the vertex on the screen of the rectangle Target area determining means for storing,
A corresponding target area determining unit that calculates an area corresponding to the rectangle as a corresponding target area in the other of the two images, and stores a vertex position on the screen of the corresponding target area;
Target area actual size calculating means for calculating the three-dimensional coordinates of the vertices of the target area using the projection matrix, the vertex position in the target area, and the vertex position in the corresponding target area;
Program to function as.   The computer further uses the at least one of the two images, the vertex position on the screen of the target area, and the three-dimensional coordinates of the vertex position of the target area to generate image data in the target area. The program according to claim 9, wherein the program is made to function as a target area display unit that synthesizes and displays the image on another image or a three-dimensional scene. The corresponding target area determining means includes
Region shape calculation for calculating and evaluating a vertex position on the screen of the corresponding target region by linear combination of images of three sides intersecting at one vertex of the image of the reference object using the calculated coupling coefficient Means,
Area position calculating means for calculating and evaluating a corresponding position in the other image of the vertex of the target area using a matrix that gives a constraint between the calculated two screens;
A feature amount for calculating and evaluating a similarity between a feature point obtained from luminance information in the target area in the one image and a feature point obtained from luminance information in the corresponding target area in the other image A calculation means;
A corresponding region moving / evaluating unit that determines the corresponding target region while changing the position and shape of the corresponding target region based on the evaluation values obtained by the region shape calculating unit, the region position calculating unit, and the feature amount calculating unit; ,
The image program according to claim 9 or 10, wherein the program functions as the region shape calculating unit, the region position calculating unit, the feature amount calculating unit, and the corresponding region moving / evaluating unit.   The computer further determines a corresponding target area in the other image of the target area specified in the one image, regards the corresponding target area as a new target area, and again determines a new target area in the one image. The program according to any one of claims 9 to 11, which functions as means for obtaining a corresponding target area.   A computer-readable recording medium that stores the program according to any one of claims 9 to 12.

Images