JP2005275789A - Three-dimensional structure extraction method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a three-dimensional structure extraction method allowing easy and high-accuracy extraction of a three-dimensional structure of a photographic target over 360° around. <P>SOLUTION: The photographic target 10 over 360° around is projected on the surface 2 of a sphere 1 to form a first spherical image 20 displayed by a first coordinate system 21, and the photographic target is photographed from a position different from the first spherical image to form a second spherical image 30 displayed by a second coordinate system 31. Next, a translation vector t and a rotation matrix R converting the first coordinate system to the second coordinate system are calculated on the basis of an epipolar constraint condition, and position vectors m<SB>1i</SB>, m<SB>2i</SB>of a plurality of object point images p<SB>1i</SB>. p<SB>2i</SB>corresponding to each other in each of the first and second spherical images. Next, by calculating a distance ρ<SB>1</SB>from the origin of the first coordinate system to each object point of the photographic target corresponding to each of the plurality of object point images on the basis the translation vector, the rotation matrix, and the position vectors of the plurality of object point images corresponding to each other, the three-dimensional structure of the photographic target is extracted. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a method for extracting a three-dimensional structure from a spherical image in which a subject to be photographed having a 360-degree circumference is projected on the surface of a sphere.

As a technique for extracting a three-dimensional structure to be imaged, for example, a technique disclosed in Patent Document 1 is known. In this technique, a subject to be photographed is photographed from different positions by changing the position of the camera. A partial three-dimensional structure to be photographed is extracted from the planar image obtained by each photographing. A plurality of partial three-dimensional structures are synthesized to obtain a three-dimensional structure to be photographed.
JP 2003-14421 A

  However, if the partial three-dimensional structures to be photographed are calculated from a large number of partial field images (planar images) and merged into one as in the conventional technique, the calculation processing and the like become complicated. Further, when extracting a three-dimensional structure, it is necessary to estimate various parameters indicating the posture of the photographed camera, etc., but the parameters must be estimated for each photographed image. For this reason, there is a risk of a decrease in accuracy.

  Therefore, an object of the present invention is to provide a three-dimensional structure extraction method capable of easily and accurately extracting a three-dimensional structure of a subject to be photographed around 360 degrees.

  In order to solve the above-described problem, a three-dimensional structure extraction method according to the present invention forms a first spherical image that is displayed on a first coordinate system by projecting an object to be photographed around 360 degrees around the surface of a sphere. And a second spherical image obtained by photographing the subject from a position different from the first spherical image, the subject being projected on the surface of the sphere and displayed in the second coordinate system. A plurality of objects corresponding to each other in the first and second spherical images, the step of forming the second spherical image and the translation vector and the rotation matrix for converting from the first coordinate system to the second coordinate system. Based on the position vector of the point image and the step of calculating based on the epipolar constraint condition, and the position vector, rotation matrix and translation vector of a plurality of object point images corresponding to each other in each of the first and second spherical images, Multiple Characterized in that it comprises a step of extracting the three-dimensional structure of the imaging target by calculating the distance from the origin of the first coordinate system to each object point to be imaged corresponding to each object point image.

  In the above method, first, a first and second spherical images are formed by photographing a subject to be photographed around 360 degrees at different positions. Thereby, the 1st and 2nd spherical image on which the same imaging | photography object was projected is obtained. Then, the translation vector and the rotation matrix for converting the first coordinate system described above to the second coordinate system are (1) each object point image onto which a plurality of object points included in the object to be imaged are projected. Thus, the calculation is performed based on position vectors of a plurality of object point images corresponding to each other in the first and second spherical images and (2) epipolar constraint conditions. The distance from the origin of the first coordinate system to each object point based on the calculated translation vector and rotation matrix, and each object point image corresponding to each other in each of the first and second spherical images. Is calculated.

  The first and second spherical images are obtained by projecting a subject to be photographed over 360 degrees. Then, by using the first and second spherical images, distances to a plurality of object points included in the photographing target are calculated. Therefore, it is possible to easily extract the three-dimensional structure of the subject to be photographed over 360 degrees in comparison with the case of extracting the three-dimensional structure for each of the plurality of planar images and combining them.

  In the three-dimensional structure extraction method according to the present invention, the first spherical image is obtained by installing an imaging device having a fisheye lens having an angle of view of 180 degrees or more at a predetermined position and capturing an image of 360 degrees around. Among them, a planar image on which an imaging area covering at least 180 degrees around is projected and a planar image on which an imaging area including the remaining area is projected are acquired, each planar image is converted into a hemispherical image, and the hemispherical images are combined. The second spherical image is a flat image obtained by installing the imaging device at a position different from the predetermined position and projecting an imaging region of at least 180 degrees out of the imaging target of 360 degrees around the second spherical image. And a plane image on which the imaging area including the remaining area is projected, each plane image is converted into a hemispherical image, and these hemispherical images are combined to form It is preferable.

  In this case, since the field of view of the fisheye lens is 180 degrees or more, it is possible to obtain a planar image in which at least a surrounding area of 180 degrees is projected and a planar image in which the remaining areas are projected out of an imaging target of 360 degrees around. . For this reason, it is possible to acquire a large amount of image information to be captured by capturing twice. Since the first or second spherical image is formed from the planar image thus obtained, it is possible to acquire a spherical image with higher accuracy.

  ADVANTAGE OF THE INVENTION According to this invention, the three-dimensional structure extraction method which can extract the three-dimensional structure of 360 degree | times surrounding circumference | surroundings easily and accurately can be provided.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. In the following description, the same reference numerals are used for the same elements, and redundant descriptions are omitted.

  First, a spherical image for extracting a three-dimensional structure to be imaged in the present embodiment will be described. The spherical image in the present embodiment is obtained by projecting a shooting target located on the entire sky outside the system (around 360 degrees) on the surface of a sphere. A more specific description will be given with reference to FIG.

FIG. 1 is a diagram for explaining a spherical image. As shown in FIG. 1, a sphere 1 having a unit radius is considered in space, and an X axis, a Y axis, and a Z axis that are orthogonal to each other with the center O as an origin are set. A point on the surface 2 of the sphere connecting an arbitrary point P _i (i is an arbitrary integer) in space and the center O of the sphere is defined as _pi .

また、点ｐ_ｉと原点Ｏとを結ぶ直線のＺ軸からの仰角をθ_ｉとし、Ｘ軸からの方位角をφ_ｉとすると、点ｐ_ｉの位置ベクトルｍ_ｉは、次のように表される。

上記点ｐ_ｉは点Ｐ_ｉが球体の表面へ投影された点に相当する。すなわち、点Ｐ_ｉからの光が中心Ｏに向かって進んでいるときに球体１の表面２と交わる点がｐ_ｉであり、点ｐ_ｉは点Ｐ_ｉからの光が担う画像を形成するために必要な情報（画像情報）を有している。 The position vector M _i of the point P _i is expressed as follows.

Further, if the elevation angle from the Z axis of the straight line connecting the point p _i and the origin O is θ _i and the azimuth angle from the X axis is φ _i , the position vector _mi of the point p _i is expressed as follows: Is done.

The point p _i corresponds to the point where the point P _i is projected onto the surface of the sphere. That is, when the light from the point P _i is traveling toward the center O, the point that intersects the surface 2 of the sphere 1 is p _i , and the point p _i forms an image carried by the light from the point P _i. Necessary information (image information).

が成り立つ。ここで、ρ_iは原点Ｏから点Ｐ_ｉまでの距離であって、

で定義される。 In addition, since the point on the surface 2 of the sphere 1 that connects the point P _i and the center O of the sphere is p _i ,

Holds. Where ρ _i is the distance from the origin O to the point P _i ,

Defined by

In (Expression 3), 1 / ρ _i functions as a scale factor. Further, from (Equation 3) and (Equation 4), if 1 / ρ _i is separated (corresponding to P _i being located sufficiently far from the origin O), the position vectors m _i and M _i coincide with each other. To do. This indicates that an imaging object that can be seen from the center O and located around 360 degrees outside the system can be projected onto the surface 2 of the sphere 1. Thus, the surface 2 of the sphere 1 having image information corresponds to a spherical image. In the following, the point P _i is also referred to as an object point, and the point p _i is also referred to as an object point image as a point on which the object point P _i is projected.

  Next, a method for forming the spherical image will be described with reference to FIGS. This spherical image is formed using two planar images formed by a fisheye lens having an angle of view of 185 degrees.

  FIG. 2 is a diagram for explaining a photographing method of a photographing target. In FIG. 2, it is assumed that the subject 10 is schematically represented by a circle shown in the drawing, and a fisheye lens having an angle of view w of 185 degrees is positioned at the center of the circle. A 185 degree imaging region 11 included in the field of view of the fish-eye lens is imaged by the fish-eye lens among the imaging object 10 extending 360 degrees around to obtain a planar image. Next, the imaging area 12 including the remaining area of the imaging object 10 is imaged by a fisheye lens to obtain a planar image. These planar images are converted into hemispherical images and then combined to form a spherical image.

  The principle for converting a planar image (subject image) into a hemispherical image will be described with reference to FIG.

FIG. 3A is a diagram for explaining the imaging characteristics of the fisheye lens. FIG. 3A shows a state in which the fisheye lens is located at the center O, and external light from an object point P _i (corresponding to P _i in FIG. 1) in the visual field of the fisheye lens is incident on the fisheye lens. Yes. Here, in order to simplify the explanation, the angle of view of the fish-eye lens shown in FIG. 3 is 180 degrees.

ここで、ｆは、魚眼レンズの焦点距離である。（式５）より、仰角θ_ｉは、以下の式から算出される。

（式６）を（式２）に代入すると、以下の式が得られる。

図３に示す点ｑ_ｉは、図１の物点Ｐ_ｉの平面上への投影点であって、図１において物点Ｐ_ｉを球体１の表面２上に投影した点ｐ_ｉに対応している。ところで、点ｐ_iと点ｑ_iとで方位角φ_iは一致している。したがって、（式７）は、点ｑ_ｉの位置座標から点ｐ_iの位置座標を得ることができることを示している。 FIG. 3B is a diagram illustrating the image plane 3 of the fisheye lens. In the equidistant projection type fisheye lens, an object point P _i shown in FIG. 3A is projected onto a point q _i represented by position coordinates (r _i , φ _i ) on the image plane 3 in FIG. Is done. That is, the object point P _i is projected at a point q _i where the azimuth angle φ _i does not change and the distance r _i from the image center O _I is expressed by the following equation.

Here, f is the focal length of the fisheye lens. From (Expression 5), the elevation angle θ _i is calculated from the following expression.

Substituting (Expression 6) into (Expression 2) yields the following expression.

Q _i points shown in FIG. 3 is a projected point on a plane of the object point P _i in FIG. 1, the object point P _i corresponding to p _i that has been projected onto the surface 2 of the sphere 1 in FIG. 1 ing. By the way, the azimuth angle φ _i coincides with the point p _i and the point q _i . Therefore, (Expression 7) indicates that the position coordinates of the point p _i can be obtained from the position coordinates of the point q _i .

A planar image obtained with a fisheye lens is a collection of points q _i . Therefore, the plane image can be converted into a hemispherical image by specifying the position coordinates of each pixel (that is, the point q _i ) constituting the plane image.

  As described above, as the spherical image, first, a planar image in which each of the imaging regions 11 and 12 included in the imaging object 10 shown in FIG. 2 is formed is acquired, and then, these planar images are obtained based on the above principle. The image is converted into a hemispherical image and the two hemispherical images are combined to form.

  When combining the hemispherical images, it is preferable that the image center of the hemispherical image, the focal length of the fisheye lens, and the relative orientation of the coordinate system displaying the two hemispherical images are calibrated. In this embodiment, it is assumed that calibration is performed by the calibration method described below. In the present embodiment, calibration is performed using the fact that a fisheye lens having an angle of view of 185 degrees is used.

  First, the center of the image is obtained as the intersection of the vanishing point pair included in the planar image obtained by imaging with the fisheye lens. The vanishing point is a virtual intersection at infinity of a group of straight lines that are substantially parallel to each other. Since the angle of view w of the fisheye lens is 185 degrees, a pair of vanishing points of the parallel line group included in the photographing target appears in the planar image. Therefore, the image center is calibrated as an intersection of different vanishing point pairs.

Further, the focal length of the fisheye lens, calculates the distance r _[pi between a pair of vanishing points as divided by [pi. Next, each of the two planar images is converted into a hemispherical image based on the image center and the focal length obtained as described above. Then, two hemispherical images are displayed based on the vanishing point for the straight line group substantially parallel to the horizontal line and the vanishing point of the straight line group substantially parallel to the straight line orthogonal to the horizontal line included in each hemispherical image. A transformation matrix between coordinate systems is calculated. This transformation matrix corresponds to the relative posture between coordinate systems displaying two hemispherical images.

  Next, since the angle of view w of the fisheye lens is 185 degrees, the focal length and the transformation matrix calculated as described above are obtained using the fact that there are overlapping image areas in the two hemispherical images. Further calibrate.

  As described above, two hemispherical images are formed based on the calibrated image center, focal length, and relative attitude between the two hemispherical images, and combined to obtain one spherical image.

By the way, the spherical image has image information of the imaging object 10 of the entire sky because the imaging object 10 extending 360 degrees around is projected. However, the imaging object 10, because it is projected onto the surface 2 of the sphere 1, (the distance from the origin O to the object point P _i) three-dimensional structure has not been obtained.

  The three-dimensional structure extraction method according to the present embodiment extracts the three-dimensional structure of the shooting target 10 using two spherical images on which the same shooting target 10 is shot at different positions and the shooting target 10 is projected. It is a feature.

  First, the principle of extracting a three-dimensional structure from two spherical images will be described.

FIG. 4 is a diagram for explaining a method for extracting a three-dimensional structure to be photographed from two spherical images. FIG. 4 shows a first spherical image 20 and a second spherical image 30 obtained by photographing the same subject 10 at different positions. A first coordinate system 21 for displaying the first spherical image 20 is represented by X ₁ axis, Y ₁ axis, and Z ₁ axis, and a second coordinate system 31 for displaying the second spherical image 30 is represented by X ₂ axis, Y ₂ axis, Z ₂ axis.

The same subject 10 is projected on the first and second spherical images 20 and 30. Therefore, the object point P _i included in the subject 10 appears as object point images p _1i and p _2i in the first and second spherical images 20 and 30.

とする。なお、回転行列Ｒ及び並進ベクトルｔは、第１の座標系２１と第２の座標系３１との間の相対姿勢情報に相当する。 Here, the position vector of the object point P _i in the first coordinate system 21 is M _1i, and the position vector of the object point P _i in the second coordinate system 31 is M _2i . Also, let R be a rotation matrix for conversion from the second coordinate system 31 to the first coordinate system 21. Furthermore, a translation vector for converting from the first coordinate system 21 to the second coordinate system 31 is

And Note that the rotation matrix R and the translation vector t correspond to relative posture information between the first coordinate system 21 and the second coordinate system 31.

が成り立つ。 Since the first and second spherical images 20 and 30 are obtained by photographing the same subject 10 at different positions, the vectors M _2i , t and (RM _1i + t) are on the same plane (epipolar constraint condition) ). Therefore,

Holds.

とすると、（式９）は、

と表される。 Here, a distortion symmetric matrix T expressed using each translation component (t _x , t _y , t _z ) of the translation vector t is expressed as follows:

Then, (Equation 9) becomes

It is expressed.

From (Expression 3), the position vector M _1i and the position vector m _1i are the same _except for the scale factor (1 / ρ _1i ) (that is, the vector directions are the same). Similarly, the position vector M _2i and the position vector m _2i are equal except for the scale factor.

となる。 Therefore, (Equation 9) is

It becomes.

(Equation 12) is obtained when the position vectors m _li and m _{2i of} the plurality of object point images p _1i and p _2i corresponding to each other are found in the first and second spherical images 20 and 30, respectively. It shows that the vector t can be calculated.

In other words, the rotation matrix R and the translation vector t are calculated based on the position vectors m _li and m _2i and the epipolar constraint conditions (that is, the condition that the vectors m _2i , t and (Rm _1i + t) are on the same plane). can do. At this time, the number of object point images p _1i (p _2i ) necessary for calculating the rotation matrix R and the translation vector t differs in the method of solving (Equation 12). For example, when using a linear solution method, 8 or more.

Next, a method for calculating the distance from the origin O ₁ to the object point P _i in the first coordinate system 21 when the rotation matrix R and the translation vector t are calculated as described above will be described.

が成立する。 In each of the first and second coordinate systems 21 and 31, if the distances to the object point P _i are ρ _1i and ρ _2i ,

Is established.

と表される。 When the position vector m _2i is applied to both sides of (Equation 13), that is, the outer product of each term on both sides and m _2i is taken, the distance ρ _1i from the origin O _I to the object point P _i in the first coordinate system 21 is ,

It is expressed.

Therefore, the position vectors m _1i and m _2i of the object point images p _{1i and} p _2i in the first and second spherical images 20 and 30 corresponding to the object point P _i included in the subject 10, the rotation matrix R, and the translation vector. If t is known, the distance ρ _1i is calculated by (Equation 14). In addition, on the first spherical image 20, the imaging target 10 extending 360 degrees around is projected. Therefore, the three-dimensional structure of the subject 10 can be extracted from the first spherical image 20.

  Next, the imaging system 40 for extracting the three-dimensional structure of the imaging target 10 by applying the above principle will be described.

  FIG. 5 is a block diagram illustrating a configuration of the imaging system 40. The imaging system 40 includes an imaging device 41 and an analysis device 42 that analyzes an image obtained by the imaging device 41.

  The imaging device 41 includes a fisheye lens 41A having an angle of view of 185 degrees and an imaging element 41B. The projection method of the fisheye lens 41A is an equidistant projection method. The fisheye lens 41A does not necessarily mean a single lens, but also includes a lens system having fisheye lens characteristics. The imaging element 41B has a light receiving surface in which cells (pixels) made of photodiodes and the like are two-dimensionally arranged. The imaging element 41B captures an image (subject image) of the subject 10 imaged by the fisheye lens 41A. Output as (image data). The image sensor 41B is, for example, a CCD image sensor. The electrical signal from the image sensor 41B is input to the analysis device 42.

  The imaging device 41 inverts the planar image obtained by shooting the fisheye lens 41A in the direction A shown in the drawing and rotating the imaging device 41 about 180 degrees out of 360 degrees around the imaging device 41 in the drawing. Image data of a planar image obtained by photographing the subject 10 with the fisheye lens 41 </ b> A directed in the B direction shown in FIG.

  The analysis device 42 is configured using microcomputer hardware and software. The analyzing device 42 includes a spherical image forming unit 42A, a relative posture calculating unit 42B, and a three-dimensional structure extracting unit 42C.

  The spherical image forming unit 42A uses the image data corresponding to each of the two planar images input from the imaging device 41 to form a spherical image based on the principle described above. The spherical image forming unit 42A forms the first spherical image 20 on the basis of a planar image obtained by photographing the subject 10 at a position (predetermined position) where photographing is to be performed. Further, the spherical image forming unit 42A forms the second spherical image 30 based on the planar image obtained by photographing the subject 10 at a position different from the predetermined position. The formation of the first and second spherical images 20 and 30 means that the image data of the planar image is converted into image data that can be displayed as a spherical image.

Origin O ₁ of the first coordinate system _21, and the origin O ₂ of the second coordinate system 31 corresponds to the origin of the coordinate system formed by the imaging device 41 at each imaging position (so-called camera coordinate system) .

The relative attitude calculation unit 42B calculates a translation vector t and a rotation matrix R for converting from the first coordinate system 21 to the second coordinate system 31 shown in FIG. 4 based on (Expression 12). A plurality of object point images p _1i and p _2i used when calculating the translation vector t and the rotation matrix R may be automatically extracted from the first and second spherical images 20 and 30. For example, the operator may manually extract based on an image displayed on a display or the like.

  Here, extracting the object point image from the first and second spherical images 20 and 30 is not necessarily limited to extracting directly from the first and second spherical images 20 and 30. An object point image is extracted from each planar image used to form the first and second spherical images 20 and 30 and converted into an object point image of a spherical image (image data for displaying a spherical image). (Converted to).

The three-dimensional structure extraction unit 42C is the object point image _{p 1i,} with the position vector _m 1i of _{p 2i, m 2i,} a translation vector t and rotation matrix R, from equation (14), each object point image _{p 1i} The distance ρ _1i to the object point P _i of the corresponding photographing object 10 is calculated.

  The analysis device 42 is a RAM (Random Access Memory) that temporarily stores image data and the like input from the image sensor, each process performed by the spherical image forming unit 42A, the relative orientation calculating unit 42B, and the three-dimensional structure extracting unit 42C. ROM (Read Only Memory) storing a program and data for the CPU, and a CPU (Central Processing Unit) that performs various arithmetic processing based on data stored in the ROM and RAM by executing the program stored in the ROM (Not shown) such as Processing Unit).

  A three-dimensional structure extraction method using the imaging system 40 will be described.

  FIG. 6 shows the shooting position of the imaging device 41 with respect to the shooting target 10 for acquiring the first and second spherical images 20 and 30. A 360 degree landscape around the imaging device 41 is the subject 10 to be photographed. For example, in FIG.

First, an imaging device 41 disposed on the first shot point 51 (a predetermined position), photographs the imaging area of the imaging target 10 included in the field of view of the fisheye lens 41A toward the fisheye lens 41A in the A ₁ direction in FIG. 6 . Then, the image data of the planar image obtained by photographing is input to the analysis device 42. Further, the imaging device 41, rotates along an axis passing through the focal point of the fish-eye lens 41A, photographs the imaging area including the remaining area of the imaging target 10 toward a fisheye lens 41A in B ₁ direction in FIG. Then, the image data of the planar image obtained by the photographing is input to the analysis device 42. For convenience of explanation, A ₁ direction and is divided into A ₂ direction, which are substantially shown the same direction. B ₁ direction and the _{B 2} direction of relationship is the same.

  The spherical image forming unit 42 </ b> A of the analysis device 42 forms the first spherical image 20 displayed on the first coordinate system 21 by projecting the imaging object 10 around 360 ° around the imaging device 41.

  Next, the imaging device 41 is moved to a second imaging point 52 different from the first imaging point 51 in the room that is the imaging object 10, and the second spherical surface is obtained in the same manner as in the case of the first imaging point 51. An image 30 is formed. On the second spherical image 30, a subject to be photographed 360 degrees around the photographing device 41 is projected. Further, the second spherical image 30 is displayed in the second coordinate system 31.

  The first and second spherical images 20 and 30 are obtained by projecting the subject 10 covering 360 degrees around the surface 2 of the sphere 1 (see FIG. 1). 30, the same subject 10 is projected.

Next, the relative posture calculation unit 42B of the analysis device 42 extracts object point images p _1i and p _2i that correspond to each other in each of the first spherical image 20 and the second spherical image 30. This may accept points extracted by the operator. The relative orientation calculation unit 42B of the analyzer 42, by substituting the object point image _{p 1i,} a position vector _m 1i of _{p 2i,} the _{m 2i} in (Equation 12), to calculate a translation vector t and rotation matrix R.

Subsequently, the three-dimensional structure extraction unit 42C _obtains the position vectors m _1i and m _{2i of} the object point images p _1i and p _2i corresponding to each other, the translation vector t and the rotation matrix R calculated by the relative posture calculation unit 42B. used than in (equation 14), to calculate a distance [rho _1i to the object point _{P i} of the imaging subject 10 corresponding to each object point image _{p 1i.}

The subject 10 is projected on the first spherical image 20. The distance ρ _1i to the object point P _i in the first coordinate system 21 is calculated by the above process. That is, the three-dimensional structure of the subject 10 is extracted. Note that the final fusion of the surrounding three-dimensional structure is realized by matching feature points (object point images) included in the spherical image.

  The three-dimensional structure may be extracted as follows. FIG. 7 is a flowchart showing a three-dimensional structure extraction method.

First, the planar images taken at the first and second photographing points 51 and 52 (see FIG. 6) to form the first and second spherical images 20 and 30 (see FIG. 4) correspond to each other. Feature points (that is, points corresponding to the planar images of the object point images m _1i and m _2i ) are searched (S10). Here, for the sake of explanation, for example, eight feature points are extracted. This search may be configured as software, and a feature point search filter that extracts feature points based on image data of a planar image may be used. For example, an operator may extract feature points while confirming on a display.

Next, position vectors of feature points (object point images m _{11 to} m ₁₈ , m _{21 to} m ₂₈ ) on the first and second spherical images 20 and 30 corresponding to the feature points obtained in S10 are expressed by 12), the translation vector t and the rotation matrix R are calculated (S11).

Subsequently, by using the translation vector t and the rotation matrix R, other feature points m _{1 (8 + t)} and m _{2 (8 + t)} (t are arbitrary corresponding to each other in the first and second spherical images 20 and 30. (Integer) is searched and extracted (S12). Here, the search for other feature points is performed based on the epipolar line.

Next, the position vector of the feature points m _1n and m _2n (n is an arbitrary integer from 1 to (8 + t)) is substituted into (Expression 14), and the cubic of the object point P _1n corresponding to the feature point m _1n The original position is calculated. (S13). That is, to calculate the distance [rho _1n from the origin _{O 1} in a first coordinate system 21 of the object point _{P 1n.}

In addition, while performing step S13, the feature point m _1n in the first spherical image 20 is projected onto a planar image, and a Delaunay triangle is calculated based on the projected feature points (S14). Further, image information (texture) of the first spherical image 20 corresponding to the Delaunay triangle is pasted on the Delaunay triangle. That is, a texture image is generated by converting an image of a corresponding region of the first spherical image 20 into a planar image (S15).

Then, the texture image generated in S15 is mapped (texture mapping) according to the distance ρ _1n of the object point P _1n corresponding to each feature point m _1n to form a planar image displaying the three-dimensional structure (S16).

  Next, the result of performing a simulation based on the three-dimensional structure extraction method according to the present embodiment will be described.

次に、シミュレーションのために設定している第１の球面画像２０において、表２に示す１１点の物点像ｐ_１ｊ（ｊは、１〜１１の整数）を設定した。

そして、表２に示す１１点の物点像ｐ_１ｊ、及び、その物点像に対応する第２の球面画像３０の物点像に基づいて、（式１２）より並進ベクトルｔ及び回転行列Ｒを算出した。 Table 1 shows the translation components (t _x , t _y , t _z ) of the translation vector t of the first and second coordinate systems 21, 31 and the rotation angles (Roll, Pitch, Yaw) that specify the rotation matrix R. (See the "True Value" line). Here, the rotation angles Roll, Pitch, and Yaw are rotation angles of the coordinate axes orthogonal to each other.

Next, in the first spherical image 20 set for simulation, 11 object point images p _1j (j is an integer of 1 to 11) shown in Table 2 were set.

Then, based on the object point image p _1j of 11 points shown in Table 2 and the object point image of the second spherical image 30 corresponding to the object point image, the translation vector t and the rotation matrix R from (Equation 12). Was calculated.

  The rotation angle (Roll, Pitch, Yaw) for specifying each component of the calculated translation vector t and the rotation matrix R is as shown in Table 1 (see “Estimation” row). According to Table 1, the calculated ratio between the components almost coincides with the set ratio between the components. That is, the relative orientation information of the first and second coordinate systems 21 and 31 can be obtained by calculating the translation vector t and the rotation matrix R using (Equation 12).

表３の結果より、実際に設定している値（True Value）と、（式１４）に基づいて算出した値（Estimation）とはほぼ一致している。 Table 3 shows the calculation result of the distance ρ _1j from the origin O _{1 of} the first spherical image 20 to each object point corresponding to the 11 object image shown in Table 2 according to (Equation 14). is there. In the calculation, the scale factor of the translation vector t was set as appropriate.

From the results in Table 3, the actually set value (True Value) and the value (Estimation) calculated based on (Equation 14) are almost the same.

  Next, experimental results using spherical images obtained by actual photographing will be described.

  In order to form a spherical image, the fish-eye lens 41A used is FCON-02 (manufactured by Olympus Corporation) having an angle of view of 185 degrees. As in the case of FIG. 6, one room is set as the shooting target 10, the imaging device 41 is shot at the first shooting point 51, and then the imaging device 41 is moved in parallel and shot at the second shooting point 52. Carried out.

FIG. 8 is a diagram of a planar image obtained by setting the imaging device 41 at the first shooting point 51 and shooting. 8 (a) is a diagram of a case taken toward a fisheye lens 41A in the A ₁ direction shown in FIG. In the first shot point 51. 8 (b) is a diagram of a case taken toward a fisheye lens 41A in B ₁ direction shown in the figure by inverting the image pickup device 41.

FIG. 9 is a diagram of a planar image obtained by setting the imaging device 41 at the second shooting point 52 and shooting. 9 (a) is a diagram of a case taken toward a fisheye lens 41A in the A ₂ direction shown in FIG. In the second shot point 52. 9 (b) is a diagram of a case taken toward a fisheye lens 41A in the B ₂ direction shown in the figure by inverting the image pickup device 41.

  Comparing FIG. 8 and FIG. 9, it can be seen that images in the same direction can be taken. That is, in FIGS. 8A and 9A, the window side of the room is photographed, and in FIG. 8B and FIG. 9B, the door side of the room is photographed.

  FIG. 10 is a diagram showing a part of the first spherical image 20 formed from the two planar images shown in FIG. FIG. 11 is a diagram showing a part of the second spherical image 30 formed from the two planar images shown in FIG.

  FIG. 12 is a diagram in which a plurality of object point images are extracted from the planar image of FIG. The round dots in FIG. 12 indicate the positions of the extracted object point images. 102 object points were extracted. An object point image corresponding to this object point image is also extracted from the planar image shown in FIG.

Based on the position vector of the object point image in the first and second spherical images 20 and 30 corresponding to the extracted 102 object point images and (Expression 12), from the first coordinate system 21 A translation vector t and a rotation matrix R for conversion to the second coordinate system 31 were obtained. And distance (rho) _1i to each object point of the imaging | photography object 10 corresponding to the several object point image shown in FIG. 12 was computed based on (Formula 14).

FIG. 13 is a diagram illustrating a three-dimensional structure of the imaging target 10 reconstructed based on the distance to the imaging target 10 calculated as described above. 13 (a) shows, the window side has a three-dimensional structure of _{(A 1} or _{A 2} direction in FIG. 6) are extracted, FIG. 13 (b), _{B 1} or _{B 2} direction of the door-side (in FIG. 6 ) Is extracted. Further, in FIG. 13C, the three-dimensional structure of the photographing object 10 located in the C direction in FIG. 6 is extracted, and in FIG. 13D, the photographing object 10 located in the D direction in FIG. The three-dimensional structure is extracted. As shown in FIGS. 13C and 13D, the three-dimensional structure in the direction different from the shooting direction in the shooting target 10 can also be extracted from the first and second spherical images 20 and 30.

As described above, the first and second spherical images 20 and 30 are obtained by projecting the imaging target 10 extending around 360 degrees. Then, by using the first and second spherical images 20 and 30, the distances to the plurality of object points P _i included in the imaging target 10 are calculated. Therefore, as compared with the case of extracting a three-dimensional structure for each of a plurality of planar images and synthesizing them, the three-dimensional structure of the imaging object 10 extending 360 degrees around the periphery can be extracted more easily.

  The preferred embodiment of the present invention has been described above, but the present invention is not limited to the above embodiment. For example, since the imaging device 41 has a single fisheye lens 41A, the imaging device 41 is rotated to shoot the object 10 to be photographed. In FIG. 5, a fisheye lens that can shoot the A direction and the B direction are photographed. A possible fisheye lens may be provided. In this case, it is sufficient to shoot once. In addition, the fisheye lens 41 has an angle of view of 185 degrees, but the angle of view may be 180 degrees or more (that is, the field of view is 180 degrees or more).

In the above embodiment, the position vectors m _1i and m _2i of the object point images p _1i and p _2i used for calculating the translation vector t and the rotation matrix R are used for calculating the distance ρ _1i . . However, for example, in order to calculate the distance ρ _1i , object point images corresponding to each other in the first and second spherical images 20 and 30 may be extracted.

  The three-dimensional structure extraction method according to the present invention can be used for a monitoring device, a visual simulator, and the like.

It is a figure explaining a spherical image. It is a figure explaining the imaging | photography method of imaging | photography object. FIG. 3A is a diagram for explaining the imaging characteristics of the fisheye lens. FIG. 3B is a diagram illustrating an image plane of the fisheye lens. It is a figure explaining the extraction method of the three-dimensional structure of imaging | photography object from two spherical images. 2 is a block diagram illustrating a configuration of an imaging system 40. FIG. It is a figure which shows the imaging position of the imaging device 41 with respect to the imaging object 10. It is a flowchart which shows the extraction method of a three-dimensional structure. 4 is a diagram of a planar image photographed at a first photographing point 51. FIG. 5 is a diagram of a planar image photographed at a second photographing point 52. FIG. It is a figure which shows a part of 1st spherical image 20 formed actually. It is a figure which shows a part of 2nd spherical image 30 formed actually. It is a figure which shows the process of extracting a some object point image. It is a figure which shows the three-dimensional structure of the imaging | photography object 10.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Sphere, 2 ... Surface, 10 ... Object to be imaged, 20 ... First spherical image, 30 ... Second spherical image, 21 ... First coordinate system, 31 ... Second coordinate system, 41A ... Fisheye lens, t ... translation vector.

Claims (2)

Forming a first spherical image that is projected on the surface of a sphere around 360 degrees around it and displayed in a first coordinate system;
A second spherical image obtained by photographing the subject from a position different from the first spherical image, wherein the subject is projected on the surface of a sphere and is displayed in a second coordinate system. Forming a spherical image of 2;
A translation vector and a rotation matrix for converting from the first coordinate system to the second coordinate system, a position vector of a plurality of object point images corresponding to each other in each of the first and second spherical images, and Calculating based on epipolar constraint conditions;
Each object to be imaged corresponding to each of the plurality of object point images based on the position vector, the rotation matrix, and the translation vector of a plurality of object point images corresponding to each other in each of the first and second spherical images. Extracting the three-dimensional structure of the object to be imaged by calculating a distance from the origin of the first coordinate system to a point. In the first spherical image, an imaging device including a fisheye lens having an angle of view of 180 degrees or more is installed at a predetermined position, and an imaging area extending at least 180 degrees is projected from the imaging object covering 360 degrees around the first spherical image. It is formed by acquiring a planar image on which an imaging region including a planar image and the remaining region is projected, converting each planar image into a hemispherical image, and combining these hemispherical images,
The second spherical image includes a planar image in which an imaging device is installed at a position different from the predetermined position, and a photographing area extending at least 180 degrees around the imaging object covering 360 degrees is projected and the remaining area. 2. The three-dimensional image according to claim 1, wherein the three-dimensional image is formed by acquiring a planar image on which a photographing area is projected, converting each planar image into a hemispherical image, and combining the hemispherical images. Structure extraction method.

Images