JP2005252482A - Image generating apparatus and three-dimensional distance information acquisition apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image generating apparatus capable of detecting the attitude of a photographing apparatus without being affected by various operating environments and processing generation of an image in a short processing time in real time. <P>SOLUTION: An acceleration sensor and a gyro sensor as a position attitude information acquisition means for acquiring position attitude information are provided at the middle of the inside of an active sensor for acquiring an omnidirectional stereoscopic image. The image generating apparatus is provided with a conversion means for acquiring three-dimensional distance information from image information temporally synchronously with the obtained attitude information and converting furthermore the three-dimensional distance information into world coordinate system corresponding three-dimensional distance information wherein the active sensor is located on the basis of the position attitude information. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image generation device and a three-dimensional distance information acquisition device.

  In the fields of virtual reality, robot navigation, and computer vision, various researches using image processing have been actively conducted. For example, when virtually participating in a conference held at a remote place, it is possible to give the user a sense of realism if the image of the actual conference hall can be reproduced around the user. Also, when performing remote control with a robot in a dangerous area or special area where people cannot enter, if the 3D environment model of the area can be reproduced from the image acquired by the robot, the user will have a very realistic environment. Remote operation can be performed inside. In the generation of such a three-dimensional environment model, it is important to acquire a wide range of environment information by a photographing device such as a camera.

  For example, Patent Document 1 discloses a three-dimensional distance information acquisition device that can be used as an image input-side device for generating such a three-dimensional environment model. This device is composed of 20 stereo units mounted on a regular icosahedron. Each stereo unit is composed of three CCD cameras arranged in a T-shape, so that two types of stereo images, upper and lower and left and right, can be obtained. Here, when the posture (inclination) of the device itself is different for each photographing, the device coordinate system is different for each omnidirectional photographed image depending on the posture of the device itself at the time of photographing, and the unit coordinates are not uniform. Therefore, it is necessary to match the device coordinate system of each omnidirectional image with the world coordinate system where the device is located.

In Non-Patent Document 1, the inclination of such a device with respect to the world coordinate system is estimated by detecting a vertical edge in the acquired image. The vertical edge is a straight line that exists in a lot of scenes such as indoor environments and urban areas. Usually, an environment such as a room includes many vertical edges, and the direction of these vertical edges is the same as the direction of the vertical axis Zw of the world coordinate system. Therefore, if the distribution in the vertical edge direction in the apparatus coordinate system is examined and the distribution in the largest vertical edge direction is detected, the direction of the vertical axis Zw of the world coordinate system in the apparatus coordinate system can be estimated. If the vertical axis Zw of the world coordinate system in the device coordinate system is known, the direction of the Z axis of the device coordinate system in the world coordinate system can also be known. By the above method, the inclination of the device with respect to the world coordinate system is estimated, corrected based on the estimated value, the device coordinate system of each omnidirectional image is matched with the world coordinate system, and a highly accurate three-dimensional distance is obtained. It was supposed to be able to get information.
JP 2001-285692 A (Claim 1, FIG. 1) Wang, Tanahashi, Sato, Hirayu, Niwa, Yamamoto, “Estimation of sensor position and orientation using edge histograms of omnidirectional images”, IEICE D-II, Vol. J86-D-II, no. 10, pp. 1400-1410, 2003

  However, since the estimation process using the vertical edge takes a long processing time in calculation, there is a problem that it is difficult to generate a corrected two-dimensional image or a three-dimensional environment model in real time.

  Also, in the indoor environment, there is usually some kind of vertical edge. It was difficult to estimate. As described above, when the device is used in a special place or when the use environment of the apparatus is not fixed, it cannot be used effectively.

  Therefore, the present invention provides an image generation apparatus and a three-dimensional distance information acquisition apparatus that can detect the posture of the apparatus without being affected by various usage environments and can perform processing in a short time and in real time. Objective.

  In order to achieve the above object, an image generation apparatus according to claim 1 is provided with an image photographing apparatus that obtains an image in all directions, and posture information acquisition that is provided in the image photographing apparatus and obtains posture information of the image photographing apparatus. Means, synchronization means for temporally synchronizing the image information photographed by the image photographing device and the posture information obtained by the posture information obtaining means, and image information synchronized with the posture information by the synchronizing means, The gist of the present invention is to have a two-dimensional image generation means for generating a two-dimensional image having no inclination with respect to a predetermined axis of a three-dimensional world coordinate system of the environment where the image capturing device is located based on the posture information.

A gist of an image generation apparatus according to a second aspect is the image generation apparatus according to the first aspect, wherein the predetermined axis is a vertical axis.
A three-dimensional distance information acquisition device according to claim 3 is an image photographing device that obtains an omnidirectional stereo image, a posture information obtaining unit that is provided in the image photographing device and obtains posture information of the image photographing device, Synchronizing means for temporally synchronizing the image information captured by the image capturing apparatus and the orientation information acquired by the orientation information acquiring means, and the three-dimensional distance information from the image information synchronized with the orientation information by the synchronizing means. A three-dimensional distance information acquisition means for acquiring the three-dimensional distance information, and converting the three-dimensional distance information into three-dimensional distance information based on the three-dimensional world coordinate system of the environment where the image capturing device is located based on the posture information Having a means.

A three-dimensional distance information acquisition device according to a fourth aspect is the three-dimensional distance information acquisition device according to the third aspect, wherein the posture information acquisition means includes a gyro sensor.
The three-dimensional distance information acquisition device according to claim 5 is the three-dimensional distance information acquisition device according to claim 3 or 4, wherein the three-dimensional distance information acquisition device is a three-dimensional distance converted by the conversion means. The gist of the invention is to further include a three-dimensional environment model generating means for generating a three-dimensional environment model from the information.

  A three-dimensional distance information acquisition device according to a sixth aspect is the three-dimensional distance information acquisition device according to any one of the third to fifth aspects, wherein the three-dimensional distance information acquisition device is connected to the image photographing device. It has a position information acquisition unit that is provided and acquires the position information of the image capturing apparatus, and the synchronization unit synchronizes the image information and the position information.

The three-dimensional distance information acquisition device according to claim 7 is the three-dimensional distance information acquisition device according to claim 6, wherein the position information acquisition means includes an acceleration sensor.
The three-dimensional distance information acquisition device according to claim 8 is the three-dimensional distance information acquisition device according to any one of claims 3 to 7, wherein the three-dimensional distance information acquisition device is the image photographing device. Further comprising estimation means for estimating an inclination of the image photographing device with respect to a world coordinate system where the image photographing device is located based on an edge of the obtained image and its direction, and based on the posture information, The gist is to estimate the posture of the image capturing apparatus.

  According to the present invention, it is possible to detect the attitude of the apparatus without being affected by various use environments, and to perform processing in real time with a short processing time.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described with reference to FIGS.
FIG. 1 is a schematic diagram illustrating a system configuration of an image processing apparatus according to the present embodiment. As shown in FIG. 1, an image processing apparatus (image generation apparatus) 11 according to the present embodiment includes an active sensor (an omnidirectional stereo imaging apparatus, an image imaging apparatus) 12 and a computer 13. In the description of this embodiment, the simple sensor refers to the active sensor 12. The active sensor 12 includes a plurality of, in this embodiment, 20 units of a three-eye stereo unit (hereinafter simply referred to as “stereo unit”) 18, an acceleration sensor 14, a gyro sensor 15, a memory unit 16, a synchronization signal generator 17, and the like. ing. In FIG. 1, one plane conceptually shows acquisition of omnidirectional images and data of the acceleration sensor 14 and the gyro sensor 15 by 20 stereo units at the same time, and these are given at predetermined time intervals. This conceptually shows that the acquisition is continuously performed.

  FIG. 2 is a schematic diagram of the mechanical configuration of the active sensor 12. The active sensor 12 includes a vehicle body 100 having a plurality of wheels, and the wheels are driven by an electric motor (not shown) provided in the vehicle body 100 to automatically travel toward an arbitrary position in the environment K (straight line and Including travel along a curve). In FIG. 2, the environment K is shown smaller than the vehicle body 100 for convenience of explanation. The memory unit 16, the computer 13, the synchronization signal generator 17, etc. are stored in the vehicle body 100.

  FIG. 3 is an overall external view of the active sensor 12. The stereo unit 18 provided in the active sensor 12 includes three video cameras, and each stereo unit 18 is disposed on each surface of a regular icosahedron (see FIG. 4). That is, in this embodiment, there are 20 stereo units 18. The aggregate of these stereo units 18 is supported by a stand 19.

  In addition, each stereo unit 18 has the same characteristics, and the stereo unit 18 arranged on each surface can acquire an omnidirectional color image and a monochrome image (stereo image for obtaining a distance image) in real time at the same time. ing. As a result, color images and three-dimensional information in all directions in the three-dimensional space can be obtained at the same time.

  Furthermore, by arranging the stereo units 18 having the same characteristics on each surface of the regular icosahedron, it is possible to divide the three-dimensional space equally and acquire high-resolution information. The active sensor 12 is described in ““ Omnidirectional Stereo System (SOS) for Real Environment Sensing ”, IEEJ Transactions C. Vol. 121-C, No. 5, pp. 876-1811, 2001”. Has been described.

  As shown in FIG. 1 (FIG. 3), the stereo unit 18 includes one standard video camera VCs and a pair of reference video cameras VC. The reference video camera VC is arranged so as to be included in a pair of planes orthogonal to each other with the optical axis of the standard video camera VCs as an intersection line. These cameras are arranged so as to form two stereo pairs. In the present embodiment, since there are 20 stereo units 18, there are also 20 reference video cameras VCs.

  From each stereo unit 18, a stereo image composed of one color image and two monochrome images is acquired, and 20 color images in all directions and 40 monochrome images are set as one set at 15 sets / second. Transfer to the memory unit 16. The memory unit 16 stores the transferred omnidirectional image data (hereinafter referred to as an omnidirectional image). The omnidirectional image includes image data obtained by the reference video camera VCs.

  FIG. 4 is a diagram for explaining how the acceleration sensor 14 and the gyro sensor 15 provided in the active sensor 12 are attached. As shown in FIG. 4, an acceleration sensor 14 and a gyro sensor 15 are provided at a substantially central portion inside the active sensor 12. The acceleration sensor 14 acquires the position of the active sensor 12, and the gyro sensor 15 acquires the attitude (tilt) of the active sensor 12. Processing related to detailed acquisition will be described later. The acceleration sensor 14 constitutes position information acquisition means, and the gyro sensor 15 constitutes posture information acquisition means.

  Also, an external synchronization signal based on a common absolute time is supplied from the synchronization signal generator 17 to each video camera of each stereo unit 18, and time information is added to the image data. Thus, completely synchronized image data can be obtained in the digitized frame. The synchronization signal generator 17 also adds time information to the position and orientation information from the acceleration sensor 14 and the gyro sensor 15.

  The computer 13 executes a program stored in advance in a ROM 13a (see FIG. 1) provided in the computer 13 at predetermined intervals. Further, the computer 13 accesses the memory unit 16 and acquires the omnidirectional image and the position / orientation information from time to time. Further, the computer 13 synchronizes the omnidirectional image and the position and orientation information with respect to time, a two-dimensional image generation unit that generates a two-dimensional image, and the inclination of the active sensor 12 based on the edge of the obtained image. This corresponds to estimation means for estimation. The processing of each means will be described in detail later.

Next, the operation of the image processing apparatus 11 configured as described above will be described. First, terms used in the following description will be described.
1. Sensor coordinate system (3D coordinate system)
The sensor coordinate system is a coordinate system determined by the viewpoint position and the orientation of the viewpoint when imaged from the center of the sensor.

2. World coordinate system (3D coordinate system)
It is the coordinate system of the environment where the sensor itself is located. The position and orientation of the sensor are specified by this world coordinate system. A three-dimensional environment model is generated based on this world coordinate system. In the present embodiment, the world coordinate system is determined with the starting point of the movement of the sensor as the origin, the vertical direction as the Z-axis direction, and a predetermined direction, for example, magnetic north as the X-axis direction, but is not limited thereto.

3. Three-dimensional vertical edge A three-dimensional vertical edge is an edge having a direction perpendicular to a horizontal reference plane such as the ground or floor in the world coordinate system. The edge thus has an edge direction parallel to the Zw axis, assuming that the world coordinate system has a Zw axis perpendicular to the reference plane. The three-dimensional vertical edge point means a point on the three-dimensional vertical edge when viewed discretely.

  In the image processing apparatus 11 of the present embodiment, after obtaining a rough posture of the active sensor 12 from the gyro sensor 15, a vertical straight line (hereinafter referred to as "vertical edge") that is frequently present in scenes such as indoor environments and urban areas. Is used to estimate the exact posture (vertical tilt and horizontal rotation) of the active sensor 12. By using the information of the gyro sensor 15 and the image information together, the attitude of the active sensor 12 can be obtained efficiently and accurately. Then, the image is corrected based on the obtained posture information to obtain image data in a state without rotation.

  FIG. 5 is a flowchart showing a series of processes of the image processing apparatus 11. First, attitude information of the active sensor 12 is acquired from the gyro sensor 15 (step 1, hereinafter “step” is abbreviated as “S”), and a conversion parameter from the sensor coordinate system to the world coordinate system is obtained (S2). Next, using the conversion parameter, the edge and the vertical edge are detected from the omnidirectional image obtained by the active sensor 12 based on the direction and the direction thereof, and the attitude of the active sensor 12 is estimated (S3). Then, the obtained image is corrected by the posture parameter to generate a two-dimensional image without rotation (S4).

  Next, regarding the posture estimation (S3) of the active sensor, the relationship between the edge direction and the posture of the active sensor 12, the tilt estimation using the edge direction, the estimation of the horizontal rotation of the active sensor 12, and the smoothing by the Kalman filter are further performed in this order. Describe in detail.

  First, regarding the principle of inclination estimation, in order to estimate the inclination of the active sensor 12, the direction of the vertical axis (hereinafter referred to as the Z axis) of the sensor coordinate system in the world coordinate system may be obtained. On the other hand, since the tilt estimation of the active sensor 12 in the world coordinate system and the tilt estimation of the world coordinate system in the sensor coordinate system are the same problem, if the direction Zw of the world coordinate system in the sensor coordinate system is known, the world coordinate The direction of the Z axis of the sensor coordinate system in the system is also known.

Regarding the posture estimation (S3) of the active sensor, first, the relationship between the edge direction and the posture of the active sensor 12 will be described. The three-dimensional vertical edge point Sw and its edge direction Ew in the world coordinate system are S and E in the sensor coordinate system, respectively. Further, the Z axis of the world coordinate system in the sensor coordinate system (X, Y, Z) is defined as Zw. In the three-dimensional space, since the vertical edges S + λE and Zw are parallel, they are on the same plane. That is, a homogeneous plane (that is, a plane passing through the origin of the coordinate system) (S × E) · X = 0, which is constituted by the three-dimensional vertical edge point S and the edge direction E, passes through the Z axis Zw of the world coordinate system. As shown in FIG. 6, if the plane (S × E) · X = 0 intersects with the plane Z = 1, the intersection line is the intersection p = (x, y, 1) between Zw and the plane Z = 1. Pass through ^? When a two-dimensional space (x, y) on the Z = 1 plane in the sensor coordinate system is a voting space, each edge point S and its edge direction E in the three-dimensional space are expressed by the following formula: voting space (x, y) Voted for.

シーンの中に垂直エッジが多数存在する場合、各エッジから作成された平面（Ｓ×Ｅ）^?Ｘ＝０を上述のように平面Ｚ＝１に投影すれば、その交差線はすべて共通点ｐを通るため、投票空間（ｘ，ｙ）中のｐ＝（ｘ_ｐ，ｙ_ｐ）において大きなピークが形成される。したがって、投票空間（ｘ，ｙ）から最も高いピークｐを検出することで、ワールド座標系のＺ軸の方向Ｚｗを推定することができ、その値は（ｘ_ｐ，ｙ_ｐ，１）^?になる。

When there are many vertical edges in the scene, if the plane (S × E) ^? X = 0 created from each edge is projected onto the plane Z = 1 as described above, all the intersection lines have a common point p. A large peak is formed at p = (x _p , y _p ) in the voting space (x, y). Therefore, by detecting the highest peak p from the voting space (x, y), the Z-axis direction Zw of the world coordinate system can be estimated, and its value is (x _p , y _p , 1) ^? Become.

  Next, the inclination estimation using the edge direction of the omnidirectional image will be described. The inclination is estimated by the above-described voting using the edge and the direction of the image obtained from each camera of the active sensor 12. A two-dimensional edge line segment composed of the edge s obtained from the image of each camera of the active sensor 12 and its direction e, and a three-dimensional edge line segment composed of the three-dimensional point S and the three-dimensional direction E of the edge. Are on the same homogeneous plane, and when each camera coordinate system of the active sensor 12 is coincident with the sensor coordinate system of the active sensor 12, s and e satisfy the following expressions.

一般的に、アクティブセンサ１２の各カメラの座標系とアクティブセンサの１２のセンサ座標系とは一致しないが、垂直エッジの消失点はカメラの並進に対して不変であり、各カメラの回転Ｒｃのみに関係する。この性質から次の関係式が得られる。

Generally, the coordinate system of each camera of the active sensor 12 and the sensor coordinate system of the active sensor 12 do not match, but the vanishing point of the vertical edge is invariant to the translation of the camera, and only the rotation Rc of each camera. Related to. From this property, the following relational expression is obtained.

ここで、ｓ_ｃとｅ_ｃはそれぞれカメラの画像におけるエッジとその方向である。

Here, s _c and e _c are an edge and its direction in the image of the camera, respectively.

The k-th edge point on the image of the c-th camera of the active sensor 12 is set to s _ck = (x _ck , y _ck ), and the edge direction is set to e _ck = (d _yck , −d _xck ). Here, k = 1,..., N _c , N _c are the number of edge pixels in each camera image. If the voting space on the plane of Z = 1 in the sensor coordinate system of the active sensor 12 is (x, y), the following voting formula can be obtained from the formula (3).

ただし、

However,

式（４）から明らかなように、アクティブセンサ１２の各カメラの画像上の一つのエッジ点は投票空間の１本の直線に対応する。すなわち、エッジ点ｓ_ｃｋとその方向ｅ_ｃｋが与えられたとき、投票空間においてはこれに対応する直線上に投票を行えば良い。すべてのエッジ点に対して投票を行った後、投票空間において最も高いピークを検出することによって、垂直エッジに対応する消失点を求めることができる。

As apparent from the equation (4), one edge point on the image of each camera of the active sensor 12 corresponds to one straight line in the voting space. That is, when an edge point s _ck and its direction e _ck are given, voting may be performed on a corresponding straight line in the voting space. After voting for all edge points, the vanishing point corresponding to the vertical edge can be determined by detecting the highest peak in the voting space.

  In this way, by estimating the inclination of the active sensor 12, an omnidirectional edge image of the active sensor 12 when there is no inclination can be generated. Since an omnidirectional edge image with no inclination can be virtually generated in this way, the camera position / posture estimation method (horizontal only) using the omnidirectional edge histogram already proposed by the present applicant is used. Other position / posture parameters can also be obtained. Specifically, it is based on ““ Position and Posture Estimation of Sensor Using Edge Histogram of Omnidirectional Image ”, Science D-II, Vol. J86-D-II, No. 10, pp. 1400-1410, 2003”. Next, how to determine the horizontal rotation component of the active sensor will be described. Here, the attitude parameter means a parameter for determining the entire attitude including the vertical direction and the horizontal direction.

  When the active sensor 12 is not tilted, translation or rotation of the active sensor 12 causes a shift of the vertical projection edge histogram of the omnidirectional image. The shift amount of the histogram caused by the horizontal rotation of the active sensor 12 is constant in all azimuth angles and becomes an offset of the entire shift amount. However, the shift amount of the histogram caused by the translation of the active sensor 12 is the movement direction and the edge. It has a sin curve characteristic in relation to the azimuth angle.

  When the orientation of the active sensor 12 at the initial time is used as a reference, the omnidirectional edge histogram at the initial time is set as a reference histogram. The omnidirectional edge histogram obtained at the current time is matched with the reference histogram to determine the shift amount of the histogram. The rotation angle of the active sensor 12 is obtained from the overall offset of the shift amount of the omnidirectional edge histogram. For details, refer to the document “Estimation of position and orientation of sensor using edge histogram of omnidirectional image”, Science theory D-II, Vol. J86-D-II, no. 10, pp. 1400-1410, 2003.

  There is a random estimation error in the posture parameter of the active sensor 12 obtained as described above. In general, the posture change of the active sensor 12 is smooth, but due to the random estimation error, the temporal variation of the restored posture is not necessarily a smooth locus in the parameter space. Therefore, a small vibration may occur in the omnidirectional image restored using the estimated posture parameter of the active sensor 12.

To solve this problem, the Kalman filter (literature “A New Approach to Linear Filtering and Prediction Problems”, Trans. Of ASME-Journal of Basic Engineering, Vol. 82-D, pp. 35-45, 19) is used. Thus, the temporal variation (trajectory) of the obtained posture parameter of the active sensor 12 is smoothed. Here, it is assumed that the temporal fluctuations of the two degrees of freedom α and β of the inclination and the one degree of freedom γ of the horizontal rotation are independent of each other. Further, it is assumed that the change in the posture parameters is a constant speed change. The posture parameter at time t is Pt = (α _t , β _t , γ _t ) ^T , and the parameter change amount Δt = P _t −P _t−1 . In that case, smoothing by the Kalman filter is as follows.

ここで、ＲとＱはそれぞれ対角要素の値がＲとＱである３×３の対角行列である。Ｑはパラメータ推定誤差の分散を表し、本実施形態ではＱ＝２とする。Ｒは仮定した等速モデルから実際の姿勢変化量の偏差値であり、本実施形態ではＲ＝４とする。

Here, R and Q are 3 × 3 diagonal matrices whose diagonal elements are R and Q, respectively. Q represents the variance of the parameter estimation error. In this embodiment, Q = 2. R is a deviation value of the actual posture change amount from the assumed constant velocity model, and in this embodiment, R = 4.

Further, as initial values, Δ ₀ = 0, and P ₀ is an initial estimated value of the posture. Let M ₀ be a diagonal matrix whose diagonal element value is R. The value obtained on the left side of Equation (9) is used as the estimated value of the posture at time t.

  As described above, in this embodiment, both the inclination obtained from the gyro sensor 15 and the inclination obtained from the vertical edge are used in combination. In the gyro sensor 15, there is a time error accumulation in the obtained data. However, since the inclination obtained from the vertical edge is an absolute inclination with respect to the world coordinate system of the active sensor 12, the gyro sensor 15 is used by using it. The drift error has been corrected.

  Next, generation of a two-dimensional image (FIG. 5, S4) will be described in detail. If the attitude of the active sensor 12 is known, an image without rotation can be generated from the omnidirectional image of the active sensor using the attitude parameter. Here, a method for generating an all-spherical spherical developed image of the active sensor 12 will be described.

  The attitude parameters of the active sensor 12 obtained as described above can be represented by rotation angles α, β, and γ corresponding to the X, Y, and Z axes, respectively. If the entire spherical surface developed image when the posture is changed is represented by the three-axis angular coordinate system (A, B, Γ), the whole spherical surface developed image when the active sensor 12 is not changed is (A−α). , B-β, Γ-γ). From this relationship, the rotation amount of the X, Y, and Z axes is the horizontal shift amount of the image in the cylindrical developed image with the respective axes as the vertical axes. Therefore, by utilizing this feature, a rotation-invariant two-dimensional cylindrical development image obtained by correcting the posture change of the active sensor 12 from the omnidirectional image obtained by each camera of the active sensor 12 in the following procedure at high speed is obtained. Generate.

First, the correspondence relationship between the cylindrical developed image with the X axis of the sensor coordinate system of the active sensor 12 as the vertical axis and the image of each camera of the active sensor 12 is calculated. Since the relationship between each camera coordinate system of the active sensor 12 and the sensor coordinate system is known, an image of each camera of the active sensor and a cylindrical developed image with the X axis as the vertical axis are used based on the planar projection model using the relationship. The correspondence can be easily obtained, and the correspondence is represented by c _x (θ _x , φ _x ), i _x (θ _x , φ _x ), j _x (θ _x , φ _x ). (Θ _x , φ _x ) are the coordinates of the cylindrical developed image with the X axis as the vertical axis. c _x is a camera number, and i _x and j _x are image coordinates of the camera c _x .

Next, the correspondence relationship between the cylindrical developed image (θ _y , φ _y ) with the Y axis in the sensor coordinate system of the active sensor 12 as the vertical axis and the cylindrical developed image (θ _x , φ _x ) with the X axis as the vertical axis. Is expressed by _xθ (θ _y , φ _y ), x _φ (θ _y , φ _y ). Similarly, the correspondence relationship between the cylindrical developed image (θ _z , φ _z ) with the Z axis as the vertical axis and the cylindrical developed image (θ _y , φ _y ) with the Y axis as the vertical axis is represented by y _θ (θ _z , φ _z ), y _φ (θ _z , φ _z ). Using the posture parameters (α, β, γ) of the active sensor 12, the corresponding relationship c _r (θ, φ), i _r ( θ, φ), j _r (θ, φ) can be obtained by multiple indexes from the above correspondence. The following formula shows a calculation formula for the correspondence relationship c _r (θ, φ) of the camera number.

式（１１）〜（１３）と同様に、ｉ_ｒ（θ，φ）とｊ_ｒ（θ，φ）についても多重インデックスを構築することができる。

Similar to equations (11) to (13), multiple indexes can be constructed for i _r (θ, φ) and j _r (θ, φ).

  Through each process described in detail above, a rotation-invariant two-dimensional image corresponding to the world coordinate system can be generated from the omnidirectional image. “It corresponds to the world coordinate system” means that the Z axis of the sensor coordinate system and the Zw axis of the world coordinate system are parallel and always in a constant state without being inclined with respect to the Z axis of the world coordinate system. Show. In other words, the sensor coordinate system of the active sensor 12 changes variously in the world coordinate system depending on the attitude of the active sensor 12 when viewed from the world coordinate system. Each obtained image can always be generated as a rotation-invariant two-dimensional image in which the sensor coordinate system in each image is unified. FIG. 7 is a diagram showing the result of restoring the image as a cylindrical image. FIG. 7A shows an image without correction, and FIG. 7B shows posture correction using posture parameters obtained through the above processing. The image is shown. As shown in FIG. 7A, in the uncorrected cylindrical image Pa, the image is distorted so as to draw a sin curve, but in the corrected cylindrical image Pb, the image is distorted as shown in FIG. 7B. Instead, an image as if the surrounding environment was viewed in a horizontal state was generated. In this way, it can be seen that accurate posture estimation is performed based on the obtained posture parameters and is corrected with high accuracy. That is, according to the present embodiment, even when the active sensor 12 moves within the environment and changes its posture, it is possible to always generate a two-dimensional image with no inclination with respect to the Zw axis of the world coordinate system. it can. Further, since the inclination from the gyro sensor 15 is first obtained and the data is used, the calculation amount is much smaller than that of estimating the posture only by the method of detecting the vertical edge from the omnidirectional image, and the rotation invariant 2 A three-dimensional image can be generated at high speed in real time.

According to the above embodiment, the following effects can be obtained.
(1) By using the gyro sensor 15 to acquire posture information, the posture of the active sensor 12 can be easily estimated with a small amount of calculation. In the present embodiment, both the inclination obtained from the gyro sensor 15 and the inclination obtained from the vertical edge are used in combination. For this reason, accurate estimation is difficult with only the gyro sensor 15 due to the accumulation of time errors in the obtained data, but the inclination obtained from the vertical edge is an absolute inclination with respect to the world coordinate system of the active sensor 12. Therefore, it is possible to correct the drift error of the gyro sensor 15 by using it.

  (2) Furthermore, a two-dimensional image corresponding to the world coordinate system can be generated from the captured omnidirectional image using the obtained posture parameter. That is, even when the active sensor 12 moves within the environment and changes its posture, it is possible to stably obtain an image when the environment is always viewed in the same posture. For example, when photographing an environment where a person cannot enter, such as a danger zone or a narrow cave, if the captured image is corrected using the active sensor 12 and displayed on a monitor or the like as a rotation-invariant two-dimensional image, distortion is generated. Since it is generated as an image, it is very easy to see and effective in grasping the situation.

  (3) Furthermore, although the posture can be estimated only by the method of detecting the vertical edge from the omnidirectional image, the processing speed is relatively slow because the processing is performed on the image. In this respect, in this embodiment, since the amount of calculation is less than that in the case of estimating the posture only by the method of detecting the vertical edge, the generation of the rotation invariant two-dimensional image is efficiently processed in real time. Can do.

  (4) In addition, since the imaging region of the active sensor 12 according to the present embodiment covers all directions of 360 ° × 180 ° (entire sky) and has no blind spots, the active sensor 12 itself is in any posture. Even if there is, it can always cover all fields of view. Sufficient information can be obtained when monitoring the entire environment or grasping the situation.

  (5) In the present embodiment, since the inclination is obtained from the gyro sensor 15, even when there is no vertical edge on the image, the posture of the active sensor 12 can be estimated and a rotation-invariant two-dimensional image can be generated. Of course, when the movement of the gyro sensor 15 advances and the drift error is accumulated, high accuracy cannot be expected, but a certain degree of posture correction can be made possible.

(Second Embodiment)
Next, a second embodiment of the present invention will be described with reference to FIG. In the second embodiment, the processing of the image processing apparatus 11 in the first embodiment is partly changed to function as a three-dimensional distance information acquisition apparatus. In other respects, the second embodiment differs from the first embodiment. It has the same configuration. Accordingly, only changes in the processing will be described below. Since the configuration of the image processing apparatus 11 is the same as that of the first embodiment, a description thereof will be omitted, and the description will be made assuming that the same components are denoted by the same reference numerals. Note that the computer 13 of the present embodiment includes a synchronization unit that synchronizes position information and image information, a three-dimensional distance information acquisition unit, a conversion unit, and a three-dimensional environment model generation unit in addition to the estimation unit similar to the first embodiment. It is equivalent to.

  FIG. 8 is a flowchart showing a series of processes of the image processing apparatus (three-dimensional distance information acquisition apparatus) 11. First, position and orientation information is acquired from the gyro sensor 15 and the acceleration sensor 14 (S11). Details will be described later. Next, three-dimensional distance information is acquired from the image synchronized with the position and orientation information by the synchronization means (S12). In the present embodiment, the three-dimensional distance information is acquired by photographing a three-dimensional image in the direction in which each surface of the regular icosahedron faces with a plurality of cameras constituting the plurality of stereo units 18. Next, a conversion parameter from the sensor coordinate system to the world coordinate system is obtained (S13). Next, using the conversion parameter, an edge is detected from the omnidirectional image obtained by the active sensor 12 based on the direction and its direction, and the attitude of the active sensor 12 is estimated (S14). Then, the three-dimensional distance information is converted into three-dimensional distance information based on the world coordinate system based on the position and orientation information (S15). Here, the position / orientation information refers to the position / orientation information obtained from the data obtained directly from the acceleration sensor 14 and the gyro sensor 15 through the estimation using the vertical edge, and is used as the attitude parameter in the first embodiment. Equivalent to. Next, a three-dimensional environment model is generated based on the three-dimensional distance information obtained by the conversion (S16).

  Next, each processing will be described by paying attention to differences from the first embodiment. First, acquisition of position and orientation information (S11) will be described. In the present embodiment, position information is acquired by the acceleration sensor 14 in addition to the posture information. The acceleration sensor 14 measures the sum of acceleration vectors of the active sensor 12. This acceleration vector can be converted from the sensor coordinate system to the world coordinate system using the conversion parameter obtained from the gyro sensor 15. The position information is obtained by subtracting the influence of gravity from the measured acceleration and double-integrating the value starting from a known initial position.

  Next, conversion from the sensor coordinate system to three-dimensional distance information based on the world coordinate system (S15) and generation of a three-dimensional environment model (S16) based on the position and orientation information will be described. The active sensor 12 acquires three-dimensional distance information at each point on the moving route while moving, and converts the sensor coordinate system to the world coordinate system using the position and orientation information of the active sensor 12 at each point ( S15). Here, for example, the center of the room or the start point of movement can be set as the origin. The world coordinate system with the start point as the origin is the common coordinate system, and each 3D point acquired at each time t is

とする。ここで、Ｎは視点の数である。時刻ｔで得られた３軸回転角度をα_ｔ，β_ｔ，γ_ｔとし、スタート地点に対する平行移動をＴ＝（Ｔ_ｘ，Ｔ_ｙ，Ｔ_ｚ）^?とする。すると、次の式のように、時刻ｔの３次元データを共通座標系に変換することができる。

And Here, N is the number of viewpoints. The triaxial rotation angles obtained at time t are α _t , β _t , and γ _t, and the parallel movement with respect to the start point is T = (T _x , T _y , T _z ) ^? Then, the three-dimensional data at time t can be converted into a common coordinate system as in the following equation.

ここで、Ｒ_ｘ（θ），Ｒ_ｙ（θ），Ｒ_ｚ（θ）はそれぞれＸ，Ｙ，Ｚ軸に対して角度θだけ回転したときの回転マトリクスである。

Here, R _x (θ), R _y (θ), and R _z (θ) are rotation matrices when rotated by an angle θ with respect to the X, Y, and Z axes, respectively.

  According to this processing, the three-dimensional distance information obtained at each point is unified in the world coordinate system with the start point as the origin (reference point). Then, by integrating the three-dimensional distance information at each point converted to data based on the world coordinate system, a three-dimensional environment model of the entire route that has passed can be generated (S16).

According to the said embodiment, in addition to the effect (1) and (4) in 1st Embodiment, the following effects can be acquired further.
(6) In the above embodiment, since the position information of the active sensor 12 is calculated using the gyro sensor 15 and the acceleration sensor 14 in addition to the posture information, when the active sensor 12 moves, each position after the movement is calculated. 3D distance information can be obtained.

  (7) Further, since the calculation amount is smaller than that in the case of estimating the posture only by the method of detecting the vertical edge from the omnidirectional image, and the above effect (6) allows the active sensor 12 to be moved while moving. It is possible to build a three-dimensional environment model of the passage route in time.

  (8) Furthermore, if a three-dimensional environment model is constructed, for example, when participating in a remote conference, the user can have a sense of realism. Further, when performing a remote operation by a robot in a dangerous area where a person cannot enter, the user can perform the remote operation in a highly realistic environment.

In addition, you may change each said embodiment into the following other forms (another example).
In the first embodiment, the acceleration sensor 14 may not be provided. A two-dimensional image can be generated only by the gyro sensor 15 as posture information acquisition means.

  In the second embodiment, the acceleration sensor 14 may not be provided. Also in this case, since posture information at a certain point can be obtained, a three-dimensional environment model at that point can be created. In this case, instead of constructing a model from the origin in the world coordinate system while moving the active sensor 12, it is possible to create a three-dimensional environment model at one point of the active sensor 12 in a stopped state.

  In each of the above embodiments, the acceleration sensor 14 and the gyro sensor 15 are provided in the substantially central portion of the active sensor 12, but the position is not limited to this position. In addition, for example, the same can be implemented by attaching the active sensor 12 to the stand 19 (see FIG. 3) or the vehicle body 100 (see FIG. 2).

  In the above embodiments, the active sensor 12 has a configuration in which a plurality of stereo units 18 are arranged on each surface of a regular icosahedron, but other regular tetrahedrons, regular hexahedrons, regular octahedrons, and regular dodecahedrons. Or the like. According to this, the division of the three-dimensional space becomes uniform, and the characteristics of the plurality of support members that support the camera and the cameras that are supported by the support member can be made uniform.

-It can implement as an image processing apparatus which combined the said 1st Embodiment and 2nd Embodiment.
In each of the above embodiments, the acceleration sensor 14 is used as the position information acquisition unit that acquires the position information of the active sensor 12, and the gyro sensor 15 is used as the posture information acquisition unit that acquires the posture information. It can be set as various sensors, such as a magnetic type, an ultrasonic type, and an infrared type. In short, any sensor that can detect position information and posture information can be changed as appropriate.

1 is a schematic diagram illustrating a system configuration of an image processing apparatus according to an embodiment. The schematic of the mechanical structure of an active sensor. The whole external view of an active sensor. The figure explaining the attachment aspect of the gyro sensor and acceleration sensor which are provided in an active sensor. The flowchart which shows the process which concerns on 1st Embodiment. The figure explaining the relationship between the vertical edge and the inclination of an active sensor. It is a figure which shows the result of having decompress | restored the two-dimensional image as a cylindrical image, (a) shows the image without correction | amendment, (b) shows the image after performing attitude | position correction | amendment using an attitude | position parameter. The flowchart which shows the process which concerns on 2nd Embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 11 ... Image processing apparatus (image generation apparatus, three-dimensional information acquisition apparatus), 12 ... Active sensor (image photographing apparatus), 13 ... Computer (synchronization means, estimation means, two-dimensional image generation means, three-dimensional distance information acquisition means, (3D environment model generation means), 14 (acceleration sensor (position information acquisition means)), 15 (gyro sensor (posture information acquisition means)), 18 (stereo unit).

Claims (8)

An image capturing device for acquiring images in all directions;
Posture information acquisition means provided in the image photographing device for obtaining posture information of the image photographing device;
Synchronization means for temporally synchronizing the image information photographed by the image photographing device and the posture information obtained by the posture information obtaining means;
Image information synchronized with posture information by the synchronization means is generated as a two-dimensional image having no inclination with respect to a predetermined axis of a three-dimensional world coordinate system of the environment where the image capturing device is located based on the posture information. An image generation apparatus comprising: a two-dimensional image generation unit. The image generating apparatus according to claim 1, wherein the predetermined axis is a vertical axis. An image capturing device that acquires stereo images in all directions;
Posture information acquisition means provided in the image photographing device for obtaining posture information of the image photographing device;
Synchronization means for temporally synchronizing the image information photographed by the image photographing device and the posture information obtained by the posture information obtaining means;
3D distance information acquisition means for acquiring 3D distance information from image information synchronized with posture information by the synchronization means;
Conversion means for converting the three-dimensional distance information into three-dimensional distance information based on the three-dimensional world coordinate system of the environment in which the image capturing device is located based on the posture information. Distance information acquisition device. The three-dimensional distance information acquisition apparatus according to claim 3, wherein the posture information acquisition unit includes a gyro sensor. The three-dimensional distance information acquisition device further includes three-dimensional environment model generation means for generating a three-dimensional environment model from the three-dimensional distance information converted by the conversion means. The three-dimensional distance information acquisition device described. The three-dimensional distance information acquisition device includes a position information acquisition unit that is provided in the image shooting device and acquires position information of the image shooting device;
The three-dimensional distance information acquisition apparatus according to claim 3, wherein the synchronization unit synchronizes the image information and the position information. The three-dimensional distance information acquisition apparatus according to claim 6, wherein the position information acquisition unit includes an acceleration sensor. The three-dimensional distance information acquisition device includes an estimation unit that estimates an inclination of the image photographing device with respect to a world coordinate system where the image photographing device is located based on an edge of the image obtained by the image photographing device and its direction. 8. The three-dimensional distance information acquisition device according to claim 3, further comprising: estimating the posture of the image capturing device by the estimating unit based on the posture information. 9.

Images