JP2005284992A - Electronic apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an electronic apparatus for composing a virtual object and a real video image with high positioning accuracy while reducing a labor of a user. <P>SOLUTION: A display device 10 converts an optical image including feature points photographed by a camera 11 in different attitudes into a plurality of image data. Coordinates of the feature points photographed in the image data are then acquired by a pointing device. Furthermore, a position of the camera corresponding to a tripod 14 is determined by an arithmetic section 18 based on coordinate values and a rotating angle of the camera 11. Therefore, the position of the camera 11 can be accurately known. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an electronic device that realizes augmented reality.

  When creating something new in a park or street, it is important to know how the newly created object affects the surrounding landscape. Considering a construction problem as an example, when creating a new construction structure, it is necessary to consider whether or not the structure damages the surrounding landscape. At present, in order to satisfy this need, there are a first method for showing the environment after construction with pictures and photographs and a second method for showing the environment after construction with a model. In this first method, it is difficult to grasp the post-construction situation viewed from various angles. In the second method, there is a limit in restricting all surrounding environments.

  In order to satisfy the above-described requirements, recently, it is also possible to check a post-construction situation using a virtual reality that builds a virtual world in a computer. However, in this artificial reality, since all of the objects provided to the user are virtual objects, it is difficult to reproduce the entire surrounding environment.

As a method of compensating for the disadvantages of CG, there is an augmented reality technology (see Non-Patent Document 1 below). Augmented reality is a technology that superimposes a virtual object on the real world. With this technique, the user can obtain more realistic visual information as compared with the other methods described above, and therefore can more accurately predict changes in the landscape after construction.
http://www.tobi-tech.com/tech/mr.htm (searched on February 19, 2004)

  However, the conventional augmented reality technology has the following three problems. Specifically, there are a first problem to obtain geometric matching, a second problem to obtain temporal matching, and a third problem to obtain optical matching.

  The first problem is a three-dimensional alignment problem between the real world and the virtual object. It is necessary to match the virtual object with the real world so that the virtual object is displayed at the correct position in the real world and its size is also accurate.

  The second problem is a problem of time delay of virtual object display with respect to the real world. When the position and orientation of the camera is changed, it is necessary to make the movement of the virtual object and the movement of the real world the same.

  The third problem is the problem of the difference in image quality and shadow between the real world and the virtual world. It is necessary to match so that the optical conditions such as the brightness and shadow of the virtual object displayed on the display or the like are the same as those in the real world.

  In the augmented reality technology, each of the above three problems is important. Among them, the most important one is the first problem of geometric matching. When comparing a structure to be constructed with the surrounding landscape, there is a desire to check images using cameras from various angles and locations. However, there has been a problem that a suitable method for performing alignment between the real world and the virtual object when the position of the camera is moved has not been developed.

  Furthermore, although the camera is usually fixed to a support means such as a tripod to take a picture of the real world, there is a problem that it is difficult to obtain the relative positional relationship between the camera and a fixing portion for fixing the camera.

  The present invention has been made in view of the above-described problems, and a main object of the present invention is to provide an electronic device that combines a virtual object and a live-action image with high positional accuracy while reducing user effort. There is.

  An electronic apparatus according to the present invention includes a photographing unit that photographs a real-world optical image including a feature point and converts the image into image data, a display unit that displays an image based on the image data, and a fixing unit that fixes the photographing unit. Obtaining a coordinate on the display means of the feature point displayed on the display means; a support means for rotatably supporting the fixing means; a measuring means for measuring a rotation angle of the fixing means; And a calculation means for performing a calculation based on the coordinates of the feature points. The shooting means has a first image data and a posture different from that when the first image data is shot. 2 is captured, and the acquisition means is a first coordinate that is a coordinate of the feature point that appears in the first image data, and a coordinate of the feature point that appears in the second image data. A second coordinate and the measuring hand Measures the rotation angle of rotation of the fixing means when the second image data is photographed, and the calculation means is based on the first coordinates, the second coordinates and the rotation angle, The position of the photographing means relative to the fixing means is obtained.

  Furthermore, in the electronic device of the present invention, the measuring means includes a first measuring means for measuring a horizontal rotation angle of the fixing means, and a second measuring means for measuring a vertical rotation angle of the fixing means. It is characterized by comprising.

  Furthermore, in the electronic apparatus of the present invention, the acquisition unit is a pointing device, and the feature point displayed on the display unit is selected by the pointing device, whereby coordinates of the feature point on the display unit are selected. It is characterized by acquiring.

  Furthermore, in the electronic apparatus of the present invention, the calculation means calculates the position of the photographing means by a statistical method.

  Furthermore, in the electronic device of the present invention, the calculation means calculates a change in position and orientation of the photographing means and the fixing means when the first image data is photographed and when the second image data is photographed. Then, the position of the photographing means relative to the fixing means is obtained.

  An electronic apparatus according to the present invention includes a photographing unit that photographs a real-world optical image including a feature point and converts the image into image data, a display unit that displays an image based on the image data, and a supporting unit that supports the photographing unit. Acquisition means for acquiring the coordinates of the feature points displayed on the display means on the display means, and calculation means for calculating the position of the support means relative to the real world based on the coordinates of the feature points It is characterized by comprising.

  Furthermore, in the electronic apparatus of the present invention, the calculation means calculates the coordinates of the support means based on the coordinates of the plurality of feature points included in one image data.

  Furthermore, in the electronic apparatus according to the present invention, the photographing unit is rotatably fixed to the support unit, and includes a measuring unit that measures a rotation angle of the photographing unit, and the acquiring unit rotates the photographing unit. Obtains the coordinates of the feature point on the display means, and the computing means calculates from the coordinates of the feature point obtained by the obtaining means and the rotation angle of the photographing means when the feature point is earned. The position of the support means is calculated.

  Furthermore, in the electronic apparatus of the present invention, the acquisition unit is a pointing device, and the feature point displayed on the display unit is selected by the pointing device, whereby coordinates of the feature point on the display unit are selected. It is characterized by acquiring.

  Furthermore, in the electronic apparatus of the present invention, the calculation means calculates the position of the photographing means by a statistical method.

  Furthermore, in the electronic device of the present invention, the measuring means includes a first measuring means for measuring a horizontal rotation angle of the fixing means, and a second measuring means for measuring a vertical rotation angle of the fixing means. It is characterized by comprising.

  Furthermore, in the electronic apparatus of the present invention, the photographing unit is fixed to a fixing unit, and the fixing unit is rotatably fixed to the supporting unit.

  According to the electronic apparatus of the present invention, the image capturing means and the image capturing means are obtained by capturing the feature points arranged in the real world and acquiring the coordinate values on the display means of these feature points by the acquiring means. The relative positional relationship with the supporting means to support can be detected. Therefore, it is possible for the user to specify the position of the photographing means by a simple method.

  Furthermore, according to the electronic apparatus of the present invention, the imaging unit is supported by imaging a plurality of feature points by the imaging unit and acquiring the coordinate values on the display unit at these feature points by the acquisition unit. It becomes possible to easily calculate the coordinates of the support means in the real world. The feature points can be photographed using a plurality of feature points reflected in one field of view of the photographing means. Furthermore, it is also possible to photograph a plurality of feature points arranged discretely while rotating the photographing means. This allows the user to move the fixing means to a more free place.

  Hereinafter, an electronic apparatus according to an embodiment of the present invention will be described with reference to the drawings.

<First form>
In this embodiment, a configuration of an electronic device 10 that superimposes a real image and a virtual object, a method of using the electronic device 10, and the like will be described.

  With reference to FIG. 1, the mechanical and electrical configuration of the electronic apparatus 10 of the present embodiment will be described. In the electronic apparatus 10 of this embodiment, the camera 11 is fixed to a tripod 14 via a pan head 13 so as to be rotatable. The user 16 can freely rotate the camera 11 with the operation lever 15 while recognizing the monitor 12 as display means fixed to the camera platform 13.

  A computer 17 as a calculation means is connected to the above-described camera 11, monitor 12, and the like, and performs calculation based on a predetermined program. Moreover, each component of this form is electrically connected.

  The tripod 14 has a role of supporting the entire apparatus, and the camera 11 and the like are fixed to the upper part of the tripod 14 via the pan head 13. Further, the tripod 14 is fixed so as to be movable with respect to the ground so that the entire apparatus can be moved according to the user's request. Furthermore, the leg of the tripod 14 can be expanded and contracted, and the height of the camera 11 and the like can be adjusted by extending and contracting the leg.

  The pan head 13 is interposed in the upper part of the tripod 14, and fixes the camera 11 etc. to the tripod rotatably. The pan head 13 is provided with an angle sensor (not shown). Further, there is a part that functions as a fixing means for fixing the camera 11 at the upper part of the camera platform 13, and an operation lever 15 is connected to this part.

  Details of the angle sensor will be described with reference to FIG. First detection means 20A for detecting a change in the angle in the horizontal direction with respect to the head grounding surface on the tripod, and a second detection for detecting a change in the angle in the vertical direction with respect to the head grounding surface on the tripod. The display device 10 is equipped with an angle sensor comprising means 20B. By detecting the angle by these detection means, it is possible to know a change in the posture of the camera fixed on the top of the pan head 13. The resolution of the detection means described above is, for example, 24000 steps / rotation, which is a very high-performance sensor. In the following description, a direction parallel to the pan / tilt ground contact surface on the tripod may be referred to as a pan direction, and a direction perpendicular to the pan / tilt head ground surface may be referred to as a tilt direction. Further, the angle information obtained by the first measuring unit 20A and the first measuring unit 20B is transmitted to the calculation unit 18. These angle information may be stored in the storage unit 19.

  The camera 11 is fixed to the top of the camera platform 13 and has a function of converting a light image in the real space into image data. As the camera 11, a camera including an individual photographing element that converts an optical signal into an electric signal is suitable. Specifically, a camera 11 incorporating an individual photographing element having a CCD (charge-coupled device) or an individual photographing element having a CMOS (complementary MOS) can be applied to this embodiment. With reference to FIG. 1B, the electrical signal constituting the image data converted by the camera 11 is transmitted to the calculation unit 18.

  The monitor 12 is fixed on the camera platform 13 together with the camera 11 described above. The monitor 12 displays the video output from the calculation unit 18. Specifically, image data captured by the camera 11 is processed by the calculation unit 18 to generate video data, and a video based on the video data is displayed on the monitor 12. The monitor 12 continuously displays video based on image data captured by the camera 11 in real time. As a result, the user 16 can accurately know the optical image captured by the camera.

  Examples of the process described above include a process of superimposing CG (computer graphics) on image data captured by the camera 11. The monitor 12 is preferably a thin liquid crystal display in consideration of portability. Further, the monitor 12 is not necessarily fixed to the camera platform 13 and may be installed separately from the tripod 14.

  The operation lever 15 is a rod-like lever fixed to the upper part of the pan head 13. The user 16 can freely rotate the camera 11 fixed to the camera platform 13 by operating the operation lever 15. The behavior of rotation at this time is measured by the first measuring means 20A and the second measuring means 20B described above, and is stored in the storage unit 19.

  The storage unit 19 is a recording medium having a plate shape such as a hard disk or a semiconductor recording medium, and is a part having a function of recording information. Information stored in the storage unit 19 includes information on the CG to be superimposed, setting information of the entire apparatus, coordinates of feature points used for specifying the position, and angle change measured by the first measuring unit 20A. Information on the angle change measured by the second measuring means 20B, or a program for controlling the behavior of each part.

  Next, with reference to FIG. 2 (A), the coordinate system used when describing this form is demonstrated. The alignment between the real world and the virtual object requires a transformation matrix from the real world coordinates to the camera coordinates. In this embodiment, the following coordinate system is used.

  The world coordinate C1 is a coordinate system set in the real world, and is a coordinate system used for expressing the positions of CG structures and feature points. Here, the feature point is a point for which a three-dimensional coordinate value used as a reference when calculating a transformation matrix between coordinate systems is known.

  The tripod coordinate system C2 is a coordinate system based on the tripod head, and the coordinate system changes when the user moves the position recognition device.

  The tripod PT coordinate system C3 has the same origin position as the tripod coordinate system, but is a coordinate system rotated by a user's operation in the tilt and pan directions. Therefore, when the tilt angle and the pan angle are zero degrees, the tripod PT coordinate system C3 coincides with the tripod coordinate system C2.

  The camera coordinate system C4 is a coordinate system based on the focus of the camera. Therefore, there is a possibility that this coordinate system is moved by changing the lens of the camera.

  The screen coordinate system C5 is a coordinate system of a secondary original image displayed on the screen of the monitor 12 or the like. In addition, since distortion caused by the lens is generated in the image observed by the camera, the image captured by the camera is corrected to remove this distortion.

  When the relationship between the coordinates is clarified, it becomes possible to superimpose a CG whose coordinate value is known in the world coordinate system and an optical image reflected on the lens. Here, the relationship between the coordinate systems means that a conversion matrix between the coordinates is specifically obtained. With reference to FIG. 2B, a description will be given of a transformation matrix that performs transformation of each coordinate system.

  The matrix M1 is a transformation matrix from the world coordinate system to the tripod coordinate system. Specifically, when positioning the real world and the virtual object, the position and orientation of the tripod in the real world is necessary. The transformation matrix from the world coordinate system to the tripod coordinate system need only be obtained once when the tripod is installed. This can be realized by using several points of image data and three-dimensional coordinate values in the real world and performing calibration. If you change the position of the tripod in the real world, you will need to do this again. The calibration for obtaining the matrix M1 will be described later. Furthermore, since the user moves to a desired location, the relationship between the world coordinate system and the tripod coordinate system cannot be measured in advance. Therefore, it is obtained when the user installs a tripod in the real world site.

  The matrix M2 is a transformation matrix from tripod PT coordinates to tripod coordinates. Specifically, the conversion from the tripod PT coordinate system to the tripod coordinate system is obtained from angle information in the pan / tilt direction obtained from the first and second measuring means 20A and 20B which are sensors mounted on the tripod. Can do. If the rotations in the pan / tilt directions are 1P [rad] and 2P [rad], respectively, this conversion is expressed by the following equation (1). As described above, the measuring means of this embodiment is highly accurate. By this measuring means, the user's operation in the pan / tilt direction is input to the calculation unit 18 in real time and with high accuracy.

行列Ｍ３は、カメラ座標系から三脚ＰＴ座標系までの変換行列である。具体的には、本システムでは三脚１４の雲台１３にカメラ１１を固定しているが、雲台１３のパン・チルト方向の回転軸とカメラ座標系の原点がずれている。ゆえに雲台１３のパン・チルト方向への回転とカメラの位置姿勢変化の関係を表現するための変換行列をあらかじめ求めておく必要がある。更に詳述すると、カメラ座標系の原点はカメラの焦点位置にあり、三脚ＰＴ座標系とは、原点位置も向きもことなる。しかし、カメラ１１は三脚１４上にしっかりと固定されているので、ねじが緩んだり、壊れない限り変化しない。実際は、カメラは三脚から容易に取り外したり、取り付けたりすることあるが、取り付け自体がしっかりしているので、毎回の取り付けによって位置や角度が変わることがない。つまり、この関係は、事前に一度求めておけばよい。この三脚とカメラのオフセットを表す変換行列を求めるキャリブレーションは後述する。

The matrix M3 is a transformation matrix from the camera coordinate system to the tripod PT coordinate system. Specifically, in this system, the camera 11 is fixed to the pan head 13 of the tripod 14, but the rotation axis of the pan head 13 in the pan / tilt direction is shifted from the origin of the camera coordinate system. Therefore, it is necessary to obtain in advance a transformation matrix for expressing the relationship between the rotation of the camera platform 13 in the pan / tilt direction and the change in the position and orientation of the camera. More specifically, the origin of the camera coordinate system is at the focal position of the camera, and the origin position and orientation of the tripod PT coordinate system are different. However, since the camera 11 is firmly fixed on the tripod 14, it does not change unless the screw is loosened or broken. In practice, the camera may be easily removed from or attached to the tripod, but since the attachment itself is firm, the position and angle will not change with each attachment. In other words, this relationship may be obtained once in advance. The calibration for obtaining a conversion matrix representing the tripod and camera offset will be described later.

  The matrix M4 is a transformation matrix from the camera coordinate system to the screen coordinate system. Specifically, a function provided by the augmented reality construction tool ARToolKit [6] is used to derive this transformation matrix. A specific method for calculating M4 will be described later. Since the relationship between the camera coordinate system and the screen coordinate system does not change unless the camera 11 is replaced, it may be obtained once in advance.

  Next, a method for obtaining the transformation matrix M3 from the camera coordinate system C4 to the tripod PT coordinate system will be described with reference to FIGS. FIG. 3 is a flowchart for explaining this method in detail, and FIG. 4 is a schematic diagram showing the relationship between the coordinate systems. In this embodiment, a plurality of images are taken while rotating the camera in the pan direction and the tilt direction, and the transformation matrix M3 is obtained from the feature points reflected in the images.

  With reference to FIG. 3, specific steps for calculating the conversion matrix M3 will be described.

  In step S1, the three-dimensional coordinates of feature points located in the real world are measured. Here, the characteristic point is preferably a characteristic point that can be visually recognized by the user. Specifically, any object whose position can be specified, such as a corner of a building, the base of a sign, or an antenna, can be used as a feature point. Further, when there is no such object, an appropriately arranged marker can be employed as the feature point. Furthermore, the three-dimensional position of the feature point can be measured using a measuring instrument called a total station. Here, an object having a visually distinct color, an object to be issued, or the like can be employed as the marker.

  In step S2, the feature point is photographed by a camera. That is, the feature point is photographed by adjusting the position and orientation of the camera so that the feature point is photographed.

  In step S3, coordinates on the monitor of the feature point imaged in step S2 are acquired. This acquisition operation can be performed using a pointing device such as a mouse. Specifically, coordinates can be acquired by the user operating the mouse to move the mouse cursor to the location of the feature point displayed on the monitor and clicking. Here, as the pointing device, a tablet or the like can be used in addition to the mouse. In addition, the image data acquired by the camera may be analyzed by a program to calculate the position of the feature point on the monitor.

  In step S4, the camera is rotated. In this embodiment, the camera is rotated in the pan direction and the tilt direction. This makes it possible to perform calibration for obtaining a transformation matrix in which both the pan direction and the tilt direction are considered. The rotation angle of the camera in the pan direction and the tilt direction in this step is accurately measured by the first measurement unit 20A and the second measurement unit 20B, and the information is stored in the storage unit 19.

  And after rotating a camera in step S4, it returns to step S2 and image | photographs a feature point. That is, the feature point that appears on the monitor is shot a plurality of times after the camera is rotated.

  In this embodiment, the transformation matrix is obtained by a statistical method. Therefore, generally, the more the number of times of shooting described above, the higher the accuracy of the calculated conversion matrix. However, when the coordinates of the feature points on the monitor are manually acquired by the user, if the number of acquired coordinates is too many, the user gets tired and the accuracy of the acquired coordinates itself is reduced. There is a fear. For this reason, the feature points are photographed within a range in which accuracy is ensured without imposing a burden on the user.

  In step S5 and later described in detail below, the transformation matrix M3 is calculated from the information on the coordinate pair and the rotation angle obtained in the above step. That is, the relative positional relationship between the camera 11 and its fixed part (the upper part of the pan head 13) is clarified by photographing while rotating a plurality of feature points arranged in the real world.

In step S5, a transformation matrix T _i from the world coordinate system C1 _i to the camera coordinate system C4 _i is obtained for the coordinate pair. Further, the transformation matrix T _i is a matrix as shown in the following Equation 2. Here, i indicates the number of images to be captured. In order to obtain this conversion matrix, a function provided by ARToolKit [6] is used. As a specific method, referring to FIG. 5, a plurality of black spots 25 arranged on the surface of the board 24 are photographed using a camera, and operations such as obtaining the center of gravity of the black spots 25 are performed. A transformation matrix of Formula 2 is obtained.

ステップＳ６では、座標ペアに対して、三脚ＰＴ座標系Ｃ３から三脚座標系Ｃ２までの変換を行う変換行列Ｕ_ｉを求める。ここで、変換行列Ｕ_ｉは以下の数３に示すような行列である。この変換行列は、上述した数１を用いて求めることができる。

In step S6, a transformation matrix U _i for performing transformation from the tripod PT coordinate system C3 to the tripod coordinate system C2 is obtained for the coordinate pair. Here, the transformation matrix U _i is a matrix as shown in Equation 3 below. This conversion matrix can be obtained using the above-described equation (1).

ステップＳ７では、三脚ＰＴ座標系の位置姿勢の変化を示す変換行列Ｓ_ｊを求め、更に、カメラ座標系の位置姿勢の変化を示す変換行列Ｃ_ｊを求める。変換行列Ｓ_ｊを下記数４に示し、変換行列Ｃ_ｊを下記数５に示す。ここで、ｉ＝１、２、３・・・ｎ＋１のとき、ｊ＝１、２、３・・・ｎである。

In step S7, a transformation matrix S _j indicating the change in position and orientation of the tripod PT coordinate system is obtained, and further, a transformation matrix C _j showing the change in position and orientation in the camera coordinate system is obtained. The transformation matrix S _j is shown in the following equation 4, and the transformation matrix C _j is shown in the following equation 5. Here, when i = 1, 2, 3,... N + 1, j = 1, 2, 3,... N.

上記した変換行列Ｃ_ｊの具体的な算出方法を、図４を参照して説明する。変換行列Ｃ_１は、１番目と２番目に於けるＴ_１とＴ_２から導出することができる。世界座標系Ｃ１は常に静止して動かない座標系である。そして、変換行列Ｔ_１は世界座標系Ｃ１からカメラ座標系Ｃ４_１への変換を行う行列である。また、変換行列Ｔ_２は、世界座標Ｃ１からカメラ座標系Ｃ４_２への変換行列である。ここで、求めたい変換行列Ｃ_１は、カメラ座標系Ｃ４_１からカメラ座標系Ｃ４_２への変換行列である。この変換行列Ｃ_１の変換ルートは、Ｃ４_１→Ｃ４_２である。更にこの変換ルートは、Ｃ４_１→Ｃ１→Ｃ４_２とみなすことができる。図４では、このルートをＲ１で示している。このことから、ｊ番目の変換行列Ｃ_ｊは下記数６で求めることができる。

A specific method for calculating the transformation matrix C _j will be described with reference to FIG. The transformation matrix C ₁ can be derived from T ₁ and T _{2 in the first} and _second . The world coordinate system C1 is a coordinate system that is always stationary and does not move. The transformation matrix T ₁ is a matrix for performing transformation from the world coordinate system C 1 to the camera coordinate system C 4 ₁ . The conversion matrix _{T 2} are a transformation matrix from a world coordinate C1 to the camera coordinate system C4 _2. Here, the transformation matrix C _{1 to be} obtained is a transformation matrix from the camera coordinate system C 4 ₁ to the camera coordinate system C 4 ₂ . The conversion route of this conversion matrix C ₁ is C 4 ₁ → C 4 ₂ . Further, this conversion route can be regarded as C4 ₁ → C1 → C4 ₂ . In FIG. 4, this route is indicated by R1. From this, the j-th transformation matrix C _j can be obtained by the following _equation (6).

またＣ_ｊを求めた上記と同様の考え方で、下記数７でｊ番目の変換行列Ｓ_ｊは下記数７で求めることができる。

Further, based on the same idea as described above for _obtaining C _j , the j-th transformation matrix S _j can be obtained by the following _equation (7).

以上の計算で、カメラ座標系および三脚ＰＴ座標系の姿勢変化を示す変換行列Ｃ_ｊおよびＳ_ｊが計算される。

With the above calculation, the transformation matrices C _j and S _j indicating the posture change of the camera coordinate system and the tripod PT coordinate system are calculated.

In step S8, R, which is a conversion matrix from the camera coordinate system C4 _i to the tripod PT coordinate system C3 _i , is calculated using the conversion matrices C _j and S _j calculated in the above step. Here, R is a matrix as shown in Equation 8 below.

図４を参照して、変換行列Ｒを算出するために、カメラ座標系Ｃ４_１から三脚ＰＴ座標系Ｃ３_２を算出する経路を考えてみる。この経路は二通が考えられる。１つの経路は、Ｃ４_１→Ｃ４_２→Ｃ３_２の経路Ｒ２である。それに対して、もう一つの経路は、Ｃ４_１→Ｃ３_１→Ｃ３_２の経路Ｒ３である。どちらの経路でも同じ座標系に到達する。更に、Ｃ_ｊおよびＳ_ｊは上記のステップにより既に求められているので、以下の数９が得られる。

Referring to FIG. 4, in order to calculate a transformation matrix R, consider the path from the camera coordinate system C4 ₁ calculates a tripod PT coordinate system C3 _2. There are two possible routes. One path is a path R2 of C4 ₁ → C4 ₂ → C3 ₂ . On the other hand, another route is a route R3 of C4 ₁ → C3 ₁ → C3 ₂ . Both paths reach the same coordinate system. Further, since C _j and S _j have already been obtained by the above steps, the following formula 9 is obtained.

上記数９を変形すると、以下の数１０を得る。

When the above formula 9 is transformed, the following formula 10 is obtained.

上記数１０を展開すると下記の数１１を得る。

When the above formula 10 is expanded, the following formula 11 is obtained.

ｎ＋１枚の画像を取り込んだ場合、上記数１１をｎ個得ることができる。そこで、ｎ個の上記数１１をまとめた物を以下の数１２と考えると、Ｊは１２ｎ×１行列、Ｋは１２ｎ×１１行列、Ｌは１１×１行列となる。

When n + 1 images are captured, n of the above equation 11 can be obtained. Therefore, when the above-described n number 11 is summarized as the following number 12, J is a 12n × 1 matrix, K is a 12n × 11 matrix, and L is an 11 × 1 matrix.

このことから、最小二乗法を用いて下記数１３が得られる。数１３では、正方行列の逆行列を用いた最小二乗法による演算を行っている。

From this, the following Equation 13 is obtained using the least square method. In Equation 13, an operation is performed by the least square method using an inverse matrix of a square matrix.

この数１３を計算することで、変換行列Ｒの全ての成分を求めることができる。即ち、本形態では、カメラ座標系Ｃ４から三脚座標ＰＴ系Ｃ３への変換行列を計算する際に、先ず、これらの座標系の位置姿勢の変化を表す変換行列（Ｃ_ｊおよびＳ_ｊ）を求める。その後に、Ｃ_ｊおよびＳ_ｊを用いた統計学的手法により、変換行列Ｒを算出している。

By calculating Equation 13, all components of the transformation matrix R can be obtained. That is, in this embodiment, when calculating the conversion matrix from the camera coordinate system C4 to the tripod coordinate PT system C3, first, conversion matrices (C _j and S _j ) representing changes in the position and orientation of these coordinate systems are obtained. . Thereafter, the transformation matrix R is calculated by a statistical method using C _j and S _j .

  As a method for obtaining the transformation matrix R, a method for solving simultaneous equations mathematically instead of the statistical method described above is also conceivable. However, the statistical method described above is suitable as a method for obtaining the transformation matrix R in this embodiment. Considering the case where the transformation matrix R is calculated by a mathematical method, there is a problem that one must be selected from a plurality of obtained solutions, a problem that a stable solution may not be obtained, etc. This is because there is a risk of occurrence. In comparison, the method of calculating the transformation matrix R using the statistical method employed in this embodiment has an advantage that a stable solution can be obtained. Furthermore, this embodiment has an advantage that the accuracy of the obtained solution is improved when the number of feature points to be observed is increased.

  With reference to FIG. 6 and FIG. 7, a method for calculating a transformation matrix for specifying the position when the display device 10 is moved will be described. FIG. 6 is a flowchart for obtaining a transformation matrix, and FIG. 7 is a conceptual diagram showing how the virtual object 22 is observed by moving the display device 10.

  For example, in order to know in detail how a virtual object such as a structure to be constructed has a visual influence on the surrounding landscape, it is necessary to install the display device 10 at a location desired by the user. In such a case, every time the display device 10 is moved, it is necessary to recalculate the transformation matrix M1 for performing transformation from the world coordinate system C1 to the tripod coordinate system C2. Hereinafter, each step for recalculating the transformation matrix M1 will be described.

  In step S11, the display device 10 is moved. That is, the user moves the display device 10 to a desired position so that the comparison between the virtual object and the surrounding landscape can be suitably performed. By performing this step multiple times, the user can know in detail how the virtual object affects the surroundings.

  In step S12, the feature points arranged in the real world are photographed. There are two possible methods for photographing the feature points. The first method is a method of photographing a plurality of feature points that fit in one video without rotating the camera. The second method is a method of photographing a plurality of feature points arranged over a wide range while rotating the camera. In order to improve accuracy, the second method in which feature points can be arranged in a wide range is preferable. In the following description, the second method is used for explanation, but the present embodiment can also be implemented by the first method. The details of the feature points are the same as those described with reference to FIG.

  In step S13, the coordinate value of the feature point on the screen is acquired. Specifically, the planar coordinates of the feature points displayed on the monitor 12 are acquired using a pointing device. This makes it possible to obtain a plurality of coordinate pairs consisting of the three-dimensional coordinate value of the feature point in the real world and the two-dimensional coordinate value on the monitor. A specific method for acquiring coordinate values on the monitor is the same as the method described with reference to FIG. Further, in order to distinguish between feature points, an ID is specified when acquiring coordinates on the monitor of each feature point. As a result, a plurality of coordinate pairs consisting of the three-dimensional coordinate value of the feature point in the real world and the two-dimensional coordinate value on the monitor are obtained.

  In step S14, the camera is rotated. In this embodiment, since the transformation matrix is calculated using the coordinate values of the feature points, the feature points are preferably distributed over a wide range. Furthermore, depending on the relative positions of the display device 10 and the feature points, the number of feature points that can be displayed in one field of view of the camera may change. Therefore, in this embodiment, the feature point is photographed while rotating the camera. The rotation angle of the camera in the pan direction and tilt direction at this step is always measured and stored.

  After rotating the camera so that other feature points are reflected on the monitor, the process returns to step S12 to shoot other feature points, and in step S13, the coordinate values of the feature points on the monitor are obtained. To do.

  With reference to FIG. 7, a method for acquiring coordinates on a feature point monitor will be described using the display device 10 </ b> A located on the left side of the drawing as an example. First, the first and second feature points P1 and P2 are photographed, and coordinate values on the monitor of each feature point are acquired. Next, the camera of the display device 10A is rotated so that the third feature point P3 to the fifth feature point P5 are reflected on the monitor. And the coordinate value on the monitor of the 3rd feature point P3 to the 5th feature point P5 is acquired. Through the above steps, even when a plurality of feature points are arranged over a wide range, coordinate values on the monitor of those feature points can be acquired and coordinate pairs can be acquired.

  In step S15, a transformation matrix M1 from the world coordinate system C1 to the tripod coordinate system C2 is calculated using the coordinate pair obtained in the above step. That is, the position and orientation of the tripod in the world coordinate system C1 are obtained. The specific method is described below.

  First, consider the following equation 14 which performs coordinate transformation from the world coordinate system C1 to the screen coordinate system C5.

数１４にて、Ｘ_Ｓを含む左辺の行列は、モニタ上に映し出された特徴点の位置である。換言すると、ユーザーがクリックを行ったマウスポインタの位置である。そして、Ｘ_Ｗを含む右辺の行列は、特徴点の世界座標系での位置であり、予め測定されている。また、左辺のｈは、行列式の変換を行う際に、３次元から２次元への変換を行う変数である。この変数ｈは、一般的に同次座標系と呼ばれている。尚、上記数１４に現れるＭ１からＭ４は、図２を参照して説明された変換行列である。

At number 14, the left side of the matrix containing the X _S is the position of the projected feature points on the monitor. In other words, it is the position of the mouse pointer where the user clicked. The matrix on the right side containing X _W is a position in the world coordinate system of the feature points is previously measured. In addition, h on the left side is a variable for performing conversion from three dimensions to two dimensions when performing determinant conversion. This variable h is generally called a homogeneous coordinate system. Note that M1 to M4 appearing in Equation 14 are the transformation matrices described with reference to FIG.

  Here, by measuring the n feature points while rotating the camera, the coordinate value of the feature point in the screen coordinate system, the coordinate value of the feature point in the world coordinate system, and the combination of M3 is n. Can be obtained. Here, when the above equation 14 is rewritten, the following equation 15 is obtained.

上記数１５にて、Ｍ（４−２）はＭ４・Ｍ３^−１・Ｍ２^−１である。そして、添え字のｉは、１、２、・・・・ｎである。即ち、ｎ点の特徴点について上記ステップを行った場合は、ｎ個の数１５を得ることができる。そして、ｎ個の数１５を用いた誤差最小化法により変換行列Ｍ１を算出する。具体的には、下記数１６のような定義を行い、下記数１７を最小化するようなＭ１を求めればいい。数１６および数１７にて、ハットマークが上部に付された変数は、ユーザーの操作により取得されたスクリーン座標系の特徴点の座標値を示している。

In the above equation 15, M (4-2) is M4 · M3 ⁻¹ · M2 ⁻¹ . The subscript i is 1, 2,... N. That is, when the above steps are performed for n feature points, n number 15 can be obtained. Then, a transformation matrix M1 is calculated by an error minimization method using n number 15. Specifically, a definition like the following formula 16 is made, and M1 that minimizes the following formula 17 may be obtained. In Equations 16 and 17, the variable with a hat mark at the top indicates the coordinate value of the feature point of the screen coordinate system acquired by the user's operation.

即ち、数１６及び数１７を用いた算出方法では、特徴点の世界座標系に於ける座標値から算出される特徴点のスクリーン座標系での計算値が、実際の値に近くなるように、変換行列Ｍ１を求める。

That is, in the calculation method using Equations 16 and 17, the calculated value of the feature point in the screen coordinate system calculated from the coordinate value of the feature point in the world coordinate system is close to the actual value. A transformation matrix M1 is obtained.

  Through the above steps, the transformation matrix M1 for performing transformation from the world coordinate system C1 to the tripod coordinate system C2 is calculated. As a result, all transformation matrices for performing coordinate transformation from the world coordinate system C1 to the screen coordinate system C5 were obtained.

  In step S16, the transformation matrix obtained as described above is applied to the virtual object to be displayed, so that the virtual object is superposed on the image showing the real world while maintaining the geometric consistency of the virtual object. Further, when shooting from a different location, the process returns to step S11 and the display device 10 is moved.

  Through the steps of the present embodiment described above, the user can move the display device 10 to a desired location and obtain an image in which the virtual object 22 and the surrounding landscape are combined. Therefore, the user can know more precisely how the virtual object 22 such as a structure to be constructed affects the surrounding landscape. Specifically, even when the display device 10 is moved, a plurality of feature points are observed using the display device, thereby obtaining a conversion matrix for performing conversion from the world coordinate system C1 to the tripod coordinate system C2. be able to. Further, other conversion matrices necessary for conversion from the world coordinate system C1 to the screen coordinate system C5 are known. Therefore, by using these transformation matrices, the virtual object 22 can be displayed while maintaining geometric consistency.

  Furthermore, in this embodiment, it is possible to capture the feature points while rotating the camera equipped in the display device 10. Therefore, since the feature points can be arranged in a wide range, the accuracy of the transformation matrix calculated using the feature points can be improved.

<Second form>
In this embodiment, the description will focus on the video display method using the display device described in the first embodiment.

  With reference to FIG. 8 and FIG. 9, a method of superimposing a virtual object on a real photographed image of the real world using the electronic device 10 whose positional relationship has been clarified by the first embodiment will be described. 8A shows an image displayed on the monitor 12 shown in FIG. 1, and FIG. 8B is an enlarged view of a portion where the box 27 is projected. FIG. 8C shows an image in which a virtual object is superimposed on a real image, and FIG. 8C is an enlarged view thereof.

  Referring to FIG. 8A, first, planar coordinates on a monitor of a plurality of feature points arranged in the real world are acquired. In the figure, a box 27 is placed on the top of the desk 26. Then, the three-dimensional positions in the real world of the corners of the box 27 that are feature points are measured. The position of the feature point can be measured with a surveying instrument such as a total station.

  With reference to FIG. 8B, the planar coordinates of the feature points on the monitor 12 are acquired. This coordinate value is acquired using a pointing device such as a mouse. That is, the user 16 aligns the tip of the mouse pointer with the feature point displayed on the monitor 12 and clicks to obtain the planar coordinates of each feature point 28. Further, in order to record which feature point the clicked point corresponds to, the point ID is designated. As a result, a plurality of coordinate pairs consisting of the three-dimensional coordinates in the real world and the two-dimensional coordinate values on the monitor can be acquired. In this embodiment, conversion coordinates from the coordinate system in the real world to the coordinate system on the monitor are obtained based on these coordinate pairs. Therefore, the user 16 can obtain this transformation matrix only by designating a plurality of coordinate pairs.

  With reference to FIG. 8C, a video in which the virtual object 29 is superimposed on the real image is displayed on the monitor 12. Here, an image of a virtual object 29 such as an airplane placed on the upper surface of the box 27 is displayed on the monitor 12. Here, the virtual object 29 does not exist in the real world and is a so-called CG (Computer Graphic). The shape of the virtual object 29 is expressed by three-dimensional coordinates in the real world. Specifically, the three-dimensional coordinates of each part constituting the virtual object 29 are determined so that the virtual object 29 is placed on the top of the box 27. Accordingly, by applying the transformation matrix obtained from the coordinate pair to the three-dimensional coordinates of the virtual object 29, it is possible to perform superposition while maintaining three-dimensional geometric consistency. FIG. 8D is an enlarged view of a portion where the virtual object 29 is displayed, and it can be understood that the state in which the virtual object 29 is placed on the top of the box 27 is accurately visualized. .

  With reference to FIG. 9, a change in the image displayed on the monitor 12 when the camera that captures the box 27 and the like is rotated will be described. FIG. 9A shows images taken with the camera installed so that the box 27 is reflected in the front, and FIGS. 9B to 9D show state images taken by rotating the camera. Yes.

  With reference to FIG. 9A, the virtual object 29 displayed in a state of being placed on the box 27 and its upper part is displayed near the center of the video.

  Referring to FIG. 9B, here, an image displayed on the monitor 12 when the camera is rotated in the right direction is shown. As the camera rotates, the box 27 and the like are displayed on the left side. Furthermore, in this embodiment, the rotation angle of the camera is measured in real time. Then, using the measured rotation angle, a conversion matrix for performing conversion from the real-world coordinate system to the coordinate system on the monitor is sequentially calculated. And the coordinate on the monitor of the virtual object 29 is calculated | required using the calculated | required conversion matrix. Therefore, in this embodiment, even when the camera is rotated, the virtual object 29 can be displayed while maintaining geometric consistency.

  Referring to FIGS. 9C and 9D, these drawings show images when the camera is rotated leftward and upward. In these drawings, it can be understood that the geometric consistency of the virtual object 29 is maintained.

  With reference to FIG. 10 and FIG. 11, an image composition method using the display device 10 of the present embodiment will be described in more detail.

  First, with reference to FIG. 10, conditions necessary for superimposing a CG on a real image captured by the camera 11 provided in the display device 10 will be described. FIG. 10A is a diagram showing the arrangement of the buildings 30A and the like in the real world, and FIG. 10B is a diagram showing the state of the CG building 31. FIG. Here, a case where a CG building 32 is synthesized between a building 30A and a building 30B facing the road 32 is considered.

  Referring to FIG. 10A, in this embodiment, buildings 30A and 30B that are already constructed facing road 32 are photographed using camera 11. Here, a building that actually exists is drawn with a solid line, and a building drawn with CG is drawn with a dotted line.

  In order to synthesize CG while maintaining geometric consistency with a live-action image, two pieces of information are required: three-dimensional shape data of an object to be drawn (here, building 31) and parameters relating to the camera. Here, the parameter relating to the camera is information related to the matter of drawing the CG viewed from which position.

  Parameters related to the camera include parameters representing the relationship between the camera and the external environment, such as the camera's own parameters (internal parameters) such as viewing angle and image size, and from which position the camera is installed in which direction. External parameters).

  The internal parameters are parameters that are made constant at that time when a camera to be used for live-action shooting is determined. Therefore, it can be obtained from the specification of the lens provided in the camera, or can be measured in advance. The internal parameters obtained in this way are set as parameters for performing CG drawing.

  Regarding the external parameters, in the present embodiment, the user can move the camera 11 freely, and therefore the following steps are performed. First, the same thing as the coordinate system expressing CG shape data is assumed in the real world. Specifically, the world coordinate system for drawing the CG of FIG. 10B is the same as the world coordinate system of FIG. Then, a coordinate transformation matrix between this world coordinate system and the actual camera is obtained, and this transformation matrix is set as a parameter for CG synthesis. Through the above steps, it is possible to draw a CG that is geometrically consistent with the actual image. That is, in the present application, every time the actual camera position or orientation changes, this coordinate transformation matrix is recalculated and set.

  In the display device 10 of the present application, the user can freely move the tripod and freely rotate the camera on the tripod in the pan direction and the tilt direction. Furthermore, the display device 10 can synthesize CG without degrading geometric consistency by obtaining a coordinate transformation matrix between the actual camera and the real camera in real time.

  Further, with reference to FIG. 10B, an image superimposing method will be described. Considering the case where the CG is simply superimposed on the live-action image, an image is generated in which the lower part of the CG structure 31 actually hides the building 30A in front of it. Such an image is an unnatural image. Accordingly, in the present application, for each pixel constituting the video, it is determined which of the actual world structure and the CG world structure is in front of the camera.

  Specifically, if the CG structure is in front of the actual structure, CG is synthesized for the pixel. On the other hand, if the CG structure is behind the actual structure, the CG is not synthesized for the pixel. These operations are necessary.

  Next, with reference to FIG. 11, a method for synthesizing an image in consideration of lens distortion will be described. FIG. 11 is a flowchart showing a method for synthesizing images.

  As described above with reference to FIG. 10, it is possible to draw a CG that is geometrically consistent by matching all the CG parameter settings with the actual camera imaging conditions. Furthermore, in this embodiment, a CG combining method is performed in consideration of lens distortion, which is a parameter inside the camera.

  Lens distortion means that a straight line is observed as a curved line due to optical characteristics of the lens. That is, an image taken with an actual camera is distorted. This lens distortion occurs remarkably in a fisheye lens or the like. If a CG image without distortion is superimposed on a real image with distortion, geometrical consistency is lost. Therefore, in this embodiment, a CG image without distortion is synthesized with the actual image that has been subjected to the lens distortion correction processing. The above problem can also be solved by synthesizing distorted CG in accordance with the distortion of the lens. However, considering a reduction in the amount of calculation, it is preferable to synthesize CG into a real image with distortion corrected.

  The specific method is demonstrated with reference to the flowchart of FIG. First, in step S21, the real environment is photographed with an actual camera. As a result, a live-action image including distortion caused by a lens mounted on the camera can be obtained. In step S22, lens distortion is corrected. That is, geometric correction is performed on a portion indicated as a curve due to lens distortion. Through the above operation, a live-action image from which lens distortion has been eliminated is obtained.

In step S24, CG is generated from CG data prepared in advance. As a result, a CG image in which distortion is not considered is obtained. In step S23, similarly, a CG image without distortion is combined with a real image without distortion. As a result, it is possible to synthesize an image in which geometric matching is maintained.

1A and 1B are a conceptual diagram and a conceptual diagram illustrating a display device of the present invention. It is the conceptual diagram (A) explaining the coordinate system used for this invention, and the conceptual diagram (B) explaining the relationship between each coordinate system. It is a flowchart which shows the calculation method of the conversion matrix using the display apparatus of this invention. It is a conceptual diagram which shows the relationship from a world coordinate system to a tripod coordinate system. It is a perspective view which shows the whole board. It is a flowchart which shows the calculation method of the conversion matrix using the display apparatus of this invention. It is a conceptual diagram which shows the usage method of the display apparatus of this invention. It is a conceptual diagram (A)-(D) which shows the usage method of the display apparatus of this invention. It is a conceptual diagram (A)-(D) which shows the usage method of the display apparatus of this invention. FIGS. 4A and 4B are a perspective view and a perspective view illustrating a state in which an image is synthesized using the display device of the present invention. It is a flowchart which shows the composition method of the image using the display apparatus of this invention.

Explanation of symbols

10 Display device
11 Camera
12 Monitor
13 Head
14 Tripod
15 Control lever
16 users
17 Calculator
18 Calculation unit
19 Memory
20A First measuring means
20B Second measuring means
C1-C5 coordinate system
M1-M4 transformation matrix

Claims (12)

Photographing means for photographing a real-world optical image including feature points and converting it into image data;
Display means for displaying video based on the image data;
Fixing means for fixing the photographing means;
Supporting means for rotatably supporting the fixing means;
Measuring means for measuring the rotation angle of the fixing means;
Obtaining means for obtaining coordinates on the display means of the feature points displayed on the display means;
Computation means for performing computation based on the coordinates of the feature points,
The imaging means captures the first image data and the second image data in a posture different from when the first image data is captured,
The acquisition means acquires first coordinates that are coordinates of the feature points that appear in the first image data, and second coordinates that are coordinates of the feature points that appear in the second image data,
The measuring means measures a rotation angle of the fixing means when the second image data is captured;
The electronic apparatus is characterized in that the calculation means obtains a position of the photographing means relative to the fixing means based on the first coordinates, the second coordinates, and the rotation angle.   The measuring means comprises first measuring means for measuring a horizontal rotation angle of the fixing means, and second measuring means for measuring a vertical rotation angle of the fixing means. Item 1. An electronic device according to Item 1. The acquisition means is a pointing device;
2. The electronic apparatus according to claim 1, wherein coordinates of the feature point on the display unit are acquired by selecting the feature point displayed on the display unit with the pointing device.   The electronic apparatus according to claim 1, wherein the calculation unit calculates a position of the photographing unit by a statistical method. The computing means is
After calculating the change in position and orientation of the photographing means and the fixing means when the first image data is photographed and when the second image data is photographed, the fixing means of the photographing means is 2. The electronic device according to claim 1, wherein all positions are obtained. Photographing means for photographing a real-world optical image including feature points and converting it into image data;
Display means for displaying video based on the image data;
Support means for supporting the imaging means;
Obtaining means for obtaining coordinates on the display means of the feature points displayed on the display means;
An electronic apparatus comprising: an operation unit that calculates a position of the support unit with respect to the real world based on the coordinates of the feature point.   The electronic apparatus according to claim 6, wherein the calculation unit calculates the coordinates of the support unit based on the coordinates of the plurality of feature points included in one image data. The photographing means is rotatably fixed to the support means,
Comprising measuring means for measuring the rotation angle of the photographing means;
The acquisition unit acquires the coordinates of the feature points on the display unit while rotating the photographing unit,
The calculation means calculates the position of the support means from the coordinates of the feature points acquired by the acquisition means and the rotation angle of the photographing means when the feature points are obtained. The electronic device described. The acquisition means is a pointing device;
The electronic device according to claim 6, wherein coordinates on the display unit of the feature point are acquired by selecting the feature point displayed on the display unit with the pointing device.   The electronic apparatus according to claim 6, wherein the calculation unit calculates a position of the photographing unit by a statistical method.   The measuring means comprises first measuring means for measuring a horizontal rotation angle of the fixing means, and second measuring means for measuring a vertical rotation angle of the fixing means. Item 9. An electronic device according to Item 8. The photographing means is fixed to a fixing means;
The electronic device according to claim 8, wherein the fixing unit is rotatably fixed to the support unit.

Images