JP6931267B2 - A program, device and method for generating a display image obtained by transforming the original image based on the target image. - Google Patents

Description

The present invention relates to a technique for transforming an original image based on a target image.

Conventionally, there is a technique for displaying contents in a photographed image in real time according to the relative position and orientation of the camera with respect to the object to be photographed (see, for example, Patent Document 1 and Non-Patent Document 1). This content is a virtual image that can be displayed superimposed on the captured image as an Augmented Reality image.

FIG. 1 is an explanatory diagram for displaying an augmented reality virtual image superimposed on a captured image.

According to FIG. 1A, a marker (reference target) is attached to the object to be photographed. The marker may be, for example, a two-dimensional QR code (registered trademark). The object to be photographed is photographed by four cameras cam1, cam2, cam3, and cam4 having different positions and orientations.
According to FIG. 1B, images taken by each camera are shown. When a marker appears in the captured image, the marker appears from a different angle depending on the orientation of the camera. The position and orientation are extracted from the shape of the reference image of the marker reflected in the captured image.
According to FIG. 1C, a virtual image corresponding to the position / orientation (reference world coordinate value) is stored in advance for each reference image in which the reference object is displayed. Then, the position / orientation of the reference image in the captured image and the position / orientation vector of the reference image in which a plurality of reference objects stored in advance are reflected are compared based on the “distance function”. A virtual image associated with a reference target estimated to have a short distance (similar position / orientation) is displayed superimposed on the captured image. When the content has a three-dimensional shape, virtual images having different shapes are selected according to the line-of-sight vector of the camera.
According to FIG. 1D, for the display image, for example, a rectangular parallelepiped virtual image is superimposed and displayed on the marker displayed on the target captured image.
By such processing, a high-quality virtual image can be displayed in real time with a relatively low processing load.

Japanese Unexamined Patent Publication No. 2016-71496

"Augmented Reality System with Pre-Captured Images for Realistic Reproduction of Real Objects with Complex Optical Characteristics", IPSJ 100th AVM Workshop Lecture No.13, [online], [2018 6 Search on March 13], Internet <URL: https://www.ipsj.or.jp/kenkyukai/event/avm100.html> Kato, M. Billinghurst, Asano, Tachibana, "Augmented Reality System Based on Marker Tracking and Its Calibration", Journal of The Virtual Reality Society of Japan, vol.4, 20 no.4, pp.607-617, 1999., [online], [Search on June 13, 2018], Internet <URL: http://intron.kz.tsukuba.ac.jp/tvrsj/4.4/kato/p-99_VRSJ4_4.pdf> DGLowe, Distinctive image features from scale-invariant key points, Proc. Of Int. Journal of Computer Vision (IJCV), 60 (2) pp.91-110 (2004), [online], [June 13, 2018 Day search], Internet <URL: https://www.cs.ubc.ca/~lowe/papers/ijcv04.pdf> H.Bay, T.Tuytelaars, and LVGool, SURF: Speed Up Robust Features, Proc. Of Int. Conf. Of ECCV, (2006), [online], [Search June 13, 2018], Internet < URL: http://www.vision.ee.ethz.ch/~surf/eccv06.pdf ＞ Ethan Rublee, Vincent Rabaud, Kurt Konolige, Gary R. Bradski: ORB: An efficient alternative to SIFT or SURF. ICCV 2011: 2564-2571., [Online], [Search June 13, 2018], Internet <URL : http://www.willowgarage.com/sites/default/files/orb_final.pdf ＞

According to the above-mentioned prior art, the virtual image is further projected and transformed so that the position / orientation of the predetermined plane of the virtual image stored in advance matches the position / orientation of the predetermined plane of the captured image for the selected virtual image. ing. As a result, the deviation between the position and orientation associated with the registered virtual information and the position and orientation of the captured image is corrected.
However, all the pixels included in the virtual image are treated as points on a predetermined plane, and the projective transformation that corrects the position and orientation of the predetermined plane is applied to a three-dimensional object located in a space other than the predetermined plane. Therefore, the virtual image represented as a three-dimensional object is transformed so that its shape is distorted.
This problem is also the same when trying to convert a general original image into a target image regardless of the difference in position and orientation between the captured image and the virtual image.

FIG. 2 is an explanatory diagram in which the original image in the prior art is projected and transformed based on the target image.

According to FIG. 2, the reference image based on the reference object and the three-dimensional object other than the reference object are reflected in the original image representing the three-dimensional space. In addition, at least a reference image based on the reference target is reflected in the target image.
As is clear from FIG. 2, when the original image is transformed into the position and orientation of the target image from the viewpoint by the projective transformation based on the predetermined plane, the shape of the three-dimensional object is transformed as if it were distorted. ..
Further, when the target image is an image that changes in time series, the distortion method due to the projective transformation also changes in time series. Therefore, when the original image contains a rigid three-dimensional object, the shape of the three-dimensional object, which should not change originally, seems to change in time series. This gives the user an unnatural impression.

Therefore, the present invention provides a program, an apparatus, and a method for generating a displayed image that is deformed to match the position and orientation of a target image without causing distortion in the geometric shape of a three-dimensional object reflected in the original image. The purpose.

According to the present invention, it is a program that functions a computer mounted on a device that generates a display image in which an original image in which a reference object is reflected is deformed based on a target image in which the reference object is reflected.
For each original image and the target image, the position and orientation estimation means a certain unit vector based on the reference image in which the reference object is reflected, estimated as the position and orientation vector,
A transformation matrix calculation means for calculating a transformation matrix that matches the position / orientation vector of the original image with the position / orientation vector of the target image.
It is characterized in that a computer functions as a display image generation means for generating a display image obtained by transforming an original image by a transformation matrix.
Also, according to other embodiments in the program of the present invention.
The position-posture estimation means uses the position-posture vector,
Vertical unit vector of the world coordinate system,
Any horizontal unit vector in the world coordinate system, or
Any unit vector set by the user
Is a vector calculated by perspective projection transformation.
It is also preferable to make the computer function as such.

According to other embodiments in the program of the present invention
Position and orientation estimation means
The reference image showing the reference object and the reference world coordinates of the reference object are stored in advance in association with each other.
From the original image and the target image, feature points based on the local features of the reference image are extracted.
It is also preferable to make the computer function to estimate the relative position / orientation vector of the reference object by the perspective projection transformation between the feature point coordinates of the reference image and the reference world coordinates of the reference object.
Also, according to other embodiments in the program of the present invention.
The position / orientation estimation means uses the starting point of the position / orientation vector as the starting point.
Any point set by the user, or
The intersection of a straight line extending in the shooting direction from the world coordinate value of the camera viewpoint and a predetermined periphery of the world coordinate value of the reference target.
It is also preferable to make the computer function so as to.

According to other embodiments in the program of the present invention
Transformation matrix is represented by the rotational theta, translation t and scale s,
It is also preferable that the transformation matrix calculation means causes the computer to function so as to remove the error included in the rotation θ and the translation t and calculate the transformation matrix in which the position / orientation vector is matched so that the error is included only in the scale s.

According to other embodiments in the program of the present invention
The transformation matrix calculation means is
Rotational alignment means that aligns the rotation direction in the image plane,
A translational alignment means that aligns the translational directions in the image plane,
It is also preferable to have the computer function to include scale matching means for matching the scales in the image plane.

According to other embodiments in the program of the present invention
It is intended to function the translational matching means following the rotational matching means.
The rotation matching means rotates and matches the rotation direction of the position / orientation vector of the original image in the image plane so as to match the rotation direction of the position / orientation vector of the target image, and after rotation matching based on the rotation matrix. Output the original image of the above to the translational matching means,
The translational alignment means translates and aligns the translational direction of the position-orientation vector of the original image after rotational alignment in the image plane so as to match the translational direction of the position-orientation vector of the target image, a rotation matrix, and rotation. It is also preferable to make the computer function so as to output the original image after the rotation translation matching based on the matrix and the translation vector to the scale matching means.

According to other embodiments in the program of the present invention
The translational matching means is followed by the rotational matching means.
The translational alignment means has a translational vector that translates and aligns the translational direction of the position and orientation vector of the original image in the image plane so as to coincide with the translational direction of the position and orientation vector of the target image, and after translational alignment based on the translational alignment. The original image of is output to the rotation matching means,
The rotation matching means rotates and matches the rotation direction of the position / orientation vector of the original image after translational alignment in the image plane so as to match the rotation direction of the position / orientation vector of the target image, and the translation vector and rotation. It is also preferable to make the computer function so as to output the original image after the rotation translation matching based on the matrix and the translation vector to the scale matching means.

According to other embodiments in the program of the present invention
It is also preferable that the scale matching means causes the computer to function so as to scale-match the original image after the rotation-translation matching so that the scale of the original image after the rotation-translation matching matches the scale of the target image.

According to other embodiments in the program of the present invention
The target image is an image that changes in time series at a predetermined frame rate.
It is also preferable that the scale matching means causes the computer to function so as to smooth the scale conversion values calculated continuously in time series.

According to other embodiments in the program of the present invention
The reference target is a predetermined object or a predetermined marker.
The original image is an original image taken in advance by the camera.
It is also preferable to make the computer function so that the target image is a target captured image captured in real time by the camera.

According to other embodiments in the program of the present invention
In order to generate a target augmented reality image for the target captured image as a display image from the original augmented reality image in which a virtual image is superimposed on a predetermined position based on the reference image in which the reference target appears in the original captured image.
The display image generation means causes the computer to function so as to transform the virtual image by a transformation matrix and generate a target augmented reality image superimposed on a predetermined position of the reference image reflected in the target captured image as a display image. Is also preferable.

According to other embodiments in the program of the present invention
It is also preferable to make the computer function so that the virtual image in the original augmented reality image has a shape viewed from the camera viewpoint three-dimensionally with respect to the reference image in which the reference object appears in the original captured image.

According to the present invention, it is a display image generation device that generates a display image obtained by transforming an original image in which a reference object is reflected based on a target image in which the reference object is reflected.
For each original image and the target image, the position and orientation estimation means a certain unit vector based on the reference image in which the reference object is reflected, estimated as the position and orientation vector,
A transformation matrix calculation means for calculating a transformation matrix that matches the position / orientation vector of the original image with the position / orientation vector of the target image.
It is characterized by having a display image generation means for generating a display image obtained by transforming the original image by a transformation matrix.

According to the present invention, there is a display image generation method of an apparatus for generating a display image obtained by transforming an original image in which a reference object is reflected based on a target image in which the reference object is reflected.
The device is
For each original image and the target image, a first step of estimating the predetermined unit vector based on the reference image in which the reference object is reflected, as the position and orientation vector,
The second step of calculating the transformation matrix that matches the position / orientation vector of the original image with the position / orientation vector of the target image, and
It is characterized by executing the third step of generating a display image obtained by transforming the original image by a transformation matrix.

According to the program, apparatus and method of the present invention, it is possible to generate a display image deformed to match the position and orientation of the target image without causing distortion in the geometric shape of the three-dimensional object displayed in the original image.

It is explanatory drawing which superimposes and displays the virtual image of augmented reality on the photographed image. It is explanatory drawing which projected-transformed the original image in the prior art based on a target image. It is explanatory drawing which deformed the original image in this invention based on the target image. It is a functional block diagram of the apparatus in this invention. It is explanatory drawing which made the unit vector of a position-posture vector a gaze point. It is explanatory drawing which shows the position | posture vector of the original image and the target image calculated by perspective projection transformation. It is explanatory drawing which deformed the original image based on the target image. It is explanatory drawing which deformed the original image including a virtual image based on a target image.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings.

FIG. 3 is an explanatory view obtained by deforming the original image in the present invention based on the target image.

According to FIG. 3, as compared with FIG. 2, a displayed image is shown which is deformed to match the position and orientation of the target image without causing distortion in the geometric shape of the three-dimensional object reflected in the original image. By converting the original image corresponding to the target image in this way, the user does not have an unnatural impression about the shape of the three-dimensional object that should not change originally.

FIG. 4 is a functional configuration diagram of the display image generator according to the present invention.

According to FIG. 4, the display image generation device 1 of the present invention generates a display image obtained by deforming an original image in which a reference object is reflected based on a target image in which the reference object is reflected. Here, the display image generation device is represented as a terminal 1 equipped with a camera and a display, such as a smartphone. It is not limited to a smartphone, but may be a tablet or a personal computer.
Of course, the function of the present invention may be mounted on a server via a network. For example, the server receives a captured image (target image) taken by the camera of the user's smartphone via the communication interface, and displays a display image obtained by transforming the original image based on the target image to the user's smartphone. It may be a reply.
It should be noted that, without realizing the present invention with only one device, the functions of the present invention can be divided and configured as a system of a server and a terminal via a network.

According to FIG. 4, the terminal 1 has a position / orientation estimation unit 11, a transformation matrix calculation unit 12, and a display image generation unit 13. These functional components are realized by executing a program that makes the computer mounted on the device function. Further, the processing flow of these functional components can be understood as a display image generation method of the device.
According to FIG. 4, it is assumed that the terminal 1 holds the original image in which the reference image of the reference target is reflected, and acquires the target image in which the reference image of the reference target is reflected from the camera. At this time, a display image obtained by transforming the original image into a target image is generated.

[Position / Posture Estimating Unit 11]
The position / orientation estimation unit 11 estimates the position / orientation vector based on the reference image in which the reference target is reflected for each of the original image and the target image.
The "position / orientation vector" is, for example, a relative position / orientation (translation t and rotation θ) with reference to a marker (reference object) attached to the object to be photographed when viewed from the camera (for example, Non-Patent Document 2). reference).

The position / orientation estimation unit 11 stores in advance the reference image in which the reference object is displayed and the reference world coordinates of the reference object in association with each other (for example, a marker in Non-Patent Document 2). Then, the position / orientation vector is estimated by the following steps.
(S1) From the original image and the target image, feature points based on the local features (for example, relative luminance gradient) of the reference image are extracted by the local feature extraction algorithm.
(S2) For each of the original image and the target image, the relative position / orientation vector of the reference object is estimated based on the coordinate conversion between the feature point coordinates of the reference image and the reference world coordinates of the reference object.
Then, the original image, the target image, and the respective position / orientation vectors are output to the transformation matrix calculation unit 12.

In addition, for example, when the reference target is not reflected in the target image or when the reference image of the reference target cannot be extracted due to the influence of noise of the target image or the like, the process is terminated. In such a case, the target image captured by the camera is displayed as it is on the display of the terminal 1 as a display image.

The feature information of the reference image may be a feature quantity of SIFT (Scale-Invariant Feature Transform) or SURF (Speeded Up Robust Features), which is robust to rotation and scale change, or compact local feature at high speed. It may be a feature amount of ORB (Oriented FAST and Rotated BRIEF) to be extracted (see, for example, Non-Patent Documents 3, 4, and 5).
For example, in the case of SIFT, a set of 128-dimensional local features is extracted from one image. SIFT is a technique for analyzing a characteristic local region using a scale space and describing a local feature that is invariant to the scale change and rotation.
Further, in the case of SURF, faster processing than SIFT is possible, and a set of 64-dimensional local features is extracted from one image. While SIFT has a high processing cost and difficult real-time matching, SURF speeds up processing by using an integrated image.
Further, in the case of ORB, as mobile terminals such as smartphones and tablets become widespread, it is suitable for memory saving and high-speed matching for augmented reality applications.

The position / orientation estimation unit 11 calculates the position / attitude vector gwld in the world coordinate system. The position / orientation vector is at least a three-dimensional vector having a length, and is a predetermined unit vector.
The position / orientation vector refers to a vector obtained by, for example, any of the following unit vectors calculated by a perspective projection transformation.
(1) Vertical unit vector of the world coordinate system (2) Arbitrary horizontal unit vector of the world coordinate system (3) Arbitrary unit vector set by the user

Further, the starting point (world reference point Gwld) of the position / orientation vector gwld in the world coordinate system may be any point on the three-dimensional space of the reference target. For example, it may be any of the following points.
(1) Arbitrary point set by the user and included in both the original image and the target image (2) A straight line (line-of-sight vector) extending from the world coordinate value of the camera viewpoint in the shooting direction and the world coordinate of the reference target The intersection with the specified periphery of the value

FIG. 5 is an explanatory diagram with the unit vector of the position / orientation vector as the gazing point.

According to FIG. 5, the “shooting line-of-sight vector” from the camera viewpoint Pcam (x, y, z) toward the reference target is represented by using the world coordinate values of the camera. The intersection Plookat (u, v, w) between the half line extending from the camera viewpoint Pcam (x, y, z) in the shooting direction and the plane of symmetry based on the reference object is the gazing point, and the gazing point is the starting point of the unit vector. It is supposed to be.

FIG. 6 is an explanatory diagram showing the position / orientation vectors of the original image and the target image calculated by the perspective projection conversion.

According to FIG. 6, for the original image Is, the position / orientation vector gwld of the world coordinate system is projected onto the plane of the original image Is by fluoroscopic projection conversion, and the position / orientation vector gs (start point Gs) of the original image Is is calculated. ..
Similarly, for the target image Id, the position / orientation vector gwld in the world coordinate system is projected onto the plane of the target image Id by fluoroscopic projection conversion to calculate the position / orientation vector gd (start point Gd) of the target image.
Then, the position / orientation estimation unit 11 outputs the position / orientation vector gs of the original image Is and the position / orientation vector gd of the target image Id to the transformation matrix calculation unit 12.

[Transformation matrix calculation unit 12]
The transformation matrix calculation unit 12 calculates a transformation matrix that matches the position / orientation vector gs of the original image Is with the position / orientation vector gd of the target image Id.

FIG. 7 is an explanatory view obtained by deforming the original image based on the target image.
According to FIG. 7A, the position / orientation vector based on the reference image of the original image is rotationally translated and matched with the position / orientation vector based on the reference image of the target image.

Here, there are the following two methods for calculating the transformation matrix.
<< Matching of position and orientation based on similarity transformation >>
<< Position-orientation matching that eliminates rotation and translation errors and includes errors only in the scale >>

<< Matching of position and orientation based on similarity transformation >>
The transformation matrix calculation unit 12 starts from the feature point coordinates (xi, yi) of the reference image reflected in the original image Is (i ∈ I, I is a subscript set of feature points) to the feature point coordinates of the reference image reflected in the target image Id. The similarity conversion matrix to Xi, Yi) is calculated as a conversion matrix. The similarity transformation is defined as follows.

原画像Ｉsと目標画像Ｉdとから抽出された特徴点のうち、対応の存在する特徴点の添字集合をＪ＝｛ｊ₀,・・・,ｊ_N｝とすると、前述の式［１］は、以下のように書き換えられる。

Here, it is set as follows.

Of the feature points extracted from the original image Is and the target image Id, if the index set of the feature points having a correspondence is J = {j ₀ , ..., j _N }, the above equation [1] is , Is rewritten as follows.

Next, assuming that the equation [3] is Ax = b, x is calculated by the least squares method as follows.
x = (A ^T A) ^-1 A ^T b equation [4]
The following similarity transformation matrix M is calculated by the parameters a, b, c, and d.

As shown in FIG. 3 described above, the original image in which a three-dimensional object in the three-dimensional space is reflected is converted so that it can be seen from different viewpoints by a homothety conversion based on a plane according to the target image. As a result, the positions and orientations can be aligned so that the positions and orientations of the predetermined planes are approximately the same while maintaining the geometric shape of the three-dimensional object in the image.

Here, according to the above-mentioned equation [5], an error is included in all three elements of translation, rotation, and scale in order to calculate an approximate solution that minimizes the error. On the other hand, as in Non-Patent Document 1 described above, when a virtual image whose position and orientation are matched is presented to the user in place, the error included in the scale of the virtual image is perceived as the influence of the angle of view of the camera. Because it is done, it does not give an unnatural impression. However, the error included in the rotation and translation is perceived as a deviation from the real world, which gives the user an unnatural impression.

<< Position-orientation matching that eliminates rotation and translation errors and includes errors only in the scale >>
The transformation matrix calculation unit 12 removes the error included in the rotation θ and the translation t, and calculates the transformation matrix that matches the position and orientation so that the error is included only in the scale s.

The transformation matrix calculation unit 12 has the following three functions.
Rotational matching unit that aligns the rotation directions in the image plane Translational matching unit that aligns the translational directions in the image plane Scale matching unit that aligns the scales in the image plane According to the present invention, the transformation matrix calculation unit 12 is a rotational matching unit. The part and the translation matching part are made to function preferentially over the scale matching part.

In the transformation matrix calculation unit, the rotation matching unit and the translation matching unit function in the following order.
<Rotation matching unit 1211-> Translation matching unit 1222>
<Translation matching part 1221-> Rotation matching part 1212>

<Rotation matching unit 1211-> Translation matching unit 1222>
[Rotation matching unit 1211]
The rotation matching unit is based on a rotation matrix R that rotationally aligns the rotation direction of the position / orientation vector gs of the original image in the image plane so as to coincide with the rotation direction of the position / orientation vector gd of the target image, and the rotation matrix R. The original image Is (R) after rotation matching is output to the translation matching section.
This rotation angle θ is calculated as follows, with the counterclockwise rotation direction as positive.
θ = sgn (gs × gd) arccos ((gs · gd) / (| gs || gd |)) Equation [6]

これは、原画像Ｉsの中心座標値（ｃx,ｃy）を用いて、原画像Ｉsの同次座標系に積算したものである。
これによって、回転整合部は、回転角θを用いて原画像Ｉsを回転整合し、回転整合後の原画像Ｉs（Ｒ）を算出する。
そして、回転整合部は、回転行列Ｒと回転整合後の原画像Ｉs（Ｒ）とを、並進整合部へ出力する。 The rotation matching unit calculates the rotation matrix R as follows.

This is integrated into the homogeneous coordinate system of the original image Is using the center coordinate values (cx, cy) of the original image Is.
As a result, the rotation matching unit rotates and matches the original image Is using the rotation angle θ, and calculates the original image Is (R) after the rotation matching.
Then, the rotation matching unit outputs the rotation matrix R and the original image Is (R) after the rotation matching to the translation matching unit.

[Translation matching unit 1222]
Next, the translational matching unit translates and aligns the translational direction of the position-orientation vector of the original image Is (R) after rotation-matching in the image plane so as to coincide with the translational direction of the position-orientation vector of the target image Id. The vector T and the original image Is (RT) after the rotation translation matching based on the rotation matrix R and the translation vector T are output to the scale matching unit.

The translation matching unit rotates the vector GsGd from the point Gs of the original image Is to the point Gd of the target image Id using the rotation matrix R input from the rotation matching unit, and calculates it as the translation vector T. It is expressed as follows with the point O as the origin.
T = R (GsGd) = R (OGd-OGs) Equation [8]
By adding this translation vector T to the coordinate system of the original image Is (R) after rotation matching input from the rotation matching unit, translation matching is performed, and the original image Is (RT) after the rotation translation matching is calculated. ..
Then, the translation matching unit outputs the original image Is (RT) after the rotation translation matching, the rotation matrix R, and the translation vector T to the scale matching unit.

<Translation matching part 1221-> Rotation matching part 1212>
[Translation matching part 1221]
The translation matching unit is based on a translation vector T that translates and aligns the translation direction of the position / orientation vector gs of the original image in the image plane so as to coincide with the translation direction of the position / orientation vector gd of the target image, and the translation vector T. The original image Is (T) after translational matching is output to the rotation matching section.
This translation vector T is expressed as follows with the point O as the origin.
T = GsGd = OGd-OGs equation [9]
The translational vector T is added to the coordinate system of the original image Is for translational matching, and the image Is (T) after the translational matching and the translational vector T are output to the rotation matching unit.

[Rotation matching unit 1212]
Next, the rotation matching unit rotates and aligns the rotation direction of the position / orientation vector of the original image Is (T) after translational alignment in the image plane so as to coincide with the rotation direction of the position / orientation vector of the target image. The R and the original image Is (RT) after the rotation translation matching based on the rotation matrix R and the translation vector T are output to the scale matching unit.
This rotation angle θ is calculated as follows, with the counterclockwise rotation direction as positive.
θ = sgn (gs × gd) arccos ((gs · gd) / (| gs || gd |)) Equation [10]

これは、引数に対応する座標値を返す関数Ｇd（・）を用いて、原画像のＩsの同次座標系に積算したものである。
これによって、回転整合部は、回転角θを用いて並進整合後の原画像Ｉs（Ｔ）を回転整合し、回転並進整合後の原画像Ｉs（ＲＴ）を算出する。
そして、回転整合部は、回転並進整合後の原画像Ｉs（ＲＴ）と、回転行列Ｒ及び並進ベクトルＴとを、スケール整合部へ出力する。 The rotation matching unit calculates the rotation matrix R as follows.

This is integrated into the homogeneous coordinate system of Is of the original image using the function Gd (・) that returns the coordinate values corresponding to the arguments.
As a result, the rotation matching unit rotates and matches the original image Is (T) after the translational matching using the rotation angle θ, and calculates the original image Is (RT) after the translational translation matching.
Then, the rotation matching unit outputs the original image Is (RT) after the rotation translation matching, the rotation matrix R, and the translation vector T to the scale matching unit.

[Scale matching part]
The scale matching unit scale-matches the scale of the original image Is (RT) after the rotational translation matching so as to match the scale of the target image Id, and outputs the original image Is (sRT).

i∈Ｉ（Ｉは、特徴点の添字集合） The scale matching unit performs rotation translation transformation on the feature point coordinates extracted from the original image Is using the input rotation matrix R and translation vector T. With respect to the feature point coordinates (xi, yi), the feature point coordinates (xi (RT), yi (RT)) after the rotational translation transformation are calculated as follows.

i ∈ I (I is the index set of feature points)

j∈Ｊ（Ｊは、対応が存在する特徴点の添字集合）
尚、目標画像の点Ｇdを中心とするスケール変換をすることによって、目標画像の点Ｇdの位置を固定したスケール変換をしていることに注意すべきである。 The feature point coordinates (Xi, Yi) extracted from the target image are calculated as follows using the scale conversion value s.

j ∈ J (J is an index set of feature points for which a correspondence exists)
It should be noted that the scale conversion is performed with the position of the point Gd of the target image fixed by performing the scale conversion centered on the point Gd of the target image.

The above equation [13] is set as follows.
bs: A matrix in which the vertical vectors on the left side are subscripted and arranged in the horizontal direction Ad: A matrix in which the vertical vectors on the right side are subscripted and arranged in the horizontal direction In this case, the scale conversion value s is the least squares method. It is calculated as follows.
s = (Ad ^T Ad) ^-1 Ad ^T bs equation [14]

The scale matching unit calculates by integrating the scale conversion value s into the coordinate system of the image.
Then, the scale matching unit of the transformation matrix calculation unit 12 finally outputs the transformation matrix (rotation, translation, scale) to the display image generation unit 13.

<Embodiment for a target image that changes in time series>
When the feature point coordinates (Xi (t), Yi (t)) of the target image change in time series, the scale conversion value s (t) may also fluctuate erratically. As a result, the scale for matching the position / orientation vectors also fluctuates erratically, giving the user an unnatural impression.

Therefore, the scale matching unit of the transformation matrix calculation unit 12 smoothes the scale conversion value s (t) calculated continuously in time series.
The scale matching unit stores from the scale conversion value s (t) calculated by itself to the scale conversion value s (tK) before the K step time, and the smoothed scale conversion value s (t) is as follows. Is calculated.
s (t) = 1 / (K + 1) Σ _{k = 0} ^K s (tk) Equation [15]
The scale matching unit integrates the scale conversion value s (t) into the coordinate system of the original image Is (RT (t)) after the rotation translation matching, so that the position and orientation of the original image Is (sRT (t)) are matched. )) Is calculated.

As a result, even when the target image changes in time series at a predetermined frame rate, a display image in which the position and orientation of the three-dimensional object appear to be stable is generated without causing distortion of the geometric shape of the three-dimensional object reflected in the image. do. That is, it is possible to prevent the user who browses the displayed image in real time continuously in time series from feeling uncomfortable as much as possible.

The position / orientation estimation unit 11, the transformation matrix calculation unit 12, and the display image generation unit 13 described above are executed for each target image in the time series.

[Display image generation unit 13]
The display image generation unit 13 generates a display image in which the position and orientation are matched by integrating the transformation matrix M into the coordinate system of the original image Is.
According to FIG. 7B, the original image represents a display image in which rotation, translation, and scale are matched based on a transformation matrix.

FIG. 8 is an explanatory view obtained by deforming an original image including a virtual image based on a target image.

According to FIG. 8, it is as follows.
Reference target: Predetermined object or predetermined marker Original captured image: Original captured image including a reference image previously captured by the camera Original image: Original virtual image to be superimposed on the original captured image Target image: Reference captured in real time by the camera Target captured image including image Display image: Target augmented real image in which the target virtual image obtained by transforming the original virtual image is superimposed on the target captured image.

According to FIG. 8A, the position / orientation vector of the predetermined position based on the reference image in which the reference target is projected is estimated for the original captured image, and the position of the predetermined position in the target captured image based on the reference image in which the reference target is projected. The attitude vector is estimated. Then, the position / orientation vector of the original captured image is rotationally translated and matched with the position / orientation vector of the target captured image, and the transformation matrix is calculated.
According to FIG. 8B, the transformation matrix is used to transform the original virtual image to generate a target virtual image.
According to FIG. 8C, the target virtual image is superimposed on the target image to generate a display image as the target augmented reality image.

As described in detail above, according to the program, apparatus, and method of the present invention, the geometric shape of the three-dimensional object reflected in the original image is deformed in accordance with the position and orientation of the target image without causing distortion. A display image can be generated.
In addition, without causing distortion in the geometric shape of the three-dimensional object of the original virtual image, it is deformed to match the position and orientation of the target captured image and displayed in a superimposed manner to generate a display image as a target augmented reality image. You can also.

According to the various embodiments of the present invention described above, those skilled in the art can easily make various changes, modifications and omissions within the scope of the technical idea and viewpoint of the present invention. The above explanation is just an example and does not attempt to restrict anything. The present invention is limited only to the claims and their equivalents.

1 Device 11 Position / orientation estimation unit 12 Transformation matrix calculation unit 13 Display image generation unit

Claims (15)

A program that functions a computer mounted on a device that generates a display image obtained by transforming an original image in which a reference object is reflected based on a target image in which the reference object is reflected.
For each original image and the target image, the position and orientation estimation means a certain unit vector based on the reference image in which the reference object is reflected, estimated as the position and orientation vector,
A transformation matrix calculation means for calculating a transformation matrix that matches the position / orientation vector of the original image with the position / orientation vector of the target image.
A program characterized in that a computer functions as a display image generation means for generating a display image obtained by transforming an original image by the transformation matrix. The position / orientation estimation means uses the position / attitude vector.
Vertical unit vector of the world coordinate system,
The program according to claim 1 , wherein the computer functions so that an arbitrary horizontal unit vector in the world coordinate system or an arbitrary unit vector set by the user is a vector calculated by fluoroscopic projection transformation. .. The position / orientation estimation means
The reference image showing the reference object and the reference world coordinates of the reference object are stored in advance in association with each other.
From the original image and the target image, feature points based on the local features of the reference image are extracted.
Claim 1 or 2 characterized in that the computer functions to estimate the relative position / orientation vector of the reference object by the perspective projection conversion between the feature point coordinates of the reference image and the reference world coordinates of the reference object. The program described in. The position / orientation estimation means uses the start point of the position / orientation vector as a starting point.
Any point set by the user, or
According to any one of claims 1 to 3 , the computer functions so as to be an intersection of a straight line extending from the world coordinate value of the camera viewpoint in the shooting direction and a predetermined periphery of the world coordinate value of the reference target. Described program. Before SL transformation matrix is represented by the rotational theta, translation t and scale s,
The transformation matrix calculation means is characterized in that the computer functions to calculate a transformation matrix in which the position / orientation vector is matched so that the error included in the rotation θ and the translation t is removed and the error is included only in the scale s. The program according to any one of claims 1 to 4. The transformation matrix calculation means
Rotational alignment means that aligns the rotation direction in the image plane,
A translational alignment means that aligns the translational directions in the image plane,
The program according to claim 5 , wherein the computer functions to include a scale matching means for matching the scales in the image plane. Following the rotation matching means, the translation matching means functions.
The rotation matching means has a rotation matrix for rotating and matching the rotation direction of the position / orientation vector of the original image in the image plane so as to coincide with the rotation direction of the position / orientation vector of the target image, and rotation matching based on the rotation matrix. The later original image is output to the translational matching means,
The translational matching means includes a translational vector that translates and aligns the translational direction of the position-orientation vector of the original image after rotation-matching in the image plane so as to coincide with the translational direction of the position-orientation vector of the target image, and the rotation matrix. The program according to claim 6 , wherein the computer functions to output the rotation matrix and the original image after rotation translation matching based on the translation vector to the scale matching means. Following the translational matching means, the rotational matching means is made to function.
The translational matching means includes a translational vector that translates and aligns the translational direction of the position and orientation vector of the original image in the image plane so as to coincide with the translational direction of the position and orientation vector of the target image, and a translational matching based on the translational vector. The latter original image is output to the rotation matching means, and the image is output to the rotation matching means.
The rotation matching means includes a rotation matrix that rotationally aligns the rotation direction of the position / orientation vector of the original image after translational alignment with the rotation direction of the position / orientation vector of the target image in the image plane, and the translation vector. The program according to claim 6 , wherein the computer functions to output the rotation matrix and the original image after rotation translation matching based on the translation vector to the scale matching means. The scale matching means is characterized in that the computer functions to scale-match the original image after the rotation-translation matching so that the scale of the original image after the rotation-translation matching matches the scale of the target image. Item 2. The program according to any one of items 7 or 8. The target image is an image that changes in time series at a predetermined frame rate.
The program according to claim 9 , wherein the scale matching means causes a computer to function so as to smooth a scale conversion value calculated continuously in a time series. The reference object is a predetermined object or a predetermined marker.
The original image is an original image taken in advance by the camera.
The program according to any one of claims 1 to 10 , wherein the target image causes the computer to function as if it were a target captured image captured in real time by a camera. In order to generate a target augmented reality image for the target captured image as the display image from the original augmented reality image in which a virtual image is superimposed on a predetermined position based on the reference image in which the reference target appears in the original captured image.
The display image generation means is a computer that transforms the virtual image by the transformation matrix and generates a target augmented reality image superimposed on a predetermined position of a reference image reflected in the target captured image as the display image. The program according to claim 11 , wherein the program is made to function. The virtual image in the original augmented reality image is characterized in that the computer functions so that the reference image in which the reference target is reflected in the original captured image has a three-dimensional shape as viewed from the camera viewpoint. The program according to claim 12. It is a display image generation device that generates a display image obtained by transforming an original image in which a reference object is reflected based on a target image in which the reference object is reflected.
For each original image and the target image, the position and orientation estimation means a certain unit vector based on the reference image in which the reference object is reflected, estimated as the position and orientation vector,
A transformation matrix calculation means for calculating a transformation matrix that matches the position / orientation vector of the original image with the position / orientation vector of the target image.
A display image generation device comprising a display image generation means for generating a display image obtained by transforming an original image by the transformation matrix. It is a display image generation method of a device that generates a display image obtained by transforming an original image in which a reference object is reflected based on a target image in which the reference object is reflected.
The device is
For each original image and the target image, a first step of estimating the predetermined unit vector based on the reference image in which the reference object is reflected, as the position and orientation vector,
The second step of calculating the transformation matrix for matching the position / orientation vector of the original image with the position / orientation vector of the target image, and
A display image generation method comprising executing a third step of generating a display image obtained by transforming an original image by the transformation matrix.

Images