JP6820821B2 - Devices, programs and methods for calculating the degree of similarity between camera positions and orientations from captured images - Google Patents

Description

The present invention relates to a technique for calculating the degree of similarity in the position and orientation of a plurality of cameras.

Conventionally, there is a technique of displaying another content on a photographed image in real time according to the relative position and orientation of the object to be photographed and the camera (see, for example, Patent Document 1).
According to this technique, a "marker" as a reference target is attached to an object to be photographed, and the position and orientation of the camera are estimated using a photographed image in which the marker is reflected. In addition, for each reference image in which the reference target is displayed, the content (presentation information) corresponding to the position and orientation of the camera is stored in advance. The content here can be displayed as an Augmented Reality image superimposed on the captured image.
Then, from the captured image in which the reference object is reflected, the position and orientation of the camera and the position and orientation of the reference image in which the plurality of reference objects stored in advance are reflected are compared based on the "distance function". The presentation information associated with the reference target estimated to have a short distance (similar in position and orientation) is superposed on the captured image.

There is also a technique for estimating the relative position / orientation (translation vector t and rotation vector θ) between the camera and the reference object (see, for example, Non-Patent Document 1). According to this technique, the reference image of the marker to be referred to is stored in advance, and the position and orientation are estimated by matching the reference image from the captured image.

Japanese Unexamined Patent Publication No. 2016-71496

Kato, M. Billinghurst, Asano, Tachibana, "Augmented Reality System Based on Marker Tracking and Its Calibration", Journal of the Virtual Reality Society of Japan, vol.4, 20 no.4, pp.607-617, 1999., [online], [Search on October 7, 2017], Internet <URL: http://intron.kz.tsukuba.ac.jp/tvrsj/4.4/kato/p-99_VRSJ4_4.pdf> D.G.Lowe, Distinctive image features from scale-invariant key points, Proc. Of Int. Journal of Computer Vision (IJCV), 60 (2) pp.91-110 (2004) H.Bay, T.Tuytelaars, and L.V.Gool, SURF: Speed Up Robust Features, Proc. Of Int. Conf. Of ECCV, (2006) Ethan Rublee, Vincent Rabaud, Kurt Konolige, Gary R. Bradski: ORB: An efficient alternative to SIFT or SURF. ICCV 2011: 2564-2571.

FIG. 1 is an explanatory diagram showing an evaluation of position and posture in the prior art.

For example, using the following distance function, it is determined that the shorter the distance, the more similar (closer) the positions and orientations of the cameras are.
First distance function based on position-orientation vector Second distance function based on image features

<First distance function based on position-attitude vector>
As the first distance function, the Euclidean distance of the translation vector t and the rotation vector θ representing the relative positions and orientations of the camera and the reference object is used.

According to FIG. 1A, it is determined which of the camera cam1 and the camera2 is closer to the camera CAM on the world coordinate system.
Translational difference between CAM and cam ₁ : Δt ₁ = 100 mm
Difference in rotation between CAM and cam ₁ : Δθ ₁ = 45 °
Translational difference between CAM and cam2: Δt ₂ = 120mm
Difference in rotation between CAM and cam 1: Δθ ₂ = 10 °
Adding the weight w (= 1) and adding the rotation t and the translation θ gives, for example, the following.
D ₁ = Δt ₁ + w · Δθ ₁ = 100 + 1 * 45 = 145
D ₂ = Δt ₂ + wvΔθ ₂ = 120 + 1 * 10 = 130
In this case, D ₁ > D ₂ , and it is estimated that the position and orientation of the camera cam ₂ is closer to the position and orientation of the camera CAM.
On the other hand, when the scale of the shooting environment changes, for example, when the translation t is on the order of 10000 mm, the result is as follows.
D ₁ = Δt ₁ + w · Δθ ₁ = 10000 + 45 = 10045
D ₂ = Δt ₂ ＋ w ・ Δθ ₂ ＝ 12000 ＋ 10 ＝ 12010
In this case, D ₁ <D ₂ , and it is estimated that the position and orientation of the camera cam 1 is closer to the position and orientation of the camera CAM.

In this way, it is necessary to adjust the relative weight w given to the parameter according to the scale of the translation t and the rotation θ of the shooting environment. In addition, it is unclear how to appropriately determine the weight w in the first place.

<Second distance function based on image features>
As the second distance function, the Euclidean distance of the image coordinates between the local regions (image features) is used.

The relationship between the Euclidean distance in image coordinates and the Euclidean distance in world coordinates is non-linear. Even though the distance of the image feature amount in the image coordinates is small, the position and orientation in the world coordinates may be significantly different. That is, if there is a lateral shift or a change in enlargement / reduction in the captured image, it is estimated that the distance is long (the degree of similarity is low).

According to FIG. 1B, an attempt is made to estimate the camera closest to the position and orientation of the camera CAM (high degree of similarity).
It is presumed that the camera cam1 has a long distance (low similarity) between image features due to the influence of enlargement in the image.
It is presumed that the camera cam2 has a long distance (low similarity) between image features due to the effect of translation in the image.
On the other hand, in the camera cam3, it is estimated that the distance between the image features is short (high degree of similarity) even though the image features are geometrically greatly distorted in the image.
As a result, according to FIG. 1B, it is estimated that the camera cam3 is the closest to the camera CAM in terms of position and orientation. At this time, if the presented information (content) is superimposed on the captured image in a form deviating from the actual position / posture, it gives an unnatural impression to the user.

On the other hand, the inventors of the present application wondered if the following two problems could be eliminated in the evaluation of the similarity between the positions and orientations of the cameras.
(1) When using the Euclidean distance of the vector representing the position and orientation of the camera relative to the object to be photographed, the problem of dependence on the scale of translation and rotation in the photographing environment (2) In the image coordinates of the image feature amount Problem of non-linearity with distance in the world coordinate system when using the Euclidean distance of

Therefore, the present invention is a device and a program for calculating the similarity between the position and orientation of the camera from the captured image while maintaining the linearity with the distance in the world coordinate system without depending on the scale of translation and rotation in the photographing environment. And to provide a method.

According to the present invention, it is a device that calculates the similarity of the position and orientation of different cameras from a captured image in which a reference target having a known world coordinate value Ptarget is displayed.
A reference target storage means in which a reference image showing a reference target and a reference world coordinate value of the reference target are associated and stored in advance, and
Based on the coordinate transformation between the image coordinate value of the captured image in which the reference target is reflected and the reference world coordinate value of the reference image of the reference target, the relative shooting position / orientation vector (translation t) between the camera and the reference target in the captured image. And the shooting position / orientation estimation means for estimating the rotation θ),
Shooting from the camera viewpoint Pcam (x, y, z), which represents the world coordinate values of the camera, to the camera gazing point Plookat (u, v, w) with respect to the world coordinate value Ptarget to be referenced, using the shooting position / orientation vector. A shooting line-of-sight calculation means for calculating the line-of-sight vector,
It is characterized by having a similarity calculation means for calculating the similarity of shooting line-of-sight vectors in different captured images as the similarity of the position and orientation of the camera.

According to other embodiments of the apparatus of the present invention.
A presentation line-of-sight storage means that stores the presentation information in advance by associating the presentation information with the presentation line-of-sight vector of the presentation information.
It further has a presentation information display means for displaying the presentation information corresponding to the presentation line-of-sight vector searched by the similarity calculation means in a superimposed manner on the captured image.
It is also preferable that the similarity calculation means searches the presentation line-of-sight storage means for the presentation line-of-sight vector that is most similar to the shooting line-of-sight vector for the reference target used in the shooting position / orientation estimation means.

According to other embodiments of the apparatus of the present invention.
It is also preferable that the presented line-of-sight storage means previously stores the presented line-of-sight vector in each of a plurality of different presentation information for the same reference object.

According to other embodiments of the apparatus of the present invention.
The shooting line-of-sight calculation means connects the camera viewpoint Pcam and the camera gazing point Plookat, which is the intersection of the camera viewpoint Pcam and the perpendicular line drawn from the reference target world coordinate value Ptarget with respect to the half line extending in the shooting direction from the camera viewpoint Pcam. It is also preferable to calculate the shooting line-of-sight vector.

According to other embodiments of the apparatus of the present invention.
The world coordinate value Ptarget of the reference image is based on the plane of symmetry Starget.
It is also preferable that the shooting line-of-sight calculation means calculates the shooting line-of-sight vector connecting the camera viewpoint Pcam and the camera gazing point Plookat which is the intersection of the half line extending in the shooting direction from the camera viewpoint Pcam and the plane of symmetry Starget. ..

According to other embodiments of the apparatus of the present invention.
The similarity of the photographed line-of-sight vector in the similarity calculation means is the sum of the distance between the camera viewpoints (x, y, z) and the distance between the camera gazing points (u, v, w) in the line-of-sight vector. Is also preferable.

According to other embodiments of the apparatus of the present invention.
It is also preferable that the similarity of the photographing line-of-sight vector in the similarity calculation means is the Euclidean distance between the line-of-sight vectors.

According to other embodiments of the apparatus of the present invention.
The similarity of the shooting line-of-sight vector in the similarity calculation means is the sum of the Euclidean distance between different camera viewpoints Pcam (x, y, z) and the Euclidean distance between different camera gazing points Plookat (u, v, w). Is also preferable.

According to other embodiments of the apparatus of the present invention.
The captured image is continuous at a predetermined frame rate on the time axis.
The similarity calculation means is the similarity between the captured line-of-sight vector of the captured image at time t and the presented line-of-sight vector for which the similarity is calculated, the captured line-of-sight vector of the captured image at time t, and the time t-1. It is also preferable to weight the similarity with the presentation line-of-sight vector searched by the captured image of.

According to other embodiments of the apparatus of the present invention.
The captured image is continuous at a predetermined frame rate on the time axis.
The similarity calculation means is the similarity between the captured line-of-sight vector of the captured image at time t and the presented line-of-sight vector for which the similarity is calculated, the captured line-of-sight vector of the captured image at time t-1, and the time t. Calculation of similarity with the predicted value of the presented line-of-sight vector at time t obtained by adding the amount of movement of the captured image with the shot line-of-sight vector to the presented line-of-sight vector searched by the captured image at time t-1. It is also preferable to weight the similarity with the target presented line-of-sight vector.

According to the present invention, it is a program that causes a computer to function so as to calculate the degree of similarity between the positions and orientations of different cameras from captured images showing a reference target having a known world coordinate value Ptarget.
A reference target storage means in which a reference image showing a reference target and a reference world coordinate value of the reference target are associated and stored in advance, and
Based on the coordinate transformation between the image coordinate value of the captured image in which the reference target is reflected and the reference world coordinate value of the reference image of the reference target, the relative shooting position / orientation vector between the camera and the reference target in the captured image is estimated. Shooting position / orientation estimation means and
Shooting from the camera viewpoint Pcam (x, y, z), which represents the world coordinate values of the camera, to the camera gazing point Plookat (u, v, w) with respect to the world coordinate value Ptarget to be referenced, using the shooting position / orientation vector. A shooting line-of-sight calculation means for calculating the line-of-sight vector,
It is characterized in that a computer functions as a similarity calculation means for calculating the similarity of shooting line-of-sight vectors in different captured images as the similarity of the position and orientation of the camera.

According to the present invention, there is a method for calculating the position-posture similarity of a device that calculates the similarity between the positions and orientations of different cameras from a captured image showing a reference object having a known world coordinate value Ptarget.
The device is
It has a reference target storage unit that stores in advance a reference image in which a reference target is displayed and a reference world coordinate value of the reference target in association with each other.
Based on the coordinate conversion between the image coordinate value of the captured image in which the reference target is reflected and the reference world coordinate value of the reference image of the reference target, the relative shooting position / orientation vector between the camera and the reference target in the captured image is estimated. The first step and
Shooting from the camera viewpoint Pcam (x, y, z), which represents the world coordinate values of the camera, to the camera gazing point Plookat (u, v, w) with respect to the world coordinate value Ptarget to be referenced, using the shooting position / orientation vector. The second step of calculating the line-of-sight vector and
It is characterized in that the third step of calculating the similarity of the captured line-of-sight vectors in different captured images as the similarity of the position and orientation of the camera is executed.

According to the apparatus, program and method of the present invention, the similarity of the position and orientation of the camera from the captured image is maintained while maintaining the linearity with the distance in the world coordinate system without depending on the scale of translation and rotation in the imaging environment. Can be calculated.

It is explanatory drawing which shows the evaluation of the position | posture in the prior art. It is explanatory drawing which displays the presentation information of the virtual reality in this invention. It is a functional block diagram of the apparatus in this invention. It is a flowchart in this invention. It is explanatory drawing which shows the method of determining the camera gaze point in this invention. It is explanatory drawing which shows the similarity of the position | position | posture of this invention corresponding to FIG. It is explanatory drawing which shows the similarity of the position | posture of this invention in orthorhombic photography.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings.

FIG. 2 is an explanatory diagram for displaying the presentation information of the virtual reality in the present invention.

According to FIG. 2A, a marker (reference target) is attached to the object to be photographed. The marker may be, for example, a two-dimensional QR code (registered trademark). The object to be photographed is photographed by four cameras cam1, cam2, cam3, and cam4 having different positions and orientations.
According to FIG. 2B, images taken by each camera are shown. When a marker is reflected in the captured image, the marker is reflected from a different angle for each camera. According to the present invention, the position and orientation of the camera are extracted from the captured image in which the marker is reflected.
According to FIG. 2C, presentation information (content) corresponding to the position and orientation of the camera is represented. When the content has a three-dimensional shape, presentation information having a different shape is selected according to the position and orientation of the camera.
According to FIG. 2D, in the virtual reality image, for example, rectangular parallelepiped content is superimposed and displayed on the marker displayed in the captured image.
According to FIG. 2E, the user can visually recognize the captured images captured by the cameras cam1, cam2, and cam4 as if the content is superimposed on the captured object.

FIG. 3 is a functional configuration diagram of the device according to the present invention.
FIG. 4 is a flowchart of the present invention.

According to FIG. 3A, the device 1 of the present invention is represented as a terminal equipped with a camera and a display, such as a smartphone. Of course, it is not limited to smartphones, but may be tablets or personal computers.
According to FIG. 3B, the device 1 of the present invention is represented as a server. The server receives the captured image captured by the camera of the user terminal via the communication interface, and returns the virtual reality image on which the presentation information (content) is superimposed to the user terminal.
It should be noted that, without realizing the present invention with only one device, the functions of the present invention can be divided and configured as a system of a server and a terminal via a network.

The device 1 includes a reference target storage unit 101, a presentation line-of-sight storage unit 102, a shooting position / orientation estimation unit 11, a shooting line-of-sight calculation unit 12, a similarity calculation unit 13, and a presentation information display unit 14. .. These functional components are realized by executing a program that makes the computer mounted on the device function. In addition, the processing flow of these functional components can be understood as a method of calculating the degree of similarity between the positions and orientations of the cameras.

[Reference target storage unit 101]
The reference target storage unit 101 stores in advance the "reference image" in which the reference target is displayed and the "reference world coordinate value" of the reference target in association with each other.
According to FIG. 4, the reference image and the reference world coordinates are stored in association with each other in the table of the reference target storage unit 101.

The reference image may be a feature quantity of SIFT (Scale-Invariant Feature Transform) or SURF (Speeded Up Robust Features) that is robust to rotation and scale change, or an ORB (ORB) that extracts compact local features at high speed. It may be a feature quantity of Oriented FAST and Rotated BRIEF) (see, for example, Non-Patent Documents 2, 3 and 4).
For example, in the case of SIFT, a set of 128-dimensional local features is extracted from one image. SIFT is a technique for analyzing a characteristic local region using a scale space and describing a local feature that is invariant to the scale change and rotation.
Further, in the case of SURF, faster processing than SIFT is possible, and a set of 64-dimensional local features is extracted from one image. While SIFT has a high processing cost and difficult real-time matching, SURF speeds up processing by using an integrated image.
Further, in the case of ORB, as mobile terminals such as smartphones and tablets become widespread, it is suitable for memory saving and high-speed matching for augmented reality applications.

[Shooting position / orientation estimation unit 11]
The shooting position / orientation estimation unit 11 first extracts the reference image stored in the reference target storage unit 101 from the shot image taken by the camera by the local feature amount extraction algorithm.
The local feature extraction algorithm is a well-known existing technique such as SIFT, SURF, and ORB described above. Specifically, a local region similar to the feature amount of the reference image is extracted from the captured image by using the local feature amount based on the relative luminance gradient. The extracted local region has properties that are invariant to any or any combination of rotation, scaling, projective change (distortion due to projective transformation).
However, when the reference image cannot be extracted, there is no presentation information (content) superimposed on the captured image, and the captured image is output as it is from the presentation information display unit 14.

Then, the shooting position / orientation estimation unit 11 determines the camera and the reference target in the shot image based on the coordinate conversion between the “image coordinate value of the shot image” in which the reference target is projected and the “reference world coordinate value” of the reference target. Estimate the "shooting position / orientation vector" relative to.
The “shooting position / orientation vector” is the relative position / orientation (translation t and rotation θ) of the camera with reference to the marker (reference target) attached to the shooting object.
Then, the shooting position / posture vector is output to the shooting line-of-sight calculation unit 12.

[Shooting line-of-sight calculation unit 12]
The shooting line-of-sight calculation unit 12 registers the world coordinate value Ptarget of the marker (reference target) in advance by the user or by another function.
Then, the shooting line-of-sight calculation unit 12 uses the shooting position / orientation vector from the camera viewpoint Pcam (x, y, z) representing the world coordinate value of the camera to the camera gazing point Plookat (x, y, z) with respect to the reference target world coordinate value Ptarget. Calculate the "shooting line-of-sight vector" for u, v, w). The line-of-sight vector consists of 6 elements (x, y, z, u, v, w).
Then, the photographing line-of-sight calculation unit 12 outputs the calculated line-of-sight vector (x, y, z, u, v, w) to the similarity calculation unit 13.

FIG. 5 is an explanatory diagram showing a method of determining a camera gaze point in the present invention.

<First method of determining the camera gaze point>
According to FIG. 5A, the shooting line-of-sight calculation unit 12 refers to the camera viewpoint Pcam (x, y, z) and the half straight line extending in the shooting direction from the camera viewpoint Pcam (x, y, z). The camera gaze point Plookat (u, v, w), which is the intersection with the perpendicular line drawn from the world coordinate value Ptarget (u, v, w) of the reference target, is connected to the "shooting line-of-sight vector (x, y, z, u, v, w) ”is calculated.
<Second method for determining the camera gaze point>
According to FIG. 5B, it is assumed that the world coordinate value Ptarget of the reference image is based on the plane of symmetry Starget (plane). Here, the shooting line-of-sight calculation unit 12 is at the intersection of the camera viewpoint Pcam (x, y, z) and the half straight line extending in the shooting direction from the camera viewpoint Pcam (x, y, z) and the plane of symmetry Starget. The "shooting line-of-sight vector (x, y, z, u, v, w)" connecting a certain camera gaze point Plookat (u, v, w) is calculated.

[Present line-of-sight storage unit 102]
The presentation line-of-sight storage unit 102 stores the "presentation line-of-sight vector" and the "presentation information" in advance.
According to FIG. 4, the presentation line-of-sight storage unit 102 stores the presentation line-of-sight vector in advance for each of a plurality of different presentation information for the same reference target. The “presentation line-of-sight vector” is for the presentation information (content). , The line-of-sight vector of the camera. This is the information presented in the same manner as in the above-described photographing line-of-sight calculation unit 12.

[Similarity calculation unit 13]
The similarity calculation unit 13 calculates the similarity of the captured line-of-sight vectors in different captured images as the similarity of the position and orientation of the camera.
The similarity calculation unit 13 searches the presentation line-of-sight storage unit 102 for a presentation line-of-sight vector that is most similar to the shooting line-of-sight vector for the reference target used in the shooting position / orientation estimation unit 11. At this time, the similarity calculation unit 13 calculates the similarity d (C _i ', C) between the photographed line-of-sight vector C and the presented line-of-sight vector C _i '.
i_min ＝ arg _i min d (C _i ', C)
C: Shooting line-of-sight vector
C _i ': Present line-of-sight vector to be searched (i ∈ N, total number to be searched)
C _{i_min} ': _Present line-of-sight vector most similar to the shooting line-of-sight vector
arg _i min d: A set of i that minimizes d The similarity calculation unit 13 outputs the presentation information corresponding to the _searched presentation line-of-sight vector C _{i_min'to} the presentation information display unit 14.

The captured image may be continuous at a predetermined frame rate on the time axis. In this case, the captured image I (t) of the camera is input for each time range Δt, and each time, the shooting position / orientation estimation unit 11, the shooting line-of-sight calculation unit 12, the similarity calculation unit 13, and the presentation information display are performed in real time. Part 14 is executed.
i_min (t) = arg _i min d (C _i ', C (t))

Here, different embodiments in the similarity calculation will be described.
<First Embodiment of similarity calculation>
The similarity of the photographed line-of-sight vector in the similarity calculation unit 13 is the sum of the distance between the camera viewpoints (x, y, z) and the distance between the camera gazing points (u, v, w) in the line-of-sight vector. ..
d (C', C) = r + g
= | P'cam (x, y, z) -Pcam (x, y, z) |
＋ ｜ P'lookat (u, v, w) －Plookat (u, v, w) ｜
r: Distance between camera viewpoints (x, y, z) | P'cam (x, y, z) -Pcam (x, y, z) |
g: Distance between camera gaze points (u, v, w) | P'lookat (u, v, w) -Plookat (u, v, w) |

<Second embodiment of similarity calculation>
The similarity of the photographed line-of-sight vector in the similarity calculation unit 13 is the Euclidean distance between the line-of-sight vectors. That is, it can be calculated as a distance in a six-dimensional space.
d (C ', C) = √ ((x'-x) 2 + (y'-y) 2 + (z'-z) 2 + (u'-u) 2 + (v'-v) 2 + (w'-w) ² )

<Third embodiment of similarity calculation>
The similarity of the shooting line-of-sight vector in the similarity calculation unit 13 is the Euclidean distance between different camera viewpoints Pcam (x, y, z) and the Euclidean distance between different camera gazing points Plookat (u, v, w). It is a sum.
d (C ', C) = √ ((x'-x) 2 + (y'-y) 2 + (z'-z) 2) + √ ((u'-u) 2 + (v'-v ) ² + (w'-w) ² )

<Fourth Embodiment of Similarity Calculation>
When the captured image is continuous at a predetermined frame rate on the time axis, the similarity calculation unit 13 sets the captured line-of-sight vector of the captured image at time t and the presented line-of-sight vector to be calculated for the similarity. Is calculated by weighting the similarity between the photographed line-of-sight vector of the captured image at time t and the presented line-of-sight vector searched by time t-1.

When processing time-series captured images in real time, the presented line-of-sight vector C _{i_min (t)} 'that most closely resembles the captured line-of-sight vector C (t) of the captured image is searched for every time t. On the other hand, presents visual line portion vector C _i 'are those that are stored in advance based on irregular and discrete position and orientation. Therefore, the presented line-of-sight vector C _{i_min (t)} 'determined to be the most similar is significantly different from the already determined presented line-of-sight vector C _{i_min (t-1)} ' at the latest past time t-1. There is a possibility that Such a sudden fluctuation causes the presentation line-of-sight vector C _{i_min (t)} 'to be selected in a form that suddenly deviates from the past shooting line-of-sight vector C (t-1) at time t. At this time, when the presented line-of-sight vector is continuously selected in real time on the time series, the selection result may fluctuate in a sawtooth shape.

Therefore, the presented line-of-sight vector C _{i_min (t)} 'determined at each time t fluctuates smoothly on the time axis, so that the presented line-of-sight vector C already determined at the latest past time t-1 _{From i_min (t-1)} ', the following equation with a cost term added is used to prevent the presented line-of-sight vector C _{i_min (t)} 'at time t from fluctuating extremely.
i_min (t) = arg _i min {αd (C _i ', C (t)) + (1-α) d (C _i ', C _{i_min (t-1)} ')}
α: 0 <α <1

<Fifth Embodiment of similarity calculation>
The similarity calculation unit 13 has already determined the presentation at the latest past time t-1 so that the presentation line-of-sight vector C _{i_min (t)} 'determined at each time t fluctuates smoothly on the time axis. In order to suppress the extreme fluctuation of the presented line-of-sight vector C _{i_min (t)} 'at time t from the line-of-sight vector C _{i_min (t-1)} ', the line-of-sight vector C of the captured image at time t-1 _{Add the} amount of movement (C (t) -C (t-1)) between (t-1) and the shooting line-of-sight vector C (t) of the shot image at time t to C _{i_min (t-1)} '. The similarity between the predicted value of C _{i_min (t)} 'obtained by the _above and the presented line-of-sight vector to be calculated for the similarity is added as a cost term to the following equation.
i_min (t) ＝ arg _i min {αd (C _i ', C (t)) ＋
(1−α) d (C _i ', C _{i_min (t-1)} ' + {C (t) −C (t-1)})}
α: 0 <α <1
As a result, the fluctuation of the presentation line-of-sight vector C _{i_min (t)} 'at time t can be smoothed. In this case, the presentation information corresponding to the smoothed presentation line-of-sight vector C _{i_min (t)} 'is selected in the presentation line-of-sight storage unit 102.

[Presentation information display unit 14]
The presentation information display unit 14 superimposes and displays the presentation information corresponding to the presentation line-of-sight vector searched by the similarity calculation unit 13 on the captured image.
As a result, a high-quality virtual reality image can be provided to the user by drawing the content on the captured image by computer graphics.

FIG. 6 is an explanatory diagram showing the degree of similarity of the position and orientation of the present invention corresponding to FIG.

In FIG. 6A, as compared with FIG. 1A, the rotation is treated as the coordinates of the camera gazing point, so that the weighting between the rotation and the translation becomes unnecessary.
The similarity between the line-of-sight vector of the camera CAM and the line-of-sight vector of the camera cam1 is calculated as follows, for example.
D ₁ = r ₁ + g ₁ = 100mm + 300mm = 400mm
The similarity between the line-of-sight vector of the camera CAM and the line-of-sight vector of the camera cam2 is calculated as follows, for example.
D ₂ = r ₂ + g ₂ = 120mm + 100mm = 220mm
In this case as well, D ₁ > D ₂ and it is estimated that the position and orientation of the camera cam ₂ are closer.
On the other hand, even if the scale of the shooting environment changes, for example, the translation t is on the order of 10000 mm, the result is as follows.
D ₁ = r ₁ + g ₁ = 10000 + 30000 = 40000
D ₂ = r ₂ + g ₂ = 12000 + 10000 = 22000
In this case as well, D ₁ > D ₂ and it is estimated that the position and orientation of the camera cam2 are closer, there is no change in the comparison of the position and orientation, and the position and orientation of the camera cam2 is closer without considering the scale. It is estimated to be.

In FIG. 6B, as compared with FIG. 1B, it is determined that cam1 and cam2 have the same degree of similarity to the camera CAM, and cam3 has a low degree of similarity (long distance). .. According to the present invention, the similarity of position and orientation is not lowered due to the influence of enlargement / reduction, lateral displacement, and rotation with respect to the line-of-sight axis.

FIG. 7 is an explanatory diagram showing the degree of similarity between the positions and postures of the present invention in orthorhombic photography.

According to FIG. 7, the change in the similarity between the camera CAM and the cameras cam1, cam2, and cam3 has a nearly linear relationship with the change in the angle of rotation. The contribution rate to the similarity of rotation does not change easily depending on the angle of the camera, and the sensitivity to the change in rotation does not deteriorate even when facing the camera.

As described in detail above, according to the apparatus, program and method of the present invention, imaging is performed while maintaining linearity with the distance in the world coordinate system without depending on the scale of translation and rotation in the imaging environment. The degree of similarity between the position and orientation of the camera can be calculated from the image.

According to the various embodiments of the present invention described above, those skilled in the art can easily make various changes, modifications and omissions within the scope of the technical idea and viewpoint of the present invention. The above explanation is just an example and does not attempt to restrict anything. The present invention is limited only to the scope of claims and their equivalents.

1 Device 101 Reference target storage unit 102 Presented line-of-sight storage unit 11 Shooting position / posture estimation unit 12 Photographed line-of-sight calculation unit 13 Similarity calculation unit 14 Presented information display unit

Claims (12)

It is a device that calculates the similarity of the position and orientation of different cameras from the captured image showing the reference target whose world coordinate value Ptarget is known.
A reference target storage means in which a reference image showing a reference target and a reference world coordinate value of the reference target are associated and stored in advance, and
Based on the coordinate transformation between the image coordinate value of the captured image in which the reference target is reflected and the reference world coordinate value of the reference image of the reference target, the relative shooting position / orientation vector (translation t) between the camera and the reference target in the captured image. And the shooting position / orientation estimation means for estimating the rotation θ),
Using the shooting position / orientation vector, the camera viewpoint Pcam (x, y, z) representing the world coordinate value of the camera is directed toward the camera gazing point Plookat (u, v, w) with respect to the world coordinate value Ptarget to be referenced. The shooting line-of-sight calculation means for calculating the shooting line-of-sight vector,
An apparatus characterized by having a similarity calculation means for calculating the similarity of shooting line-of-sight vectors in different captured images as the similarity of the position and orientation of a camera. A presentation line-of-sight storage means that stores the presentation information in advance by associating the presentation information with the presentation line-of-sight vector of the presentation information.
Further, it has a presentation information display means for displaying the presentation information corresponding to the presentation line-of-sight vector searched by the similarity calculation means superimposed on the captured image.
The similarity calculation means is characterized in that, with respect to the reference target used in the shooting position / orientation estimation means, a presentation line-of-sight vector most similar to the shooting line-of-sight vector is searched from the presentation line-of-sight storage means. The device according to claim 1. The apparatus according to claim 2, wherein the presented line-of-sight storage means stores a presented line-of-sight vector in advance for each of a plurality of different presentation information for the same reference object. The shooting line-of-sight calculation means includes a camera gazing point Plookat, which is an intersection of the camera viewpoint Pcam and a perpendicular line drawn from the world coordinate value Ptarget of the reference target with respect to a half straight line extending in the shooting direction from the camera viewpoint Pcam. The apparatus according to any one of claims 1 to 3, wherein the photographing line-of-sight vector connecting the two is calculated. The world coordinate value Ptarget of the reference image is based on the plane of symmetry Starget.
The shooting line-of-sight calculation means calculates a shooting line-of-sight vector connecting the camera viewpoint Pcam and the camera gazing point Plookat, which is the intersection of the half line extending in the shooting direction from the camera viewpoint Pcam and the symmetry plane Starget. The apparatus according to any one of claims 1 to 3, wherein the apparatus is characterized by. The similarity of the photographed line-of-sight vector in the similarity calculation means is the sum of the distance between the camera viewpoints (x, y, z) and the distance between the camera gazing points (u, v, w) in the line-of-sight vector. The device according to any one of claims 1 to 5, wherein the device is provided. The apparatus according to any one of claims 1 to 5, wherein the similarity of the photographed line-of-sight vector in the similarity calculation means is an Euclidean distance between the line-of-sight vectors. The similarity of the photographing line-of-sight vector in the similarity calculation means is the Euclidean distance between different camera viewpoints Pcam (x, y, z) and the Euclidean distance between different camera gazing points Plookat (u, v, w). The apparatus according to any one of claims 1 to 5, wherein the device is the sum of the above. The captured image is continuous at a predetermined frame rate on the time axis.
The similarity calculation means sets the similarity between the captured line-of-sight vector of the captured image at time t and the presented line-of-sight vector to be calculated for the similarity, the captured line-of-sight vector of the captured image at time t, and the time t-. The apparatus according to any one of claims 1 to 8, wherein the similarity with the presented line-of-sight vector searched by the captured image of 1 is weighted and calculated. The captured image is continuous at a predetermined frame rate on the time axis.
The similarity calculation means sets the similarity between the captured line-of-sight vector of the captured image at time t and the presented line-of-sight vector to be calculated for the similarity, the captured line-of-sight vector of the captured image at time t-1, and the time. Similarity with the predicted value of the presented line-of-sight vector at time t obtained by adding the amount of movement of the captured image at t with the captured line-of-sight vector to the presented line-of-sight vector searched by the captured image at time t-1. The apparatus according to any one of claims 1 to 8, wherein the degree of similarity with the presentation line-of-sight vector to be calculated is weighted and calculated. It is a program that makes a computer function to calculate the similarity of the position and orientation of different cameras from the captured image showing the reference target whose world coordinate value Ptarget is known.
A reference target storage means in which a reference image showing a reference target and a reference world coordinate value of the reference target are associated and stored in advance, and
Based on the coordinate transformation between the image coordinate value of the captured image in which the reference target is reflected and the reference world coordinate value of the reference image of the reference target, the relative shooting position / orientation vector between the camera and the reference target in the captured image is estimated. Shooting position / orientation estimation means and
Using the shooting position / orientation vector, the camera viewpoint Pcam (x, y, z) representing the world coordinate value of the camera is directed toward the camera gazing point Plookat (u, v, w) with respect to the world coordinate value Ptarget to be referenced. The shooting line-of-sight calculation means for calculating the shooting line-of-sight vector,
A program characterized in that a computer functions as a similarity calculation means for calculating the similarity of shooting line-of-sight vectors in different captured images as the similarity of the position and orientation of a camera. It is a method of calculating the position / orientation similarity of a device that calculates the similarity of the position / orientation of different cameras from a photographed image showing a reference target whose world coordinate value Ptarget is known.
The device
It has a reference target storage unit that stores in advance a reference image in which a reference target is displayed and a reference world coordinate value of the reference target in association with each other.
Based on the coordinate conversion between the image coordinate value of the captured image in which the reference target is reflected and the reference world coordinate value of the reference image of the reference target, the relative shooting position / orientation vector between the camera and the reference target in the captured image is estimated. The first step and
Using the shooting position / orientation vector, the camera viewpoint Pcam (x, y, z) representing the world coordinate value of the camera is directed toward the camera gazing point Plookat (u, v, w) with respect to the world coordinate value Ptarget to be referenced. The second step of calculating the shooting line-of-sight vector
A method for calculating the position / orientation similarity of an apparatus, which comprises executing a third step of calculating the similarity of the captured line-of-sight vectors in different captured images as the similarity of the position / orientation of the camera.

Images