JP6860620B2 - Information processing equipment, information processing methods, and programs - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and a program.

The measurement of the position and orientation of an image pickup device based on an image is used for various purposes. An example of a purpose is the alignment of a real space with a virtual object in mixed reality (MR) technology / augmented reality (AR) technology. Another example of purpose is self-position estimation for autonomous movement of robots or autonomous driving of automobiles.

Non-Patent Document 1 discloses a method of measuring the position and orientation of an image pickup apparatus using feature points detected from an image. In this method, the 3D coordinates of the feature points in the scene are measured with high accuracy by the Structure from motion technology in advance, and this information is saved as a 3D map. At runtime, the feature points on the captured image corresponding to the feature points in the three-dimensional map are searched, and the position and orientation of the image pickup device are calculated based on the correspondence between the image coordinates of the feature points and the three-dimensional coordinates.

Z. Dong, G. Zhang, J. Jia, and H. Bao, “Keyframe-based real-time camera tracking,” Proc. 2009 IEEE 12th International Conference on Computer Vision (ICCV), pp. 1538-1545, 2009.

The method disclosed in Non-Patent Document 1 has a problem that high-precision position / orientation measurement cannot be performed when the imaging device is out of the range in which position / orientation can be measured by a three-dimensional map created in advance at runtime. Existed.

An object of the present invention is to extend the measurable range of the position and orientation of the imaging device while maintaining the measurement accuracy of the position and orientation in the position and orientation measurement of the imaging device based on the captured image.

In order to achieve the object of the present invention, for example, the image processing apparatus of the present invention has the following configurations. That is,
Obtaining means position to acquire a captured image of an object from a variable of the imaging device,
A three-dimensional map containing information indicating a three-dimensional position in a reference coordinate system set in the space where the subject is located, which is a first feature point obtained from a captured image of the subject in a predetermined area. A holding means for holding a three-dimensional map created from a set of captured images captured in an area, and
A derivation means for deriving the position of the image pickup apparatus by using the captured image acquired by the acquisition means and the three-dimensional map held by the holding means.
And information indicating the obtained based on subject region outside the predetermined region in the captured image captured, the three-dimensional position of the second feature points in the three-dimensional map in which the holding means for holding said first Information indicating the three-dimensional position of the feature point 1 and information indicating the reliability of the feature point with respect to the true three-dimensional position in the reference coordinate system , and information indicating the three-dimensional position of the second feature point. A correction means for correcting the three-dimensional map by using the information indicating the reliability set with high reliability of the information indicating the three-dimensional position of the first feature point.
To be equipped.

In the position / orientation measurement of the imaging device based on the captured image, the range in which the position / orientation of the imaging device can be measured can be expanded while maintaining the measurement accuracy of the position / orientation.

The figure which shows an example of the hardware configuration of the information processing apparatus which concerns on one Embodiment. The figure which shows an example of the functional structure of the information processing apparatus 1 which concerns on Embodiment 1. FIG. The figure explaining the 3D map. An example of a flowchart of the information processing method according to the first embodiment. An example of a flowchart of 3D map correction processing. The figure which shows an example of the graphical user interface which concerns on one Embodiment. The figure explaining the auxiliary index. The figure which shows an example of the functional structure of the information processing apparatus 3 which concerns on Embodiment 3. The conceptual diagram of the embodiment.

Prior to explaining each embodiment of the present invention, a hardware configuration capable of realizing the information processing apparatus shown in each embodiment will be described with reference to FIG. FIG. 1 is a hardware configuration diagram of an information processing device according to an embodiment. In FIG. 1, the CPU 10 controls each unit connected to the bus 60 via the bus 60. The input I / F40 acquires an input signal in a format that can be processed by the information processing device 1 from an external device (display device, operating device, or the like). Further, the output I / F50 outputs an output signal in a format that can be processed by the external device to an external device (display device or the like).

The program for realizing the functions of each embodiment can be stored in a storage medium such as a read-only memory (ROM) 20. The ROM 20 can also store an operating system (OS) and device drivers. A memory such as the random access memory (RAM) 30 can temporarily store these programs. Then, by executing the program stored in the RAM 30 by the CPU 10, the processing according to each flowchart described later can be executed, and the function of each embodiment can be realized. However, instead of the software processing using the CPU 10, the functions of each embodiment can be realized by using hardware having a calculation unit or a circuit corresponding to the processing of each functional unit.

In the embodiment illustrated below, the information processing apparatus acquires an image from the imaging apparatus and performs processing. In these embodiments, the imaging device is a monocular camera. In addition, the image pickup device shall capture a shade image. However, the type of imaging device is not limited to this. The position and orientation of the image pickup device are variable, and it is possible to acquire a captured image of the subject. In the following, a three-dimensional coordinate system in which the optical center of the image pickup device is the origin, the optical axis direction is the Z axis, the horizontal direction of the image is the X axis, and the vertical direction of the image is the Y axis, is the image pickup device coordinate system or the image pickup coordinate system. It is stipulated as. The position and orientation of the imaging device is the position and orientation of the imaging coordinate system (for example, the position of the origin and the Z axis) with respect to the reference coordinate system (hereinafter referred to as the world coordinate system) defined in the space (scene) for imaging. Direction). The position and orientation of the image pickup device have six degrees of freedom (position 3 degrees of freedom and posture 3 degrees of freedom). Further, in the following description, an object existing in the scene is referred to as a subject. The type of subject is not particularly limited and may be, for example, a building or a room.

[Embodiment 1]
In the first embodiment, the reliability of the pre-created portion of the three-dimensional map of the scene is increased, and the reliability of the extended portion at runtime is decreased to correct the three-dimensional map.

FIG. 2 shows the configuration of the information processing device 1 according to the present embodiment. The information processing device 1 includes a holding unit 110, an input unit 120, a derivation unit 130, a creating unit 140, a setting unit 150, and a correction unit 160. The input unit 120 is connected to the image pickup apparatus 170.

The holding unit 110 holds a three-dimensional map of the scene used by the lead-out unit 130 to derive the position and orientation of the image pickup apparatus 170. In the present embodiment, the three-dimensional map includes information indicating the three-dimensional position of the feature of the subject. Further, in one embodiment, the three-dimensional map includes information indicating the observation result of the subject at each of the plurality of viewpoints. The holding unit 110 can hold a three-dimensional map created in advance, and can sequentially hold the corrected three-dimensional map created at runtime.

FIG. 3 is a diagram illustrating an example of a three-dimensional map. The three-dimensional map of the scene shown in FIG. 3 contains a set of keyframes. Keyframes refer to captured images captured by an imaging device at various locations in the scene. This key frame has information indicating the observation result of the subject from various viewpoints. For example, from the key frame, the position of the feature can be obtained by the feature extraction process, and the color information at a specific image position can be obtained.

Further, the three-dimensional map of FIG. 3 includes a position and orientation (hereinafter, referred to as "keyframe position and orientation") in the world coordinate system of the imaging device (that is, a viewpoint) at the time of keyframe imaging. Further, the three-dimensional map has image coordinates (u, v) of features (feature points in this embodiment) on keyframes and depth d (z coordinates in the keyframe imaging coordinate system) based on the position and orientation of keyframes. ) Is included. These pieces of information represent information indicating the three-dimensional position of the feature of the subject. In this embodiment, known methods (eg, C. Forster, M. Pizzoli, and D. Scaramuzza, “SVO: fast semi-direct monocular visual odometry,” Proc. 2014 IEEE International Conference on Robotics and Automation (ICRA)) , Pp. 15-22, 2014.), the three-dimensional coordinates of the feature points in the imaging coordinate system are calculated from the image coordinates (u, v) and the depth d of the feature points. Then, the three-dimensional coordinates of the feature points calculated in this way are used for the position / orientation measurement of the image pickup apparatus 170, which will be described later.

In the present embodiment, the holding unit 110 holds, as a three-dimensional map, three-dimensional information of the position and orientation of the key frame and the geometric feature on the key frame in addition to the key frame. Such a three-dimensional map can be generated by using a known technique. In this embodiment, a three-dimensional map of the scene is created in advance. The holding unit 110 can acquire a pre-created three-dimensional map stored in a storage unit (not shown). In the following, the pre-created part of the 3D map will be referred to as the "pre-map". That is, the pre-map contains pre-made information indicating the three-dimensional position of the feature (sometimes referred to as the first feature). On the other hand, information indicating the three-dimensional position of the feature (sometimes referred to as the second feature) is also added to the three-dimensional map by the creation unit 140 described later.

In the case of the present embodiment, the pre-map includes information on a plurality of pre-created keyframes, the position and orientation of each keyframe, and feature points in each keyframe. Here, the feature point information includes the image coordinates (u, v) of the feature point on the key frame and the depth d of the feature point. On the other hand, in the present embodiment, the three-dimensional map is expanded at the time of runtime (when the position of the image pickup apparatus 170 is measured by the information processing apparatus 1). For example, the three-dimensional map is expanded when there is a possibility that the image pickup device is out of the range in which the position / orientation measurement using the pre-map is possible according to a predetermined condition according to the position / orientation of the image pickup device. Ru. That is, information on a new key frame, the position / orientation of this key frame, and the feature points in this key frame is added to the three-dimensional map. By such a method, the measurement of the position and orientation of the image pickup apparatus 170 can be continued. For example, when there is an obstacle in the area targeted for pre-mapping and the image pickup apparatus 170 avoids the obstacle, the image pickup apparatus 170 may be out of the range in which the position and orientation can be measured. Further, when the image pickup device 170 moves out of the range in which the position / orientation can be measured using the pre-map, the image pickup device 170 may move out of the range in which the position / orientation can be measured.

The input unit 120 acquires an image captured by the imaging device 170 (hereinafter referred to as an input image). The input unit 120 can acquire a moving image from the image pickup apparatus 170, and can acquire, for example, a time-series image of 30 frames per second. The storage unit (not shown) of the information processing device 1 holds internal parameters (focal length, image center position, lens distortion parameter, etc.) of the image pickup device 170. The internal parameters of the imaging device 170 are described by known methods (eg, Z. Zhang, “A flexible new technique for camera calibration,” IEEE Trans. On Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp. 1330- It can be pre-calibrated using 1334, 2000.).

The derivation unit 130 derives the position and orientation of the image pickup apparatus 170 by using the input image and the three-dimensional map. The derivation unit 130 derives the position and orientation of the image pickup device 170 at the time of image capturing for each image input in time series from the input unit 120.

The creating unit 140 adds information indicating the three-dimensional position of the further feature of the subject, which is obtained based on the input image and the position / orientation of the imaging device 170, to the three-dimensional map. In this way, the 3D map is expanded. In the present embodiment, the creation unit 140 creates a three-dimensional map by adding keyframes and information indicating the three-dimensional positions of further features included in the keyframes (information on the position and orientation of the keyframes and feature points). Expand. The creation unit 140 can add keyframes, for example, when the position of the image pickup apparatus 170 may deviate from the range in which the position / orientation can be measured by the existing three-dimensional map.

The setting unit 150 sets the reliability (information indicating reliability) in the information indicating the three-dimensional position of the feature. In the present embodiment, the information indicating the three-dimensional position of the feature (first feature) included in the pre-map indicates the three-dimensional position of the feature (second feature) added by the creation unit 140. Higher reliability than information is given. In the present embodiment, the setting unit 150 sets the reliability of the keyframe included in the three-dimensional map. That is, the setting unit 150 can set the reliability of the keyframes included in the pre-map to be high and the reliability of the keyframes added by the creation unit 140 at runtime to be low.

The correction unit 160 corrects the information indicating the three-dimensional position of the feature included in the three-dimensional map based on the reliability of this information. For example, the correction unit 160 can update the information indicating the three-dimensional position of the feature included in the three-dimensional map according to the reliability of the information. According to such processing, the correction unit 160 can improve the consistency of the three-dimensional map. Such processing is known as optimization of a three-dimensional map (or pose graph). In the present embodiment, the correction unit 160 updates the information indicating the three-dimensional position of the feature by correcting the position and orientation of the key frame so that the entire three-dimensional map is consistent. The correction unit 160 can perform correction based on the reliability of the key frame set by the setting unit 150. In the present embodiment, the correction unit 160 fixes the position and orientation of the key frame with high reliability. For example, the correction unit 160 fixes the position and orientation of the keyframes included in the pre-map. Therefore, it is possible to expand the range in which the position and orientation of the image pickup apparatus 170 can be measured while maintaining the accuracy.

Next, the process according to the present embodiment will be described with reference to the flowchart of FIG. In step S1010, the holding unit 110 reads the pre-map from an external storage device (not shown). The method of creating the pre-map is not particularly limited. For example, a pre-map can be created by using a plurality of captured images (keyframes) and the position / orientation of the viewpoint of the captured image (position / orientation of the keyframe). The image coordinates (u, v) and depth d of the feature points can be determined by extracting the features from the captured image and matching the captured images. An example of a method for creating a pre-map is SLAM (Simultaneous Localization and Mapping) technology. In this embodiment, the premap is J. Engel, T. Schoeps, and. D. Cremers, “LSD-SLAM: Large-scale direct monocular SLAM,” Proc. 14th European Conference on Computer Vision (ECCV), pp. It can be created using the method described in 834-849, 2014.

The holding unit 110 sets the reliability _B i of each key frame included in advance map (i = 1, ···, Np ) to to HIGH. Here, Np is the number of keyframes included in the pre-map.

In the present embodiment, the reliability of the key frame is an index indicating whether or not the position and orientation of the key frame are reliable. When the reliability of the key frame is high, the correction unit 160 does not correct the position and orientation of the key frame. The position and orientation of the keyframe with high reliability is used as a constraint condition for correcting the position and orientation of the keyframe with low reliability. Pre-maps can be prepared in advance over time. For example, a pre-map can be created in a static state where moving objects such as people or automobiles do not enter the field of view. It is also possible to create a pre-map by performing an optimization process that takes too long to be performed in real time. In this way, it is possible to create a highly accurate pre-map. Therefore, in the present embodiment, the reliability of the keyframe included in the pre-map is set high (high).

In step S1020, the input unit 120 acquires the input image captured by the image pickup apparatus 170. The input unit 120 can acquire an input image of one frame.

In step S1030, the derivation unit 130 derives the position and orientation of the image pickup device 170 when the input image is captured by using the three-dimensional map held by the holding unit 110 and the input image. As the derivation method, various known methods can be used. For example, the difference between the image position of the feature point on the input image calculated based on the three-dimensional position of the feature point and the derived position / orientation and the image position of the feature point on the input image is reduced. The repeated position and orientation can be corrected. In this way, it is possible to derive the position and orientation of the image pickup apparatus 170. In addition, the keyframe and the imaging are performed so that the color difference (for example, the luminance difference) between the feature point of the keyframe and the corresponding point on the input image corresponding to the feature point that can be determined according to the three-dimensional position of the feature point becomes small. It is also possible to derive the relative position / orientation with the device 170. From this relative position / orientation and the position / orientation of the key frame, the position / orientation with the image pickup apparatus 170 can be obtained. Here, the three-dimensional position of the feature point can be determined from the position and orientation of the key frame and the information of the feature point (image coordinates (u, v) and depth d). In this embodiment, the method for deriving the position and orientation disclosed by Angel et al. As described above is used.

In step S1040, the creation unit 140 creates (extends) a three-dimensional map using the position and orientation of the image pickup apparatus 170 derived in step S1030. The creation unit 140 can extend the three-dimensional map by adding keyframes to the three-dimensional map. In addition, the creation unit 140 can expand the three-dimensional map depending on the condition that a predetermined condition is satisfied. Details of this step will be described later. _{Let N k} be the number of keyframes in the 3D map after the keyframes have been added in this step.

In step S1050, the setting unit 150 sets the reliability B _j (j = N _k ) of the keyframe added to the three-dimensional map in step S1040 to be low (LOW). Here, the reliability of the key frame refers to the reliability of the information (the position and orientation of the key frame or the information of the feature point in the key frame) stored in the three-dimensional map in relation to the key frame. At runtime, the reliability of keyframes is limited because it is difficult to control the situation in the scene to keep it static (for example, to keep moving objects out of the field of view). Therefore, in the present embodiment, the reliability of the keyframe added at runtime is set low.

In step S1060, the correction unit 160 corrects the three-dimensional map. For example, the correction unit 160 can correct the position and orientation of the keyframes included in the three-dimensional map so that the entire three-dimensional map is aligned. Various known methods can be used as the correction method, but in the present embodiment, the correction is performed according to the method of Engel et al. described above.

On the other hand, in the present embodiment, the second reliability (for example, HIGH) lower than the first reliability (for example, HIGH) is lower than the first reliability without updating the information indicating the three-dimensional position of the first feature. For example, the information indicating the three-dimensional position of the second feature having LOW) is updated. For example, the reliability _{B i (i = 1, ···} , N k) position and orientation of the key frame is set to LOW is subject to correction, keyframes reliability _{B i} is set to HIGH Position and orientation are not subject to correction. In this way, the correction unit 160 updates the information indicating the three-dimensional position of the feature while fixing the information indicating the three-dimensional position of the feature that has been created in advance and has not been created by the creation unit 140. The integrity of the map can be improved. Therefore, the measurable range of the position and orientation can be expanded without changing the pre-map created with high accuracy. Details of this step will be described later.

In step S1070, the out-licensing unit 130 determines the end of the position / posture measurement. For example, when an end instruction is input by the user from an input device such as a mouse or a keyboard via the input I / F40, the position / posture measurement ends. If the measurement is not completed, the process returns to step S1020, and the position / orientation measurement is continued.

(Details of step S1040)
In step S1040, the creation unit 140 first determines whether or not to extend the three-dimensional map, that is, whether or not to add a new keyframe to the three-dimensional map. Here, the creating unit 140 can make the determination as follows based on the position and orientation of the image pickup apparatus 170 derived in step S1030.

First, the creating unit 140 selects a key frame (hereinafter, a nearest neighbor key frame or simply a neighborhood key frame) in the three-dimensional map based on the position and orientation of the derived imaging device 170. For example, the creating unit 140 is based on the position and line-of-sight direction of the derived image pickup device 170 (Z-axis direction of the image pickup coordinate system in the world coordinate system) and the position and line-of-sight direction of the key frame, and the neighborhood key frame is determined according to a predetermined condition. Can be selected. The creation unit 140 can select a key frame having a position close to the image pickup apparatus 170 and a line-of-sight direction as a nearby key frame. As an example, the creation unit 140 can select a keyframe group from the three-dimensional map based on the visual axis direction of the image pickup apparatus 170. Here, the angle difference between the visual axis direction of the image pickup apparatus 170 and the visual axis direction of the selected keyframe group in the world coordinate system is within the threshold value Tv. Next, the creation unit 140 selects a neighboring keyframe from the keyframe group. Here, the position of the neighboring keyframe is closest to the position of the image pickup apparatus 170 among the keyframes included in the keyframe group.

Next, it is determined whether or not to add a new keyframe based on the number of feature points of the neighboring keyframes included in the input image acquired in step S1020. For example, the creation unit 140 calculates the image coordinates of each feature point of the neighboring keyframe on the input image acquired in step S1020. For example, in order to calculate the image coordinates, the creating unit 140 first calculates the three-dimensional coordinates X _Key of the feature points in the imaging coordinate system of the neighboring keyframes by the method described above. Next, the creating unit 140 uses the position and orientation of the neighboring keyframes and the position and orientation of the image pickup device 170 to set the three-dimensional coordinate X _Key to the three-dimensional coordinate X in the image pickup coordinate system of the image pickup device 170 at the time of input image acquisition. Convert to _{Cam. Finally, the creating unit 140 converts the three-dimensional coordinates X Cam} into the image coordinates (u, v) of the input image using the internal parameters of the image pickup apparatus 170. In this way, the creating unit 140 can obtain _{the ratio R inc of the} feature points whose calculated image coordinates are included in the input image. When the R _inc is small, there is little overlap between the nearest neighbor key frame and the input image, so that the image pickup apparatus may deviate from the position / orientation measurement range. Therefore, when R _inc is less than the threshold value T _inc , the creation unit 140 determines that a new key frame is to be added.

When it is determined that a new keyframe is to be added, the creation unit 140 adds the input image as a new keyframe by using the method of Angel et al. described above. When the method of Angel et al. Is used, the information of the feature points on the immediately preceding keyframe (or on the neighboring keyframes) is projected onto the input image and propagated to create the information of the feature points of the new keyframe. can do. For example, by obtaining the three-dimensional coordinates of the feature points from the information of the feature points of the neighboring keyframes and projecting them on the input image, the image coordinates (u, v) and the depth d of the feature points of the new keyframes are obtained. Can be determined.

On the other hand, if it is not determined that a new keyframe is to be added, the creation unit 140 may update the feature point information (image coordinates (u, v) and depth d) of the keyframe created immediately before. it can. For example, the creation unit 140 can add new feature point information or update depth d information by extracting features from captured images and matching between captured images. This process can be performed, for example, by using the method of Angel et al. described above.

(Details of step S1060)
FIG. 5 is a flowchart showing a processing procedure of the three-dimensional map correction in step S1060. In the optimization process, in general, the error between the observation result observed at the viewpoint (position and orientation of the key frame) when the feature is in the three-dimensional position and the observation result actually observed at the viewpoint is reduced. , The information indicating the three-dimensional position of the feature is updated. In the present embodiment, the information indicating the three-dimensional position of the feature is updated so that the error between the color information of the feature and the color information observed for the three-dimensional position at the viewpoint becomes small. Specifically, the color information and the three-dimensional position information of the feature points are acquired from the existing keyframes. In addition, the color information of the observed feature points at the corresponding points on the new key frame, which can be determined according to the three-dimensional position of the feature points, is also acquired. Then, the relative position / orientation between the existing key frame and the new key frame is calculated so that the color difference (for example, the luminance difference) of the feature points between the existing key frame and the new key frame becomes close. After performing such processing in steps S121-10 to S1220, in step S1230, the position / orientation information of the new keyframe (and in some cases the existing keyframe) is updated to match this relative position / orientation. To. Hereinafter, the process of step S1060 will be specifically described.

In step S1210, the correction unit 160 searches for an existing keyframe whose position and orientation are close to those newly added keyframes (hereinafter, new keyframes) in step S1040. For example, the correction unit 160 has an angle difference of the visual axis direction (Z axis of the imaging coordinate system) in the world coordinate system _{within the threshold value T Angle} and a position difference within the _{threshold value T Dist with respect to the new key frame.} You can select all the keyframes. Next, the correction unit 160 derives the relative position / orientation between the keyframes. Specifically, the correction unit 160 calculates the relative position / orientation between the selected key frame and the new key frame in the same manner as in step S1030. The obtained relative position / orientation can be stored in the holding unit 110 as keyframe correspondence information.

In step S1220, the correction unit 160 searches for an existing keyframe having a high degree of image similarity with respect to the new keyframe. For example, the correction unit 160 selects an existing keyframe having the minimum SSD (Sum of Squared Distance) between the new keyframe and the brightness value and having a threshold value T _{SSD or less among the existing keyframes.} Then, the correction unit 160 calculates the relative position / orientation between the selected key frame and the new key frame, as in step S1210. The obtained relative position / orientation can be stored in the holding unit 110 as keyframe correspondence information.

In step S1230, the correction unit 160 corrects the position and orientation of the keyframe by using the keyframe correspondence information obtained in steps S1210 and S1220. In this way, the correction unit 160 corrects the three-dimensional map. In this step, only the position / orientation of the key frame whose reliability is set to LOW is corrected, and the position / orientation of the key frame whose reliability is set to HIGH is not corrected.

To correct the position / orientation, the relative position / orientation between the keyframes calculated from the position / orientation in the world coordinate system that the keyframe has as an attribute and the relative position / orientation between the keyframes calculated in steps S1210 and S1220. Position and orientation, and the error of can be used. In this case, the sum of these relative position and orientation errors can be used as the evaluation function. In this way, the position / orientation of the keyframe included in the three-dimensional map is updated so as to reflect the relative position / orientation calculated in steps S1210 and S1220 based on the information of the feature points of the keyframe.

Here, the 4 × 4 matrices representing the positions and orientations of the keyframes i and j in the world coordinate system are referred to as M (i) and M (j), respectively. Further, a 4 × 4 matrix representing the relative position and orientation between the key frame i and the key frame j is defined as _Meter (i, j). The matrices M (i) and M (j) are matrices that convert a three-dimensional position in the world coordinate system into a three-dimensional position in the imaging coordinate system. The matrix M _rel (i, j) is a matrix that converts the three-dimensional position of the keyframe j in the imaging coordinate system into the three-dimensional position of the keyframe i in the imaging coordinate system. In the matrices M (i), M (j), and M _rel (i, j), as shown in the following equation, the upper left 3 × 3 matrix is the rotation matrix R, and the rightmost column is the translation component t. It is a matrix to represent.

The correction unit 160 corrects M (i) and M (j) so that the following equation (1) is minimized, assuming that _{the matrix M rel (i, j) is known.} In equation (1), || matrix || _F represents the Frobenius norm (sum of squares of each element of the matrix). Further, the equation (1) is the sum of all the keyframes included in the three-dimensional map for the keyframes whose reliability is set to LOW. The information of the keyframe whose reliability is set to HIGH is used to calculate the relative position and orientation between the keyframes, which is necessary to correct the position and orientation of the keyframe whose reliability is set to LOW. Will be done.

Since M (i) and M (j) are non-linear transformations, the Gauss-Newton method for performing iterative calculations can be used to minimize the equation (1), for example. As the initial values of M (i) and M (j) for the new key frame, the position / orientation derived in step S1030 can be used. The corrected M (i) and M (j) are saved again as keyframe attributes in the 3D map, and are used as initial values when the next 3D map is corrected.

FIG. 9 is a conceptual diagram relating to the correction of the three-dimensional map. FIG. 9A illustrates the prior art. As shown on the left side of FIG. 9A, it is assumed that a pre-map has been created. When the image pickup device is out of the range in which the position and orientation can be measured using the pre-map, map data is added as shown in the center of FIG. 9A. No processing for consistency is performed here. Next, as shown on the right side of FIG. 9A, the position adjustment of the feature is performed based on the entire pre-map and the expanded map (extended area). Since the positions of the features in the pre-map are also adjusted, the accuracy of the pre-map is reduced. In the present embodiment, as shown in FIG. 9B, since the reliability of the position of the feature is high in the map created in advance, the degree of fixation of the feature is high. On the other hand, in the map added at runtime, the reliability of the feature position is low, so the degree of feature fixation is low. That is, consistency can be obtained while maintaining accuracy.

As described above, in the first embodiment, the three-dimensional map is corrected by increasing the reliability of the portion of the three-dimensional map created in advance. Therefore, the position and orientation measurable range of the image pickup apparatus can be expanded while maintaining the accuracy of the portion of the three-dimensional map created in advance.

(Modified Example of Embodiment 1)
In Embodiment 1, the 3D map was a set of keyframes. The information indicating the three-dimensional position of the feature includes the position / orientation information (keyframe position / orientation) of the viewpoint in the reference coordinate system set in the space where the subject is located and the feature in the coordinate system based on this viewpoint. Position information (image coordinates (u, v) and depth d) was used. Then, for each keyframe, the image coordinates (u, v) of the feature points and the depth d with respect to the keyframes are converted into three-dimensional coordinates in the imaging coordinate system, and the three-dimensional coordinates of the feature points are the positions and orientations. It was used for measurement. Further, the correction unit 160 optimizes the three-dimensional map by updating the position / orientation information (position / orientation of the key frame) of the viewpoint in the reference coordinate system. However, the format of the three-dimensional map is not limited to this. For example, an arbitrary type of three-dimensional map capable of measuring the position and orientation of the imaging device using the features on the image can be used.

As an example, in the three-dimensional map, as information indicating the three-dimensional position of the feature, information indicating the three-dimensional position of the feature (for example, the three-dimensional coordinate of the feature point) in the reference coordinate system set in the space where the subject is located is displayed. It may be included. Even if such a three-dimensional map is used, the position and orientation of the image pickup apparatus 170 can be derived using the input image. For example, the three-dimensional map may record the three-dimensional coordinates of the feature points, and the image and the position / orientation for each key frame. In this case, in step S1030, the derivation unit 130 inputs so that the error between, for example, the position of the feature detected from the input image and the back projection position of the feature point registered in the three-dimensional map on the input image becomes small. The position and orientation of the image can be determined. Further, in step S1040, the creation unit 140 detects the corresponding feature from the input image registered as the key frame and another captured image (for example, another key frame), and determines the three-dimensional coordinates of the feature point representing this feature. Then, it can be registered in the three-dimensional map. In addition, the creation unit 140 can also register the position and orientation of the input image registered as a key frame in the three-dimensional map.

In this case, in step S1060, the correction unit 160 can update the information indicating the three-dimensional position of the feature (for example, the three-dimensional coordinate of the feature point) in the reference coordinate system. For example, the correction unit 160 reduces the error between the direction from the viewpoint (position and orientation of the key frame) to the three-dimensional position of the feature and the direction toward the feature observed at the viewpoint (image coordinates). Update the information (keyframe position / orientation) indicating the three-dimensional position of the feature. As a specific example, the correction unit 160 corrects the three-dimensional coordinates of the feature points so that the error between the position of the feature detected from the input image and the back projection position of the feature point on each key frame becomes small. Can be (commonly known as bundle adjustment). In this case, in step S1050, the setting unit 150 can set the reliability LOW with respect to the three-dimensional coordinates of the feature points newly registered in step S1040. On the other hand, the reliability HIGH can be set for the feature points registered in the pre-map. The correction unit 160 corrects only the three-dimensional coordinates of the feature points whose reliability is set to LOW, and does not correct the three-dimensional coordinates of the feature points whose reliability is set to HIGH.

Further, the correction unit 160 may correct the position and orientation of the key frame in addition to the three-dimensional coordinates of the feature point or instead of the three-dimensional coordinates of the feature point. In this case, in step S1050, the setting unit 150 can set the reliability LOW for the key frame newly registered in step S1040. On the other hand, the reliability HIGH can be set for the keyframes registered in the pre-map. The correction unit 160 corrects only the position / orientation of the key frame whose reliability is set to LOW, and does not correct the position / orientation of the key frame whose reliability is set to HIGH.

As a method of registering the 3D coordinates of the feature points in the 3D map as described above, for example, Klein et al.'S PTAM method (G. Klein and D. Murray, "Parallel Tracking and Mapping for Small AR Workspaces," Proc . 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, pp.225-234, 2007.). In this example, feature points are registered in the 3D map, and the 3D map holds the 3D coordinates in the world coordinate system for each feature point. Klein et al. Align the 3D map as a whole by correcting the 3D coordinates of the feature points in the world coordinate system by bundle adjustment. When the process according to the present embodiment is performed by applying the method of Klein et al., In this bundle adjustment, the three-dimensional coordinates of the feature points included in the pre-map are fixed, and the three-dimensional coordinates of the feature points added at runtime are fixed. Only corrected. According to such a method, the measurement range of the position and orientation of the image pickup apparatus can be expanded while maintaining the accuracy of the pre-map.

(Variation of Embodiment 1)
In Embodiment 1, the pre-map was created by SLAM technology. However, the method of creating the pre-map is not limited to SLAM technology. Any method can be used to create a pre-map that can be used to derive the position orientation at runtime. For example, Structure from motion technology (eg, I. Skrypnyk and DG Lowe, “Scene modeling, recognition and tracking with invariant image features,” Proc. 3rd IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR), pp.110-119, 2004.) may be used to create a pre-map.

In the first embodiment, the three-dimensional map was created and the position and orientation of the image pickup apparatus were derived using the Angel method. However, any method can be used as a method for creating a three-dimensional map and a method for deriving the position and orientation of the imaging device. For example, as a method of using the feature points held by the three-dimensional map for deriving the position and orientation of the image pickup apparatus, a method of using the feature points explicitly detected from the image as in the method of Skrypnyk et al. described above is used. May be good. Moreover, the type of features used is not particularly limited. For example, as a feature, a geometric feature such as an edge, a straight line, or a curve detected from an image may be used. Further, as a feature point, a point having a large gradient of the luminance value on the shade image of the key frame may be used. The three-dimensional map may include the position information (image coordinates (u, v) and depth d) of the features with respect to the points constituting these feature points or geometric features. Furthermore, the three-dimensional map may include feature position information (image coordinates (u, v) and depth d) for all pixels of the keyframe. In this way, any pixel can be treated as a feature point.

[Embodiment 2]
In the first embodiment, by increasing the reliability of the pre-created portion of the three-dimensional map of the scene, the measurable range of the position and orientation is expanded while maintaining the accuracy of the three-dimensional map. In the second embodiment, an auxiliary index (hereinafter referred to as an auxiliary index) for measuring the position and orientation of the image pickup apparatus 170 based on the captured image is arranged in the space (scene) where the subject is located. This auxiliary index is different from the features that originally exist in the scene. Then, the information indicating the three-dimensional position of the feature is given a reliability based on the information of the auxiliary index. For example, the information indicating the three-dimensional position of the feature is given a reliability according to the distance from the auxiliary index of the feature. As a specific example, a feature close to the auxiliary index can be given higher reliability than a feature away from the auxiliary index. In this way, the reliability of the peripheral portion of the auxiliary index in the three-dimensional map can be increased. By such a method, the measurable range of the position and orientation can be expanded while maintaining the accuracy of the three-dimensional map.

The configuration and processing of the information processing apparatus according to the present embodiment will be described with reference to FIGS. 1 and 2. The configuration and processing of the information processing apparatus according to the present embodiment are similar to those of the first embodiment, and different parts will be described below.

In this embodiment, the three-dimensional map of the scene created in advance is not used. Instead, as shown in FIG. 7, auxiliary indicators (markers) are placed in the scene. In the present embodiment, an auxiliary index having a predetermined shape in which an identifier that can be read by image analysis is written is used. In the example of FIG. 7, a black-and-white square index having an individual identifier is arranged as an auxiliary index inside. The placement information of the auxiliary index, that is, the position and orientation of the auxiliary index in the world coordinate system can be calibrated in advance. Calibration methods include, for example, the methods disclosed in Japanese Patent No. 4532982 or US Pat. No. 7529387. The setting unit 150 holds pre-calibrated placement information about auxiliary indexes placed in the scene.

The lead-out unit 130 derives the position and orientation of the image pickup apparatus 170 as in the first embodiment. However, in the present embodiment, when the information processing device 1 is activated, the three-dimensional map does not include keyframes, so the derivation unit 130 derives the position and orientation of the image pickup device 170 using the auxiliary index. For example, the derivation unit 130 can derive the position and orientation of the imaging device according to the position of the auxiliary index detected from the input image and the arrangement information of the auxiliary index. In this embodiment, as an example, the method of Kato et al. (H. Kato, M. Billinghurst, I. Poupyrev, K. Imamoto, and K. Tachibana, “Virtual object manipulation on a table-top AR environment,” Proc. IEEE and ACM International Symposium on Augmented Reality 2000, pp. 111-119, 2000.) Is used.

Similar to the first embodiment, the setting unit 150 sets the reliability of the information indicating the three-dimensional position of the feature. In the present embodiment, the setting unit 150 trusts the information indicating the three-dimensional position of the feature created by the creating unit 140 based on the input image based on the number of auxiliary indexes in the captured image (in the input image). Set the degree. For example, as the number of auxiliary indexes increases, the accuracy of determining the position and orientation of the image pickup apparatus 170 increases, so that the reliability of the information indicating the three-dimensional position of the feature (which can be expressed by the information of the position and orientation of the key frame and the feature point) The degree also increases. Therefore, when the number of auxiliary indexes in the input image is equal to or greater than the threshold value, higher reliability can be set for the information indicating the three-dimensional position of the feature as compared with the case where the number is small. In the case of the present embodiment, when the setting unit 150 holds the arrangement information about the auxiliary index detected from the key frame (that is, when the auxiliary index is detected), the reliability of this key frame is set high. .. On the other hand, when the setting unit 150 cannot detect the auxiliary index holding the arrangement information from the key frame (for example, when the auxiliary index is not detected), the setting unit 150 lowers the reliability of this key frame.

Similar to the first embodiment, the correction unit 160 corrects the position and orientation of the key frame based on the reliability of the key frame set by the setting unit 150 so that the entire three-dimensional map can be consistent. The position and orientation of the key frame with high reliability is calculated based on the auxiliary index by using the method of Kato et al. described above, and is fixed at the time of correction by the correction unit 160.

In the present embodiment, the processes of steps S1020 to S1040 and S1070 are the same as those in the first embodiment. Hereinafter, the processes of steps S1010 and S1050 to S1060 will be described.

In step S1010, the setting unit 150 acquires the arrangement information of the auxiliary index calibrated in advance from the external storage device (not shown).

In step S1050, the setting unit 150 sets the reliability of the keyframe added to the three-dimensional map in step S1040. First, the setting unit 150 detects the auxiliary index from the added key frame. The detection of the auxiliary index can be performed, for example, as follows. First, the setting unit 150 binarizes the key frame and detects a quadrangular region from the binarized image. Next, the setting unit 150 calculates the projective transformation from the quadrilateral region to the square region based on the image coordinates of each vertex of the quadrilateral region, and converts the quadrilateral region into the square region. Further, the setting unit 150 identifies the identifier of the auxiliary index by reading out the brightness value of the image at a predetermined position in the square area. When the identified identifier is the identifier of the auxiliary index in which the arrangement information is held, the setting unit 150 determines that the auxiliary index has been identified on the key frame.

If the auxiliary index is not identified on the keyframe, the setting unit 150 sets the reliability of this keyframe to LOW. Further, when the auxiliary index is identified on the key frame, the setting unit 150 can set the reliability of the key frame to HIGH. On the other hand, in the present embodiment, the setting unit 150 can determine the identification status of the auxiliary index and set the reliability of the key frame according to the index indicating the determination result. For example, the setting unit 150 sets the reliability of the key frame to HIGH when the auxiliary index is accurately identified, for example, when the dispersion degree of the arrangement of the auxiliary index on the key frame is equal to or more than the threshold value, and is less than the threshold value. If, the keyframe reliability can be set to LOW.

As an example of such a method, when it is determined that the auxiliary index has been identified, the setting unit 150 can further calculate the distribution of the vertices of the auxiliary index on the image. For example, the setting unit 150 can calculate _{the minimum eigenvalue λ min} of the covariance matrix of the image coordinates of each vertex of all the identified auxiliary indexes as the evaluation value of the distribution of the vertices of the auxiliary indexes. λ _min is large when the vertices of the auxiliary index are widely distributed on the image and small when the vertices of the auxiliary index are narrowly distributed. When λ _min is small, the position and orientation of the key frame calculated from the identified auxiliary index may be less accurate because it is easily affected by the detection error of the auxiliary index. Therefore, when λ _min is smaller than the threshold value T _λ , the setting unit 150 sets the keyframe reliability B to LOW. On the other hand, when λ _min is large, the position and orientation of the key frame calculated from the identified auxiliary index is not easily affected by the detection error of the auxiliary index, so that the accuracy is considered to be high. Therefore, when λ _min is equal to or greater than the threshold value T _λ , the setting unit 150 sets the keyframe reliability B to HIGH.

In step S1060, the correction unit 160 corrects the three-dimensional map by correcting the position and orientation of the keyframes so that the entire three-dimensional map is aligned, as in the first embodiment. That is, the correction unit 160 corrects only the position / orientation of the key frame whose reliability is set to LOW, and does not correct the position / orientation of the key frame whose reliability is set to HIGH. As described above, the correction unit 160 provides information indicating the three-dimensional position of the feature generated based on the auxiliary index arranged in the space where the subject is located for measuring the position and orientation of the image pickup device based on the captured image. To fix. By updating the information indicating the three-dimensional position of the feature while performing such fixing, the consistency of the three-dimensional map is improved.

As described above, in the second embodiment, by increasing the reliability of the key frame in which the auxiliary index is detected, the measurable range of the position and orientation can be expanded while maintaining the accuracy. That is, since the position / orientation of the key frame determined by using the auxiliary index is considered to have high accuracy, the accuracy can be maintained by not correcting this position / orientation in the correction of the three-dimensional map.

(Modified Example of Embodiment 2)
Similar to the modification of the first embodiment, the feature points may be registered in the three-dimensional map, and the three-dimensional coordinates of each feature point in the world coordinate system may be held in the three-dimensional map. In this case, the setting unit 150 may set the reliability of the three-dimensional coordinates of the feature points close to the auxiliary index to HIGH and the reliability of the other three-dimensional coordinates to LOW. For example, the three-dimensional coordinates of the feature points close to the auxiliary index are calculated based on the position and orientation of the image pickup device estimated based on the auxiliary index, and when the bundle adjustment is performed, the feature points close to the auxiliary index are calculated. The three-dimensional coordinates of the above may be fixed and only the three-dimensional coordinates of other feature points may be corrected. Further, as described above, the reliability of the keyframe in which the auxiliary index is detected may be set to HIGH, and the reliability of other keyframes may be set to LOW. That is, when the bundle adjustment is performed, the position / orientation of the keyframe in which the auxiliary index is detected may be fixed and only the position / orientation of the other keyframes may be corrected. Also by such a method, it is possible to expand the measurement range of the position and orientation of the image pickup apparatus while utilizing the highly accurate information obtained by using the auxiliary index.

(Variation of Embodiment 2)
The shape of the auxiliary index is not limited to a square (quadrangle). For example, triangular, hexagonal, circular, or elliptical auxiliary indicators may be used. A shape that can be stably detected from the image can be appropriately selected. Further, it is not essential to specify the identifier of the auxiliary index from the image. For example, when a circular auxiliary index is used, the position of the auxiliary index on the image is calculated based on the position and orientation of the imaging device to be derived by the out-licensing unit 130, and the auxiliary index is associated with the auxiliary index detected in the vicinity of this position. Can identify auxiliary indicators. Further, the auxiliary index with an identifier and the auxiliary index without an identifier may be used together. In this case, the position of the auxiliary index without the identifier on the image may be calculated based on the position and orientation of the imaging device calculated from the auxiliary index with the identifier, and may be associated with the auxiliary index detected in the vicinity.

The calibration of the arrangement information of the auxiliary index does not necessarily have to be performed in advance. For example, the arrangement information may be calibrated in parallel with the creation of the three-dimensional map of the scene by using the image captured by the image pickup apparatus 170. In this case, the accuracy of the 3D map is corrected by correcting the 3D map using the auxiliary index after the reliability of the arrangement information of the auxiliary index exceeds the threshold value without correcting the 3D map at first. Can be prevented from decreasing. The reliability of the arrangement information of the auxiliary indexes can be determined based on, for example, the average value or the minimum value of the number of times each auxiliary index is detected in the image group used for calibration. The larger the average value or the minimum value of the number of detections, the higher the reliability of the placement information. As an example, when the auxiliary index is detected from the position on the input image corresponding to the three-dimensional position indicated by the arrangement information of the auxiliary index, it can be determined that the auxiliary index is detected.

The index indicating the identification status of the auxiliary index is not limited to the evaluation value of the distribution of the vertices of the auxiliary index or the minimum eigenvalue of the covariance matrix of the image coordinates of the vertices of the auxiliary index. For example, the number of auxiliary indexes detected on the image may be used. In this case, the larger the number of detected auxiliary indexes, the lower the evaluation value can be. Further, an index based on both the number of detected auxiliary indexes and the distribution of the vertices of the auxiliary indexes may be used.

Further, various methods can be used as a method for determining the reliability. For example, in addition to the method of determining the reliability based on whether the auxiliary index is detected on the image, whether motion blur is occurring, or whether the whole or part of the image is too bright or too dark. Or, the reliability may be determined based on. For example, keyframe reliability can be reduced if motion blur is occurring, or if the image is too bright or too dark. In the present embodiment, the reliability of the three-dimensional map created in advance and the reliability based on the auxiliary index can be used together. For example, the reliability of keyframes included in the pre-map and the reliability of keyframes for which auxiliary indicators are sufficiently detected can be set to HIGH, and the reliability is set to HIGH. Can be excluded from the correction target.

(Variations common to Embodiments 1 and 2)
In embodiments 1 and 2, the reliability of keyframes or feature points was binary of HIGH or LOW. In addition, when correcting the 3D map, the reliability was used to determine whether to fix the position / orientation of the keyframe or the 3D position of the feature point. That is, the position / orientation of the key frame having a reliability of HIGH or the three-dimensional position of the feature point was fixed, and only the position / orientation of the key frame having a reliability of LOW or the three-dimensional position of the feature point was corrected. However, the reliability of information indicating the three-dimensional position of a feature (for example, a keyframe or a feature point) is not limited to a binary value, and can be a multi-valued value or a real value.

In such an example, the correction unit 160 can update the information indicating the three-dimensional position of the feature with an update amount according to the reliability of the information indicating the three-dimensional position of the feature. For example, the correction unit 160 may correct the position / orientation of the key frame or the three-dimensional position of the feature point so that the correction amount becomes smaller according to the higher reliability by using the reliability as a weight. ..

In the first and second embodiments, after correcting the three-dimensional map, the next input image is acquired and the position and orientation of the image pickup apparatus 170 are derived. However, the timing of correction of the three-dimensional map is not particularly limited. For example, as in the Angel method and Klein method described above, parallel processing that corrects the three-dimensional map may be used in parallel with the acquisition of the input image and the derivation of the position and orientation of the image pickup apparatus 170. In one embodiment, the derivation of the position and orientation of the image pickup apparatus 170 (steps S1020 to S1030), the creation of keyframes (steps S104 to S1050), and the correction of the three-dimensional map (step S1060) are independently performed at arbitrary timings. Can be done.

[Embodiment 3]
In the first and second embodiments, when the image pickup device is about to go out of the measurable range of the position and orientation, a new key frame is added to the 3D map, and correction is performed so that the entire 3D map can be matched. In the third embodiment, when the image pickup device is about to deviate from the measurable range of the position and orientation, a temporary map (hereinafter, temporary map) is created based on the image captured by the image pickup device in addition to the prior map, and the image pickup device is created. Position and orientation measurement is continued. In the present embodiment, the position and orientation are measured by using the temporary map by dead reckoning.

The configuration and processing of the information processing apparatus according to the present embodiment will be described with reference to FIGS. 8 and 2. The configuration and processing of the information processing apparatus according to the present embodiment are similar to those of the first embodiment, and different parts will be described below. FIG. 8 shows the functional configuration of the information processing apparatus 3 according to the third embodiment. The information processing device 3 includes a holding unit 110, an input unit 120, a derivation unit 130, and a creating unit 140, as in the first embodiment. Further, the input unit 120 is connected to the image pickup apparatus 170. The information processing device 3 further includes a selection unit 390. The configuration and processing of the information processing apparatus according to the present embodiment are similar to those of the first embodiment, and different parts will be described below.

The holding unit 110 holds the pre-map as in the first embodiment, and also holds the temporary map created at runtime. The selection unit 390 selects a pre-map or a temporary map as a three-dimensional map for deriving the position and orientation of the image pickup device 170 based on the predicted value of the position and orientation of the image pickup device 170. The derivation unit 130 derives the position and orientation of the image pickup apparatus 170 by using the input image and the three-dimensional map selected by the selection unit 390. The creation unit 140 creates a temporary map based on the position / orientation of the image pickup apparatus 170 and the input image.

In this embodiment, the processing of steps S101 to S1020 and S1070 is the same as that of the first embodiment. Further, the processes of steps S1050 to S1060 are not performed. Hereinafter, the processing of steps S103 to S1040 will be described.

In step S1030, the derivation unit 130 derives the position and orientation of the image pickup apparatus 170 by using the input image and the pre-map or the temporary map. The process of step S1030 can be performed as follows. First, the selection unit 390 selects a three-dimensional map to be used for deriving the position / orientation of the image pickup device 170 based on the predicted value of the position / orientation of the image pickup device 170. In the present embodiment, the selection unit 390 selects whether to use the pre-map or the temporary map held by the holding unit 110. The details of this process will be described later.

Next, the derivation unit 130 derives the position and orientation of the image pickup device 170 when the image is taken, based on the three-dimensional map selected by the selection unit 390 and the input image. The treatment method is not particularly limited, and can be carried out in the same manner as in the first embodiment. In the present embodiment, the relative position / orientation between the position / orientation of the image pickup device 170 of the key frame or the front frame and the position / orientation of the image pickup device 170 of the current frame as disclosed by Angel et al. The method of deriving the position and orientation used is used. In addition, when using a temporary map, dead reckoning is performed. That is, the derivation unit 130 calculates the relative position / orientation change of the image pickup device 170 from the previous frame and integrates it with the position / orientation in the world coordinate system of the previous frame, so that the image pickup device 170 in the world coordinate system of the current frame Derivation of the position and orientation of.

In step S1040, the creation unit 140 creates a temporary map including information indicating the three-dimensional position of the feature of the subject based on the position / orientation of the image pickup apparatus 170 derived in step S1030 and the input image. Details of this step will be described later.

(Details of 3D map selection in step S1030)
First, the selection unit 390 selects a neighborhood keyframe in the pre-map based on the predicted value of the position and orientation of the image pickup apparatus 170. As the predicted value of the position / orientation of the image pickup apparatus 170, for example, the position / orientation derived in the previous step S1030 can be used. The selection of the neighborhood keyframe can be performed by using the same method as in step S1040 of the first embodiment. Further, the selection unit 390 determines whether or not the image pickup apparatus 170 is out of the position / orientation measurement range. This determination can be made based on _{the ratio R inc} of the feature points of the neighboring keyframes included in the input image, as in step S1040 of the first embodiment. When the R _inc is less than the threshold value T _inc , the selection unit 390 determines that the image pickup apparatus 170 is out of the measurement range of the position / orientation according to the pre-map.

When the image pickup apparatus 170 determines that the position / orientation measurement range is out of the pre-map, the selection unit 390 selects a temporary map as a three-dimensional map to be used for deriving the position / orientation. When the image pickup apparatus 170 determines that the measurement range of the position / orientation based on the pre-map is not deviated, the selection unit 390 selects the pre-map as a three-dimensional map to be used for deriving the position / orientation.

(Details of step 1040)
In the present embodiment, the creation unit 140 creates a temporary map having information indicating the three-dimensional position of the feature of the subject. The temporary map can be created as follows. First, the creation unit 140 selects an _{image I pair} that is a stereo pair with _{the input image Current} in order to create a temporary map. If the premap is selected in step S1030, the selected neighborhood keyframe is selected as the _Ipair. On the other hand, when the temporary map is selected in step S1030, the past input image is selected as _{Ipair. Here, I pair,} of the past input _image, the difference between the imaging position of the _{I current} is not less threshold _{T p} above, and the imaging time is closest to the input image and _{I current.}

Next, the creation unit 140 searches for the feature points on the _{I current} corresponding to each feature point on the _{I pair.} For this search, _{epipolar lines calculated from the positions and orientations of I current} and I _{pair at} the time of imaging can be taken into consideration. For example creating unit _140, the epipolar line on the _{I current} corresponding to the feature points on the _{I pair,} a difference is smallest position of the luminance values in a window of N × N pixels between _{I pair} and _{I current} Explore. The creation unit 140 determines that the position thus searched is _{the image coordinates on the I current} corresponding to the image coordinates of the feature points on the _{I pair} .

Next, the creating unit 140 calculates the depth of each feature point in the imaging coordinate system of _{Icurlent by triangulation using the obtained pair of image coordinates.} Then, the creating unit 140 holds the image coordinates (u, v) and the depth d of the feature point cloud on the _{ICurent obtained in this way in the temporary map.} Further, the creation unit 140 holds the position / orientation in the world coordinate system at the time of _{Icurent imaging as an attribute in the temporary map.}

The derivation unit 130 can derive a new position and orientation of the image pickup apparatus 170 with respect to the image pickup coordinate system of the _Ilcurent by using the temporary map created in this way. Then, the derivation unit 130 can convert the new position / orientation of the image pickup apparatus 170 with respect to the image pickup coordinate system of the _ICurent into the position / orientation in the world coordinate system by using the position / orientation in the world coordinate system at the time of _{Icurent imaging.} it can.

As described above, in the third embodiment, when the imaging device deviates from the measurable range of the position / orientation, the measurement of the position / orientation is continued by using the temporary map. In this way, the measurable range can be expanded as compared with the case where only the pre-map is used. On the other hand, since the pre-map is not corrected, the accuracy of the pre-map can be maintained.

(Variation of Embodiment 3)
In the third embodiment, a temporary map is created every time the input image is acquired, regardless of whether the image pickup apparatus 170 is within the measurable range or is likely to be out of the measurable range. However, the timing of creating a temporary map is not limited to this. For example, in step S1040, based on the position / orientation derived in the immediately preceding step S1030, the imaging device 170 is likely to deviate from the measurement range of the position / orientation based on the pre-map by using the same method as in step S1030. It can be determined whether or not it is within the range of. Then, when it is determined that the image pickup apparatus 170 is about to deviate from the measurement range of the position / orientation based on the prior map, the creation unit 140 may create a temporary map.

Further, in step S1040, based on the position / orientation derived in the immediately preceding step S1030, it is possible to determine whether or not the image pickup apparatus 170 is out of the measurement range of the position / orientation based on the prior map by using the same method as in step S1030. .. Then, when it is determined that the image pickup apparatus 170 is out of the measurement range of the position / orientation by the prior map, the creation unit 140 can create a temporary map. In this case, the selection unit 390 may select the three-dimensional map to be used for deriving the position and orientation of the captured image in the next frame in step S1040 instead of step S1030.

In the third embodiment, the temporary map was used only for deriving the position and orientation of the image pickup apparatus 170 in one frame. However, the temporary map may be used to derive the position and orientation of the image pickup apparatus 170 in a plurality of frames. For example, the holding unit 110 can hold a plurality of temporary maps as a three-dimensional map. Then, when it is determined that the image pickup apparatus 170 is out of the measurement range of the position / orientation based on the prior map, the selection unit 390 selects a temporary map in the vicinity from a plurality of temporary maps in order to derive the position / orientation. Can be done. Here, the temporary map in the vicinity can be selected, for example, based on the imaging position of the image used to create the temporary map. As an example, it is possible to select a temporary map created by using an image in which the difference in the imaging position from the input image is within the threshold value and the imaging time is the closest. Further, the temporary map may be corrected by using the three-dimensional map correction method described in the first and second embodiments so that the plurality of temporary maps can be matched.

In the third embodiment, the position / orientation of the image pickup device 170 derived in the previous frame was used as the predicted value of the position / orientation of the image pickup device 170 in the current frame. However, the method for deriving the predicted value of the position and orientation is not limited to this. For example, the position and orientation of the image pickup apparatus 170 can be predicted using a motion model. As the motion model, for example, a motion model representing isokinetic motion, isoacceleration motion, or equiangular velocity motion can be used. By estimating the parameters representing the motion model based on the derivation result of the past position and orientation, the position and orientation in the current frame can be predicted using the motion model. Alternatively, the position and orientation of the image pickup device 170 in the current frame may be predicted using the measured values of the sensor fixed to the image pickup device 170. As the sensor, for example, a sensor that measures position, speed, acceleration, posture, or angular velocity can be used.

[Embodiment 4]
In the first and second embodiments, the information indicating the three-dimensional position of the feature given the high reliability is fixed, and the information indicating the three-dimensional position of the feature given the low reliability is updated to maintain the accuracy. However, the measurable range of position and orientation has been expanded. On the other hand, the position / orientation of the key frame or the three-dimensional position of the feature point is corrected so that the correction amount increases according to the high reliability of the information indicating the three-dimensional position of the feature (for example, the key frame or the feature point). You may.

In the fourth embodiment, the information indicating the three-dimensional position of the feature given the high reliability is updated, and the information indicating the three-dimensional position of the feature given the low reliability is excluded from the update target. In the present embodiment, the reliability of the information indicating the three-dimensional position of the feature on the subject whose position changes in the three-dimensional space is set low, and the reliability of the information indicating the three-dimensional position of the feature on the subject whose position does not change is set low. Is set high. For example, the reliability of the key frame indicating the three-dimensional position of the feature on the subject whose position changes in the three-dimensional space, or the feature point on the subject whose position changes in the three-dimensional space can be set low, and other The reliability of keyframes or feature points can be set high. According to such a configuration, it is possible to suppress a decrease in the accuracy of the three-dimensional map due to the existence of a feature whose position changes.

The hardware configuration of the information processing apparatus according to the fourth embodiment may be the same as that of the first embodiment. Further, the processing in the fourth embodiment is similar to that in the first embodiment, and mainly different points will be described below.

In this embodiment, the SLAM technology is used to create a map and measure the position and orientation of the image pickup apparatus at the same time. As the SLAM technique, the method of Engel et al. Described in the first embodiment shall be used. Also, when a new keyframe is added to the map, the reliability of this keyframe is set to HIGH. The key frame whose reliability is set to HIGH is the target for updating the position and orientation when optimizing the three-dimensional map as in the first embodiment.

As described above, in the present embodiment, it is determined whether or not the position of the feature on the subject changes in the three-dimensional space, and the reliability of the information indicating the three-dimensional position of the feature determined to change is set to be low. Will be done. As an example, the following method can be used. In the method of Engel et al., When the position and orientation are measured, the position and orientation of the keyframes added in the past are measured. At this time, it is possible to determine whether or not the feature included in the key frame is on the subject whose position changes. As a specific example, first, the luminance difference between the feature points of the key frame and the corresponding points on the input image corresponding to the feature points is obtained, and then the feature points of the entire key frame having the luminance difference of the threshold value Tout or more are obtained. The ratio of Rout to Rout is calculated. When Rout is equal to or higher than a certain value, it is determined that the position of the subject in the key frame has changed, and the reliability of this key frame is changed to LOW. The key frame whose reliability is set to LOW is excluded from the update target of the optimization of the three-dimensional map in this embodiment. Therefore, it is possible to avoid a decrease in the accuracy of the three-dimensional map due to a change in the position of the subject.

(Variation of Embodiment 4)
The method for determining whether or not the position of the subject in the key frame has changed is not limited to the above-mentioned method. For example, based on the difference between the image coordinates of the feature points of the keyframes calculated based on the position and orientation of the derived imaging device on the input image and the image coordinates of the corresponding feature points detected on the input image. , This determination may be made. In addition, the three-dimensional position of the feature point corresponding to the feature point of the key frame is calculated based on the image taken after the key frame is added, and the calculated three-dimensional position and the three-dimensional position registered in the key frame are used. This determination may be made based on the difference between.

In this embodiment, the reliability of the feature whose position changes is set low. However, the reliability setting does not necessarily have to be based on the change in position. For example, the reliability may be set based on the change in lighting. In this case, the average luminance value can be compared between the key frame whose position or the direction of the optical axis is close to the input image and the input image, and when the difference is large, the reliability of the key frame can be set to LOW. With such a configuration, it is possible to prevent a decrease in map accuracy due to a change in lighting.

[Example of user interface of each embodiment]
FIG. 6 shows an example of a GUI (window) that presents a three-dimensional map created according to the first embodiment to a user. This window shows a three-dimensional map of the scene. In the example of FIG. 6, the feature points of the three-dimensional map when viewed from a predetermined virtual viewpoint are displayed. This viewpoint can be arbitrarily selected according to, for example, a user instruction. In addition, an object representing a key frame and an object representing the current imaging device are drawn in this window. In the present embodiment, the feature points are represented by round dots, and the position and orientation of the key frame and the image pickup device are represented by a cone-shaped object imitating a camera.

In this embodiment, the user can use an input device such as a mouse or a trackball to change the position, orientation, or internal parameters of the virtual viewpoint. In this way, the user can observe and confirm the three-dimensional map and the current position and orientation of the imaging device from various viewpoints.

In this embodiment, the keyframes or feature points are displayed with different appearances depending on the reliability. For example, in this GUI, the keyframes included in the pre-map and the keyframes added at runtime can be represented by objects having different appearances. By changing the drawing method of these keyframes in this way, it becomes easy to confirm whether or not the image pickup device is within the range in which the position and orientation can be measured using the pre-map. In FIG. 6, the keyframes included in the pre-map are represented by solid cones, and the keyframes added at runtime are represented by dashed cones. Similarly, the feature points included in the pre-map and the feature points added at runtime can be represented by objects having different appearances.

[Variation example of each embodiment]
In the first and second embodiments, the reliability of the pre-created portion of the three-dimensional map of the scene or the peripheral portion of the auxiliary index is set to be high, and the other portion is corrected to improve the accuracy of the position / orientation measurement. The measurable range of position and posture was expanded while maintaining it. However, the method of expanding the measurable range of the position / orientation while maintaining the accuracy of the position / orientation measurement is not limited to this. For example, the 3D information of a pre-created part (for example, a keyframe or a geometric feature) of the 3D map of the scene may be fixed, and the other parts may be corrected. Further, the position / orientation of the key frame detected from the image by the auxiliary index may be derived and fixed based on the auxiliary index, and the position / orientation of other key frames may be corrected at the time of the three-dimensional map correction. Similarly, the three-dimensional coordinates of the feature points whose three-dimensional coordinates are calculated based on the auxiliary index may be fixed, and the three-dimensional coordinates of the other feature points may be derived at the time of the three-dimensional map correction.

In each of the above embodiments, the input unit 120 has acquired an image captured by the image pickup apparatus 170. However, the image acquired by the input unit 120 is not limited to this. For example, the input unit 120 may acquire an image of a scene that has been previously imaged and stored in a storage device (not shown). Further, the information processing apparatus according to the embodiment may use the Structure from motion technique to create a three-dimensional map of the scene offline instead of in real time. In this case, the correction unit 160 may read the pre-map, which is a pre-created three-dimensional map, and perform bundle adjustment while fixing the three-dimensional coordinates of the geometric features included in the pre-map. Further, as in the second embodiment, when the auxiliary index is detected on the image, the correction unit 160 derives the position and orientation of the image based on the auxiliary index, and derives the position and orientation based on the auxiliary index. Bundle adjustment may be performed while fixing the position and posture.

In each of the above embodiments, the imaging device 170 has been described as being a monocular camera. However, the type of the image pickup apparatus 170 is not limited to this. As the image pickup device 170, any image pickup device capable of measuring the position and orientation based on the captured image can be used. For example, the image pickup apparatus 170 may be a stereo camera composed of a plurality of cameras, a multi-lens camera, a depth camera, or a combination thereof. Also, the input image is not limited to the shade image. For example, the input image may be a color image, a depth image, a combination of a gray image and a depth image, or a combination of a color image and a depth image (RGBD image). Further, the imaging device 170 may be a three-dimensional measuring device that obtains a three-dimensional image of the subject by using a method such as LIDAR, and the input unit 120 may acquire the measured three-dimensional point cloud information.

For some embodiments, the criteria for determining whether to add a new keyframe have been described. For example, it is determined whether or not the image pickup apparatus 170 is out of the measurable range or is likely to be out of the measurable range based on the ratio of the feature points whose image coordinates are in the input image among the feature points in the neighboring keyframes. Based on this determination, it was determined whether or not to add a keyframe. However, the determination method is not limited to this. For example, when the difference between the position of the nearby key frame and the position of the image pickup device 170 in the current frame is larger than a predetermined threshold value, it may be determined that the image pickup device 170 is out of the measurable range. Further, when the angle difference between the optical axis direction of the neighboring key frame and the optical axis direction of the image pickup device 170 in the current frame is larger than a predetermined threshold value, it is determined that the image pickup device 170 is out of the measurable range. You may.

(Other Examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It is also possible to realize the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

110 Holding unit; 120 Input unit; 130 Derivation unit; 140 Creation unit; 150 Setting unit; 160 Correction unit; 170 Imaging unit

Claims (15)

Obtaining means position to acquire a captured image of an object from a variable of the imaging device,
A three-dimensional map containing information indicating a three-dimensional position in a reference coordinate system set in the space where the subject is located, which is a first feature point obtained from a captured image of the subject in a predetermined area. A holding means for holding a three-dimensional map created from a set of captured images captured in an area, and
A derivation means for deriving the position of the image pickup apparatus by using the captured image acquired by the acquisition means and the three-dimensional map held by the holding means.
And information indicating the obtained based on subject region outside the predetermined region in the captured image captured, the three-dimensional position of the second feature points in the three-dimensional map in which the holding means for holding said first Information indicating the three-dimensional position of the feature point 1 and information indicating the reliability of the feature point with respect to the true three-dimensional position in the reference coordinate system , and information indicating the three-dimensional position of the second feature point. A correction means for correcting the three-dimensional map by using the information indicating the reliability set with high reliability of the information indicating the three-dimensional position of the first feature point.
An information processing device characterized by being equipped with. The first aspect of the present invention is characterized in that the information indicating the three-dimensional positions of the first feature point and the second feature point is further provided with a setting means for setting the reliability as the information indicating the reliability. The information processing device described. 2. The setting means is characterized in that the information indicating the three-dimensional position of the first feature point is set with higher reliability than the information indicating the three-dimensional position of the second feature point. The information processing device described in. The space where the Utsushitai is positioned, the auxiliary indicators for measuring the position of the imaging device based on the captured image is arranged,
The information processing apparatus according to claim 2, wherein the setting means sets reliability based on the information of the auxiliary index in the information indicating the three-dimensional position of the feature point. The setting means is characterized in that the reliability based on the number of the auxiliary indexes in the captured image is set for the information indicating the three-dimensional position of the feature point created based on the captured image. , The information processing apparatus according to claim 4. Said correction means updates said first without updating the information indicative of the three-dimensional position of the feature point, information indicating the three-dimensional position of the first lower and the second feature points more reliable feature point The information processing apparatus according to any one of claims 1 to 5, wherein a three-dimensional map including information indicating the updated three-dimensional position of the second feature point is created. The correction means reduces the correction amount as the reliability of the information indicating the three-dimensional positions of the first feature point and the second feature point becomes higher, and the first feature point and the second feature The information processing apparatus according to any one of claims 1 to 5, wherein the information indicating the three-dimensional position of the point is updated. The correction means according to any one of claims 1 to 7, wherein the correction means updates information indicating three-dimensional positions of the first feature point and the second feature point in the reference coordinate system. The information processing device described. Wherein the information indicating the three-dimensional position of the first feature points and the second feature point, and the position and orientation information of the viewpoint before Kimoto quasi coordinate system, the first feature in the coordinate system based on the viewpoint It is represented by a combination of a point and the position information of the second feature point .
The information processing apparatus according to any one of claims 1 to 7, wherein the correction means updates the position / orientation information of the viewpoint in the reference coordinate system. The three-dimensional map, which includes information indicating the Utsushitai observations that put each of the plurality of viewpoints,
The correction means includes observation results observed at the viewpoint when the first feature point and the second feature point are at the three-dimensional position, and observation results actually observed at the viewpoint. The information according to any one of claims 1 to 9, wherein the information indicating the three-dimensional positions of the first feature point and the second feature point is updated so that the error becomes small. Processing equipment. The correction means is such that the error between the color information of the first feature point and the second feature point and the color information observed for the three-dimensional position at the viewpoint is reduced . The information processing apparatus according to claim 10, wherein the information indicating the feature points and the three-dimensional positions of the second feature points is updated. Wherein the correction means includes a direction to the three-dimensional position of the viewpoint and the direction from the viewpoint, which is observed in the viewpoint to the feature point, like the error is small, the three-dimensional position of the feature point The information processing apparatus according to claim 10, wherein the information shown is updated. An acquisition means for acquiring a captured image of a subject from an imaging device, and
A three-dimensional map containing information indicating a three-dimensional position in a reference coordinate system set in the space where the subject is located, which is a feature point obtained from a captured image of the subject in a predetermined area. A holding means for holding a three-dimensional map created from a set of captured images, and
Using a three-dimensional map stored in the captured image and the holding means which is acquired by the acquisition unit, a deriving means for deriving a position of the imaging device,
Wherein for measuring the position of the image pickup device, to secure the information indicating the three-dimensional position of the feature point that was created on the basis of the auxiliary indicator which is disposed in a space under Utsushitai it is positioned on the basis of an imaging image However, the correction means for correcting the three-dimensional map and
An information processing device characterized by being equipped with. It is an information processing method performed by an information processing device.
A step of acquiring a captured image of an object position from a variable of the imaging device,
A three-dimensional map including information indicating a three-dimensional position in a reference coordinate system set in the space where the subject is located, which is a first feature point obtained from a captured image of the subject in a predetermined area. A step of deriving the position of the imaging device by using a three-dimensional map held by the holding means created from a set of the captured images captured in the region and the captured image acquired by the imaging device. When,
And information indicating the obtained based on subject region outside the predetermined region in the captured image captured, the three-dimensional position of the second feature points in the three-dimensional map in which the holding means for holding said first Information indicating the three-dimensional position of the feature point 1 and information indicating the reliability of the feature point with respect to the true three-dimensional position in the reference coordinate system , and information indicating the three-dimensional position of the second feature point. The step of correcting the three-dimensional map by using the information indicating the reliability set with high reliability of the information indicating the three-dimensional position of the first feature point, and the step of correcting the three-dimensional map.
An information processing method characterized by having. Program for causing a computer to function as each unit of the information processing apparatus according to any one of claims 1 to 1 3.

Images