JP6922348B2 - Information processing equipment, methods, and programs - Google Patents

Description

Disclosure techniques relate to information processing devices, methods, and programs.

An image processing device for accurately detecting the observation position is known. This image processing device generates normal information of the scene at the observation position, and estimates the observation position based on the generated normal information.

There are also known methods for merging multiple maps for computer vision based tracking. This method simultaneously performs self-location estimation and map construction using multiple maps of a scene from multiple mobile devices to generate a Simultaneous Localization and Mapping (SLAM) map. Then, in this method, the SLAM map is shared among a plurality of mobile devices.

Further, there is known a technique for estimating the position and orientation of a terminal based on an image captured by a camera mounted on the terminal and a loop formed from a movement locus of the terminal.

International Publication No. 2016/181678 Special Table 2016-58085

Ra´ul Mur-Artal, J.M.M. Montiel, "ORB-SLAM: A Versatile and Accurate Monocular SLAM System", IEEE TRANSACTIONS ON ROBOTICS, VOL. 31, NO. 5, OCTOBER 2015

However, the movement locus of the terminal equipped with the camera may not form a loop. In this case, there is a high possibility that the estimation error of the position and orientation of the image pickup device provided in the terminal will increase.

In one aspect, the disclosed technique is aimed at reducing estimation errors in the position and orientation of the imaging device.

In one embodiment of the disclosed technique, the information processing apparatus acquires an image captured by an imaging apparatus mounted on a terminal. Then, the information processing device estimates the position and orientation of the imaging device based on the acquired image, and when a predetermined index is detected from the acquired image, the imaging with respect to the index is performed. Estimate the position and orientation of the device. When the index is detected from the image, the information processing device estimates each of the position and the posture of the image pickup device when each of the images is captured based on the loop. The loop is formed from each of the positions of the image pickup device estimated based on each of the images acquired up to the previous time and the position of the image pickup device with respect to the estimated index.

As one aspect, it has an effect that the estimation error of the position and orientation of the image pickup apparatus can be reduced.

It is a schematic block diagram of the information processing apparatus which concerns on 1st Embodiment. It is explanatory drawing for demonstrating the estimation error of the position and posture of a camera. It is a figure which shows an example of a keyframe table. It is a figure which shows an example of a map point table. It is a figure which shows an example of a feature point and a feature quantity corresponding to a feature point. It is a figure which shows an example of an index and a pattern. It is explanatory drawing for demonstrating the optimization of the position and posture of a camera when a keyframe image is imaged. It is a figure for demonstrating the loop formed in this embodiment. It is a block diagram which shows the schematic structure of the computer which functions as the control part of the information processing apparatus which concerns on 1st Embodiment. It is a flowchart which shows an example of the posture estimation process in 1st Embodiment. It is a flowchart which shows an example of the map generation processing in 1st Embodiment. It is a flowchart which shows an example of the optimization process in 1st Embodiment. It is a schematic block diagram of the information processing apparatus which concerns on 2nd Embodiment. It is a block diagram which shows the schematic structure of the computer which functions as the control part of the information processing apparatus which concerns on 2nd Embodiment. It is a flowchart which shows an example of the optimization process in 2nd Embodiment. It is a schematic block diagram of the information processing apparatus which concerns on 3rd Embodiment. It is a figure which shows an example of the display screen displayed on the display device. It is a block diagram which shows the schematic structure of the computer which functions as the control part of the information processing apparatus which concerns on 3rd Embodiment. It is a flowchart which shows an example of the display control processing in 3rd Embodiment. It is a schematic block diagram of the information processing apparatus which concerns on 4th Embodiment. It is a figure for demonstrating the loop formed in 4th Embodiment. It is a block diagram which shows the schematic structure of the computer which functions as the control part of the information processing apparatus which concerns on 4th Embodiment. It is a flowchart which shows an example of the posture estimation process in 4th Embodiment. It is a flowchart which shows an example of the optimization process in 4th Embodiment.

Hereinafter, an example of an embodiment of the disclosed technology will be described in detail with reference to the drawings.

<First Embodiment>
FIG. 1 shows a schematic diagram showing a configuration example of the information processing device 10.

As shown in FIG. 1, the information processing device 10 of the present embodiment includes a camera 12 and a control unit 14. The camera 12 is an example of an imaging device according to the disclosed technology. The camera 12 can be mounted on a moving body such as a vehicle or carried by a person. The position of the camera 12 can be changed by being mounted on another device or carried by a person. Both the camera 12 and the control unit 14 may be included in the information processing device 10, or the information processing device 10 is equipped with the control unit 14, and the camera 12 can communicate with the control unit 14. It may be a device. In the present embodiment, a case where the information processing device 10 is mounted on a vehicle will be described as an example.

The camera 12 sequentially captures images around the vehicle.

The control unit 14 includes a data storage unit 15, an image acquisition unit 16, a feature point extraction unit 18, a posture estimation unit 20, a map generation unit 22, an index detection unit 24, an optimization unit 26, and an adjustment unit. 28 and. The data storage unit 15 is an example of a storage unit of the disclosed technology. Further, the optimization unit 26 is an example of an estimation unit of the disclosed technology.

The data storage unit 15 stores map information representing information on the surrounding environment of the vehicle. The map information is generated based on the image captured by the camera 12 mounted on the vehicle. The map information will be described below.

For example, as shown in 2A of FIG. 2, a case where a camera mounted on a mobile terminal or the like moves around the building R will be described as an example. 2A in FIG. 2 represents the area around the building R as seen from above. In 2A of FIG. 2, the movement locus L1 is generated by the movement of the camera. When the camera moves on the movement locus L1, the camera 12 mounted on the terminal sequentially captures images around the camera. Among the sequentially captured images around the camera, an image satisfying a predetermined condition is stored in the data storage unit 15 as a keyframe image. A keyframe image is an image that satisfies a predetermined condition.

Feature points are extracted from the image. The feature point is, for example, an edge point in an image representing the shape of an object existing in the target area. In addition, map points representing three-dimensional coordinates corresponding to the feature points in the image are generated. For example, a map point M as shown in 2B of FIG. 2 is generated for the feature point F. Further, the feature point F extracted from the image 2C is associated with the map point M. Therefore, the position and orientation of the camera 12 mounted on the terminal are sequentially estimated according to the correspondence between the map point M of the map information stored in the data storage unit 15 and the feature point F extracted from the image 2C. Will be done.

However, when estimating the position and orientation of the camera 12 in the area where the map information is not stored in the data storage unit 15, if a plurality of images captured at the same location cannot be acquired, the movement locus of the camera Loop is not formed. Therefore, for example, the optimization process as shown in Reference 1 below cannot be performed. Therefore, as shown in 2B of FIG. 2, the estimation result of the position and the posture of the camera 12 is a movement locus L2 different from the original movement locus L1. As a result, in the movement locus L2 representing the estimation result of the position and orientation of the camera 12, the estimation error of the position and orientation of the camera 12 increases.

Reference 1: Ra´ul Mur-Artal, J.M.M. Montiel, "ORB-SLAM: A Versatile and Accurate Monocular SLAM System", IEEE TRANSACTIONS ON ROBOTICS, VOL. 31, NO. 5, OCTOBER 2015

Therefore, in the present embodiment, a predetermined index is set in the environment, and the position and posture of the camera 12 are optimized every time the index is detected. Hereinafter, a specific description will be given.

The data storage unit 15 includes each of the keyframe images, each of the positions and orientations of the camera 12 when the keyframe image is captured, and map points which are three-dimensional coordinates of each of the feature points of the keyframe image. Map information representing is stored.

Specifically, the map information is represented by a key frame table and a map point table, and the key frame table and the map point table are stored in the data storage unit 15 as map information.

The keyframe table shown in FIG. 3 includes a keyframe ID representing keyframe identification information, a position and orientation of the camera 12, a keyframe image, feature points of the keyframe image, and map points corresponding to the feature points. It is stored in association with the ID. For example, the position and orientation of the camera 12 corresponding to the keyframe ID “001” in the keyframe table of FIG. 3 indicates (0.24,0.84,0.96,245.0,313.9,23.8) as shown in FIG. It is a dimensional real value. Of the 6-dimensional real values, (0.24,0.84,0.96) represents the attitude of the camera 12, and (245.0,313.9,23.8) represents the 3D position of the camera. One row of information in the keyframe table represents one keyframe.

The keyframe image (24,46, ...) Corresponding to the keyframe ID "001" in the keyframe table of FIG. 3 represents the pixel value of each pixel of the keyframe image. Further, the feature points "(11,42), (29,110) ..." Corresponding to the keyframe ID "001" in the keyframe table of FIG. 3 represent the pixel positions corresponding to the positions of the feature points in the keyframe image. .. Further, the map point ID "3,5,9,32 ..." Corresponding to the keyframe ID "001" in the keyframe table of FIG. 3 represents the map point ID corresponding to each feature point. The map point ID corresponds to the map point ID in the map point table.

In the map point table shown in FIG. 4, the map point ID representing the identification information of the map point, the three-dimensional position coordinates (X [m], Y [m], Z [m]) of the map point, and the map point The feature amount is stored in association with it. For example, the feature amount of the map point table in FIG. 4 is, for example, Oriented FAST and Rotated BRIEF (ORB) described in Reference 2, and the feature amount of ORB is a 32-dimensional feature amount representing 0 or 1. Represented by.

Reference 2: E. Rublee et al., "ORB: An efficient alternative to SIFT or SURF", In Proc. Of International Conference on Computer Vision, pp. 2564-2571, 2011.

In the present embodiment, the control unit 14 of the information processing device 10 has a posture estimation function, a map generation function, and an optimization function. Hereinafter, each functional unit corresponding to each function will be described.

The internal parameters of the camera 12 are acquired in advance by calibration based on, for example, the method described in Reference 3. The internal parameters of the camera 12 include, for example, a matrix including the focal length and the optical center, and a distortion coefficient (for example, five dimensions).

Reference 3: Z. Zhang et al., "A flexible new technique for camera calibration.", IEEE Transactions on Pattern Analysis and Machine Intelligence, 22 (11): 1330-1334, 2000.

[Posture estimation function]

The image acquisition unit 16 sequentially acquires images captured by the camera 12. Next, the image acquisition unit 16 converts the image captured by the camera 12 into a grayscale image. Then, the image acquisition unit 16 outputs a grayscale image.

The feature point extraction unit 18 acquires feature points from the grayscale image output from the image acquisition unit 16. For example, the feature point extraction unit 18 extracts feature points from the grayscale image by using the method described in Reference 1 above. Then, the feature point extraction unit 18 calculates the feature amount for each feature point. For example, when the ORB described in Reference 2 is used as a feature amount, a 32-dimensional feature amount representing 0 or 1 is extracted.

FIG. 5 shows an example of a feature amount data structure corresponding to each feature point. As shown in FIG. 5, the feature point ID representing the identification information of the feature points, the pixels u [pixel] and v [pixel] representing the positions of the feature points, and the feature amount are associated with each other.

The posture estimation unit 20 estimates the position and posture of the camera 12 based on the feature points extracted by the feature point extraction unit 18 and the feature quantities corresponding to the feature points. When the map points are not obtained, the posture estimation unit 20 generates initial map points by using, for example, the method described in "IV Automatic Map Initialization" of Reference 1.

Specifically, first, the posture estimation unit 20 acquires the feature points extracted by the feature point extraction unit 18 from the grayscale image captured from any two viewpoints. Next, the posture estimation unit 20 associates feature points between grayscale images captured from the two viewpoints, and the camera 12 corresponding to the second viewpoint with respect to the position and orientation of the camera 12 corresponding to the first viewpoint. Find the position and posture of. The posture estimation unit 20 sets the position and orientation of the camera 12 corresponding to the first viewpoint as the origin of the world coordinate system, and sets the position and orientation of the camera 12 corresponding to the second viewpoint as the position and orientation of the camera 12. Set as the initial value of the posture.

Further, the posture estimation unit 20 corresponds to the feature point by using triangulation based on the position and orientation of the camera 12 corresponding to the second viewpoint with respect to the position and orientation of the camera 12 corresponding to the first viewpoint. Calculate the map points that represent the dimensional coordinates.

Next, the posture estimation unit 20 sequentially estimates the position and posture of the camera 12 by using the map points when the camera 12 moves according to the movement of the vehicle. For example, first, the posture estimation unit 20 assumes that the vehicle equipped with the camera 12 performs a predetermined motion model (for example, constant velocity motion). Then, the posture estimation unit 20 associates the feature points of the grayscale image extracted by the feature point extraction unit 18 with the map points stored in the map point table of the data storage unit 15.

More specifically, the posture estimation unit 20 projects the map points stored in the map point table of the data storage unit 15 onto the grayscale image, and associates each of the feature points of the grayscale image with each of the map points.

For example, the posture estimation unit 20 estimates the position and orientation of the camera by using the PnP algorithm described in Reference 4 below. The PnP algorithm is a method of calculating the position and orientation of the camera 12 that minimizes the distance between the map points and feature points projected on the grayscale image by a nonlinear optimization algorithm such as the Levenberg-Marquardt method. Is.

Reference 4: V. Lepetit et al., "EPnP: An Accurate O (n) Solution to the PnP Problem", International Journal of Computer Vision, Vol.81, No.2, pp.155-166 (2008).

Further, the posture estimation unit 20 determines whether or not to store the grayscale image output by the image acquisition unit 16 as a keyframe image. For example, the posture estimation unit 20 determines whether or not to store the grayscale image output by the image acquisition unit 16 as a keyframe image according to the following criteria. When all of the following criteria (1) to (3) are satisfied, the posture estimation unit 20 uses a grayscale image as a keyframe image, along with the correspondence between the position and posture of the camera 12 and the feature points and map points. It is stored in the data storage unit 15.

(1) A certain frame (for example, 20 frames) has passed since the previous storage of the key frame image.
(2) Of the keyframe images stored up to the previous time, the number of corresponding points between the feature points of the keyframe image closest to the position of the grayscale image and the feature points of the grayscale image is a certain number or more ( For example, 50 points).
(3) Among the keyframe images stored up to the previous time, the corresponding points are reduced by a certain percentage (for example, 90%) as compared with the keyframe image closest to the position of the grayscale image.

[Map generation function]

When a new keyframe image is stored in the data storage unit 15 by the posture estimation unit 20, the map generation unit 22 calculates each map point of the feature points of the new keyframe image by using triangulation. For example, the map generation unit 22 uses the method described in Reference 5 based on the new keyframe image and the keyframe image stored in the data storage unit 15 up to the previous time to map points of the new keyframe image. To generate.

Reference 5: R. I. Hartley et al., "Triangulation, Computer Vision and Image Understanding", Vol. 68, No.2, pp.146-157, 1997.

Specifically, the map generation unit 22 selects the nearest keyframe image closest to the position of the new keyframe image among the keyframe images stored in the data storage unit 15 up to the previous time. Next, the map generation unit 22 identifies the feature points in the nearest keyframe image corresponding to the feature points included in the new keyframe image by epipolar search. The epipolar search is a process of finding a corresponding point only on the epipolar line of the second viewpoint where the feature point of the first viewpoint should exist by using the geometric constraint between the two viewpoints.

Then, the map generation unit 22 uses triangulation to generate map points of the new keyframe image based on the information of the feature points associated between the new keyframe image and the nearest keyframe image. calculate. For triangulation, for example, the method described in "5.1 Linear Triangulation" in Reference 5 can be used.

Then, the map generation unit 22 stores the map points of the new keyframe image in the data storage unit 15.

Based on the map information stored in the data storage unit 15, the adjustment unit 28 determines that the sum of the reprojection errors between the feature points and the map points on the keyframe image for all the keyframe images is the minimum. Correct the map points so that

Specifically, the adjusting unit 28 acquires each key frame stored in the key frame table of the data storage unit 15, a map point associated with each key frame, and an internal parameter of the camera 12. Then, the adjusting unit 28 determines the position and orientation of the camera 12 when each keyframe image is captured so that the reprojection error between the feature point and the map point on the keyframe image is minimized. Correct the coordinates of the map points associated with the feature points of each keyframe image. As an algorithm for minimizing the reprojection error between the feature point and the map point on the keyframe image, a nonlinear optimization algorithm such as the Levenberg-Marquardt method described in Reference 6 below is used. be able to. The process in the adjusting unit 28 is also referred to as bundle adjustment.

Reference 6: B. Triggs et al., "Bundle Adjustment-A Modern Synthesis", In Proc. Of International Workshop on Vision Algorithms: Theory and Practice, pp.298-392, 1999.

Then, the adjusting unit 28 replaces the position and orientation of the camera 12 when each keyframe image of the map information stored in the data storage unit 15 is captured with the corrected position and orientation. Further, the adjusting unit 28 replaces the coordinates of the map points associated with each keyframe image in the map information stored in the data storage unit 15 with the coordinates of the corrected map points.

[Optimization function]

The index detection unit 24 detects whether or not the new keyframe image stored in the data storage unit 15 includes an index having a predetermined shape and size. The index is set in advance in the target area where the camera moves. In addition, a plurality of indicators are set, and the relative positions and postures between the indicators for each of the plurality of indicators are known. For example, index 1 as shown in FIG. 6 is pre-installed in the environment. As shown in FIG. 6, the index 1 includes a predetermined pattern 2.

The index detection unit 24 uses, for example, the method described in Reference 7 to determine whether or not an index exists in the grayscale image output by the image acquisition unit 16. Specifically, the index detection unit 24 binarizes the grayscale image output by the image acquisition unit 16. Then, the index detection unit 24 detects the index by estimating the positions of the coordinates of the four corners of the index.

Reference 7: H. Kato et al., "Marker tracking and HMD calibration for a video-based augmented reality conferencing system", In Proc. Of IEEE and ACM International Workshop on Augmented Reality (IWAR), pp.85-94, 1999.

When the index is detected from the new keyframe image by the index detection unit 24, the posture estimation unit 20 estimates the position and posture of the camera 12 with respect to the index based on the keyframe image including the index. For example, the posture estimation unit 20 estimates the position and posture of the camera 12 with respect to the index according to “4. Position and pose estimation of markers” in Reference 7.

Then, the optimization unit 26 stores the map information stored in the data storage unit 15 based on the position and orientation of the camera 12 with respect to the index when the new keyframe image estimated by the attitude estimation unit 20 is captured. to correct.

The correction of the map information in the present embodiment will be specifically described.

For example, as shown in 7A of FIG. 7, a case where the position and posture (a, b, c) of the camera when each keyframe image is captured is obtained will be described as an example. In this case, a represents the position and orientation of the camera 12 when the keyframe image of the starting point is captured. Further, c represents the position and orientation of the camera when a new keyframe image is captured. b represents the position and orientation of the camera 12 when the keyframe image located between a and c is captured. Further, X represents the position and posture of the camera 12 with respect to the index 1 obtained by the posture estimation unit 20.

In this embodiment, as shown in 7B of FIG. 7, the position and orientation Y of the camera 12 in each keyframe image is based on the position and orientation a of the camera 12 and the position and orientation X of the camera 12 with respect to the index 1. To get.

Specifically, as shown in 7C of FIG. 7, the position and orientation (a, b, c) of the camera 12, the position and orientation X of the camera 12 with respect to the index 1, and the position and orientation a of the camera 12 A loop is formed that includes the position and orientation of the camera 12 with respect to other indicators. At this time, the corrected position and orientation Y of the camera 12 are obtained by recalculating the graph representing the loop. As a result, the movement locus S before correction becomes the movement locus P, and map information corresponding to the real world can be obtained.

Here, the loop formed from the position and orientation of the camera will be described in more detail.

As shown in FIG. 8, the position and orientation (a, b, c) of the camera 12 when each keyframe image is captured, and the index (1A, 1B) when a new keyframe image is captured. A loop is formed that includes the position and orientation (X1, X2) of the camera 12 with respect to the camera 12. However, the relative positions and postures between the indicators (1A, 1B) are known.

As shown in FIG. 8, when the index 1A is detected from the keyframe image, the position and posture X1 of the camera 12 with respect to the index 1A are estimated. Note that a is the position and orientation of the camera 12 estimated when stored as a keyframe image.

When the index 1B is detected from the new keyframe image, the position and orientation X2 of the camera 12 with respect to the index 1B are estimated. Note that c is the position and orientation of the camera 12 estimated when stored as a keyframe image.

Then, the relative position and orientation between the indexes (1A, 1B), the position and orientation X1 of the camera 12 with respect to the index 1A, the position and orientation b of the camera 12 when each keyframe image is captured, and the index. A loop Z is formed from the position and posture X2 of the camera 12 with respect to 1B. This makes it possible to perform the Pose Graph optimization described in Reference 8 below.

Reference 8: Ra´ul Mur-Artal and Juan D. Tard´os, “Fast Relocalisation and Loop Closing in Keyframe-Based SLAM”, 2014 IEEE International Conference on Robotics & Automation (ICRA) May 31 --June 7, 2014. Hong Kong, China

Specifically, first, the optimization unit 26 acquires the map information stored in the data storage unit 15 when the index is detected from the new keyframe image. Next, the optimization unit 26 determines the position and orientation of the camera 12 with respect to the index when the index was detected from the keyframe image last time, and the position and orientation of the camera 12 when the past keyframe image was captured. Get each and. The optimization unit 26 further acquires the position and orientation of the camera 12 with respect to the index when a new keyframe image is captured and the relative position and orientation between the indexes to form a loop. The map information is corrected based on the loop to be performed.

More specifically, the optimization unit 26 performs each of the positions and orientations of the camera 12 in the keyframe image around the new keyframe image and the new key by the Pose Graph optimization described in Reference 8 below. The position and orientation of the camera 12 in the frame image are corrected.

Then, the optimization unit 26 transforms the coordinates of the map points corresponding to the feature points of each keyframe image according to the correction of the position and posture of the camera 12 when each keyframe image is captured. Specifically, the optimization unit 26 between the position and orientation of the camera 12 before correction when each keyframe image is captured and the map points corresponding to the feature points of each keyframe image before correction. The coordinates of the map points are transformed so that the relative relationship of is maintained.

The adjustment unit 28 determines the position and orientation of the camera 12 in each keyframe image and each key so as to minimize the reprojection error of the map points to the feature points of each keyframe image corrected by the optimization unit 26. Correct the coordinates of the map points associated with the frame image. Then, the adjusting unit 28 corrects the coordinates of the map points associated with each key frame and each key frame image in the map information stored in the data storage unit 15, and each key frame image and each key frame. Replace with the coordinates of the map point associated with the image.

The control unit 14 of the information processing device 10 can be realized by, for example, the computer 50 shown in FIG. The computer 50 includes a CPU 51, a memory 52 as a temporary storage area, and a non-volatile storage unit 53. Further, the computer 50 controls reading / writing of data to the input / output interface (I / F) 54 to which the camera 12, the display device, the input / output device and the like (not shown) are connected, and the recording medium 59. A write (R / W) unit 55 is provided. Further, the computer 50 includes a network I / F 56 connected to a network such as the Internet. The CPU 51, the memory 52, the storage unit 53, the input / output I / F 54, the R / W unit 55, and the network I / F 56 are connected to each other via the bus 57.

The storage unit 53 can be realized by a Hard Disk Drive (HDD), a solid state drive (SSD), a flash memory, or the like. The storage unit 53 as a storage medium stores an information processing program 60 for causing the computer 50 to function as the control unit 14 of the information processing device 10. The information processing program 60 includes an image acquisition process 62, a feature point extraction process 63, an attitude estimation process 64, a map generation process 65, an index detection process 66, an optimization process 67, and an adjustment process 68. Further, the storage unit 53 has a data storage area 69 in which information constituting the data storage unit 15 is stored.

The CPU 51 reads the information processing program 60 from the storage unit 53, expands the information processing program 60 into the memory 52, and sequentially executes the processes included in the information processing program 60. The CPU 51 operates as the image acquisition unit 16 shown in FIG. 1 by executing the image acquisition process 62. Further, the CPU 51 operates as the feature point extraction unit 18 shown in FIG. 1 by executing the feature point extraction process 63. Further, the CPU 51 operates as the posture estimation unit 20 shown in FIG. 1 by executing the posture estimation process 64. Further, the CPU 51 operates as the map generation unit 22 shown in FIG. 1 by executing the map generation process 65. Further, the CPU 51 operates as the index detection unit 24 shown in FIG. 1 by executing the index detection process 66. Further, the CPU 51 operates as the optimization unit 26 shown in FIG. 1 by executing the optimization process 67. Further, the CPU 51 operates as the adjustment unit 28 shown in FIG. 1 by executing the adjustment process 68. Further, the CPU 51 reads information from the data storage area 69 and expands the data storage unit 15 into the memory 52. As a result, the computer 50 that executes the information processing program 60 functions as the control unit 14 of the information processing device 10. Therefore, the processor that executes the information processing program 60, which is software, is hardware.

The function realized by the information processing program 60 can also be realized by, for example, a semiconductor integrated circuit, more specifically, an Application Specific Integrated Circuit (ASIC) or the like.

Next, the operation of the information processing device 10 according to the present embodiment will be described. The information processing device 10 executes a posture estimation process, a map generation process, and an optimization process. When the terminal equipped with the information processing device 10 starts moving and the camera 12 starts capturing an image around the camera, the control unit 14 of the information processing device 10 executes the posture estimation process shown in FIG. Similarly, the control unit 14 of the information processing apparatus 10 executes the map generation process shown in FIG. 11 and the optimization process shown in FIG. Hereinafter, each process will be described in detail.

<Posture estimation process>

In step S100, the image acquisition unit 16 acquires the image captured by the camera 12. Next, the image acquisition unit 16 converts the image captured by the camera 12 into a grayscale image. Then, the image acquisition unit 16 outputs a grayscale image.

In step S102, the feature point extraction unit 18 extracts feature points from the grayscale image output in step S100. Then, the feature point extraction unit 18 calculates the feature amount for each feature point.

In step S104, the posture estimation unit 20 estimates the position and posture of the camera 12 based on the feature points extracted in step S102 and the feature quantities corresponding to the feature points.

In step S106, the posture estimation unit 20 determines whether or not to store the grayscale image output in step S100 as a keyframe image. If it is determined that the grayscale image is stored as the keyframe image, the process proceeds to step S108. On the other hand, if it is determined that the grayscale image is not stored as the keyframe image, the process returns to step S100.

In step S108, the posture estimation unit 20 stores the grayscale image output in step S100 in the data storage unit 15 as a keyframe image. Further, the position and orientation of the camera 12 estimated in step S104 are stored in the data storage unit 15.

<Map generation process>

In step S200, the map generation unit 22 determines whether or not a new keyframe image is stored in the data storage unit 15 by the posture estimation process. When the keyframe image is stored in the data storage unit 15, the process proceeds to step S202. On the other hand, if the keyframe image is not stored in the data storage unit 15, the process returns to step S200.

In step S202, the map generation unit 22 has a new key based on the new keyframe image stored in the data storage unit 15 by the posture estimation process and the keyframe image stored in the data storage unit 15 up to the previous time. Generate map points for the frame image.

In step S204, the map generation unit 22 stores the map points of the new keyframe image generated in step S202 in the data storage unit 15.

In step S206, the adjusting unit 28 determines the reprojection error between the feature points and the map points on the keyframe image for all the keyframe images based on the map information stored in the data storage unit 15. Correct the map points so that the sum is minimized. Then, the adjusting unit 28 replaces the position and orientation of the camera 12 when each keyframe image of the map information stored in the data storage unit 15 is captured with the corrected position and orientation. Further, the adjusting unit 28 replaces the map points associated with each keyframe image in the map information stored in the data storage unit 15 with the corrected map points.

<Optimization process>

In step S300, the index detection unit 24 determines whether or not the index is included in the new keyframe image stored in the data storage unit 15 by the posture estimation process. If the new keyframe image contains an index, the process proceeds to step S302.

In step S302, the posture estimation unit 20 estimates the position and posture of the camera 12 with respect to the index based on a new keyframe image including the index.

In step S304, the optimization unit 26 includes each of the position and orientation of the camera 12 in the past keyframe image stored in the data storage unit 15, and the position and orientation of the camera 12 with respect to the index obtained in step S302. Form a loop from. Then, the optimization unit 26 determines each of the positions and orientations of the camera 12 in the past keyframe image and the position and orientation of the camera 12 in the new keyframe image by Pose Graph optimization based on the formed loop. To correct.

In step S306, the optimization unit 26 coordinates the coordinates of the map points of each keyframe according to the correction of the position and orientation of the camera 12 when each keyframe image is captured, which was obtained in step S304. Convert.
For example, loop closure optimization can be used for loop-based position and orientation correction.

In step S308, the adjusting unit 28 determines the position and orientation of the camera 12 in each keyframe image so as to minimize the reprojection error of the map points obtained in step S306 onto the feature points of each keyframe image. , Correct the map points of each keyframe image. Then, the adjusting unit 28 associates the map points associated with each key frame and each key frame in the map information stored in the data storage unit 15 with each corrected key frame and each key frame. Replace with a map point.

As described above, the information processing apparatus according to the present embodiment estimates the position and orientation of the camera based on the image captured by the camera. Then, when a predetermined index is detected from the captured image, each of the keyframe images is based on a loop formed from each of the camera positions and postures and the camera position and posture with respect to the index. Estimate each of the position and orientation of the camera when the image is taken. This makes it possible to reduce the estimation error of the position and orientation of the camera.

Further, each time the index is detected, the position and orientation of the camera when each of the keyframe images is captured is optimized, so that the optimization can be performed with high frequency.

Further, since the optimization is performed with high frequency, the time until the bundle adjustment performed by the adjustment unit converges can be reduced, and the convergence to the local solution can be avoided.

<Second embodiment>
Next, the second embodiment will be described. In the second embodiment, when the index is detected from the image captured by the camera, the reliability of the image including the index is calculated according to the detection result of the index. Then, when the reliability is larger than the preset threshold value, the position and orientation of the camera when each of the keyframe images is captured is corrected, which is different from the first embodiment.

FIG. 13 shows a configuration example of the information processing device 210 of the second embodiment. As shown in FIG. 13, the information processing device 210 of the second embodiment includes a camera 12 and a control unit 214.

The control unit 214 includes a data storage unit 15, an image acquisition unit 16, a feature point extraction unit 18, a posture estimation unit 20, a map generation unit 22, an index detection unit 24, an optimization unit 226, and an adjustment unit. 28 and a reliability calculation unit 225 are provided.

When the index is detected from the keyframe image stored in the data storage unit 15, the reliability calculation unit 225 calculates the reliability according to the detection result of the index.

For example, the reliability calculation unit 225 calculates the angle θ formed by the plane normal of the index detected from the keyframe image and the optical axis of the camera 12. Then, when the angle θ formed by the reliability calculation unit 225 satisfies θ1 ≦ θ ≦ θ2, the reliability calculation unit 225 increases the reliability regarding the angle. On the other hand, when the angle θ formed does not satisfy θ1 ≦ θ ≦ θ2, the reliability regarding the angle is lowered. θ1 and θ2 are preset, and for example, θ1 = π / 18, θ2 = 4π / 9. If the angle θ between the index and the optical axis of the camera is too large or too small, the error of the detection points at the four corners of the index will have a large effect on the camera position and orientation estimation (for example, References). 9) Since the estimation accuracy deteriorates, the reliability is calculated according to the angle θ formed.

Reference 9: Y. Uematsu et al., "Improvement of Accuracy for 2D Marker-Based Tracking Using Particle Filter", In Proc. Of IEEE International Conference on Artificial Reality and Telexistence (ICAT), pp.183-189, 2007.

Further, the reliability calculation unit 225 calculates the reliability regarding the distance according to the distance d between the camera 12 and the index. The reliability calculation unit 225 calculates the reliability with respect to the distance so that the larger the distance d, the lower the reliability, and the smaller the distance d, the higher the reliability. As the distance between the camera 12 and the index increases, the identification accuracy of the detection points at the four corners deteriorates, and the estimation accuracy deteriorates. Therefore, the reliability is calculated according to the distance.

Further, the reliability calculation unit 225 calculates the degree of coincidence between the pattern included in the index detected from the keyframe image and the pattern registered in advance as the degree of reliability regarding the match. If the degree of pattern matching is low, the possibility of being recognized as a different index increases, so the reliability is calculated according to the degree of pattern matching.

The optimization unit 226 captures each of the positions and orientations of the camera 12 when the past keyframe image was captured and a new keyframe image according to the reliability calculated by the reliability calculation unit 225. The position and orientation of the camera 12 with respect to the index at that time are corrected.

For example, the optimization unit 226 captures a keyframe image when at least one of the reliability regarding the angle, the reliability regarding the distance, and the reliability regarding the match calculated by the reliability calculation unit 225 is equal to or higher than the threshold value. The position and orientation of the camera 12 at that time are corrected. Alternatively, when the optimization unit 226 captures a keyframe image when all of the reliability regarding the angle, the reliability regarding the distance, and the reliability regarding the match calculated by the reliability calculation unit 225 are equal to or higher than the threshold value. The position and orientation of the camera 12 may be corrected.

The control unit 214 of the information processing device 210 can be realized by, for example, the computer 50 shown in FIG. The storage unit 53 as a storage medium of the computer 50 stores an information processing program 260 for causing the computer 50 to function as a control unit 214 of the information processing device 210. The information processing program 260 includes an image acquisition process 62, a feature point extraction process 63, an attitude estimation process 64, a map generation process 65, an index detection process 66, a reliability calculation process 266, and an optimization process 267. It has an adjustment process 68. Further, the storage unit 53 has a data storage area 69 in which information constituting the data storage unit 15 is stored.

The CPU 51 reads the information processing program 260 from the storage unit 53, expands it in the memory 52, and sequentially executes the processes included in the information processing program 60. The CPU 51 operates as the image acquisition unit 16 shown in FIG. 13 by executing the image acquisition process 62. Further, the CPU 51 operates as the feature point extraction unit 18 shown in FIG. 13 by executing the feature point extraction process 63. Further, the CPU 51 operates as the posture estimation unit 20 shown in FIG. 13 by executing the posture estimation process 64. Further, the CPU 51 operates as the map generation unit 22 shown in FIG. 13 by executing the map generation process 65. Further, the CPU 51 operates as the index detection unit 24 shown in FIG. 13 by executing the index detection process 66. Further, the CPU 51 operates as the reliability calculation unit 225 shown in FIG. 13 by executing the reliability calculation process 266. Further, the CPU 51 operates as the optimization unit 226 shown in FIG. 13 by executing the optimization process 267. Further, the CPU 51 operates as the adjustment unit 28 shown in FIG. 13 by executing the adjustment process 68. Further, the CPU 51 reads information from the data storage area 69 and expands the data storage unit 15 into the memory 52. As a result, the computer 50 that executes the information processing program 60 functions as the control unit 214 of the information processing device 210. Therefore, the processor that executes the information processing program 260, which is software, is hardware.

The function realized by the information processing program 260 can also be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC or the like.

Next, the operation of the information processing device 210 in the second embodiment will be described. The information processing device 210 executes the optimization process shown in FIG.

<Optimization process>
Steps S300 to S302 and steps S304 to S308 are executed in the same manner as in the first embodiment.

In step S403, the reliability calculation unit 225 calculates the reliability according to the detection result of the index of the keyframe image.

In step S404, the optimization unit 226 determines whether or not the reliability calculated in step S403 is equal to or greater than the threshold value. If the reliability is equal to or higher than the threshold value, the process proceeds to step S304. On the other hand, if the reliability is less than the threshold value, the process returns to step S300.

As described above, in the second embodiment, when the index is detected from the image captured by the camera, the information processing apparatus 210 relies on the keyframe image including the index according to the detection result of the index. Calculate the degree. Then, when the reliability is greater than a preset threshold value, the information processing device 210 corrects each of the position and orientation of the camera when each of the keyframe images is captured. As a result, it is possible to accurately correct each of the position and orientation of the camera when each of the keyframe images is captured by using the reliability regarding the detection of the index.

<Third embodiment>
Next, a third embodiment will be described. In the third embodiment, the point of controlling the display device so that the preset object is superimposed and displayed on the display device according to the estimated position and orientation of the camera is the first or second embodiment. Different from the form.

FIG. 16 shows a configuration example of the information processing device 310 of the third embodiment. As shown in FIG. 16, the information processing device 310 of the third embodiment includes a camera 12, a control unit 314, and a display device 326. Further, in the third embodiment, the case where the information processing device 310 is mounted on the information terminal will be described as an example. The user operates the information terminal to browse the screen displayed on the display device.

The control unit 314 includes a data storage unit 15, an image acquisition unit 16, a feature point extraction unit 18, a posture estimation unit 320, a map generation unit 22, an index detection unit 324, an optimization unit 26, and an adjustment unit. 28, an initial position estimation unit 319, and a display control unit 325 are provided.

The data storage unit 15 of the third embodiment stores map information generated in advance.

The initial position estimation unit 319 is described in the above reference 8 based on the feature points extracted by the feature point extraction unit 18, the feature amount corresponding to the feature points, and the map information stored in the data storage unit 15. The initial position and initial posture of the camera 12 are estimated by Relocalization.

Specifically, the initial position estimation unit 319 is based on the feature amount corresponding to the feature point and the feature point extracted by the feature point extraction unit 18 and the feature amount corresponding to the feature point and the feature point in the map information. , The keyframe image most similar to the image acquired by the image acquisition unit 16 is searched. Then, the initial position estimation unit 319 matches the feature points between the image acquired by the image acquisition unit 16 and the most similar keyframe image. Then, the initial position estimation unit 319 associates the feature points and the map points in the image acquired by the image acquisition unit 16 with each other based on the pair of the feature points and the map points in the most similar keyframe image. Then, the initial position estimation unit 319 estimates the initial position and the initial posture of the camera 12 by the PnP algorithm described in Reference 4 above.

The index detection unit 324 further detects whether or not the grayscale image output by the image acquisition unit 16 contains an index.

When the index is detected by the index detection unit 324, the posture estimation unit 320 estimates the position and posture of the camera 12 with respect to the index based on the grayscale image including the index. On the other hand, when the index is not detected by the index detection unit 324, the posture estimation unit 320 positions the camera 12 based on the feature points extracted by the feature point extraction unit 18 and the feature amount corresponding to the feature points. And estimate the posture. The posture estimation unit 20 may estimate the position and posture of the camera 12 by using, for example, the method described in Reference 10 below.

Reference 10: Japanese Unexamined Patent Publication No. 2015-158461

The display control unit 325 controls the display device 326 so that a preset object is superimposed and displayed on the display device 326 based on the position and orientation of the camera 12 estimated by the posture estimation unit 20.

For example, the display device 326 displays a display screen D taken by the camera 12 as shown in FIG. The display control unit 325 controls the display device 326 so that the object G is superimposed and displayed on the display device 326 based on the position and orientation of the camera 12 estimated by the posture estimation unit 20.

The control unit 314 of the information processing device 310 can be realized by, for example, the computer 50 shown in FIG. The computer 50 includes a CPU 51, a memory 52 as a temporary storage area, and a non-volatile storage unit 53. Further, the computer 50 is an R / W unit 55 that controls reading and writing of data to the input / output I / F 54 to which the camera 12, the display device 326, the input / output device and the like (not shown) are connected, and the recording medium 59. To be equipped.

The storage unit 53 can be realized by an HDD, an SSD, a flash memory, or the like. The storage unit 53 as a storage medium stores an information processing program 360 for causing the computer 50 to function as a control unit 314 of the information processing device 310. The information processing program 360 includes an image acquisition process 62, a feature point extraction process 63, a posture estimation process 364, a map generation process 65, and an index detection process 366. The information processing program 360 also includes an optimization process 67, an adjustment process 68, an initial position estimation process 370, and a display control process 371. Further, the storage unit 53 has a data storage area 69 in which information constituting the data storage unit 15 is stored.

The CPU 51 reads the information processing program 260 from the storage unit 53, expands it in the memory 52, and sequentially executes the processes included in the information processing program 60. The CPU 51 operates as the image acquisition unit 16 shown in FIG. 16 by executing the image acquisition process 62. Further, the CPU 51 operates as the feature point extraction unit 18 shown in FIG. 16 by executing the feature point extraction process 63. Further, the CPU 51 operates as the posture estimation unit 320 shown in FIG. 16 by executing the posture estimation process 364. Further, the CPU 51 operates as the map generation unit 22 shown in FIG. 16 by executing the map generation process 65. Further, the CPU 51 operates as the index detection unit 324 shown in FIG. 16 by executing the index detection process 366. Further, the CPU 51 operates as the optimization unit 26 shown in FIG. 16 by executing the optimization process 67. Further, the CPU 51 operates as the adjustment unit 28 shown in FIG. 16 by executing the adjustment process 68. Further, the CPU 51 operates as the initial position estimation unit 319 shown in FIG. 16 by executing the initial position estimation process 370. Further, the CPU 51 operates as the display control unit 325 shown in FIG. 16 by executing the display control process 371. Further, the CPU 51 reads information from the data storage area 69 and expands the data storage unit 15 into the memory 52. As a result, the computer 50 that executes the information processing program 360 functions as the control unit 314 of the information processing device 310. Therefore, the processor that executes the information processing program 360, which is software, is hardware.

The function realized by the information processing program 360 can also be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC or the like.

Next, the operation of the information processing apparatus 310 according to the third embodiment will be described. The information processing device 310 executes a posture estimation process, a map generation process, an optimization process, and a display control process. The posture estimation process, the map generation process, and the optimization process are the same as those in the first or second embodiment. Hereinafter, the display control process shown in FIG. 19 will be described in detail.

<Display control processing>
Upon receiving the instruction signal indicating that the display control process is to be executed, the information processing apparatus 310 executes the display control process shown in FIG.

In step S500, the initial position estimation unit 319 acquires the map information stored in the data storage unit 15.

In step S502, the image acquisition unit 16 acquires an initial image captured by the camera 12. Next, the image acquisition unit 16 converts the image captured by the camera 12 into a grayscale image. Then, the image acquisition unit 16 outputs a grayscale image.

In step S504, the feature point extraction unit 18 extracts feature points from the grayscale image output in step S502. Then, the feature point extraction unit 18 calculates the feature amount for each feature point.

In step S506, the initial position estimation unit 319 sets the initial position of the camera 12 and the initial position of the camera 12 based on the feature points extracted in step S504 and the feature quantities corresponding to the feature points and the map information acquired in step S500. Estimate the initial posture.

In step S508, the image acquisition unit 16 acquires the image captured by the camera 12. Next, the image acquisition unit 16 converts the image captured by the camera 12 into a grayscale image. Then, the image acquisition unit 16 outputs a grayscale image.

In step S510, the index detection unit 324 determines whether or not the grayscale image output in step S508 contains the index. If the grayscale image contains an index, the process proceeds to step S512. On the other hand, if the grayscale image does not include the index, the process proceeds to step S514.

In step S512, the posture estimation unit 320 estimates the position and posture of the camera 12 with respect to the index based on the grayscale image including the index.

In step S514, the feature point extraction unit 18 extracts feature points from the grayscale image output in step S508. Then, the feature point extraction unit 18 calculates the feature amount for each feature point.

In step S516, the posture estimation unit 320 estimates the position and posture of the camera 12 based on the feature points extracted in step S514 and the feature quantities corresponding to the feature points.

In step S518, the display control unit 325 controls the display device 326 so that the preset object is superimposed and displayed on the display device 326 based on the position and orientation of the camera 12 estimated in step S516. do.

In step S520, the display control unit 325 determines whether or not the stop signal of the display control process has been received. When the stop signal of the display control process is received, the display control process is terminated. If the stop signal of the display control process is not received, the process returns to step S508.

As described above, in the third embodiment, the information processing device 310 controls the display device so that the object is superimposed and displayed on the display device according to the estimated position and orientation of the camera. In addition, each time the index is detected, the position and orientation of the camera when each of the keyframe images is captured is optimized, so that the optimization is performed with high frequency. As a result, the object can be displayed at an appropriate position on the display screen according to the accurately estimated position and orientation of the camera.

<Fourth Embodiment>
Next, a fourth embodiment will be described. In the fourth embodiment, the first to third points are to estimate the position and orientation of the camera in the keyframe image when the index corresponding to the previously detected index is detected from the image captured by the camera. It is different from the embodiment of.

In the first embodiment, the relative positions and orientations between the plurality of indicators need to be known. In the fourth embodiment, the relative positions and orientations between the indicators need not be known.

FIG. 20 shows a configuration example of the information processing device 410 of the fourth embodiment. As shown in FIG. 20, the information processing apparatus 410 of the fourth embodiment includes a camera 12 and a control unit 414.

The control unit 414 includes a data storage unit 15, an image acquisition unit 16, a feature point extraction unit 18, a posture estimation unit 420, a map generation unit 22, an index detection unit 424, an optimization unit 426, and an adjustment unit. 28 and.

[Posture estimation process]

The index detection unit 424 further detects whether or not the grayscale image output by the image acquisition unit 16 contains an index.

The posture estimation unit 420 estimates the position and posture of the camera 12 based on the feature points extracted by the feature point extraction unit 18 and the feature quantities corresponding to the feature points. Further, when the index is detected by the index detection unit 424, the posture estimation unit 420 estimates the position and posture of the camera 12 with respect to the index based on the grayscale image including the index.

Then, when the index is detected by the index detection unit 424, the posture estimation unit 420 stores the grayscale image including the index as a keyframe image in the data storage unit 15. When the index is detected, the grayscale image including the index is stored in the data storage unit 15 as a keyframe image, but the index is stored in the image stored as the keyframe image in the data storage unit 15. Not always included. For example, as in the first embodiment, a keyframe image satisfying a predetermined condition is also stored in the data storage unit 15.

Further, the posture estimation unit 420 stores the position and posture of the camera 12 with respect to the index in the data storage unit 15 together with the key frame image. When the posture estimation unit 420 stores the keyframe image, the posture estimation unit 420 stores the index ID representing the identification information of the index in the data storage unit 15 together with the keyframe image. For example, the index area image in the keyframe image can be used as the index ID. Further, the posture estimation unit 420 does not store the grayscale image output by the image acquisition unit 16 as a keyframe image when an index having the same index ID is detected in consecutive frames.

Further, when the posture estimation unit 420 stores the grayscale image including the index as a new keyframe image, whether or not it is the same as the index included in the keyframe image already stored in the data storage unit 15. judge. Specifically, it is determined whether or not the index area image included in the new keyframe image and the index ID stored in the data storage unit 15 are the same.

When the optimization unit 426 determines that the index area image included in the new keyframe image is the same as the index ID stored in the data storage unit 15, the camera when each keyframe image is captured. A loop is formed based on the positions and postures of the twelve. Specifically, the optimization unit 426 is the position and orientation of the camera 12 when a new keyframe image is captured, and the camera when a past keyframe image already stored in the data storage unit 15 is captured. An edge is formed between the 12 positions and postures to form a loop.

For example, as shown in FIG. 21, when the index 1 is detected from the new keyframe image, the position and posture X1 of the camera 12 with respect to the index 1 are estimated. Further, the index 1 is detected from the past keyframe images already stored in the data storage unit 15, and the position and posture X2 of the camera 12 with respect to the index 1 are estimated.

In this case, a loop is formed from the position and orientation X1 of the camera 12 with respect to the index 1, the position and orientation b of the camera 12 when each keyframe image is captured, and the position and orientation X2 of the camera 12 with respect to the index 1. NS. This makes it possible to perform the Pose Graph optimization described in Reference 8 above.

Therefore, the optimization unit 426 corrects each of the position and orientation of the camera 12 in the past keyframe image already stored in the data storage unit 15 by the Pose Graph optimization described in Reference 8 above.

The control unit 414 of the information processing device 410 can be realized by, for example, the computer 50 shown in FIG. The storage unit 53 as a storage medium of the computer 50 stores an information processing program 460 for causing the computer 50 to function as a control unit 414 of the information processing device 410. The information processing program 460 includes an image acquisition process 62, a feature point extraction process 63, an attitude estimation process 464, a map generation process 65, an index detection process 466, an optimization process 467, and an adjustment process 68. Further, the storage unit 53 has a data storage area 69 in which information constituting the data storage unit 15 is stored.

The CPU 51 reads the information processing program 460 from the storage unit 53, expands it in the memory 52, and sequentially executes the processes included in the information processing program 460. The CPU 51 operates as the image acquisition unit 16 shown in FIG. 20 by executing the image acquisition process 62. Further, the CPU 51 operates as the feature point extraction unit 18 shown in FIG. 20 by executing the feature point extraction process 63. Further, the CPU 51 operates as the posture estimation unit 420 shown in FIG. 20 by executing the posture estimation process 464. Further, the CPU 51 operates as the map generation unit 22 shown in FIG. 20 by executing the map generation process 65. Further, the CPU 51 operates as the index detection unit 424 shown in FIG. 20 by executing the index detection process 466. Further, the CPU 51 operates as the optimization unit 426 shown in FIG. 20 by executing the optimization process 467. Further, the CPU 51 operates as the adjustment unit 28 shown in FIG. 20 by executing the adjustment process 68. Further, the CPU 51 reads information from the data storage area 69 and expands the data storage unit 15 into the memory 52. As a result, the computer 50 that executes the information processing program 460 functions as the control unit 414 of the information processing device 410. Therefore, the processor that executes the information processing program 460, which is software, is hardware.

The function realized by the information processing program 460 can also be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC or the like.

Next, the operation of the information processing apparatus 410 according to the fourth embodiment will be described. The information processing device 410 executes the posture estimation process shown in FIG. Further, the information processing device 410 executes the optimization process shown in FIG. 24.

<Posture estimation process>
Steps S100 to S104 are executed in the same manner as in the first embodiment.

In step S606, the index detection unit 424 detects whether or not the index is included in the grayscale image output in step S100. If it is detected that the grayscale image contains the index, the process proceeds to step S607. On the other hand, if it is detected that the grayscale image does not include the index, the process proceeds to step S100.

In step S607, the posture estimation unit 420 estimates the position and posture of the camera 12 with respect to the index based on the grayscale image including the index.

In step S608, the posture estimation unit 420 stores the grayscale image output in step S100 in the data storage unit 15 as a keyframe image. Further, the posture estimation unit 420 stores the position and posture of the camera 12 with respect to the index estimated in step S607 in the data storage unit 15 together with the key frame image. Further, when the posture estimation unit 420 stores the keyframe image, the posture estimation unit 420 stores the index ID representing the identification information of the index in the data storage unit 15.

<Optimization process>
Steps S204 to S206 are executed in the same manner as in the first embodiment.

In step S700, when the posture estimation unit 420 stores the grayscale image including the index as a new keyframe image, the posture estimation unit 420 stores the index area image included in the new keyframe image and the data storage unit 15. It is determined whether or not the index ID is the same as the index ID. If the index ID that is the same as the index area image included in the new keyframe image is stored in the data storage unit 15, the process proceeds to step S702. On the other hand, when the index ID that is the same as the index area image included in the new keyframe image is stored in the data storage unit 15, the process of step S700 is repeated.

In step S702, the optimization unit 426 has an edge between the position and orientation of the camera 12 in the new keyframe image and the position and orientation of the camera 12 in the past keyframe image already stored in the data storage unit 15. To form a loop. Then, the optimization unit 426 corrects each of the position and orientation of the camera 12 in the past keyframe image already stored in the data storage unit 15 by the Pose Graph optimization described in Reference 8 above.

In the above description, the mode in which each program is stored (installed) in the storage unit in advance has been described, but the present invention is not limited to this. The program according to the disclosed technology can also be provided in a form recorded on a recording medium such as a CD-ROM, a DVD-ROM, or a USB memory.

All documents, patent applications and technical standards described herein are to the same extent as if the individual documents, patent applications and technical standards were specifically and individually stated to be incorporated by reference. Incorporated by reference in the document.

Next, a modification of each of the above embodiments will be described.

In the first and second embodiments, the posture estimation unit 20 uses the camera based on the feature points extracted from the image and the feature quantities corresponding to the feature points and the map information stored in the data storage unit 15. The case of estimating the position and the posture of the twelve has been described as an example, but the present invention is not limited to this. For example, when the index detection unit detects the index in the grayscale image, the position and orientation of the camera 12 with respect to the index may be estimated.

Further, in each of the above embodiments, the case where Pose Graph optimization and bundle adjustment are performed for all the keyframes stored in the data storage unit 15 has been described as an example, but the present invention is not limited to this. For example, Pose Graph optimization may be performed for keyframes for which bundle adjustment has not been performed. In addition, bundle adjustment may be performed for keyframes that have not been optimized for Pose Graph. The combination of performing Pose Graph optimization and performing bundle adjustment may be changed as appropriate.

Further, in each of the above embodiments, a case where a loop is formed by the position and orientation of the camera estimated from the keyframe image and the position and orientation with respect to the index when the index is detected from the image has been described. Not limited. For example, a loop is formed and optimized by the position and orientation of the camera estimated from each of the images sequentially acquired by the image acquisition unit up to the previous time and the position and orientation with respect to the index when the index is detected from the image. May be executed. In this case, not only the position and orientation estimated from the keyframe image but also the estimated position and orientation for each of the acquired images are optimized, so that the movement trajectory of the camera can be estimated accurately. In this case, each time the image acquisition unit acquires an image, it may be determined whether or not the image includes an index.

Further, in the third embodiment described above, the case where the object is displayed on the display device has been described as an example, but additional information is superimposed on an image of a large-scale environment such as a factory or a plant. The display device may be controlled so as to display, and the work support of the worker may be provided.

The following additional notes will be further disclosed with respect to each of the above embodiments.

(Appendix 1)
An image acquisition unit that acquires an image captured by an image pickup device whose shooting position can change, and an image acquisition unit.
The position and orientation of the imaging device are estimated based on the image acquired by the image acquisition unit, and when a predetermined index is detected from the image acquired by the image acquisition unit, the index A posture estimation unit that estimates the position and posture of the image pickup device with respect to the image, and a posture estimation unit.
When the index is detected from the image acquired by the image acquisition unit, each of the positions of the imaging device estimated based on each of the images acquired up to the previous time and the posture estimation unit An estimation unit that corrects each of the position and orientation of the image pickup device when each of the images is imaged, based on a loop formed from the position of the image pickup device with respect to the estimated index.
Information processing equipment including.

(Appendix 2)
The estimation unit is based on the estimation results of the position and orientation of the imaging device when each of the keyframe images satisfying a predetermined condition is captured among the images acquired by the image acquisition unit. Generates map points that represent the 3D coordinates of the positions corresponding to each of the feature points in the keyframe image.
The information processing device according to Appendix 1.

(Appendix 3)
The estimation unit stores the images acquired by the image acquisition unit based on the estimation results of the position and orientation of the image pickup device when each of the key frame images satisfying a predetermined condition is captured. Correct each of the position and orientation of the image pickup device when each of the key frame images stored in the unit is imaged.
The information processing device according to Appendix 1 or Appendix 2.

(Appendix 4)
For each of the plurality of indicators, the relative positions and orientations between the indicators are known.
The information processing device according to any one of Supplementary note 1 to Supplementary note 3.

(Appendix 5)
When the index is detected from the image acquired by the image acquisition unit, the reliability calculation unit for calculating the reliability of the image including the index is further included according to the detection result of the index.
When the reliability calculated by the reliability calculation unit is larger than a preset threshold value, the estimation unit corrects each of the position and orientation of the image pickup device when each of the images is imaged. ,
The information processing device according to any one of Supplementary note 1 to Supplementary note 4.

(Appendix 6)
A display control unit that controls the display device is further included so that a preset object is superimposed and displayed on the display device according to the position and orientation of the image pickup device estimated by the posture estimation unit.
The information processing device according to any one of Supplementary note 1 to Supplementary note 5.

(Appendix 7)
The estimation unit is the position of the image pickup apparatus when each of the images is imaged when the index corresponding to the previously detected index is detected from the image acquired by the image acquisition unit. And correct each of the postures,
The information processing device according to any one of Supplementary note 1 to Supplementary note 6.

(Appendix 8)
Acquires an image captured by an imaging device whose shooting position changes as it moves,
The position and orientation of the imaging device are estimated based on the acquired image, and when a predetermined index is detected from the acquired image, the position and orientation of the imaging device with respect to the index are estimated. death,
When the index is detected from the acquired image, each of the positions of the imaging device estimated based on each of the images acquired up to the previous time, and the position of the imaging device with respect to the estimated index. Based on the loop formed from the position, each of the position and orientation of the image pickup device when each of the images is imaged is corrected.
An information processing program that allows a computer to perform processing.

(Appendix 9)
Among the acquired images, each of the feature points of the keyframe image corresponds to each of the feature points of the keyframe image based on the estimation results of the position and the posture of the image pickup device when each of the keyframe images satisfying a predetermined condition is captured. Generate a map point that represents the 3D coordinates of the position to be
The information processing program according to Appendix 8.

(Appendix 10)
Among the acquired images, the keyframe image stored in the storage unit based on the estimation results of the position and orientation of the image pickup device when each of the keyframe images satisfying a predetermined condition is captured. Correct each of the position and orientation of the image pickup device when each of the images is taken.
The information processing program according to Appendix 8 or Appendix 9.

(Appendix 11)
For each of the plurality of indicators, the relative positions and orientations between the indicators are known.
The information processing program according to any one of Supplementary note 8 to Supplementary note 10.

(Appendix 12)
When the index is detected from the acquired image, the reliability of the image including the index is further calculated according to the detection result of the index.
When the calculated reliability is greater than a preset threshold value, each of the positions and orientations of the imaging device when each of the images is imaged is corrected.
The information processing program according to any one of Supplementary note 8 to Supplementary note 11.

(Appendix 13)
The display device is controlled so that a preset object is superimposed and displayed on the display device according to the estimated position and orientation of the image pickup device.
The information processing program according to any one of Supplementary note 8 to Supplementary note 12.

(Appendix 14)
When the index corresponding to the previously detected index is detected from the acquired image, the position and posture of the image pickup device when each of the images is imaged is corrected.
The information processing program according to any one of Supplementary note 8 to Supplementary note 13.

(Appendix 15)
Acquires an image captured by an imaging device whose shooting position changes as it moves,
The position and orientation of the imaging device are estimated based on the acquired image, and when a predetermined index is detected from the acquired image, the position and orientation of the imaging device with respect to the index are estimated. death,
When the index is detected from the acquired image, each of the positions of the imaging device estimated based on each of the images acquired up to the previous time, and the position of the imaging device with respect to the estimated index. Based on the loop formed from the position, each of the position and orientation of the image pickup device when each of the images is imaged is corrected.
An information processing method for causing a computer to perform processing.

(Appendix 16)
Among the acquired images, each of the feature points of the keyframe image corresponds to each of the feature points of the keyframe image based on the estimation results of the position and the posture of the image pickup device when each of the keyframe images satisfying a predetermined condition is captured. Generate a map point that represents the 3D coordinates of the position to be
The information processing method according to Appendix 15.

(Appendix 17)
Among the acquired images, the keyframe image stored in the storage unit based on the estimation results of the position and orientation of the image pickup device when each of the keyframe images satisfying a predetermined condition is captured. Correct each of the position and orientation of the image pickup device when each of the images is taken.
The information processing method according to Appendix 15 or Appendix 16.

(Appendix 18)
For each of the plurality of indicators, the relative positions and orientations between the indicators are known.
The information processing method according to any one of Supplementary note 15 to Supplementary note 17.

(Appendix 19)
When the index is detected from the acquired image, the reliability of the image including the index is further calculated according to the detection result of the index.
When the calculated reliability is greater than a preset threshold value, each of the positions and orientations of the imaging device when each of the images is imaged is corrected.
The information processing method according to any one of Supplementary note 15 to Supplementary note 18.

(Appendix 20)
Acquires an image captured by an imaging device whose shooting position changes as it moves,
The position and orientation of the imaging device are estimated based on the acquired image, and when a predetermined index is detected from the acquired image, the position and orientation of the imaging device with respect to the index are estimated. death,
When the index is detected from the acquired image, each of the positions of the imaging device estimated based on each of the images acquired up to the previous time, and the position of the imaging device with respect to the estimated index. Based on the loop formed from the position, each of the position and orientation of the image pickup device when each of the images is imaged is corrected.
A storage medium that stores an information processing program for causing a computer to execute processing.

1,1A, 1B Index 10,210,310,410 Information processing device 12 Camera 14,214,314,414 Control unit 15 Data storage unit 16 Image acquisition unit 18 Feature point extraction unit 20,320,420 Posture estimation unit 22 Map Generation unit 24,324,424 Index detection unit 26,226,426 Optimization unit 28 Adjustment unit 225 Reliability calculation unit 319 Initial position estimation unit 325 Display control unit 326 Display device 50 Computer 51 CPU
53 Storage unit 59 Recording medium 60, 260, 360, 460 Information processing program M Map point

Claims (9)

An image acquisition unit that acquires an image captured by an image pickup device whose shooting position can change, and an image acquisition unit.
Based on the image acquired by the image acquisition unit, the first position and posture, which are the positions of the image pickup device, are estimated, and a predetermined index is detected from the image acquired by the image acquisition unit. In this case, a posture estimation unit that estimates the second position and the posture, which are the positions of the image pickup device with respect to the index, and the posture estimation unit.
When the index is detected from the image acquired by the image acquisition unit, each of the first positions of the image pickup apparatus and the second position estimated based on each of the images acquired up to the previous time. An estimation unit that corrects each of the first position and the posture of the image pickup device when each of the images is imaged based on the loop formed from the position.
Information processing equipment including. The estimation unit is based on the estimation results of the first position and the posture of the imaging device when each of the keyframe images satisfying a predetermined condition is captured among the images acquired by the image acquisition unit. To generate map points representing the three-dimensional coordinates of the positions corresponding to each of the feature points of the keyframe image.
The information processing device according to claim 1. The estimation unit is based on the estimation results of the first position and the posture of the imaging device when each of the keyframe images satisfying a predetermined condition is captured among the images acquired by the image acquisition unit. Then, each of the first position and the posture of the image pickup device when each of the key frame images stored in the storage unit is imaged is corrected.
The information processing device according to claim 1 or 2. For each of the plurality of indicators, the relative positions and orientations between the indicators are known.
The information processing device according to any one of claims 1 to 3. When the index is detected from the image acquired by the image acquisition unit, the reliability calculation unit for calculating the reliability of the image including the index is further included according to the detection result of the index.
In the estimation unit, when the reliability calculated by the reliability calculation unit is larger than a preset threshold value, each of the first position and the posture of the image pickup device when each of the images is imaged. To correct,
The information processing device according to any one of claims 1 to 4. A display control unit that controls the display device is further included so that a preset object is superimposed and displayed on the display device according to the position and orientation of the image pickup device estimated by the posture estimation unit.
The information processing device according to any one of claims 1 to 5. The estimation unit, from said image acquired by the image acquisition unit, if the index corresponding to the index which is previously detected is detected, the said imaging device when each of the image is captured Correct each of the first position and posture,
The information processing device according to any one of claims 1 to 6. Acquires an image captured by an imaging device whose shooting position changes as it moves,
Based on the acquired image, the first position and posture, which are the positions of the image pickup device, are estimated, and when a predetermined index is detected from the acquired image, the image pickup device with respect to the index is used. Estimate the second position and posture, which are the positions,
When the index is detected from the acquired image, it is formed from each of the first position and the second position of the imaging device estimated based on each of the images acquired up to the previous time. Based on the loop, each of the first position and the posture of the imaging device when each of the images is captured is corrected.
An information processing program that allows a computer to perform processing. Acquires an image captured by an imaging device whose shooting position changes as it moves,
Based on the acquired image, the first position and posture, which are the positions of the image pickup device, are estimated, and when a predetermined index is detected from the acquired image, the image pickup device with respect to the index is used. Estimate the second position and posture, which are the positions,
When the index is detected from the acquired image, it is formed from each of the first position and the second position of the imaging device estimated based on each of the images acquired up to the previous time. Based on the loop, each of the first position and the posture of the imaging device when each of the images is captured is corrected.
An information processing method that causes a computer to perform processing.

Images