JP2017146952A - Information processing device, method for controlling information processing device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a highly accurate three-dimensional map.SOLUTION: An information processing device comprises: an acquisition unit for acquiring a photographic image captured by an imaging unit at a plurality of position attitudes; a generation unit for generating a guidance instruction for acquiring a photographic image corresponding to almost the same position attitude as a photographic image in a normal state; an output unit for outputting the guidance instruction; a determination unit for determining whether or not one photographic image acquired by the acquisition unit after outputting the guidance instruction is imaged at almost the same position attitude as the photographic image in the normal state; and a map generation unit for, if it is determined that the photographic image is imaged at almost the same position attitude, generating a three-dimensional map from three-dimensional coordinates of features included in the photographic image or updating the same on the basis of a plurality of photographic images including one photographic image imaged at a plurality of position attitudes.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus, a control method for the information processing apparatus, and a program.

  2. Description of the Related Art In recent years, a technique for calculating the position and orientation of a photographing device based on characteristics captured in an image obtained by the photographing device is called SfM (Structure from Motion) or SLAM (Simultaneous Localization And Mapping). It is widely used for AR (Augmented Reality), MR (Mixed Reality), and the like that superimpose and present virtually generated images.

  In SfM and SLAM, when an imaging device (or a user of the device that holds the imaging device) moves around a wide space, the tertiary of feature points in the detected space increases as the moving distance of the imaging device increases. Deviations gradually occur in the original coordinates (three-dimensional map) and the estimated values of the position and orientation of the photographing apparatus. In order to solve this problem, a method called loop close is widely used in which a closed path is formed by moving the photographing apparatus in a circular manner and optimization is performed so that the position and orientation of the start point and end point of the path are matched. Patent Document 1 discloses that a three-dimensional map is generated when the loop close can be applied, and the process is terminated.

U.S. Pat. No. 8,787,614

  However, Patent Document 1 does not mention any method for guiding the user of the apparatus to the starting point of the route in order to reliably perform the loop closing. Therefore, there is a problem that it is not easy to generate a highly accurate three-dimensional map.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide a technique for realizing generation of a highly accurate three-dimensional map.

An information processing apparatus according to the present invention that achieves the above object is as follows.
Acquisition means for acquiring captured images captured by the imaging unit at a plurality of positions and orientations;
Generating means for generating a guidance instruction for acquiring a captured image corresponding to substantially the same position and orientation as the captured image in the prescribed state;
Output means for outputting the guidance instruction;
A determining unit that determines whether one captured image acquired by the acquiring unit after the output of the guidance instruction has been captured in substantially the same position and orientation as the captured image in the specified state;
When it is determined that the images are taken at the substantially same position and orientation, a third order is obtained from the three-dimensional coordinates of the features included in the photographed image based on the plurality of photographed images including the one photographed image photographed at the plurality of positions and orientations. Map generation means for generating or updating the original map;
It is characterized by providing.

  According to the present invention, it is possible to generate a highly accurate three-dimensional map.

1 is a diagram illustrating a system configuration and a configuration of an information processing apparatus according to a first embodiment. It is a figure which shows the example of the input image which concerns on 1st Embodiment, an initial position orientation image, and its positional relationship. It is a figure which shows the example of the output image which concerns on 1st Embodiment. It is a flowchart which shows the procedure of the process which the information processing apparatus which concerns on 1st Embodiment implements. It is a figure which shows the system configuration | structure and information processing apparatus structure which concern on 2nd Embodiment. It is a figure which shows the example of the guidance instruction | indication image and output image which concern on 2nd Embodiment. It is a figure which shows the method the guidance production | generation part which concerns on 2nd Embodiment determines a guidance direction. It is a flowchart which shows the procedure of the process which the information processing apparatus which concerns on 2nd Embodiment implements. It is a figure explaining the outline | summary of the additional determination process which the determination part which concerns on the modification 10 performs. It is a figure which shows the method the guidance production | generation part which concerns on 3rd Embodiment determines a guidance direction. It is a figure which shows the example of a guidance instruction | indication image and an output image in 3rd Embodiment. It is a figure which shows the system configuration | structure and information processing apparatus structure which concern on 4th Embodiment. It is a figure which shows the example of a guidance instruction | indication image, a virtual image, and an output image in 4th Embodiment. It is a flowchart which shows the procedure of the process which the information processing apparatus which concerns on 4th Embodiment implements.

  Hereinafter, embodiments will be described with reference to the drawings. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

(First embodiment)
<Overview>
In this embodiment, an information processing apparatus that can be applied to a case where a highly accurate three-dimensional map is generated and prepared in advance prior to the MR (or AR) experience will be described. A user of the information processing apparatus captures an image sequence for generating a three-dimensional map while holding and moving the connected camera. The device of the present invention guides the user to allow loop closure. Specifically, an image photographed at an initial position / posture (specified position / posture) serving as a starting point of the route is presented to the user.

  While viewing this image, the user moves the camera so that the position and orientation of the camera substantially coincide with the initial position and orientation (specified position and orientation). The information processing apparatus determines that the camera has almost reached the initial position / posture (specified position / posture), and performs a loop close process. As a result, the information processing apparatus performs optimization so that the position and orientation of the start point and end point of the route coincide with each other, and generates a highly accurate three-dimensional map.

<System configuration>
FIG. 1 is a diagram illustrating a system configuration according to the first embodiment. Each rectangular frame in FIG. 1 indicates a functional module that performs each process of the present embodiment, and an arrow indicates a data flow. The system according to the first embodiment includes an information processing device 200, and a photographing unit 101 and a display unit 102 connected to the information processing device 200. The configuration of FIG. 1 is an example, and is not limited to that illustrated.

  The imaging unit 101 captures a series of image sequences while the user changes its position and orientation in a space that is a target for generating a three-dimensional map using the information processing apparatus 200. The imaging unit 101 is used by being connected to the information processing apparatus 200, and continuously captures images at a short time period such as 1/30 second. Each time an image is taken, the image is input to the image acquisition unit 201. Specifically, the photographing unit 101 is a camera provided in a head mounted display (HMD).

  The display unit 102 displays an image to be presented to the user. The image to be presented is supplied from the information processing apparatus 200 (more specifically, the guidance output unit 205). Specifically, the display unit 102 is a display device such as a monitor, or a liquid crystal panel provided in the HMD.

<Configuration of information processing apparatus>
The information processing apparatus 200 includes an image acquisition unit 201, an image storage unit 202, a determination unit 203, a guidance generation unit 204, a guidance output unit 205, and a 3D map generation unit 206.

  The image acquisition unit 201 captures an image photographed by the photographing unit 101 into the information processing apparatus 200 and sends it as an input image 10 to the image storage unit 202, the determination unit 203, the guidance generation unit 204, and the guidance output unit 205.

  The image storage unit 202 stores and holds the input image 10 at each time acquired by the image acquisition unit 201. In addition, the image storage unit 202 uses the initial position / posture image 20 (specified position / posture) when the input image 10 is taken in a state where preparation for photographing for generating a three-dimensional map is completed or at an arbitrary timing designated by the user. Registered as a captured image).

  More specifically, the image storage unit 202 assigns and holds a label to the input image 10 that is regarded as the initial position and orientation image. When the information processing apparatus 200 generates a highly accurate three-dimensional map, the user is required to capture an image of a position and orientation that roughly matches the initial position and orientation so that the movement history of the imaging unit 101 forms a closed path. Is a guidance instruction. The image storage unit 202 holds an initial position / posture image 20 that defines the state of the guidance target for the user.

  Here, FIG. 2A and FIG. 2B show examples of the movement history of the photographing unit 101 and the initial position / posture image 20. FIG. 2A is an overhead view of the measurement space observed from above, where the black circle indicates the position of the photographing unit 101, the triangle indicates the field of view of the photographing unit 101 (the vertex of the triangle is the viewpoint position, and the base of the triangle is the field of view). Represents. 2A shows that the photographing unit 101 has moved from the position / posture A to the position / posture B and the position / posture C along the arrow. The initial position / orientation image 20 shown in FIG. 2 (b) is an image taken at the position / orientation A in FIG. 2 (a).

  As a result of guidance by the guidance generation unit 204 and the guidance output unit 205, the determination unit 203 reaches the initial position and orientation of the photographing unit 101, and the locus of the position and orientation from the initial position and orientation registration constitutes a closed path. A closed path determination process is performed to determine whether or not. In the present embodiment, the determination unit 203 has at least two modes: an activation mode and a closed path determination mode.

  Immediately after the initial position / orientation image 20 is registered, the position / orientation of the photographing unit 101 has hardly changed, so that it is substantially closed and the closed path is not formed although there is substantially no movement. There is a risk of misjudgment. Therefore, the determination unit 203 does not perform the closed path detection determination process when in the activation mode. Alternatively, the closed path detection determination process may be performed even in the activation mode, but the closed path may not be regarded as having been configured even if the determination is successful. In the present embodiment, the determination unit 203 shifts from the activation mode to the closed path determination mode when a predetermined time has elapsed since the initial position and orientation registration.

  In the closed path determination mode, the determination unit 203 compares the current input image 10 input from the image acquisition unit 201 with the initial position and orientation image 20 held by the image storage unit 202. A closed path determination process is performed. That is, the determination unit 203 determines whether or not the current position / orientation of the photographing unit 101 substantially matches the position / orientation at the time of registration of the initial position / orientation image 20. If they roughly match, the user of the information processing apparatus 200 is notified of this fact and prompted to end shooting.

  Specifically, the determination unit 203 generates a display form 30 for notifying that the closed route has been reached, as shown in FIG. 3B, and sends it to the guidance output unit 205. Further, the determination unit 203 assigns a label to the image storage unit 202 and stores the input image 10 at this time as a closed path arrival image. Otherwise, a guidance instruction for roughly matching the position and orientation of the photographing unit 101 with the initial position and orientation is given to the user of the information processing apparatus 200 by processing of the guidance generation unit 204 described later.

  Next, the content of the closed path determination process by the determination unit 203 will be described. The condition that is determined to be a closed path (conditions that can be applied to loop close) is that there is an overlap that allows a common feature to be detected in the captured area of the captured image at the start point and the end point of the closed path. The relative position and orientation between the two images can be calculated. The fact that there are many common areas between the two images means that the positions and orientations of the two images are similar and can be regarded as being continuous. Therefore, in this embodiment, Bag of Words (BoW) is applied. The closed path is determined. BoW calculates similarity between a plurality of images based on the appearance frequency of features included in the images. The determination unit 203 determines that a closed path is detected when the BoW similarity exceeds a threshold, that is, when the similarity between the start point image and the end point image is high.

  Here, FIG. 2C shows an example of the input image 10 taken from the position and orientation B in FIG. 2A in the same environment as the initial position and orientation image 20 in FIG. Since there is very little overlap in each imaging region between the input image 10 in FIG. 2C and the initial position and orientation image 20 in FIG. 2B, the determination unit 203 determines the position of the input image 10 in FIG. The posture is determined not to constitute a closed path with respect to the initial position / posture image 20.

  On the other hand, FIG. 2D shows an example of the input image 10 obtained by photographing the same environment as the initial position and orientation image 20 in FIG. 2B from the position and orientation C in FIG. Since there are many overlaps in the imaging regions of the input image 10 in FIG. 2D and the initial position / posture image 20 in FIG. 2B, the determination unit 203 succeeds in the closed path determination process. Therefore, it is determined that the position / orientation of the input image 10 in FIG. 2D forms a closed path with respect to the initial position / orientation image 20.

  The guidance generation unit 204 generates a guidance instruction image 40 as shown in FIGS. 3A and 3B when the determination unit 203 transitions from the startup mode to the closed route determination mode, and sends it to the guidance output unit 205. Send it out. The guidance instruction image 40 is an image that is presented to the user of the information processing apparatus 200 and indicates the position and orientation to be moved next. In the present embodiment, the guidance generation unit 204 uses the initial position and orientation image 20 held by the image storage unit 202 as the guidance instruction image 40. That is, the guidance instruction image 40 in the present embodiment is the same as the initial position / posture image 20.

  The guidance output unit 205 generates an image to be displayed on the display unit 102 and presented to the user. In the present embodiment, as shown in FIGS. 3A and 3B, the guidance image 40 is superimposed on the input image 10 to generate an output image 50. The output image is sent to the display unit 102 outside the information processing apparatus 200. 3A and 3B show examples of the output image 50 in the present embodiment.

  The 3D map generation unit 206 refers to the series of input images 10 stored so far from the image storage unit 202 when the shooting is finished by an operation from the user of the apparatus, and performs a 3D map by SLAM processing. Is generated. At this time, the initial position / posture image 20 (start point) and the closed path arrival image (end point) are associated with each other as being continuous, and optimization processing by loop close is performed.

<Processing>
Control of the information processing apparatus 200 according to this embodiment having the above configuration will be described below. FIG. 4 is a flowchart illustrating a procedure of processing performed by the information processing apparatus 200 according to the present embodiment.

  First, when the information processing apparatus 200 is activated, an initialization process is performed in step S1010. The initialization process in step S1010 includes a process necessary for starting each module and a process for setting the determination unit 203 to the start mode.

  In step S1020, the imaging unit 101 captures the current input image 10. The input image 10 is taken into the information processing apparatus 200 through the image acquisition unit 201. In step S <b> 1030, the guidance generation unit 204 generates the guidance instruction image 40 and sends it to the guidance output unit 205. As described above, in the present embodiment, the guidance generation unit 204 generates the initial position and orientation image 20 held by the image storage unit 202 as the guidance instruction image 40. The guidance output unit 205 generates the output image 50 by superimposing the guidance instruction image 40 on the input image 10.

  In step S1040, the determination unit 203 performs a closed path determination process on the current input image 10 when the operation mode is the closed path determination mode. When the determination unit 203 is in the activation mode, the image storage unit 202 registers the initial position / posture image 20, and after a predetermined time has elapsed, the operation mode of the determination unit 203 is changed from the activation mode to the closed path determination mode. Transition.

  In step S1050, if the closed path determination process in step S1040 is successful (closed path is configured), the determination unit 203 registers the input image 10 as the closed path arrival image in the image storage unit 202, and step S1060. Proceed to Otherwise, the process returns to step S1020.

  In step S1060, the 3D map generation unit 206 uses the input image 10, the initial position and orientation image 20, and the closed path arrival image stored so far to generate a 3D map from the 3D coordinates of the features included in the captured image. The process to generate and the optimization process by loop closing are performed. When the optimized three-dimensional map is generated, the process of the present embodiment is terminated.

  As described above, according to the present embodiment, it is possible to guide the movement of the imaging unit so as to configure a closed path to which the loop close process can be applied to the user of the apparatus. In addition, it is possible to generate a highly accurate three-dimensional map by applying a loop close process to the acquired image sequence.

[Modification 1]
In step S1030, in addition to the elapsed time from the initial position and orientation registration, the determination unit 203 further transitions from the start mode to the closed path determination mode on the condition that the determination process in step S1050 has failed more than the specified number of times. You may make it do. By doing so, it is possible to prevent the user of the information processing apparatus 200 from entering the closed path determination mode when the user is stationary for a certain period of time while maintaining the initial position and orientation.

[Modification 2]
In step S1050, the determination unit 203 does not use BoW but associates feature points from both the initial position and orientation image 20 and the current input image 10 using SIFT or the like, and performs affine transformation, homography transformation, and the like. Conversion between the two images may be obtained. In this case, one image may be converted into the other image, and the reciprocal of the sum of luminance differences between these images may be used as a similarity, and it may be determined that a closed path has been detected when this is greater than a threshold value. Further, the ratio between the number of extracted feature points and the number of feature points successfully matched may be taken, and if this is equal to or greater than a threshold value, it may be determined that a closed route has been detected.

[Modification 3]
When the user of the information processing apparatus 200 approaches the initial position and orientation, the display of the superimposed input image 10 or guidance instruction image 40 may be changed. For changing the display, any method may be used as long as the user can recognize that the user has approached the initial position and orientation. Further, any method may be used for determining whether or not the initial position and posture are approached.

  In the second modification, when the number of corresponding feature points between the initial position / posture image 20 and the current input image 10 is increased, the guidance generation unit 204 to the guidance output unit 205 are initialized by the user of the information processing apparatus 200. It may be considered that the position / orientation has been approached. When the number of corresponding feature points increases, it is possible to grasp the situation in which the user of the information processing apparatus 200 is approaching the initial position and orientation by changing the color tone of the input image 10 according to the ratio.

[Modification 4]
The guidance output unit 205 may notify the user that the photographing can be ended when the closed path determination process in step S1050 is successful. FIG. 3B shows a character string “Finished” as a display form 30 for notifying that the closed route has been reached. This is because the position and orientation of a plurality of images (input images 10) that have been captured so far constitute a closed path for the user, and all necessary images can be acquired, so that the capturing can be terminated. Is presented. The notification to the user may be anything as long as the user can recognize that the shooting can be finished. In addition to the display mode 30, voice, vibration, etc. may be presented to the user.

[Modification 5]
In step S1010, after confirming that the space for generating the three-dimensional map is included in the field of view of the photographing unit 101, the process may proceed to step S1020. In this way, it is possible to prevent the accuracy of the three-dimensional map from being lowered when the photographing unit 101 is photographing an environment that does not originally generate a three-dimensional map immediately after activation. The user of the information processing apparatus 200 may be notified of the completion of preparation to the information processing apparatus 200 by a button, a switch, or the like. For example, the photographing unit 101 can observe a specific object in the environment. As a condition, the information processing apparatus 200 may determine automatically.

[Modification 6]
The imaging unit 101 according to the present embodiment may use a device that acquires information other than brightness, such as a depth sensor or a thermography, by converting the information into a luminance value. Any material can be used as long as it is possible. Further, the closed path determination process may be executed using an image or a data group acquired from the imaging unit 101 configured by these devices.

[Modification 7]
The SLAM processing performed by the three-dimensional map generation unit 206 may use any method as long as it can calculate the position and orientation of each input image 10 from the series of input images 10 and the three-dimensional coordinates of the feature points. .

(Second Embodiment)
<Overview>
In this embodiment, an information processing apparatus that can be applied to a case where a highly accurate three-dimensional map is generated and prepared in advance prior to the MR (or AR) experience will be described. A user of the information processing apparatus captures an image sequence for generating a three-dimensional map while holding and moving the connected camera. The information processing apparatus of the present invention guides the user using the estimated position and orientation so that the loop can be closed. Specifically, an image showing the direction from the current position and orientation to the initial position and orientation that is the starting point of the route is presented to the user. While viewing this image, the user moves the camera so that the position and orientation of the camera substantially coincide with the initial position and orientation.

  In the present embodiment, since a specific moving direction is presented to the user, the position and orientation of the camera can be matched with the initial position and orientation more efficiently. The information processing apparatus determines that the camera has almost reached the initial position and orientation, and performs a loop close process. As a result, the information processing apparatus performs optimization so as to match the position and orientation of the start point and end point of the route, and generates a highly accurate three-dimensional map.

<System configuration>
FIG. 5 is a diagram illustrating a system configuration according to the second embodiment. A rectangular frame in FIG. 5 indicates a functional module that performs each process of the present embodiment, and an arrow indicates a data flow. The system according to the present embodiment includes an information processing apparatus 200, a photographing unit 101 and a display unit 102 connected thereto. The configuration of FIG. 5 is an example, and is not limited to that illustrated. In addition, different parts from the first embodiment will be mainly described, and description of the same parts will be omitted.

<Configuration of information processing apparatus>
In the present embodiment, a state storage unit 207 is provided instead of the image storage unit 202. The state storage unit 207 holds the position and orientation of the photographing unit 101 at the time of photographing the input image 10 in association with the input image 10. When the initial position / posture image 20 is registered, the state storage unit 207 also stores the current position / posture as the initial position / posture at the same time. In other words, in the present embodiment, the state storage unit 207 holds all of the initial position / posture image 20 and the input image 10 including the closed path arrival image in association with the position / posture. Detailed operation of the state storage unit 207 will be described later.

  Further, in the present embodiment, a position / orientation acquisition unit 208 is newly added. The position and orientation acquisition unit 208 uses the input image 10 to estimate the position and orientation of the imaging unit 101 when the current input image 10 is captured by the SLAM process. The estimated position and orientation are held by the state storage unit 207 in association with the input image 10. Hereinafter, these associated position and orientation are referred to as the position and orientation of the input image 10. The detailed operation of the position / orientation acquisition unit 208 will be described later.

  The determination unit 203 compares the current input image 10 from the image acquisition unit 201 with the initial position / orientation image 20 held by the image storage unit 202, and determines the position / orientation associated with them. Referring to, a closed path determination process is performed.

  More specifically, the determination unit 203 compares the position / posture of the input image 10 and the position / posture of the initial position / posture image 20 in addition to the determination of the similarity by BoW described in the first embodiment. By comparing not only the similarity of images but also the position and orientation of both, it is possible to more accurately determine the closed path. For the position, the Euclidean distance between the two three-dimensional coordinates, and for the attitude, the inner product (or angle formed) between the two attitude vectors may be compared as an index.

  That is, if the Euclidean distance between two three-dimensional coordinates is smaller than a threshold value, the positions are considered to be matched, and if the inner product between two posture vectors is larger than the threshold value, the attitudes can be regarded as matched. . Note that the position and orientation estimated by SLAM may not be the same when the same point is reached again as described above. Therefore, when comparing the initial position and posture with the current position and posture, the position threshold value may be set to be particularly large.

  When the determination unit 203 is in the closed route determination mode, the guidance generation unit 204 calculates a direction for guiding the user of the information processing apparatus 200 and generates a guidance instruction image 60 described later. The generated guidance instruction image 60 is sent to the guidance output unit 205. In order to generate the guidance instruction image 60, the guidance generation unit 204 refers to the position and orientation corresponding to the input image 10 held in the state storage unit 207, and the three-dimensional map.

  FIG. 6A shows an example of a guidance instruction image 60 and an output image 70 on which the guidance instruction image 60 is superimposed in the present embodiment. It is assumed that the input image 10 is taken at the position / orientation B in FIG. If the initial position / posture image 20 is taken at the position / posture A in FIG. 2 (a), the user of the information processing apparatus 200 with respect to the position / posture B at which the input image 10 is taken in FIG. 6 (a). Is guided to the left, a closed path can be formed.

  Therefore, in FIG. 6A, as the guidance instruction image 60, an arrow indicating the left direction is shown at the center of the output image 70, and an overhead view showing the entire measurement space is shown at the upper right. In the overhead view, an initial position and orientation (black circle), a locus of position and orientation (solid arrow), and a path to be moved (broken line arrow) are shown. A user of the information processing apparatus 200 can recognize a target position and orientation from an overhead view. Further, by referring to the arrow and the overhead view in the output image 70, it can be recognized that it is only necessary to move toward the left side.

  FIG. 6B shows another example of the guidance instruction image 60 and the output image 70 on which the guidance instruction image 60 is superimposed in the present embodiment. In FIG. 6B, a cubic virtual object is displayed at the center of the output image 70 as the guidance instruction image 60. Characters are written on the left side of the virtual object, and a message prompting the user to read the characters written on the left side is displayed. In this case, the user of the information processing apparatus 200 can be expected to perform an operation of turning around to the left side so that the characters written on the left side of the virtual object can be read well. The guidance generation unit 204 calculates the rotation angle of the virtual object based on the current position and orientation and the guidance direction.

  When the user of the information processing apparatus 200 is guided in the left direction, the virtual object is rotated in a direction in which characters written on the virtual object are hidden from the left side of the screen. Conversely, when guiding the user of the information processing apparatus 200 to the right, the virtual object is rotated in a direction to hide the character written on the virtual object from the right side of the screen to the back. Since the guidance generation unit 204 rotates the virtual object every moment according to the estimated position and orientation within a short time, the user of the information processing apparatus 200 can be guided to a desired position and orientation.

  FIG. 6C shows another example of the guidance instruction image 60 and the output image 70 on which it is superimposed in the present embodiment. FIG. 6C shows an image in which a portion excluding the guiding direction of the output image 70 is blacked out as the guidance instruction image 60. In FIG. 6C, the color is black, but a different color may be used. Also, a part of the input image 10 underneath may be made visible even in a covered part, such as making the fill color translucent. The fill color is determined so that the user can easily move toward the bright part of the output image 70.

In the present embodiment, a process in which the guidance generation unit 204 determines a direction to be guided from the current position / orientation toward the initial position / orientation will be described with reference to FIGS. 7 (a) and 7 (b). The initial position _{T 0} by black circles in FIG. 7 (a), shows the initial position _{R 0} by triangles. The triangle indicates the viewing direction at _R0 . The current position and orientation T _t and R _t are similarly indicated by white circles and triangles. In addition, the line-of-sight vector can be calculated from the current posture R _t and v _t. A moving direction vector _dt indicating a direction from the current position Tt is expressed by Expression (1).

d _t = T ₀ −T _t (1)
The guiding direction can be determined by an angle θ _t formed from v _t to _dt . If θ _t is positive, the guidance direction is left, and if θ _t is negative, the guidance direction is right. That is, the induction generator 204, the theta _t arrow in the left direction if positive may be generated a rightward arrow as guiding instruction image 60 if it is negative.

On the other hand, when the current position / posture is sufficiently close to the initial position / posture, the guidance generation unit 204 needs to guide the current posture to match the initial posture. A method for determining the direction to be guided in this case will be described with reference to FIG. If the line-of-sight vector in the initial posture in FIG. 7B is v ₀ , the direction to be guided is determined by the angle φ _t formed from v _t to v ₀ . guidance direction if it is positive is φ _t is left, if it is negative induction direction is right. That is, the induction generator 204, the phi _t arrow in the left direction if positive may be generated a rightward arrow as guiding instruction image 60 if it is negative.

  The 3D map generation unit 206 generates a 3D map of the measurement space from the input image 10 and the position / orientation held by the state storage unit 207. In the first embodiment, a 3D map is generated after a series of photographing is completed. In the present embodiment, every time an input image 10 is input and a corresponding position and orientation is estimated. Generate and update 3D maps. The generated / updated three-dimensional map is held in the state storage unit 207. When the determination unit 203 determines that the closed path is configured and the photographing is finished, the loop closing process is performed, and the optimization process is performed on the three-dimensional map generated so far.

  The state storage unit 207 stores and holds the input image 10 at each time acquired by the image acquisition unit 201. Further, the position and orientation estimated by the position and orientation acquisition unit 208 are held in association with the input image 10 taken at the same time. The state storage unit 207 also holds the 3D map generated and updated by the 3D map generation unit 206, and transmits and receives data to and from the 3D map generation unit 206 as necessary.

  The position / orientation acquisition unit 208 estimates the current position / orientation of the input image 10 by SLAM processing with reference to the past input image 10, its position / orientation, and the three-dimensional map generated so far. The estimated position / orientation is sent to the state storage unit 207 and held in association with the input image 10.

<Processing>
Control of the information processing apparatus 200 according to this embodiment having the above configuration will be described below. FIG. 8A is a flowchart illustrating a procedure of processing performed by the information processing apparatus 200 according to the present embodiment. Compared to the first embodiment, the process of step S1020 is changed to step S1025. In step S1025, the position / orientation for the current input image 10 is acquired, and the generation and update of the three-dimensional map are performed.

  FIG. 8B is a flowchart showing the flow of processing inside step S1025. In step S2010, the imaging unit 101 captures the current input image 10. The input image 10 is taken into the information processing apparatus 200 through the image acquisition unit 201. At this time, the guidance output unit 205 starts outputting the output image 70 to the display unit 102.

  In step S2020, the position / orientation acquisition unit 208 acquires the position / orientation of the input image 10 by SLAM processing using the input image 10 and the three-dimensional map from the start of activation to the present stored in the state storage unit 207.

  In step S2030, the 3D map generation unit 206 generates a 3D map using the input image 10 and the position and orientation held in the state storage unit 207. If a 3D map has already been generated, the 3D map is updated.

  About the process of step S2020 and S2030, the order of a process may be replaced and may be performed simultaneously. In any case, by completing the processing up to step S2030, the input image 10 is taken into the information processing apparatus 200, and the corresponding position and orientation and the three-dimensional map up to the present are generated and updated.

  In step S <b> 1030, the guidance generation unit 204 generates a guidance instruction image 60 and sends it to the guidance output unit 205. As described above, in the present embodiment, the guidance generation unit 204 generates the guidance instruction image 60 based on the initial position and orientation held by the state storage unit 207 and the position and orientation acquired by the position and orientation acquisition unit 208. To do. The guidance output unit 205 generates the output image 70 by superimposing the guidance instruction image 60 on the input image 10.

  In step S1040, the determination unit 203 performs a closed path determination process on the current input image 10 when the operation mode is the closed path determination mode. On the other hand, in the start mode, the state storage unit 207 registers the initial position / posture image 20 and the initial position / posture, and after a predetermined time has elapsed, the operation mode of the determination unit 203 is shifted to the closed path determination mode. To do.

  In step S1050, when the closed path determination process in step S1040 is successful (closed path is configured), the determination unit 203 registers the input image 10 as the closed path arrival image in the image storage unit 202, and the process proceeds to step S1060. move on. Otherwise, the process returns to step S1020.

  In step S1060, the 3D map generation unit 206 stores the input image 10, the initial position / posture image 20, the initial position / posture stored in step S2010, the position / posture of the input image 10 estimated in step S2020, and the closed path acquired in step S1050. Using the reached image, the optimization process by loop closing is performed using the three-dimensional map created in step S2030 as an initial value. When the optimized three-dimensional map is generated, the process of the present embodiment is terminated.

  As described above, according to the present embodiment, a closed path to which a loop close process can be applied to the user of the apparatus by using the position and orientation calculated from time to time and a 3D map. It is possible to specifically indicate the direction for constructing and to efficiently guide the movement of the photographing unit. Further, by applying a loop close process to the generated 3D map, it is possible to generate a 3D map with higher accuracy.

  In this way, it is possible to guide the user of the apparatus to a position and orientation for configuring a closed path to which the loop closing process can be applied. At this time, by presenting a specific moving direction to the user, the user can reach a position and orientation for configuring the closed path more efficiently.

[Modification 8]
The position / orientation acquisition unit 208 according to the present embodiment may directly obtain the position / orientation by installing a position / orientation sensor in the photographing unit 101 instead of obtaining the position / orientation from the input image 10 by SLAM processing. . Alternatively, a feature such as a marker is arranged on the photographing unit 101, and an image of the photographing unit 101 is photographed by a photographing device provided outside, and the feature is extracted from the image, thereby changing the position and orientation of the photographing unit 101. You may make it ask.

  When the position and orientation of the imaging unit 101 is not calculated by SLAM using the input image 10 but is estimated by an external device as described above, the 3D map generation unit 206 generates a 3D map in real time during imaging. The state storage unit 207 may not always hold the three-dimensional map. Alternatively, only the position of 3 degrees of freedom or only the position of the 6 degrees of freedom position and orientation may be measured by an external device, and the remaining parameters may be calculated by SLAM processing.

[Modification 9]
As the photographing unit 101 in the present embodiment, a device capable of measuring the depth from the viewpoint of the photographing unit 101 such as a depth sensor can be used. The 3D map generation unit 206 and the position / orientation acquisition unit 208 may perform position / orientation estimation and 3D map generation using a depth image instead of the input image 10.

[Modification 10]
The determination unit 203 may use a history (trajectory) of the position and orientation of the input image 10 in determining the closed path. At the time of loop close optimization, it is preferable that the locus of position and orientation is widely distributed, and this can be added as a determination condition of the determination unit 203. For example, in a space (measurement space) for creating a three-dimensional map, whether or not the locus of position and orientation has a sufficient spread may be added to the determination condition for closed path detection.

  An example is shown in FIG. The rectangle in FIG. 9 shows the measurement space viewed from above. The curve drawn in the inside shows the locus of the position and orientation starting from the initial position and orientation. When a circumscribed rectangle is defined with respect to the trajectory of FIG. 9, both sides in the X direction and the Y direction have a sufficient length with respect to the size of the measurement space. When configured, it can be expected that an accurate three-dimensional map can be generated. As a determination condition, for example, the ratio of the length of the measurement space in the X-axis and Y-axis directions and the extent of the circumscribed rectangle of the trajectory in the X-axis direction and the Y-axis direction may be calculated, and this ratio may be equal to or greater than a threshold value. . Further, an area may be obtained by approximating a figure drawn by the measurement space and the locus to a polygon, and an area ratio with the measurement space may be used.

[Modification 11]
The guidance generation unit 204 may generate the guidance instruction by giving priority to the direction in which the locus of the position and orientation described in Modification 7 has a sufficient spread (a direction in which the spread is expanded). The guidance generation unit 204 generates a guidance instruction image 60 so as to finally reach a position and orientation that constitutes a closed path through a direction in which at least one side becomes larger with respect to the circumscribed rectangle with respect to the locus in FIG. You may do it.

[Modification 12]
The determination unit 203 may use a three-dimensional map held by the state storage unit 207 in the closed route determination process. Since it is desirable that the point group constituting the three-dimensional map is uniformly distributed in the measurement space, the density distribution of the three-dimensional point group generated as the three-dimensional map may be added to the determination condition. For example, when an area where the three-dimensional point cloud is sparse still remains, the determination unit 203 fails the closed path determination process, and the guidance generation unit 204 instructs the user of the information processing device 200 to enter the area. Alternatively, the guidance instruction image 60 may be generated.

[Modification 13]
The determination unit 203 may use the reprojection error in each input image using the input image 10, the position and orientation, and the three-dimensional map held by the state storage unit 207 in determining the closed path. The determination unit 203 reprojects the 3D point group constituting the 3D map held by the state storage unit 207 on the input image 10 based on the position and orientation, and reprojects the feature points extracted by image processing. The difference from the coordinates is calculated. If this difference is large, it means that the position and orientation or the estimation accuracy of the 3D map is not good. Therefore, in this case, the determination unit 203 fails the closed path determination process, and the guidance generation unit 204 generates the guidance instruction image 60 to cause the user of the information processing device 200 to re-capture the input image 10 in that region. You may make it do.

[Modification 14]
The guidance generation unit 204 may change the color displayed in the guidance instruction image 60 as the user of the information processing apparatus 200 approaches or moves away from the target position and orientation that forms the closed path. For example, the user of the information processing apparatus 200 can easily approach the target position and orientation by reducing the color of the fill in FIG. Can be recognized.

  In order to determine that the guidance generation unit 204 is approaching or moving away from the target position and orientation, not only the position and orientation trajectory is used, but also correspondence between the initial position and orientation image 20 and the input image 10 is made. The number of feature points may be used. When the number of corresponding feature points increases, it can be determined that the user is approaching the initial position and orientation. On the contrary, when the number of corresponding feature points is decreasing, it can be determined that the position is away from the initial position and orientation.

[Modification 15]
The guidance generation unit 204 and the guidance output unit 205 may indicate a direction in which the user should move using a method other than an image. The direction in which the user of the information processing device 200 should move may be indicated by voice using an acoustic device (not shown). The direction of movement can be directly indicated, such as right or left. Alternatively, for example, a sound that leaves the virtual object in the direction in which the virtual object should move may be given by stereophonic sound to guide the user of the information processing apparatus 200. The user of the information processing apparatus 200 can approach the target position and orientation by performing an operation that follows the direction in which the virtual object leaves.

[Modification 16]
FIG. 6A shows both an arrow and an overhead view as an example of the guidance instruction image 60, but the guidance instruction image 60 may be composed of only one of them.

(Embodiment 3)
<Overview>
In this embodiment, an information processing apparatus that can be applied to a case where a highly accurate three-dimensional map is generated and prepared in advance prior to the MR (or AR) experience will be described. The user of the information processing apparatus 200 captures an image sequence for generating a three-dimensional map while holding and moving the connected camera. The information processing apparatus of the present invention first guides the user so that a three-dimensional map suitable for position and orientation estimation during MR experience is generated. For this purpose, the information processing apparatus captures an image sequence in which the user's movement trajectory is spatially expanded while observing a region in which a virtual object is assumed to be arranged during MR experience. Guide the user to

  Specifically, an image showing a region direction in which the virtual object is arranged and a moving direction in which the locus is spatially spread is presented to the user. While viewing this image, the user moves the camera so as to capture an image sequence suitable for position and orientation estimation during MR experience. When the information processing apparatus detects that such an image sequence has been acquired, the information processing apparatus guides the user so that the loop can be closed next. Specifically, an image showing the starting point direction of the route is presented to the user. While viewing this image, the user moves the camera so that the position and orientation of the camera substantially coincide with the initial position and orientation. The information processing apparatus determines that the camera has almost reached the initial position and orientation, and performs a loop close process. As a result, the information processing apparatus 200 performs optimization so as to match the position and orientation of the start point and end point of the route, and generates a highly accurate three-dimensional map.

<System configuration>
The overall configuration of the system according to the present embodiment is the same as that of the second embodiment shown in FIG. However, in the present embodiment, there are parts different from the second embodiment regarding the operation of the functional module. Hereinafter, only different parts from the second embodiment will be described, and description of the same parts will be omitted.

<Configuration of information processing apparatus>
The determination unit 203 performs a 3D map generation determination process for determining whether or not a 3D map for performing the MR experience has been generated, and a closed path determination process for detecting a closed path. In the present embodiment, the determination unit 203 has at least three modes: a start mode, a 3D map generation mode, and a closed route determination mode. In the present embodiment, the determination unit 203 shifts from the activation mode to the 3D map generation mode when the initial position and orientation are registered and a virtual object presentation area to be described later is detected.

  The virtual object presentation area is input to the information processing apparatus 200 as an area where the virtual object is presented during the MR experience, and the three-dimensional coordinates indicating the virtual object center and the size occupied by the virtual object (for example, virtual Information on the vertical, horizontal, and height values that define the bounding box of the object. The virtual object presentation area is input to the information processing apparatus 200 as a numerical value in advance. The state storage unit 207 holds the center coordinates and size of the input virtual object presentation area.

  When the determination unit 203 succeeds in the end determination process of the 3D map generation, the determination unit 203 shifts from the 3D map generation mode to the closed route determination mode. The 3D map generation end determination process is performed as follows.

  Similar to the seventh modification, the determination unit 203 refers to the locus of the position and orientation estimated so far. It is determined whether or not the trajectory has a sufficient length for both sides in the X direction and the Y direction with respect to the size of the measurement space (first determination). Further, the determination unit 203 refers to the number of times the virtual object presentation area is included in the input image 10 and determines whether the number of times is sufficiently large with respect to the position and orientation history (second determination). . The determination unit 203 determines that the 3D map generation end processing has been successful when both of the two determination conditions are satisfied. The operation of the determination unit 203 after shifting to the closed path determination mode is the same as in the second embodiment.

  When the determination unit 203 is in the 3D map generation mode, the guidance generation unit 204 generates a guidance instruction image 40 for guiding to a position and orientation suitable for observing a virtual object presented during the MR experience. The guidance generation unit 204 generates the guidance instruction image 40 using the position and orientation corresponding to the input image 10 held in the state storage unit 207, the three-dimensional map, and the virtual object presentation area. The generated guidance instruction image 80 (see FIG. 11) is the same as that of the second embodiment, but in this embodiment, the direction of guidance is different from that of the second embodiment, and will be described below.

  In the present embodiment, the guidance generation unit 204 guides the user of the information processing apparatus 200 in the direction in which the direction of the virtual object presentation area is directed and the position and orientation trajectory extends in the measurement space. With respect to the latter, it is only necessary to guide away from the initial position and orientation and increase the area of the circumscribed polygon of the locus of the position and orientation.

In the present embodiment, a method for determining the direction when the guidance generation unit 204 guides the virtual object presentation area from the current position and orientation will be described with reference to FIG. 10, when the center coordinates of the virtual object presentation area and T _m, the direction vector m _t facing the virtual object presentation area from the current position is represented by the formula (2).

m _t = T _m −T _t (2)
At this time, the direction to be induced is determined by the angle [rho _t from v _t to m _t. If _ρt is positive, the guidance direction is left, and if ρt is negative, the guidance direction is right. That is, the induction generator 204, the left direction of the arrow if it is positive the [rho _t may be generated right direction of arrow as guiding instruction image 80 if it is negative.

  FIG. 11 shows an example of a guidance instruction image 80 according to the present embodiment and an output image 90 on which the guidance instruction image 80 is superimposed. In FIG. 11, two types of arrows indicating the direction of guidance are drawn as the guidance instruction image 80. In FIG. 11, the black arrow indicates the direction of the virtual object presentation area, and prompts the user of the information processing apparatus 200 to visually recognize the direction. Moreover, the white arrow has shown the moving direction for producing | generating a three-dimensional map, and has shown the direction which the user of the information processing apparatus 200 should move. That is, the user of the information processing device 200 can easily generate an optimal three-dimensional map by moving in the direction indicated by the white arrow while facing the direction indicated by the black arrow. Understandable. Further, a virtual object to be actually presented may be superimposed on a part of the guidance instruction image 80 and displayed.

<Processing>
Control of the information processing apparatus 200 according to this embodiment having the above configuration will be described below. The flowchart showing the procedure of the process performed by the information processing apparatus 200 according to the present embodiment is almost the same as the flowchart of the second embodiment shown in FIG. 8, but the contents of the process are different from the second embodiment. Exists. Therefore, hereinafter, different parts from the second embodiment will be mainly described, and description of the same parts will be omitted.

  In step S1040, the guidance generation unit 204 generates a guidance instruction image 80 corresponding to the operation mode of the determination unit 203. When the operation mode of the determination unit 203 is the three-dimensional map generation mode, a guidance instruction image 80 for guiding to a position and orientation suitable for observing a virtual object presented during the MR experience is generated. On the other hand, when the mode of the determination unit 203 is the closed path determination mode, the guidance instruction image 40 or 60 that brings the photographing unit 101 to the initial position and orientation is generated as in the first and second embodiments. When the mode of the determination unit 203 is the 3D map generation mode, the guidance output unit 205 generates the output image 90 by superimposing the guidance instruction image 80 on the input image 10.

  In step S1050, the determination unit 203 performs different determination processing depending on the operation mode. If the operation mode is the 3D map generation mode, a 3D map generation end determination process is performed. If the end determination of the 3D map generation is successful, the determination unit 203 shifts its operation mode to the closed path determination mode. If the operation mode is the 3D map generation mode, the process returns to step S1025 after the 3D map generation end determination process regardless of the result. On the other hand, when the operation mode is the closed path determination mode, the closed path determination process is performed as in the first and second embodiments. If the determination is successful, the process proceeds to step S1060; otherwise, the process returns to step S1025.

  In step S1080, the guidance generation unit 204 generates different guidance instruction images 40, 60, 80, etc. depending on whether the operation mode of the determination unit 203 is the 3D map generation mode or the closed route determination mode, This is sent to the guidance output unit 205.

  As described above, according to the present embodiment, by using the position and orientation calculated from time to time and a 3D map, a user of the information processing device 200 is first displayed during MR experience. It is possible to guide to an appropriate position and orientation for observing the object. In addition, it is possible to guide to a position and orientation for configuring a closed path to which the loop closing process can be applied. By applying the loop closing process, it is possible to generate a highly accurate three-dimensional map suitable for position and orientation estimation during MR experience.

[Modification 17]
The virtual object presentation area may be specified by any method as long as it can specify an area in which a virtual object is supposed to be presented in the space. In this embodiment, a numerical value is input in advance. However, for example, a marker having a known shape or pattern may be arranged in the space, and the center coordinates may be automatically calculated from the marker image shown in the input image 10.

[Modification 18]
The guidance generation unit 204 may refer to a three-dimensional map at the time of guidance generation to the user of the device and guide the vehicle so as to avoid a direction in which it cannot move due to the presence of an obstacle or the like.

(Embodiment 4)
<Overview>
In the present embodiment, an information processing apparatus that can be applied to an MR (or AR) experience while generating a highly accurate three-dimensional map will be described. A user of the information processing apparatus 200 can perform an MR experience while wearing and moving an HMD incorporating the photographing unit 101 and the display unit 102. At this time, the information processing apparatus of the present invention generates a 3D map using a group of images photographed by the camera, and simultaneously calculates the position and orientation of the HMD using the 3D map generated so far. The information processing apparatus presents a virtual image to the user by computer graphics (CG) from the calculated position and orientation of the HMD. The information processing apparatus uses the estimated position and orientation to guide the user so that the loop can be closed. Specifically, an image showing the direction from the current position and orientation to the initial position and orientation that is the starting point of the route is presented to the user. While viewing this image, the user moves so that the position and orientation of the mounted HMD substantially coincide with the initial position and orientation.

  When the information processing apparatus determines that the HMD has almost reached the initial position and orientation, it performs a loop close process. Thereby, the information processing apparatus performs optimization so that the position and orientation of the start point and the end point of the route coincide with each other, and the three-dimensional map generated so far is corrected and highly accurate.

<System configuration>
FIG. 12 is a diagram illustrating a system configuration according to the fourth embodiment. A rectangular frame in FIG. 12 indicates a functional module that performs each process of the present embodiment, and an arrow indicates a data flow. The system according to the present embodiment includes an information processing apparatus 200, a photographing unit 101 connected thereto, a display unit 102, a virtual image generation unit 103, and a virtual image output unit 104. The configuration of FIG. 12 is an example, and is not limited to that illustrated. In addition, different parts from the second embodiment will be mainly described, and description of the same parts will be omitted.

<Configuration of information processing apparatus>
In the present embodiment, a virtual image generation unit 103 is provided outside the information processing apparatus 200. The virtual image generation unit 103 receives the position and orientation of the HMD from the position and orientation acquisition unit 208, and performs a process of generating a virtual image to be presented to the user who is performing the MR experience. More specifically, the virtual image generation unit 103 draws a virtual object by computer graphics using the position and orientation of the HMD acquired by the position and orientation acquisition unit 208 as a viewpoint, and generates a virtual image 100. The virtual image 100 generated in this way is sent to the virtual image output unit 104.

  Further, in the present embodiment, a virtual image output unit 104 is provided outside the information processing apparatus 200. The virtual image output unit 104 receives the virtual image 100 from the virtual image generation unit 103 and the output image 70 from the guidance output unit 205. The virtual image output unit 104 generates a final output image 110 by superimposing the virtual image 100 on the output image 70. The output image 110 is sent to the display unit 102 built in the HMD and presented to the user of this apparatus.

  FIG. 13 is a diagram illustrating an example of the final output image 110 output from the virtual image output unit 104. Unlike the output image 70 in the second embodiment, a virtual image 100 in which a virtual object (a chair in this example) is further drawn by CG from the estimated position and orientation is superimposed on the output image 70. That is, the user of the present apparatus refers to the guidance instruction image 40 displayed at the same time while observing the virtual image 100 during the MR experience, so that a position for configuring a closed path to which the loop closing process can be applied. Guided to posture.

  Furthermore, in this embodiment, a three-dimensional map correction unit 209 is provided. When the determination unit 203 determines that the closed route is configured, the 3D map correction unit 209 corrects the 3D map held in the state storage unit 207. The 3D map correcting unit 209 corrects the 3D map by performing a loop closing process and optimizing the 3D map.

<Processing>
Control of the information processing apparatus 200 according to this embodiment having the above configuration will be described below. FIG. 14 is a flowchart illustrating a procedure of processing performed by the information processing apparatus 200 according to the present embodiment. Compared to the second embodiment, the process of step S1030 is changed to step S1035, and the process of step S1060 is changed to step S1065. Also, step S1070 is added.

  In step S <b> 1035, the guidance generation unit 204 generates the guidance instruction image 60. The guidance generation unit 204 sends the generated guidance instruction image 60 to the guidance output unit 205. At the same time, the virtual image generation unit 103 draws a virtual object by CG from the position and orientation acquired in step S1025, and generates a virtual image 100. The virtual image generation unit 103 sends the generated virtual image 100 to the virtual image output unit 104. In step S <b> 1035, the guidance output unit 205 superimposes the guidance instruction image 60 on the input image 10, generates the output image 70, and sends it to the virtual image output unit 211. The virtual image output unit 104 generates a final output image 110 by superimposing the virtual image 100 on the output image 90 and sends it to the display unit 102.

  In step S <b> 1065, the 3D map correction unit 209 corrects the 3D map held in the state storage unit 207 when the determination unit 203 determines that the closed route is configured. The 3D map correcting unit 209 corrects the 3D map by performing a loop closing process and optimizing the 3D map. The corrected three-dimensional map is held in the state storage unit 207, and the next time the process of step S1025 is performed, the three-dimensional map corrected in step S1070 and improved in accuracy is used.

  In step S1070, when the user of the apparatus ends the MR experience, the process of the information processing apparatus 200 according to the present embodiment ends. Otherwise, the process returns to step S1025.

  As described above, according to the present embodiment, the user of the apparatus can be guided to a position and orientation for configuring a closed path to which the loop closing process can be applied during the MR experience. . Further, by applying a loop close process to the generated three-dimensional map, the three-dimensional map is corrected and highly accurate. As a result, the position and orientation of the HMD worn by the user who is experiencing the MR can be estimated with high accuracy, and a correct virtual image without positional deviation can be generated.

<Effect of Example>
According to the first embodiment, the user of the apparatus can be guided to a position and orientation for configuring a closed path to which the loop closing process can be applied. In addition, it is possible to generate a highly accurate three-dimensional map by applying a loop close process to the acquired image sequence.

  According to the second embodiment, the user of the apparatus can be guided to a position and orientation for configuring a closed path to which the loop closing process can be applied. At this time, by presenting a specific moving direction to the user, the user can reach a position and orientation for configuring the closed path more efficiently. Further, by applying a loop close process to the generated 3D map, it is possible to generate a 3D map with higher accuracy.

  According to the third embodiment, it is possible to guide to a position and orientation for capturing an image sequence for generating a three-dimensional map suitable for position and orientation estimation during MR experience. Furthermore, after shooting such an image sequence, it is possible to guide to a position and orientation for configuring a closed path to which the loop close process can be applied. By applying the loop closing process, it is possible to generate a three-dimensional map suitable for position and orientation estimation during MR experience with high accuracy.

  According to the fourth embodiment, it is possible to guide the user of the apparatus to a position and orientation for configuring a closed path to which the loop close process can be applied during the MR experience. Further, by applying a loop close process to the generated three-dimensional map, the three-dimensional map is corrected and highly accurate. As a result, the position and orientation of the HMD worn by the user who is experiencing the MR can be estimated with high accuracy, and a correct virtual image without positional deviation can be generated.

<Definition>
The image acquisition unit 201 according to the present invention may be anything as long as the image captured by the imaging unit 101 can be input to the information processing apparatus 200. The input image may be any type of image such as an RGB color image, a gray scale image, a black and white image, or a depth image.

  The image storage unit 202 in the present invention may be anything as long as it can hold data such as images and parameters while the information processing apparatus 200 is operating. Semiconductor storage devices such as RAM and flash memory, and auxiliary storage devices such as hard disks and solid state disks can be used.

  The guidance generation unit 204 and the guidance output unit 205 according to the present invention may be anything as long as they prompt the user to change the position / posture of the photographing unit 101 to approximately the same position / posture as the initial position / posture. In the first to third embodiments, the method for generating and outputting the guidance instruction image has been described. In addition, a notification method such as voice can be used, and vibration or tactile sense may be presented to the user of the information processing apparatus 200. Further, a configuration in which an operation instruction is given to the robot apparatus that holds the photographing unit 101 is also possible.

  In addition, the guidance generation unit 204 according to the present invention may generate not only a prompt for changing the position and orientation of the photographing unit 101 to a position and orientation substantially the same as the initial position and orientation, but also a further guidance. In the third embodiment and the modified example 11, the position and orientation of the photographing unit 101 are changed to the same position and orientation as the initial position and orientation while performing guidance in a direction to improve the density or accuracy of the generated three-dimensional map. I explained how to encourage you.

  The determination unit 203 according to the present invention may be anything as long as it can determine whether or not the captured image in the specified state and the captured image in the other state are captured at substantially the same position and orientation.

  In the first embodiment, the method of determining the similarity between images by BoW has been described. As described in the second modification, the geometric transformation parameter may be calculated from the corresponding feature points between the images, and the similarity may be calculated between the images to which the geometric transformation parameter is applied.

  Further, the determination unit 203 according to the present invention performs determination by adding other conditions in addition to determining whether the captured image in the specified state and the captured image in the other state are captured at substantially the same position and orientation. Also good. In the above-described modified examples 12 and 13, the method of performing the determination by adding further conditions was described in order to improve the properties of the generated three-dimensional map.

  The 3D map generation unit 206 according to the present invention may be anything as long as it calculates 3D coordinates of the target space using a plurality of captured images. In the above-described embodiment, the method using SfM or SLAM has been described. As described in the fifth modification, the position / orientation sensor may be used supplementarily, or as described in the sixth modification, a depth image may be input and used.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  101: imaging unit, 102: display unit, 200: information processing device, 201: image acquisition unit, 202: image storage unit, 203: determination unit, 204: guidance generation unit, 205: guidance output unit, 206: three-dimensional map Generation unit, 207: state storage unit, 208: position and orientation acquisition unit

Claims (16)

Acquisition means for acquiring captured images captured by the imaging unit at a plurality of positions and orientations;
Generating means for generating a guidance instruction for acquiring a captured image corresponding to substantially the same position and orientation as the captured image in the prescribed state;
Output means for outputting the guidance instruction;
A determining unit that determines whether one captured image acquired by the acquiring unit after the output of the guidance instruction is captured in substantially the same position and orientation as the captured image in the specified state;
When it is determined that the images are taken at the substantially same position and orientation, a third order is obtained from the three-dimensional coordinates of the features included in the photographed image based on the plurality of photographed images including the one photographed image photographed at the plurality of positions and orientations. Map generation means for generating or updating the original map;
An information processing apparatus comprising:   The information processing apparatus according to claim 1, wherein the generation unit generates an image including the photographed image in the specified state as the guidance instruction.   3. The determination unit according to claim 1, wherein the determination unit determines whether or not the image is captured at the substantially same position and orientation based on a similarity between the one captured image and the captured image in the specified state. The information processing apparatus described in 1.   Whether the determination unit has photographed at substantially the same position and orientation based on a geometric transformation parameter between images acquired from the feature included in the one captured image and the feature included in the captured image in the specified state. The information processing apparatus according to claim 1, wherein it is determined whether or not.   The information processing apparatus according to claim 1, further comprising an image storage unit that stores the photographed image in the specified state. Position and orientation acquisition means for acquiring the position and orientation at which the captured image acquired by the acquisition means was captured;
State storage means for storing a specified position and orientation corresponding to the specified state;
The information processing apparatus according to claim 1, wherein the generation unit generates a guidance instruction for acquiring a captured image corresponding to a position and orientation that is substantially the same as the specified position and orientation. .   The said production | generation means produces | generates the guidance instruction | indication which contains the bird's-eye view of the measurement space containing the said prescription | regulation position and orientation and the locus | trajectory of several position and orientation acquired by the said position and orientation acquisition means. Information processing device.   The generation unit includes a guidance direction from the position and orientation acquired by the position and orientation acquisition unit to the specified position and orientation based on the specified position and orientation and the position and orientation acquired by the position and orientation acquisition unit. The information processing apparatus according to claim 6, wherein a guidance instruction is generated. The state storage means further stores a plurality of positions and orientation trajectories corresponding to the plurality of captured images acquired by the acquisition means,
The said generation | production means produces | generates the said guidance instruction | indication so that the spatial expansion of the locus | trajectory of the said position / orientation with respect to measurement space may expand based on the locus | trajectory of the said position / orientation. The information processing apparatus according to any one of claims.   The determination means is substantially the same as the above based on at least one of the Euclidean distance of each position between the specified position and orientation and the position and orientation acquired by the position and orientation acquisition means, and an angle formed by each orientation vector. The information processing apparatus according to any one of claims 6 to 9, wherein it is determined whether or not the image is taken in a position and orientation.   The said determination means further calculates the estimation accuracy of the said three-dimensional map, and determines whether it image | photographed with the said substantially same position and orientation based on the said estimation accuracy. The information processing apparatus according to any one of claims.   The information processing apparatus according to claim 1, further comprising an end determination unit that determines the end of generation of the three-dimensional map by the map generation unit. Acquisition means for acquiring captured images captured by the imaging unit at a plurality of positions and orientations;
Map generating means for generating a three-dimensional map from three-dimensional coordinates of features included in the photographed image based on at least one photographed image photographed at the plurality of positions and orientations;
Generating means for generating a guidance instruction for acquiring a captured image corresponding to substantially the same position and orientation as the captured image in the prescribed state;
Output means for outputting the guidance instruction;
A determining unit that determines whether one captured image acquired by the acquiring unit after the output of the guidance instruction is captured in substantially the same position and orientation as the captured image in the specified state;
An information processing apparatus comprising: a map correcting unit that corrects the three-dimensional map when it is determined that the images are photographed at substantially the same position and orientation. A method for controlling an information processing apparatus,
An acquisition step in which an acquisition unit acquires captured images captured by the imaging unit at a plurality of positions and orientations;
A generating step for generating a guidance instruction for acquiring a captured image corresponding to a position and orientation substantially the same as the captured image in the prescribed state;
An output step in which the output means outputs the guidance instruction;
A determination step of determining whether or not one captured image acquired by the acquisition step after the output of the guidance instruction is captured at substantially the same position and orientation as the captured image in the specified state;
When it is determined that the map generation unit is photographed at the substantially same position and orientation, the feature of the feature included in the photographed image is based on a plurality of photographed images including the one photographed image photographed at the plurality of positions and orientations. A map generation process for generating a 3D map from 3D coordinates;
A method for controlling an information processing apparatus, comprising: A method for controlling an information processing apparatus,
An acquisition step in which an acquisition unit acquires captured images captured by the imaging unit at a plurality of positions and orientations;
A map generating step for generating a three-dimensional map from three-dimensional coordinates of features included in the photographed image based on at least one photographed image photographed at the plurality of positions and orientations;
A generating step for generating a guidance instruction for acquiring a captured image corresponding to a position and orientation substantially the same as the captured image in the prescribed state;
An output step in which the output means outputs the guidance instruction;
A determination step of determining whether or not one captured image acquired by the acquisition step after the output of the guidance instruction is captured at substantially the same position and orientation as the captured image in the specified state;
And a map correction step of correcting the three-dimensional map when it is determined that the map correction means is photographed at the substantially same position and orientation.   A program for causing a computer to function as the information processing apparatus according to any one of claims 1 to 13.

Images