JP7190842B2 - Information processing device, control method and program for information processing device - Google Patents

Description

The present invention relates to an information processing device, a control method for an information processing device, and a program.

Measurement of the position and orientation of an imaging device based on image information is used for various purposes such as registration of real space and virtual objects in mixed reality/augmented reality, self-position estimation of robots and automobiles, and 3D modeling of objects and spaces. used in

Non-Patent Document 1 describes a method of estimating geometric information (depth information), which is an index for calculating a position and orientation from an image, using a learning model learned in advance, and calculating the position and orientation based on the estimated depth information. is disclosed.

K. Tateno, F.; Tombari, I.; Laina andN. Navab, "CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction", IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR7), 201VPR7. Z. Zhang, "A flexible new technique for camera calibration," IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp. 1330-1334, 2000. J. Engel, T. Schps, andD. Cremers. LSD-SLAM: Large-Scale Direct Monocular SLAM. In European Conference on Computer Vision (ECCV), 2014. E. Shelhamer, J.; Long and T. Darrell, "Fully Convolutional Networks for Semantic Segmentation", Transaction on Pattern Analysis and Machine Intelligence (PAMI), Vol. 39, pp. 640-651, 2017 A. Kendall, M.; Grimes and R. Cipolla, "PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization", International Conference on Computer Vision (ICCV), 2015, pp. 2938-2946 S. Holzer, V.; Lepetit, S.; Ilic, S.; Hinterstoisser, N.; Navab, C.I. Cagniart and K. Konolige, "Multimodal Templates for Real-Time Detection of Texture-less Objects in Heavily Cluttered Scenes" In European Conference on Computer Vision (ECCV), 2011. J. Engel, T. Schps, andD. Cremers. LSD-SLAM: Large-Scale Direct Monocular SLAM. In European Conference on Computer Vision (ECCV), 2014.

In Non-Patent Document 1, there is a premise that a scene in which a learning image used for learning a learning model is captured is similar to a scene in an input image captured by an imaging device. Therefore, there has been a demand for a solution for improving the accuracy of estimating geometric information even when scenes are dissimilar.

SUMMARY An advantage of some aspects of the invention is to provide a technique for obtaining a position and orientation with high accuracy.

An information processing apparatus according to the present invention that achieves the above object includes:
A captured image that is learned using a captured image captured by an imaging device and depth information of the captured image as teacher data, and that uses a plurality of learning models for estimating depth information corresponding to an input image as the teacher data. holding means for holding in correspondence with the imaging position of
A scene captured in the input image from the plurality of learning models based on an evaluation result obtained from the degree of matching between any one of the imaging positions held in association with the learning model and the imaging position at which the input image was captured. a selection means for selecting a learning model suitable for
estimating means for estimating first depth information using the input image and the selected learning model;
with
The estimating means further estimates second depth information from the input image by a motion stereo method, further estimates third depth information based on the input image and the learning model for each of the learning models, and selects The means is characterized by calculating the evaluation result of the learning model such that the higher the degree of matching between the second depth information and the third depth information, the higher the evaluation result .

According to the present invention, geometric information can be estimated with high accuracy.

2 is a diagram showing the functional configuration of the information processing device 1 according to Embodiment 1. FIG. 4 is a diagram showing the data structure of a learning model group holding unit according to Embodiment 1. FIG. 2 is a diagram showing the hardware configuration of the information processing device 1 according to Embodiment 1. FIG. 4 is a flow chart showing a processing procedure according to the first embodiment; 4 is a flow chart showing a processing procedure in step S140 in Embodiment 1. FIG. 11 is a flow chart showing a processing procedure in step S1120 in Embodiment 1. FIG. 10 is a flow chart showing a processing procedure in step S1120 in Embodiment 2. FIG. 14 is a flow chart showing a processing procedure in step S1120 in Embodiment 3. FIG. FIG. 21 is a diagram showing an example of a GUI presenting information characterizing a learning model in Embodiment 6; FIG. 10 is a diagram showing the functional configuration of an information processing device 2 according to Embodiment 3; FIG. 20 is a diagram showing a functional configuration of an information processing device 3 in one of modifications of the fifth embodiment; FIG. 14 is a diagram showing a functional configuration of an information processing device 4 according to a ninth embodiment; FIG. 21 is a flow chart showing a processing procedure in Embodiment 9. FIG.

Hereinafter, embodiments will be described with reference to the drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

(Embodiment 1)
In this embodiment, the present invention is applied to the alignment of the real space and the virtual object in the mixed reality system, that is, the measurement of the position and orientation of the imaging device in the real space for use in drawing the virtual object. explain. A user who experiences mixed reality holds a mobile terminal such as a smartphone or tablet, and observes a real space in which virtual objects are superimposed through the display of the mobile terminal. In this embodiment, the mobile terminal is equipped with a monocular RGB camera as an imaging device, and a CG image of a virtual object drawn based on the position and orientation of the camera in the real space is superimposed on the image captured by the camera. presented to the user.

Geometric information estimated by the learning model based on the input image captured by the imaging device is used to calculate the position and orientation of the imaging device. The geometric information estimated by the learning model in this embodiment is a depth map, which is depth information estimated for each pixel of the input image. Also, the learning model is assumed to be a CNN (Convolutional Neural Network). Specifically, an image captured at a certain time t (hereinafter referred to as a current frame) and an image captured at a time t′ before the current frame (hereinafter referred to as a previous frame) are input to the learning model. Each pixel of the previous frame is projected onto the current frame based on the estimated depth information (hereinafter referred to as the previous depth map). Projection here means calculating where the pixels of the previous frame appear in the current frame. Specifically, the image coordinates (u _t-1 , v _t-1 ) and camera intrinsic parameters (f _x , f _y , c _x , c _y ) of each pixel in the previous frame, and the pixel coordinates of the previous depth map Using the depth value D, the three-dimensional coordinates (X _t−1 , Y _t−1 , Z _t−1 ) of the pixel in question in the camera coordinate system of the previous frame are calculated by Equation (1).

Next, using t _(t−1)→t , R _(t−1)→ t, which are the position and orientation of the camera that captured the current frame with respect to the position of the camera that captured the previous frame, the camera coordinate system of the current frame The three-dimensional coordinates (X _t , Y _t , Z _t ) of the feature points in are calculated by Equation (2).

Next, the three-dimensional coordinates (X _t , Y _t , Z _t ) of the feature point in the camera coordinate system of the current frame are converted into the image coordinates (u _t , v _t ) of the current frame by Expression 3.

In this embodiment, the processing of Equations 1 to 3 is called projection. Position such that the luminance difference between the luminance value of the pixel (u _t−1 , v _t−1 ) in the previous frame and the luminance value of the pixel (u _t , v _t ) in the current frame of the projection destination is the minimum. and attitudes t _(t−1)→t and R _(t−1)→t are calculated. Finally, using the positions and orientations t _w→(t−1) and R _w→(t−1) of the camera that captured the previous frame with respect to the world coordinate system, the current frame with respect to the world coordinate system was captured using Equation 4. The camera position and orientation t _w→t and R _w→t are calculated.

The learning model is trained in advance so that when an image is input, the corresponding depth map can be estimated based on a plurality of images and a plurality of depth maps of the same field of view captured at the same time. For example, if a learning model trained using training images showing indoor scenes is used, depth maps can be estimated with high accuracy when indoor images are input. However, when outdoor images are input to this learning model, the accuracy of the output depth map decreases. Therefore, in this embodiment, as a method of selecting a learning model capable of calculating geometric information with high accuracy in a scene in which an input image is captured from among a plurality of learning models learned for each scene, the learning image is similar to the input image. Explain how to select a learning model that has Scenes include, for example, indoor scenes, outdoor scenes, Japanese house room scenes, Western house room scenes, office scenes, factory scenes, and the like.

The position and orientation of the imaging device in the present embodiment represent 6 parameters including 3 parameters representing the position of the camera in the world coordinates defined in the real space and 3 parameters representing the orientation of the camera. do. In this embodiment, the position and orientation of the camera are referred to as the camera position and orientation unless otherwise specified. A three-dimensional coordinate system defined on the camera, in which the optical axis of the camera is the Z axis, the horizontal direction of the image is the X axis, and the vertical direction of the image is the Y axis, is called a camera coordinate system.

<Configuration of information processing device>
FIG. 1 is a diagram showing a functional configuration example of an information processing apparatus 1 according to this embodiment. The information processing apparatus 1 includes an image input unit 110 , a learning model selection unit 120 , a learning model group storage unit 130 , a geometric information estimation unit 140 and a position/orientation acquisition unit 150 . The image input unit 110 is connected to the imaging device 11 and the display information generation unit 12 mounted on the mobile terminal. The position/orientation acquisition unit 150 is connected to the display information generation unit 12 . The display information generation section 12 is connected to the display section 13 . However, FIG. 1 is an example of the equipment configuration, and does not limit the scope of application of the present invention. Although the display information generation unit 12 and the display unit 13 are configured outside the information processing device 1 in the example of FIG. Further, the display information generation unit 12 may be included in the information processing device 1 and the display unit 13 may be configured as an external device of the information processing device 1 .

The image input unit 110 inputs the image data of the two-dimensional images of the scene captured by the imaging device 11 in time series (for example, 60 frames per second). , to the display information generator 12 .

The learning model selection unit 120 selects each learning model held by the learning model group holding unit 130 based on the input image input by the image input unit 110 and outputs the selection result to the geometric information estimation unit 140 .

The learning model group holding unit 130 holds a plurality of learning models. Details of the data structure will be described later. The geometric information estimation unit 140 inputs the input image input by the image input unit 110 to the learning model selected by the learning model selection unit 120, and estimates geometric information. Also, the estimated geometric information is output to the position and orientation acquisition unit 150 .

The position and orientation acquisition unit 150 calculates and acquires the position and orientation of the imaging device based on the input image input by the image input unit 110 and the geometric information input by the geometric information estimation unit 140 . Then, the acquired position and orientation information is output to the display information generation unit 12 .

The display information generation unit 12 renders a CG image of the virtual object using the position and orientation acquired from the position and orientation acquisition unit 150 and internal/external parameters of the camera held by a holding unit (not shown). Then, a composite image is generated by superimposing the CG image on the input image input by the image input unit 110 . Also, the composite image is output to the display unit 13 . The display unit 13 is a display of the mobile terminal, and displays the composite image generated by the display information generation unit 12 .

2 is a diagram showing the data structure of the learning model group holding unit 130. As shown in FIG. In this embodiment, at least two learning models are held. At least one learning image used for learning of the learning model is held for each learning model. The learning model is assumed to be a data file in which, for example, a CNN (Convolutional Neural Network) discriminator is saved in binary format.

FIG. 3 is a diagram showing the hardware configuration of the information processing device 1. As shown in FIG. H11 is a CPU, which controls various devices connected to the system bus H20. H12 is a ROM that stores a BIOS (Basic Input/Output System) program and a boot program. H13 is a RAM, which is used as a main storage device for H11, which is a CPU. H14 is an external memory, which stores programs to be processed by the information processing apparatus 1. FIG. The input unit H15 is a keyboard, a mouse, and a robot controller, and performs processing related to input of information and the like. The display unit H16 outputs the calculation result of the information processing device 1 to the display device according to the instruction from H11. The display device may be of any type, such as a liquid crystal display device, a projector, or an LED indicator. Alternatively, it may be the display unit 13 . H17 is a communication interface, which performs information communication via a network, and the communication interface may be Ethernet (registered trademark), USB, serial communication, wireless communication, or the like. An input/output unit (I/O) H18 is connected to the camera H19. Note that the camera H19 corresponds to the imaging device 11 .

<Processing>
Next, a processing procedure in this embodiment will be described. FIG. 4 is a flowchart showing a processing procedure performed by an information processing system including the information processing apparatus 1 according to this embodiment.

In step S110, the system is initialized. That is, the program is read from the external memory H14 and the information processing apparatus 1 is made operable. Also, the parameters of each device (such as the imaging device 11) connected to the information processing device 1 and the initial position and orientation of the imaging device 11 are read. Internal parameters of the imaging device 11 (focal length f _x (horizontal direction of image), f _y (vertical direction of image), image center position c _x (horizontal direction of image), c _y (vertical direction of image), lens distortion parameters) are calibrated in advance by Zhang's method described in Non-Patent Document 2.

In step S<b>120 , the imaging device 11 shoots a scene and outputs it to the image input unit 110 .

In step S130, the image input unit 110 acquires an image including a scene captured by the imaging device 11 as an input image. Note that in the present embodiment, the input image is an RGB image.

In step S140, the learning model selection unit 120 uses the learning image held by the learning model group holding unit 130 to calculate the evaluation value of each learning model, and selects a learning model based on the calculated evaluation value. . Details of the evaluation value calculation process in step S140 will be described later with reference to FIGS.

In step S150, the geometric information estimation unit 140 estimates geometric information using the learning model selected in step S140. Specifically, the learning model selection unit 120 inputs the input image to the learning model and estimates a depth map, which is geometric information. In this embodiment, the previous frame image is input to the learning model to estimate the previous depth map.

In step S160, the position and orientation acquisition unit 150 calculates and acquires the position and orientation of the imaging device 11 using the geometric information (depth map) calculated in step S150. Specifically, first, each pixel of the previous frame is projected onto the current frame based on the depth map estimated by the learning model. Next, the position and orientation are calculated by the method of Engel et al. The information generation unit 12 renders the CG image of the virtual object using the position and orientation of the imaging device 11 calculated in step S160, generates a composite image superimposed on the input image, and inputs the composite image to the display unit 13. . Also, the display unit 13 displays the synthesized image on the display unit, which is the display of the mobile device.

In step S180, it is determined whether or not to terminate the system. Specifically, if the user has input an end command through an input unit (not shown), the process ends. If not, the process returns to step S120 and continues.

<Learning model selection processing>
Here, FIG. 5 is a flowchart showing the procedure of the learning model selection process of S140 in this embodiment.

In step S1110, learning model selection unit 120 determines whether or not it has already been decided which learning model to use among the learning models held by learning model group holding unit . If the learning model to be used has not been determined, the process proceeds to S1120. On the other hand, if the learning model to be used has already been determined, the process ends.

In step S1120, the learning model selection unit 120 calculates the evaluation value of each learning model held by the learning model group holding unit 130 by similar image search between the input image and the learning image. The evaluation value is the degree of similarity between the input image captured by the imaging device 11 and the learning image, and represents the degree of adaptation of the learning model to the scene captured by the imaging device. In this embodiment, the evaluation value is a continuous value from 0 to 1, and the higher the value is 1, the higher the degree of conformity. Details of the evaluation value calculation process will be described later with reference to FIG.

In step S1130, learning model selection section 120 selects a learning model including a learning image with the maximum evaluation value calculated in S1120. At this time, the learning model is read into a storage unit such as H13, which is a RAM, so that the geometric information estimation unit 140 can estimate the geometric information.

<Evaluation value calculation processing>
Next, FIG. 6 is a flowchart showing the detailed procedure of the learning model evaluation value calculation process of S1120 in this embodiment.

In step S1210, the learning model selection unit 120 loads the learning image used for learning of the learning model for which the evaluation value has not been calculated from the learning model group holding unit 130 to the RAM H13.

In step S1220, learning model selection section 120 calculates the evaluation value of the learning model based on the degree of similarity between the input image and the learning image. In this embodiment, a pHash method is used in which an image is reduced, discrete cosine transform is performed on luminance information of the image, and a hash value of low-frequency components is calculated. First, the learning model selection unit 120 calculates the hash values of the input image and the learning image, and calculates the Hamming distance between them. The degree of similarity between the input image and the learning model in this embodiment is the Hamming distance. A continuous value obtained by normalizing the Hamming distance of each model from 0 to 1 is calculated using the maximum value among the calculated Hamming distances of each model. In this embodiment, this normalized value is used as the evaluation value of each learning model.

In step S1230, learning model selection section 120 determines whether evaluation values for all learning models have been calculated. If the calculation of all evaluation values has been completed, the process ends. On the other hand, if there is a learning model whose evaluation value has not been calculated yet, the process returns to step S1210. However, it is not necessary to evaluate all the learning models held by the learning model group holding unit 130, and the evaluation value may be calculated only for the top N (N is an integer) learning models with high usage frequency. . In such a case, step S1230 determines whether evaluation of the top N learning models has been completed.

<effect>
As described above, in the first embodiment, an evaluation value is calculated for each learning model from among a plurality of learning models, and a learning model with a high evaluation value is selected. At this time, the degree of similarity between the input image and the learning image used in learning each learning model is calculated, and a high evaluation value is assigned to each learning model having a high degree of similarity. Then, the position and orientation of the imaging device are calculated using geometric information estimated using a learning model with a high evaluation value. In this way, by selecting a learning model whose input image and learning image are similar, the learning model can estimate geometric information with high accuracy. Therefore, for example, when the position and orientation of the imaging device are obtained using this estimated geometric information, the position and orientation of the imaging device can be calculated with high accuracy. Other uses of the estimated geometric information include, for example, image recognition in automatic driving of automobiles, which will be described later.

Furthermore, by selecting and using a small-scale learning model trained for each individual scene, geometric information can be estimated even on a computer with a small memory capacity. Accordingly, the position and orientation of the imaging device can be calculated even in the mobile terminal.

Furthermore, by selecting and using a small-scale learning model trained for each individual scene, geometric information can be estimated with less execution time than when using a large-scale learning model. As a result, the position and orientation of the imaging device can be calculated in a short time.

<Modification>
In the first embodiment, the learning model group holding unit 130 holds learning images used for learning the learning models. However, the images to be held are not limited to the learning images themselves as long as they are images that can characterize the learning model. For example, it may be an image obtained by cutting down a learning image, an image obtained by cutting a part of the learning image, or an image similar to the learning image.

In the first embodiment, the pHash method is used as the learning model evaluation method. However, the evaluation method is not limited to the pHash method, and any method that can calculate the degree of similarity between a learning image used for learning a learning model and an input image may be used. Specifically, the degree of similarity between color histograms calculated from the input image and the learning image may be used. Alternatively, a Hog (Histgram of Oriented Gradients) feature amount obtained by forming a histogram of the gradient directions of luminance in local regions of the input image and the learning image may be calculated and the similarity of the Hog feature amount may be used. Alternatively, the screen may be divided into small regions, and the GIST feature similarity may be used in which responses obtained by applying Gabor filters in a plurality of directions and frequencies to the small regions are used as feature amounts.

Also, the number of similar feature amounts of local features detected from each of the input image and the learning image may be used as the evaluation value. As the local feature, for example, a SIFT feature point having a gradient orientation histogram in a local region of a smoothed image as a feature quantity may be used. Alternatively, an ORB feature point may be used as a feature amount by generating a binary code from the magnitude of brightness of two points in a local image region. Further, the local feature may be an image feature amount obtained by calculating a characteristic position such as a corner in the image by the Harris corner detection method and using surrounding color information as a feature amount, or may be an image feature amount using a template of a surrounding small area as a feature amount. may be Furthermore, character information detected by character recognition may be used as a local feature. A plurality of types of image features may be used in combination.

Furthermore, the similarity of Bag-of-Visual Words converted into a histogram by vector quantization of local features may be used as an evaluation value. In addition, the Bag-of-Visual Words are calculated from the learning images for each learning model in advance, and an SVN (support vector machine) is used to calculate the discrimination boundary that maximizes the distance from the feature space where each feature value exists. The discriminator used may be used. At this time, the learning model identified by the SVN may be voted for using the feature amount calculated from the input image as an input, and the learning model with the largest number of votes may be selected. Corresponds to the output label when the input image is input to the neural network proposed by Shelhamer et al. You may select a learning model that

The input image is not limited to one image, and may be a plurality of images, and the sum, maximum value, minimum value, average value, or median of similarities between each input image and learning image may be used as evaluation values.

In the first embodiment, the learning model group holding unit 130 holds learning images, and calculates hash values and feature amounts when calculating evaluation values. However, hash values and feature amounts may be calculated from learning images in advance, and the learning model group holding unit 130 may hold them. This eliminates the need to calculate hash values and feature quantities from learning images when selecting a learning model, and allows a learning model to be selected in a short period of time.

In the first embodiment, the learning models held by the learning model group holding unit 130 are data files in which classifiers are output in binary format. However, as long as the geometric information estimation unit 140 holds the geometric information in a format that allows the geometric information to be estimated, the CNN network structure and weights may be held in the output data file. Alternatively, the network structure may be determined in advance, and only the weights thereof may be extracted and stored as an output data file. By extracting only the weights, the size of the data file can be made smaller than when the discriminator itself is output in binary format.

In Embodiment 1, the evaluation value is calculated based on the degree of similarity between the learning image and the input image. However, the evaluation value calculation method may be any method as long as it can give a high evaluation value if the scenes in which the learning image and the input image are photographed are similar. For example, scene information in which the learning image and the input image were captured is detected, and the evaluation value is calculated based on the degree of matching of the scene information. Scene information refers to scene categories such as indoors, outdoors, coasts, mountains, and roads. Scene information is detected using a scene discrimination learning model that discriminates scene categories. For example, the scene discrimination learning model is a neural network trained using deep learning so that if the input image is in the relevant category, 1 is output, otherwise 0 is output. In addition, a plurality of local features are detected from one image, and GLC features are calculated by listing averages and correlation values of the feature quantities. Then, the category of the scene may be determined by SVN (Support Vector Machine) that calculates a discrimination boundary that maximizes the distance between the feature spaces in which the GLC features of each category exist. Note that GLC is an abbreviation for generalized local correlation.

In the first embodiment, each pixel of the previous frame is projected onto the current frame based on the depth map estimated by the geometric information estimation unit 140 using the learning model, and the pixel value of the projected pixel of the previous frame and the pixel value of the current frame are calculated. The position and orientation were calculated so that the luminance difference between the However, any method may be used as long as it acquires the position and orientation based on the output of the learning model. For example, the geometric information estimating unit estimates the depth maps of the previous frame and the current frame using the learning model. Then, the ICP method is used to repeatedly calculate the position and orientation so as to minimize the distance between the three-dimensional point of each pixel in the current depth map and the closest point among the three-dimensional points of each pixel in the previous depth map. Calculate posture. ICP is an abbreviation for Iterative Closest Point. In addition, the position/orientation acquisition unit 150 calculates local features from the previous frame and the current frame, and obtains corresponding points that point to structures where they match. Then, the position and orientation may be calculated by solving the PnP problem so as to minimize the distance between corresponding points when the local features of the previous frame are projected onto the current frame using the depth values of the depth map. Note that the geometric information estimation unit may be configured to indirectly use the output of the learning model. That is, the position and orientation may be calculated based on geometric information obtained by correcting the output of the learning model by time-series filtering (described in Non-Patent Document 1).

In Embodiment 1, the geometric information estimated by the learning model was a depth map. However, the learning model that can be applied to the present embodiment may be any model that can calculate the position and orientation of the imaging device based on the output geometric information. For example, a learning model that calculates, as geometric information, saliency points for use in acquiring the position and orientation from the input image may be used. In this case, the geometric information estimation unit 140 estimates saliency points from the previous frame and the current frame using a learning model, and the position/orientation acquisition unit uses a 5-point algorithm based on the saliency points that indicate matching structures in the previous frame and the current frame. method is used to calculate the pose. Alternatively, a learning model (Non-Patent Document 5) trained to input two images of a previous frame and a current frame and estimate six degrees of freedom of position and orientation between them as geometric information may be used. At this time, instead of calculating the position and orientation by the position and orientation acquisition section 150, the position and orientation estimated by the geometric information estimation section 140 using the learning model may be directly input to the display information generation section 12 as the position and orientation acquisition result. .

In the first embodiment, an example has been described in which the present invention is applied to the registration application between the physical space and the virtual object in the mixed reality system. However, what is applicable to the information processing apparatus 1 described in the present embodiment is not limited to the application, and any application that uses the geometric information output by the learning model or the position/orientation acquisition result may be used. For example, the information processing apparatus 1 may be attached to an autonomous mobile robot or automobile and used as an autonomous mobile system that estimates the self-position of the robot or automobile. The independent movement system at this time may include a movement mechanism such as an electric motor, and a control unit that determines actions based on the position and orientation calculated by the position and orientation acquisition unit 150 and controls the movement mechanism. Moreover, it may be used as a robot system that is attached to the tip of an industrial robot hand to measure the position and orientation of the robot hand. The robot system at this time may include a manipulator such as a robot arm, a grasping device such as a suction hand, and a control unit that controls the manipulator and the grasping device based on the position and orientation calculated by the position and orientation acquisition unit 150. good.

Further, the application of the information processing apparatus 1 is not limited to position and orientation estimation, and may be used for three-dimensional reconstruction. For example, it may be used as a measurement system for generating CAD models of industrial parts and buildings. The measurement system at this time may further include a three-dimensional model generation section that generates a three-dimensional model from the geometric information estimated by the geometric information estimation section 140 . Further, it may be used as a device for acquiring a depth map with high accuracy from a camera that cannot acquire a depth map, such as an RGB camera or a camera that acquires a grayscale image.

In the first embodiment, the configuration in which the mobile device has the learning model selection unit 120 and the learning model group holding unit 130 has been described. However, the cloud server may have and execute some of the functions of the information processing apparatus shown in the first embodiment. For example, the cloud server may be configured to have the learning model selection unit 120 and the learning model group holding unit 130 . In this configuration, the mobile terminal first transfers the input image to the cloud server using a communication unit (not shown). Next, the learning model selection unit 120 on the cloud server calculates evaluation values for the learning models held by the learning model group holding unit 130 on the cloud server, and selects a learning model based on the evaluation results. Then, the learning model selected by the cloud server is transferred to the mobile terminal using the communication unit. With such a configuration, mobile devices do not need to store a large number of learning model groups and do not need to perform calculations for learning model selection. The present invention can also be applied to

Further, if the camera that captured the learning images is different from the imaging device 11, the camera parameters of the camera that captured the learning images may also be held for each learning model held by the learning model group holding unit 130. . In this case, based on the camera parameters of the imaging device 11 and the camera parameters of the camera that captured the learning image, the geometric information scale estimated by the geometric information estimation unit 140 using the learning model as in Non-Patent Document 1 is captured. Correct the geometric information to match the device 11 .

In the present embodiment, the configuration in which the imaging device that captures an image is an RGB camera has been described. However, it is not limited to the RGB camera, and any camera that captures an image of the real space may be used. A camera capable of imaging may be used. Also, it may be a monocular camera, or a camera including two or more cameras or sensors.

(Embodiment 2)
In the first embodiment, the degree of similarity between an input image and a learning image used for learning a learning model is calculated as an evaluation value. On the other hand, in the second embodiment, an example will be described in which the evaluation value of the learning model is calculated by comparing the type of object detected from the input image and the type of object detected from the learning image in advance. .

<Configuration of information processing device>
The configuration of the information processing apparatus according to the second embodiment is the same as that shown in FIG. Information held by the learning model group holding unit 130 differs from that in the first embodiment. In this embodiment, the learning model group holding unit 130 holds at least two learning models. Furthermore, it is assumed that each learning model has an object information list in which object information detected in advance from learning images used for learning the model is described. In this embodiment, one object information list is associated with each learning model, and the object information list includes object information such as "desk", "table", "bed", and "chair" (this embodiment In terms of form, the type of object) is assumed to be retained.

<Processing>
The procedure of the entire processing in the second embodiment is the same as that shown in FIG. 4 showing the processing procedure of the information processing apparatus 1 explained in the first embodiment, so the explanation is omitted. Further, the details of step S140 of the learning model in FIG. 4 are the same as those in FIG. 5, so the description is omitted. What differs from the first embodiment is the evaluation value calculation process in which the learning model selection unit calculates the evaluation value of the learning model in step S1120 of FIG. In this embodiment, the processing of FIG. 7 is performed instead of that of FIG.

Hereinafter, the learning model evaluation value calculation process according to the present embodiment will be described with reference to FIG. Note that step S1340 in FIG. 7 is the same as step S1230 in FIG. 6, so description thereof is omitted.

In step S1310, learning model selection section 120 detects an object from the input image, and stores information about the detected object in a detected object type list. For object detection, an object detection learning model that determines the presence or absence of an object is used. This object detection learning model is a neural network trained using deep learning so that if an input image contains a target object, 1 is output, and if not, 0 is output. An object detection learning model is used to detect an object from the input image, and if the target object type is included, the object information is recorded in the detected object type list.

In step S1320, the learning model selection unit 120 loads the object information list linked to the learning model for which the evaluation value has not been calculated from the learning model group holding unit 130 to the RAM H13.

In step S1330, learning model selection section 120 compares the detected object type list detected in step S1310 with the object information list loaded in step S1320, and searches for object information with a matching object type. If a matching object type is found, vote for the learning model that detected that object. Specifically, a memory holding an integer is assigned to each learning model, and if there is matching object information, the memory value is increased by +1. It is assumed that this memory has been initialized to 0 in the initialization step S110 of FIG. Note that the evaluation value in this embodiment is the number of votes.

<effect>
As described above, in the second embodiment, the object information detected from the learning image used for learning the learning model is compared with the object information detected from the input image. A higher evaluation value is assigned to the learning model. Then, the position and orientation of the imaging device are calculated using geometric information estimated using a learning model with a large evaluation value. As a result, it is possible to select a learning model in which the same type of object is shown in the input image and the learning image, and the learning model can estimate the geometric information with high accuracy, and the position and orientation of the imaging device can be determined with high accuracy. can be obtained.

<Modification>
In the second embodiment, an object detection learning model learned in advance by machine learning is used for object detection from learning images and input images. However, any object detection may be used as long as it can determine the presence or absence of an object type. For example, local features may be calculated in advance for each object type, and if the number of local features matching the local features detected from the input image is equal to or greater than a predetermined threshold, it may be determined that the local features have been detected. Alternatively, a template image obtained by clipping an image of an object may be stored in advance, and object detection may be performed by template matching from each of the learning image and the input image.

In the object detection process of the second embodiment, the learning model is given an evaluation value using the binary determination result of whether or not the object type has been detected. However, the evaluation value should be high if the same type of object appears in the input image and the learning image. For example, an object detection learning model trained to output a continuous value of 0 to 1, which is an existence probability, may be used, and an evaluation value may be assigned to the learning model based on these values. Specifically, in step S1330, memory for holding real numbers is allocated to each learning model, and the product of the existence probability of the object type detected from the learning image and the existence probability of the object type detected from the input image is stored in the memory. A value added to the value may be used as the evaluation value.

(Embodiment 3)
In the first embodiment, the degree of similarity between an input image and a learning image used for learning a learning model is calculated as an evaluation value. In the second embodiment, the evaluation value of the learning model is calculated by comparing the object type detected from the input image and the object type detected from the learning image in advance. On the other hand, in the third embodiment, an example will be described in which the evaluation value of the learning model is calculated using the position information of the input image or the learning image obtained by learning the learning model.

<Configuration of information processing device>
As shown in FIG. 10, the information processing apparatus 2 according to the third embodiment further includes a position information acquisition unit 1000 in addition to the configuration of FIG. 1 showing the configuration of the information processing apparatus 1 described in the first embodiment. The location information acquisition unit 1000 receives sensor information such as GPS signals and WiFi signals from a location information acquisition sensor (not shown), and calculates, for example, coordinate values and access point identification IDs as location information. Position information acquisition section 1000 outputs the acquired position information to learning model selection section 120 . Also, the information held by the learning model group holding unit 130 is different from that of the first embodiment. In this embodiment, the learning model group holding unit 130 has at least two learning models and a position information list that describes the position information of the learning image used for learning each model.

<Processing>
The processing procedure in the third embodiment is the same as FIG. 4 showing the processing procedure of the information processing apparatus 1 explained in the first embodiment, so the explanation is omitted. Further, the details of step S140 in FIG. 4 are the same as those in FIG. 5, so the description is omitted. What differs from the first embodiment is the evaluation value calculation process in which the learning model selection unit calculates the evaluation value of the learning model in step S1120 of FIG. In this embodiment, the processing of FIG. 8 is performed instead of that of FIG.

Hereinafter, the learning model evaluation value calculation process according to the present embodiment will be described with reference to FIG. Note that step S1340 in FIG. 8 is the same as step S1230 in FIG. 6, so description thereof is omitted.

In step S1410, the location information acquisition unit 1000 acquires the latitude and longitude and the identification ID of the access point from sensor information such as GPS signals and WiFi signals acquired from the location information acquisition sensor when the learning image held by the learning model group holding unit 130 is captured. Get location information such as

In step S1420, the learning model selection unit 120 loads the position information list associated with the learning model for which the evaluation value has not been calculated from the learning model group holding unit 130 to the RAM H13.

In step S1430, learning model selection section 120 compares the position information acquired in step S1410 with the object information list loaded in step S1420, and searches for matching position information. Specifically, if the identification ID of a WiFi access point is used as location information, when a matching access point identification ID is found, the learning model for which the location information is observed is voted for.

Further, if the latitude and longitude obtained from GPS is used as the position information, the distance between the position information list holding the position information acquired in step S1410 and the position information held by the learning model group holding unit is a predetermined threshold value. Determine if it is within If the coordinates obtained from the position information are in the same country or region, the position information is determined to match, and the learning model is voted for. A memory holding an integer is allocated to each learning model, and if there is matching position information, the memory value is increased by +1. It is assumed that this memory has been initialized to 0 in initialization step S110 in FIG. Note that the evaluation value in this embodiment is the number of votes.

<effect>
As described above, in the third embodiment, a higher evaluation value is given to the learning model as the position information of the input image and the learning image obtained by learning the learning model match. As a result, it is possible to select a learning model in which the positional information of the input image and the learning image match. The position and orientation can be calculated.

<Modification>
In the third embodiment, the latitude and longitude obtained from GPS and the identification ID of the WiFi access point are given as examples of position information. However, any position information may be used as long as the positions at which the input image and the learning image are photographed can be identified. For example, it may be a country name, region name, or address obtained from latitude and longitude, or an identification ID other than a WiFi access point, which will be described later.

As a method for measuring position information, an example is given in which the latitude and longitude are obtained from GPS signals, and the identification ID of an access point is obtained from WiFi signals. However, anything can be used as long as position information can be measured. For example, if a place name appears in the input image, position information may be calculated from the place name. The identification ID of the infrared beacon may be detected using an optical sensor and used as position information, or the identification ID of the ultrasonic beacon may be detected using a microphone and used. An identification ID of a radio beacon based on BLE (Bluetooth (registered trademark) Low Energy) may be detected and used. Also, a base station ID of a 3G or 4G mobile network may be measured and used as location information. Furthermore, only one type of the exemplified position information may be used, or a plurality of types may be used.

(Embodiment 4)
In the first embodiment, the degree of similarity between an input image and a learning image used for learning a learning model is calculated as an evaluation value. In the second embodiment, the evaluation value of the learning model is calculated by comparing the object type detected from the input image and the object type detected from the learning image in advance. In the third embodiment, the evaluation value of the learning model is calculated using the position information of the input image or the learning image used for learning the learning model. However, in the methods described so far, the training model was trained using training images taken at night, whereas the input images show scenes in the daytime, and the learning models were trained using images that were snowing in winter, even though it was spring. For example, a learning model trained using images and a learning model trained using training images on a sunny day, even though it is raining, a learning model trained using learning images with different appearances even at the same shooting location can be selected. However, if a learning model trained on learning images with different appearances is used, the accuracy of the geometric information estimated by the learning model is reduced, making it difficult to acquire the position and orientation with high accuracy. Therefore, in the fourth embodiment, a higher evaluation value is calculated for the learning model as the situation information representing the situation in which the appearance of the image can be changed, such as the shooting date, season, weather, etc. do.

<Configuration of information processing device>
The configuration of the information processing apparatus according to the fourth embodiment is the same as that shown in FIG. Information held by the learning model group holding unit 130 differs from that in the first embodiment. In this embodiment, the learning model group holding unit 130 holds at least two learning models. Furthermore, for each learning model, it holds situation information that describes the situation in which the appearance of the image when the learning image used for learning the model is captured can be changed. In this embodiment, as a specific example of situation information, a case will be described in which date and time information is used as situation information. It is assumed that one situation information list is associated with each learning model, and that the situation information of the photographed learning image is held in the situation information list as the situation information. It is also assumed that the information processing apparatus has an internal clock from which shooting times can be obtained.

<Processing>
The procedure of the entire processing in the fourth embodiment is the same as that shown in FIG. 4 showing the processing procedure of the information processing apparatus 1 explained in the first embodiment, so the explanation is omitted. Further, the details of step S140 of the learning model in FIG. 4 are the same as those in FIG. 5, so the description is omitted. What differs from the first embodiment is the evaluation value calculation process in which the learning model selection unit calculates the evaluation value of the learning model in step S1120 of FIG.

In the evaluation value calculation process, the higher the degree of matching between the date and time information acquired from the internal clock and the date and time information, which is the status information held by the learning model group holding unit, the higher the evaluation value given to the learning model. Specifically, if the time difference between the shooting time held in the status information list and the shooting time of the input image is within a predetermined time period and the shooting month/day is within a predetermined number of days, the matching time • Assume that the model is a learning model that is photographed seasonally. Each learning model is assigned a memory that can hold a binary value (1: True/0 False) representing the match/disagreement of the situation information. In this case, the memory is set to 1 (True). Note that it is assumed to have been initialized to 0 (False) in the initialization step S110 of FIG.

<effect>
As described above, in the fourth embodiment, a higher evaluation value is given to a learning model as the situation information that can change the appearance of the input image and the learning image that has learned the learning model match. As a result, it is possible to select a learning model in which the shooting conditions of the input image and the learning image match. can be calculated.

<Modification>
In the fourth embodiment, the date and time information acquired from the electronic clock held by the information processing apparatus is used as the status information. However, as long as the current time can be obtained, any method of obtaining the date and time information may be used. It may be obtained from an external server via the network via the I/F (H17), or the user may input the date and time using an input means such as a keyboard. You may

In the fourth embodiment, date and time information is used as status information. However, the situation information is not limited to date and time information, and any information representing a situation that changes the appearance of a captured image may be used. For example, instead of using the date and time information as it is, time categories classified from the date and time information such as morning/noon/evening/night/twilight may be used as the situation information. At this time, the status information holding list holds the category information of the time when the learning image was captured, and a higher evaluation value may be calculated for the learning model with which the information matches. Also, seasonal information such as spring/summer/autumn/winter may be used as the situation information. At this time, the situation information holding list holds the season information obtained by imaging the learning images. An evaluation value may be calculated. Also, weather information such as sunny/cloudy/rainy/snow obtained from a website that distributes the weather via the network via the I/F (H17) may be used as the situation information. At this time, the situation information holding list holds the weather information obtained by imaging the learning image, and the learning model having the matching information may be calculated with a higher evaluation value.

(Embodiment 5)
In the first embodiment, the degree of similarity between an input image and a learning image used for learning a learning model is calculated as an evaluation value. In the second embodiment, the evaluation value of the learning model is calculated by comparing the object type detected from the input image and the object type detected from the learning image in advance. In the third embodiment, the evaluation value of the learning model is calculated using the position information of the input image or the learning image used for learning the learning model. In the fourth embodiment, the evaluation value of the learning model is calculated from the matching degree of the situation information that can change the appearance of the captured image. On the other hand, in the fifth embodiment, an example of calculating the evaluation value of the learning model by a method combining the first to fourth embodiments will be described.
<Configuration of information processing device>
The configuration of the information processing apparatus according to the fifth embodiment is the same as that of FIG. 1 showing the configuration of the information processing apparatus 1 described in the first embodiment, so the description is omitted. The information held by the learning model group holding unit 130 is different from that of the first embodiment. In this embodiment, the learning model group holding unit 130 has at least two learning models, and for each model, holds one or more learning images used for learning the model as described in the first embodiment. do. In addition, as described in the second embodiment, an object information list containing object information detected in advance from learning images of the model is stored. As described in the third embodiment, it holds a position information list that describes the position information at which the learning images of the model were captured. Furthermore, as described in the fourth embodiment, a situation information list is held that describes the situation information when the learning image of the model was captured.

<Processing>
The processing procedure in the fifth embodiment is the same as FIG. 4 showing the processing procedure of the information processing apparatus 1 explained in the first embodiment, so the explanation is omitted. What differs from the first embodiment is the procedure in which the learning model selection unit 120 selects a learning model in step S140.

In step S<b>140 , the learning model selection unit 120 calculates the evaluation value of each learning model using the object information list held by the learning model group holding unit 130 . At this time, the evaluation value described in step S1220 of the first embodiment is calculated as evaluation value 1. FIG. Next, among the number of votes of each learning model described in Embodiment 2, Embodiment 3, and Embodiment 4, the evaluation value 2 is a continuous value of 0 to 1 normalized by dividing each number of votes by the maximum number of votes. , and an evaluation value of 3.

In step S1130 of FIG. 5, learning model selection unit 120 selects a learning model to be used based on evaluation values 1 to 4 calculated in steps S1220, S1330, and S140. In this embodiment, the learning model that maximizes the sum of the evaluation values 1 to 4 is selected. The estimation unit 140 puts the geometric information into a state that can be estimated.

<effect>
As described above, in the fifth embodiment, the degree of similarity between the input image and the learning image, the degree of matching of the object type detected from the input image and the learning image, and the degree of matching of the position information of the input image and the learning image are The evaluation value of the learning model is calculated such that the higher the evaluation value, the higher the evaluation value. A learning model in which the input image and the learning image are similar, the same type of object is captured in the input image and the learning image, the shooting positions match, and the shooting time, season, and weather match. By selecting , the geometric information can be estimated with high accuracy. Therefore, the position and orientation of the imaging device can be calculated and acquired with high accuracy.

<Modification>
In the fifth embodiment, the degree of similarity between the input image and the learning image, the degree of matching of the object type detected from the input image and the learning image, the degree of matching of the position information of the input image and the learning image, and the appearance of the captured image are changed. The evaluation value of the learning model was calculated from the matching degree of the available situation information. However, the configuration may be such that two of the above four are used. Further, in determining the learning model in step S1130, for example, a learning model that maximizes a predetermined weighted average of evaluation values 1 to 3 may be selected. Alternatively, the learning model with the largest evaluation value may be selected from among the evaluation values equal to or greater than a predetermined threshold value, or the learning model with the maximum evaluation value among all the evaluation values may be selected.

Also, the learning models may be gradually narrowed down based on each evaluation value. For example, after first narrowing down based on position information and situation information, a learning model that has been trained using learning images in which similar images or object types appear is selected. Specifically, learning models with evaluation values 3 and 4 equal to or greater than a predetermined threshold are selected, and a learning model that maximizes the sum of evaluation values 1 and 2 is selected. As a result, it is possible to reduce similar image search and object detection processing that require a large amount of calculation.

Further, when the information processing device according to the present invention is installed in an automobile, it may be used to control a moving mechanism such as an electric motor in automatic driving, or may be used to assist acceleration/deceleration and handling operations during human driving. may be used as a navigation system. In addition, the present information processing apparatus is not limited to being installed in an automobile, and the information processing apparatus may be mounted on the cloud, and the results of processing via a network may be used for automobile control, driving assistance, navigation, and the like.

When the information processing apparatus of the present invention is used for automobiles, the learning model selection unit 120 uses a car navigation system, various sensors, and various controllers mounted on the automobile to obtain driving data through the communication I/F (H17). The information can also be used to select a learning model. In the case of such a configuration, any method may be used as long as it selects a learning model using driving information obtained from a vehicle. Specifically, a scene category (urban area, mountainous area, seaside area, tunnel, highway) attached to the map information of the car navigation system is acquired as driving information, and the scene category is obtained as described in the first embodiment. A learning model may be selected based on information. As another selection method, as driving information, people and cars on the road, traffic lights, signs, their number and density, and road conditions (number of lanes, road surface: asphalt and dirt) are acquired from the camera mounted on the car. , and using them as object information, a learning model may be selected as described in the second embodiment. In addition, as driving information, the address information where the car is traveling calculated by the car navigation system, the place name information recognized from the traffic signboard photographed by the camera mounted on the car, the sensors obtained from GPS, WiFi, and various beacons Information may be acquired, and a learning model may be selected as described in the third embodiment based on the position information obtained from the information. Furthermore, the time information obtained from the car navigation system, whether or not the lights are on (which can be used to determine whether it is day or night), and whether or not the wipers are operating (which can be used to determine whether it is sunny or rainy) have been explained in the fourth embodiment. A learning model may be selected using such situation information. In addition, using the position and orientation of the camera attached to the car model and car as situational information, the evaluation value will be higher for the learning model that was shot with the camera attached in the same car model and in the same position and posture. can be calculated. Although a plurality of cases in which the methods described in Embodiments 1 to 4 are applied to automobiles are illustrated here, any one of them may be used alone, or a plurality of them may be used in combination. In addition, although the method of selecting a learning model based on the driving information of the own vehicle has been exemplified, any driving information that can be used for selecting a learning model can be used. A learning model may be selected based on driving information acquired from road signs, installed cameras installed on the side of the road, and various sensors.

(Embodiment 6)
In Embodiments 1 to 5, learning is performed based on the degree of similarity between an input image and a learning image, information on objects detected from these images, position information at the time of image capturing, and situation information that can change the appearance of the captured image. I had selected a model. In contrast, in the sixth embodiment, the geometric information (second geometric information) estimated from the input image by motion stereo and the geometric information (third geometric information) estimated using each learning model held by the learning model group holding unit The evaluation value of the learning model is calculated by comparing with the geometric information of Note that the geometric information output by the learning model selected by the learning model selection unit is the first geometric information.

That is, the first geometric information is geometric information output by the selected learning model and used to acquire the position and orientation. The second geometric information is geometric information obtained by motion stereo or the like and used for selecting a learning model. The third geometric information is geometric information output by the learning model group and used for selecting the learning model.

<Configuration of information processing device>
The configuration of the information processing apparatus according to the sixth embodiment is the same as that shown in FIG. 1 showing the configuration of the information processing apparatus 1 described in the first embodiment, so description thereof is omitted. What differs from the first embodiment is the role of each component and the input/output relationship of their data.

The geometric information estimation unit 140 inputs the input image input by the image input unit 110 to the learning model selected by the learning model selection unit 120, and estimates first geometric information. Also, the geometric information estimation unit 140 calculates second geometric information based on the plurality of images input by the image input unit 110 . A method for calculating the second geometric information will be described later. Further, the geometric information estimation unit 140 inputs an input image to each learning model held by the learning model group holding unit 130, and estimates third geometric information. Then, the second geometric information and the third geometric information are output to learning model selection section 120 . Also, the first geometric information is output to the position and orientation acquisition unit 150 .

Based on the input image input by the image input unit 110 and the second and third geometric information estimated by the geometric information estimation unit 140, the learning model selection unit 120 selects the learning model group holding unit 130 Calculate the evaluation value of each retained learning model. A learning model is selected based on the evaluation value and output to the geometric information estimation unit 140 . The learning model group holding unit 130 holds at least two learning models.

<Processing>
The processing procedure of the information processing apparatus in the sixth embodiment is the same as FIG. 4 explaining the processing procedure of the information processing apparatus 1 explained in the first embodiment and FIG. 5 explaining the details of the learning model selection process in step S140. omitted because there is Differences from the first embodiment are that the geometric information estimation unit 140 further calculates the second geometric information and the third geometric information, and the evaluation procedure of the learning model.

In step S150, the geometric information estimation unit 140 estimates a third depth map, which is third geometric information, using each learning model. Further, second geometric information is calculated based on the input image. The second geometric information in this embodiment includes the first input image captured by the imaging device 11 at the first time t, and the predetermined movement amount of the imaging device 11 (for example, 10 cm in the X-axis direction in the camera coordinate system). ) is the second depth map calculated by the motion stereo method based on the second input image captured by the imaging device 11 at the second time t+1 after the movement. It should be noted that the depth scale is defined using the above-described predetermined amount of movement as the baseline length.

Then, the learning model selection unit 120 calculates the evaluation value of each learning model held by the learning model group holding unit 130 . The learning model selection unit 120 calculates, as an evaluation value, the reciprocal of a value obtained by adding the depth difference of each pixel between the third depth map and the second depth map to all pixels in the image. Then, the learning model selection unit 120 selects the learning model with the maximum evaluation value.

<effect>
In the sixth embodiment, the evaluation value of the learning model is calculated by comparing the third geometric information estimated by the geometric information estimation unit 140 using the learning model and the second geometric information estimated from the input image by motion stereo. calculate. As a result, it is possible to select a learning model that can output geometric information similar to the second geometric information estimated by motion stereo, and to acquire the position and orientation of the imaging device with high accuracy.

In addition, even if the learning model is trained using learning images of multiple scenes, or if information that characterizes the learning model is not stored, select a learning model that can output geometric information with high accuracy. be able to. Therefore, the position and orientation of the imaging device can be obtained with high accuracy.

<Modification>
In Embodiment 6, the imaging device 11 that captures an image calculates the second geometric information by the motion stereo method using an RGB camera. However, if the imaging device 11 is a camera that can capture depth information, distance images, and three-dimensional point cloud data, the depth information obtained from them may be used as the second geometric information. Further, when using a camera including two or more cameras or sensors, the depth calculated by stereo-matching the images of the plurality of cameras may be used as the second geometric information. Furthermore, if the information processing apparatus is further equipped with sensors capable of obtaining depth information, such as LiDAR and millimeter wave radar, the depth information obtained from them may be used as the second geometric information.

In Embodiment 6, the baseline length is determined by moving the camera by a specified amount. However, if the information processing apparatus 1 is equipped with a movement amount measurement sensor such as an inertial measurement device such as an IMU that can estimate the movement amount of the camera, the sensor information (movement amount) obtained by the sensor is used as the baseline length to determine the scale. may be specified. IMU is an abbreviation for an inertial measurement unit.

Further, the learning model selection unit 120 may assign an evaluation value to each learning model held by the learning model group holding unit 130 while the baseline lengths of the two images used for motion stereo are unknown.

Specifically, first, the learning model selection unit 120 normalizes the depth map obtained by motion stereo with the baseline length unknown, for example, by the average value, the median value, the maximum value, the minimum value, etc. to obtain a second depth map. Compute the map. Next, the learning model selection unit 120 inputs the input image to the learning model held by the learning model group holding unit 130, and normalizes the obtained depth map with the average value, median value, maximum value, minimum value, etc. Compute a third depth map. Then, the reciprocal of the sum of the depth differences between the third depth map and the second depth map for the entire image is used as the evaluation value, and the learning model with the maximum evaluation value is selected.

As a result, even if the motion stereo baseline is unknown, it is possible to select a learning model in which the schematic structures of the third depth map and the second depth map match, and to acquire the position and orientation with high accuracy. can be done.

In Embodiment 6, a learning model in which the third geometric information matches the second geometric information is selected. However, it is also possible to determine a learning model using only at least the third geometric information without calculating the second geometric information. For example, the learning model selection unit 120 assigns an evaluation value to each learning model based on the residual when the position/orientation acquisition unit 150 calculates the position/orientation based on the third geometric information output by each learning model. good too. Specifically, when the position and orientation acquisition unit 150 projects each pixel of the previous frame onto the current frame based on the depth value of each pixel of the previous depth map estimated by each learning model, The position and orientation are calculated so as to minimize the luminance error between the pixels and the pixels of the current frame after projection. Also, the residual of the luminance error at this time is input to the learning model selection unit 120 . Then, the learning model selection unit 120 uses the reciprocal of the residual input by the position/orientation acquisition unit 150 as the evaluation value, and selects the learning model with the maximum evaluation value. Further, when the position/orientation acquisition unit 150 calculates the position/orientation so that the error gradually decreases by repeated calculation, the number of times and the time taken until the error converges are measured, and the learning model selection unit 120 may be entered. At this time, the learning model selection unit 120 may use the reciprocal of the number of times or time as the evaluation value, or may select the learning model with the maximum evaluation value.

Alternatively, the third depth map output by each learning model held by the learning model group holding unit 130 may be used as an initial value, and the second depth map may be calculated by time-series filtering. At this time, the learning model selection unit 120 gives an evaluation value to the learning model based on the amount of change in the depth value of each second depth map in the time-series filtering. Specifically, for example, the average value, median value, maximum value, minimum value, or the reciprocal of the sum of the amount of change in the depth of each pixel due to time-series filtering may be calculated as the evaluation value of each learning model. You may select the learning model that maximizes . Also, in time-series filtering, the variance and reliability of the depth of each pixel (Uncertainty map in Non-Patent Document 1) are obtained, and the reciprocal of the variance and reliability, for example, the average value, median value, maximum value, minimum value, and sum You may select the learning model that maximizes .

In the sixth embodiment, the case where the geometric information output by the learning model is a depth map has been described. However, it is also possible to use a learning model in which two input images at different times are input, and the obtained output geometric information is six parameters of the relative position and orientation between the two images. Note that the relative position and orientation output by the learning model will be referred to as the first relative orientation. When using such a learning model, the learning model selection unit 120 detects and matches feature points between two images, and based on the correspondence relationship between the feature points whose feature amounts match between the two images, A second relative orientation is calculated using a 5-point algorithm. At this time, the learning model selection unit 120 uses the reciprocals of the squared distances of the six parameters of the first relative position/posture and the second relative position/posture as evaluation values, and selects the learning model with the maximum evaluation value.

In the sixth embodiment, the learning model selection unit 120 compares the second depth map calculated by motion stereo with the third depth map output by the learning model, and assigns an evaluation value to the learning model. . However, if an object with a known size can be detected from the input image, a learning model that matches the size of the object with the third depth map may be given a high evaluation value.

As shown in FIG. 11, the information processing apparatus 3 in this modification further includes an object model holding unit 1110 and an object detection unit 1120 in addition to the configuration of the information processing apparatus 1 described in the first embodiment. The object model holding unit 1110 holds an object model whose shape is known. The object detection unit 1120 performs object detection from an input image and alignment of an object model. Then, the object detection unit 1120 detects the object, and the distance information to the object surface that can be calculated when the object model is aligned with the input image, and the depth value of the object area in the depth map of the area output by the learning model. are compared by the learning model selection unit 120 . An evaluation value is given to the learning model based on the comparison result. Note that the object model holding unit 1110 and the learning model holding unit 130 may be configured integrally.

The object model in this modified example is, for example, three-dimensional CAD data of an object such as a can, a PET bottle, or a human hand, which is a general object having approximately constant size and shape. Specifically, first, the object detection unit 1120 detects an object from the input image. For object detection, for example, the Line2D method (Non-Patent Document 6), which aligns the gradient image, which is the differential of the input image, with the silhouette when CAD data is observed from various directions, is used to detect objects in the input image. Align the CAD model to the existing object. Note that the method of aligning the objects is not limited to the above method. Next, the distance value from the camera to the CAD model surface is calculated based on the internal parameters of the camera. Finally, the learning model selection unit 120 inputs the input image to the learning model, and calculates the difference between the depth value of the object region of the depth map, which is the obtained geometric information, and the distance value of the region calculated from the CAD model. calculated, and the reciprocal obtained by adding over the entire object region is calculated as an evaluation value. The learning model selection unit 120 selects the learning model with the maximum evaluation value. In this modified example, the case of using a general object with a known shape has been described, but an artificial marker printed with a specific pattern of known size that is not a general object, or a three-dimensional pattern with a unique size and shape Objects may be used instead.

As a result, it is possible to select a learning model that can accurately estimate the size of an object whose size is known in the input image. can be improved.

(Embodiment 7)
In Embodiments 1 to 6, the user cannot confirm whether the selected learning model is suitable for the scene appearing in the input image. On the other hand, in the seventh embodiment, in order for the user to check, information characterizing the learning model, a synthesized image obtained by synthesizing a CG image that is a virtual object based on the output of the learning model, and an image generated based on the output of the learning model An example of displaying a three-dimensional shape of a scene on the display unit will be described.

<Configuration of information processing device>
A part of the configuration of the information processing apparatus according to the seventh embodiment is the same as that of FIG. A difference from the first embodiment is that a display information generation unit 12 and a display unit 13 are incorporated in the information processing apparatus 1 .

The learning model group holding unit 130 holds at least two learning models and an information list that characterizes the learning models. The information list that characterizes the learning model may be the learning image described in the first embodiment, the object information list described in the second embodiment, or the position information list described in the third embodiment. may be All of these lists may be retained, or only some of them may be retained. In this embodiment, the learning model group holding unit 130 holds all three types of lists as information lists that characterize learning models.

The learning model selection unit 120 includes an information list characterizing the learning model held by the learning model group holding unit 130, and geometric information obtained by inputting an input image to each learning model held by the learning model group holding unit 130. is output to the display information generation unit 12 .

The display information generation unit 12 generates a first synthesized image by rendering an information list characterizing the learning model as character information. Also, a second synthesized image is generated by synthesizing the virtual object based on the geometric information (first geometric information or third geometric information) estimated by the geometric information estimation unit 140 . Furthermore, a third synthetic image is generated by rendering the three-dimensional shape of the scene appearing in the input image created based on the geometric information (first geometric information or third geometric information) estimated by the geometric information estimation unit 140. do. These synthesized images are output to the display unit 13 as display information. Note that at least one of these may be generated as the display information.

The display unit 13 is, for example, a window of the display of the mobile terminal, and presents display information input by the display information generation unit 12 .

FIG. 9 is a diagram showing a GUI 100, which is an example of display information presented by the display unit 13 in this embodiment.

G110 is a window for presenting an information list that characterizes the learning model, G120 is a window for presenting a synthesized image obtained by synthesizing CG images, which are virtual objects, and G130 presents geometric information and three-dimensional shapes. It is a window for G140 is a window for presenting an input image or information detected from the input image. G1410 is a frame indicating the learning model selected by learning model selection section 120 .

G1110 is an example of presenting model names of learning models held by the learning model group holding unit 130 in the window G110. G1120 is an example of presenting the position information list held by the learning model group holding unit 130 . G1130 is an example of presenting an object information list held by the learning model group holding unit 130 . G1131 is an example of presenting a learning image used for learning each learning model held by the learning model group holding unit 130 . G1140 is an example of presenting the type of object appearing in the input image and the positional information of the imaged input image on the window G140. G1150 is an example of presenting the input image in window G140. The user can check whether the learning model selected by learning model selection unit 120 is appropriate by comparing the contents presented in windows G110 and G140.

G1210 is a CG image of a virtual object combined with the input image in the window G120 based on the geometric information estimated by the geometric information estimation unit 140. FIG. G1220 is an example of presenting an image obtained by synthesizing a virtual object G1210 with an input image. G1230 is an example of presenting the evaluation value of each model calculated by the learning model selection unit 120 . Here, an example is shown in which a human model is superimposed on the bed of the input image as a CG image, and a CG image of the size of the bed obtained from the third geometric information is superimposed. At this time, the user can confirm whether the input image and the CG image match.

Specifically, whether the scale of the bed and the CG image match, whether the size of the bed matches the scale of the real thing, and whether the CG image faces the bed surface, the learning model It can be determined whether the learning model selected by the selection unit 120 is appropriate.

G1310 is an example in which the result of restoring the three-dimensional shape of the scene in which the input image was captured based on the geometric information estimated by the geometric information estimation unit 140 is presented in the window G130. The user can determine whether the learning model selected by the learning model selection unit 120 is appropriate by confirming whether the presented three-dimensional shape is distorted and whether the depth scale is different from the real thing.

<Processing>
The procedure of the overall processing in the seventh embodiment is the same as FIG. 4 explaining the processing procedure of the information processing apparatus 1 explained in the first embodiment, so the explanation is omitted. A process different from that of the first embodiment is a procedure in which the display information generation unit 12 generates display information.

In step S<b>170 , the display information generation unit 12 renders the information list characterizing the learning model at the position of the window G<b>110 of the display unit 13 . Specifically, the learning image G1131 obtained by learning the learning model held by the learning model group holding unit 130, the position information list G1110, and the object information list G1120 are rendered at predetermined positions to generate display information. Also, based on the geometric information estimated by the geometric information estimation unit 140, the CG image of the virtual object is combined with the input image.

Specifically, first, a principal plane is obtained from a depth map, which is geometric information, using plane fitting in combination with the RANSAC method. Next, the normal direction of the principal plane is calculated. Finally, the CG image of the virtual object is rendered at a predetermined position (eg, G120) on the principal plane to generate display information. Note that the distance between any two points on the depth map may be rendered as shown in G1210. Furthermore, the result of restoring the three-dimensional shape of the scene in which the input image was captured, which is calculated based on the geometric information estimated by the geometric information estimation unit 140, is rendered at a predetermined position (for example, G130) to generate display information. . Specifically, each pixel of the input image is projected onto an arbitrary virtual camera based on the depth value of each pixel of the depth map to generate a projected image and add it to the display information. The display unit 13 presents the display information generated as described above on the display.

<effect>
As described above, in the seventh embodiment, information characterizing a learning model, a synthesized image obtained by synthesizing CG of a virtual object, geometric information output by the learning model, and a three-dimensional shape restored based on the geometric information are presented. . As a result, the user can confirm the suitability when using each learning model in the scene shown in the input image, and whether or not the correct learning model has been selected. Furthermore, when an inappropriate learning model is selected, the processing can be redone and an appropriate learning model can be selected again, so that the position and orientation can be obtained with high accuracy.

<Modification>
In the seventh embodiment, information characterizing a learning model, a composite image obtained by synthesizing CG of a virtual object, geometric information output by the learning model, and display information obtained by rendering a three-dimensional shape restored based on the geometric information are presented. explained. However, it is not necessary to present all three pieces of display information, and at least one piece of display information may be presented.

The learning model selection unit 120 can also select a learning model based on input information input by the user using an input unit such as a mouse or keyboard based on display information. G1420 in FIG. 9 is a radio button, and G1430 is an input button. The user presses the input button to input to the information processing apparatus that the model checked by the radio button is to be used. With such a configuration, the user can select a learning model by referring to the information that characterizes the learning model, and the learning model can calculate the geometric information with high accuracy in the scene in which the input image is captured. can. Therefore, the position and orientation can be obtained with high accuracy. Also, if the user determines that the automatically selected learning model is inappropriate, he or she can correct the selection result.

(Embodiment 8)
In Embodiments 1 to 5, the learning model is selected only once at the beginning. However, it is difficult to deal with a case where, for example, the scene shown in the input image changes due to the user's movement while experiencing the mixed reality. Therefore, in the seventh embodiment, an example will be described in which even after the learning model is selected once, the evaluation value of the learning model is calculated again.

<Configuration of information processing device>
The configuration of the information processing apparatus according to the eighth embodiment is the same as that shown in FIG. 1 showing the configuration of the information processing apparatus 1 described in the first embodiment, so the description thereof is omitted. The difference from the first embodiment is that the geometric information estimation unit 140 calculates the second geometric information based on the input image and outputs it to the learning model selection unit 120, and the learning model selection unit 120 calculates the evaluation value of the learning model. is recalculated.

The geometric information estimation unit 140 further calculates third geometric information using the learning model held by the learning model group holding unit 130 and the input image input by the image input unit 110 . Also, second geometric information is calculated by motion stereo using the input image, and is output to the learning model selection unit 120 .

Based on the third geometric information and the second geometric information input by the geometric information estimating unit 140, the learning model selecting unit 120 calculates the evaluation value of each learning model held by the learning model group holding unit 130. do. Then, the evaluation result is output to the geometric information estimation unit 140 .

<Processing>
The procedure of the overall processing in the eighth embodiment is the same as FIG. 4 explaining the processing procedure of the information processing apparatus 1 explained in the first embodiment, so the explanation is omitted. The processing different from the first embodiment is that the evaluation value of the learning model is calculated again even after the learning model is once selected in step S140, and the learning model is reselected, and in step S150 the geometric information estimating unit 140 selects the second It is a point for calculating geometric information.

In the details of the process of step S140 in this embodiment, the process of step S1110 in FIG. 5 is removed. That is, regardless of whether the learning model has been determined, the learning model selection unit 120 calculates the evaluation value of the learning model.

In step S150 in FIG. 4, the geometric information estimation unit 140 estimates the third depth map, which is the third geometric information, using the learning model. Further, second geometric information is calculated using motion stereo based on the input image.

Furthermore, in step S150 according to the present embodiment, the learning model selection unit 120 recalculates the evaluation value of the learning model based on the third geometric information and the second geometric information obtained by the geometric information estimation unit 140. do. Specifically, the learning model selection unit 120 causes the geometric information estimation unit 140 to estimate at an arbitrary time t′ after the time t when the input image used for calculating the evaluation value of the learning model in step S1120 in FIG. The reciprocal of the sum of the depth differences between the third depth map, which is the (updated) third geometric information, and the second geometric information estimated (updated) by the geometric information estimation unit 140 at the same time t′ Recalculate as the evaluation value of the learning model. Then, the learning model selection unit 120 newly selects the learning model with the maximum evaluation value.

<effect>
In the eighth embodiment, even after the learning model is selected once, the evaluation value of the learning model is calculated again, and the learning model is selected again. As a result, the learning model can be re-evaluated when, for example, the user moves while experiencing mixed reality and the scene captured in the input image changes. By selecting a learning model with a high re-evaluation result, the learning model can calculate geometric information with high accuracy in the scene captured in the input image at that time, and therefore the position and orientation of the imaging device can be obtained with high accuracy. .

<Modification>
The timing of recalculating the evaluation value of the learning model is arbitrary. That is, it may be recalculated at regular time intervals, or may be recalculated each time a key frame is added as described in Non-Patent Document 1. Further, based on the position/orientation acquisition result, recalculation may be performed when the imaging device 11 moves by a predetermined movement amount or more. Further, when the evaluation value of the learning model that has been selected once decreases or falls below a predetermined threshold value, it may be recalculated. It may be recalculated when the scene or object type captured in the input image changes, or when the position information changes (for example, when a new WiFi access point is found or when GPS position information changes). Further, it may be recalculated when a certain period of time has passed or when the weather changes.

In the eighth embodiment, the geometric information estimation unit 140 calculates the second geometric information by motion stereo. However, the second geometric information may be integrated in chronological order to improve accuracy and then used for evaluation of the learning model. For example, using the third geometric information estimated by the geometric information estimating unit 140 at time t′ as an initial value, the second geometric information is calculated by time-series filtering from the input image up to arbitrary time t′+i (non-patent described in Reference 1). Also, a second depth map is calculated by integrating a plurality of depth maps calculated in this manner at a plurality of times t'. Here, the integration is described in Non-Patent Literature 7, and the position and orientation of the camera at multiple times t′ when the input image used to generate the depth map is calculated by pose graph optimization, and the obtained camera Smoothing the plurality of second depth maps using the pose. Alternatively, you can select the learning model with the smallest residual by using the residual when optimizing the pose graph as an evaluation value, or select the learning model with the shortest processing time by using the processing time required for optimizing the pose graph as an evaluation value. You may By doing so, as the optimization progresses, the evaluation value can be calculated more accurately, and a more appropriate learning model can be selected, thereby improving the position and orientation acquisition accuracy.

In the eighth embodiment, the learning model selection unit 120 compares the second depth map, which is the second geometric information calculated by the geometric information estimation unit 140, with the third depth map output by the learning model, The evaluation value was recalculated for each model. However, the learning model selection method is not limited to this, and the evaluation value of each learning model may be recalculated by the method described in the first to seventh embodiments using the input image at time t′.

In the eighth embodiment, the learning model selection unit 120 compares the second depth map, which is the second geometric information calculated by the geometric information estimation unit 140, with the third depth map output by the learning model, The evaluation value was recalculated for each model. However, based on the degree of matching between the third geometric information and the second geometric information at multiple times t′, the higher the degree of matching, the higher the evaluation value is given, and the learning model is selected based on the evaluation value. good too. Specifically, the learning model selection unit 120 calculates a first evaluation value that is the sum of differences in depth values between the third depth map and the second depth map at a plurality of times t′. Then, for example, the average value, median value, maximum value, minimum value, or reciprocal of the sum of the first evaluation values is calculated as the second evaluation value, and the learning model with the maximum second evaluation value is selected. . This makes it possible to select a learning model capable of calculating the third geometric information with high accuracy in the input images at multiple times. Therefore, even if an incorrect learning model is initially selected, the learning model selection unit 120 can gradually reselect a learning model that can estimate geometric information with high accuracy, thereby acquiring the position and orientation with high accuracy. can do.

When the evaluation value is recalculated, the learning model selected by the learning model selection unit 120 may change. At this time, since the learning model used by the geometric information estimation unit 140 to estimate the geometric information is changed, the output of the learning model may change significantly before and after the change. In order to deal with this, in step S150, the geometric information estimation unit 140 may calculate the weighted sum of the geometric information output by the two learning models for a predetermined period of time as the third geometric information. Specifically, the depth map is corrected as in the following equation using a predetermined number of frames N representing the model switching period and the number of frames α that have elapsed since the start of switching.

However, D1 is the depth map output by the learning model before change, _D2 is the depth map _output by the learning model after change, and D is the corrected depth map.

(Embodiment 9)
In the first to eighth embodiments, a method of selecting a learning model capable of accurately estimating geometric information in a scene to which an information processing apparatus is applied from among a plurality of learning models created in advance has been described. In this embodiment, a method of generating a learning model used by an information processing device based on an RGB image and a depth map acquired by a depth sensor, which is the second imaging device 91, will be described. In particular, in this embodiment, a method of recognizing scene types from RGB images captured by the second imaging device 91 and separately creating learning models for each type will be described. In this embodiment, the second imaging device 91 is a TOF sensor, which can acquire an RGB image and a depth map.

<Configuration of information processing device>
First, the configuration of the information processing device 4 according to the ninth embodiment will be described with reference to FIG. In addition to FIG. 1 showing the configuration of the information processing apparatus 1 described in the first embodiment, the configuration of the information processing apparatus 4 in the ninth embodiment includes a second image input unit 910, a learning data classification unit 920, and a learning data holding unit. It differs from the first embodiment in that a unit 930 and a learning model generation unit 940 are added. A second image input unit 910 is connected to the second imaging device 91 .

The second image input unit 910 receives image data of a two-dimensional image of a scene captured by the second imaging device 91 (hereinafter referred to as a model learning image) and a depth map (hereinafter referred to as a model learning depth map). are input in time series (for example, 60 frames per second) and output to the learning data classification unit 920 . Note that the model learning image and the model learning depth map are collectively referred to as learning data.

The learning data classification unit 920 recognizes the scene type based on the learning data input by the second image input unit 910 , classifies the learning data by type, and outputs the learning data to the learning data holding unit 930 . A method of classifying the learning data will be described later.

The learning data holding section 930 classifies and holds the learning data classified by the learning data classification section 920 for each type of scene. The learning data holding unit 930 is, for example, an SSD (Solid State Drive). Folders are divided for each type of scene, and learning data is held in a folder corresponding to the classification result of the learning data classification unit 920 . It is assumed that a common ID (for example, serial number or time) is assigned to the model learning image and the model learning depth map acquired at the same time and are associated with each other.

The learning model generation unit 940 generates a learning model using the learning data stored in the learning data holding unit 930 based on the classification result of the learning data classification unit 920 . The generated learning model is output to the learning model group holding unit 130 .

Next, the processing procedure in this embodiment will be described with reference to the flowchart of FIG. In this embodiment, in addition to the processing procedure described in Embodiment 1, model learning image/model learning depth map imaging step S910, model learning image/depth map input step S920, image classification step S930, It differs from the first embodiment in that a learning data holding step S940, a learning data collection completion determination step S950, a learning model generating step S960, and a learning model holding step S970 are added.

In this embodiment, first, the initialization step S110 described in the first embodiment is executed to initialize the system. Next, steps S910 to S970 described below are executed to generate a learning model. Then, the processes after step S120 described in the first embodiment are executed to calculate the position and orientation of the imaging device 11 .

In step S<b>910 , the second imaging device 91 captures the scene and outputs the RGB image and depth map to the second image input section 920 . Next, the process moves to step S920.

In step S920, the second image input unit 910 acquires the image and the depth map captured by the second imaging device 91 as a model learning image and a model learning depth map. Next, the process proceeds to step S930.

In step S930, the learning data classification unit 920 recognizes the scene type from the model learning image and classifies the learning data. This embodiment uses the scene discrimination learning model described in the modified example of the first embodiment. The scene discrimination learning model is a neural network that has been learned using deep learning so that it outputs 1 if the input image belongs to the category, and 0 otherwise. That is, the model learning image is input to the scene discrimination learning model, and the obtained category is determined as the classification result of the learning data. Next, the process proceeds to step S940.

In step S940, the learning data classification unit 920 holds the learning data in the learning data holding unit 930 based on the classification result of the learning data in step S920. Specifically, an image for model learning and a depth map for model learning are stored in a folder corresponding to the classification result. Next, the process proceeds to step S950.

In step S950, it is determined whether or not the collection of learning data has been completed. Here, it is determined that the data collection is completed when the user inputs an end command using an input device (not shown). If it is determined that the data collection is completed, the process proceeds to step S960. Otherwise, proceed to step S910 to continue data collection.

In step S<b>960 , the learning model generation unit 940 uses the learning data held by the learning data holding unit 930 to generate a learning model for each category classified by the learning data classification unit 920 . That is, a learning model is generated for each folder in the learning data holding unit 930 . Specifically, a learning image is selected from the corresponding folder by random numbers, and the error between the geometric information output by the learning model using this as an input and the learning depth map corresponding to the selected learning image is minimized. Iterate to learn the learning model every time. A method of generating a learning model is described in detail in Non-Patent Document 1, which can be used as a reference.

The learning model generation unit 940 holds the learning model generated for each category classified by the learning data classification unit 920 in the learning model group holding unit 130 . At this time, the learning images are also copied and held in the learning model group holding unit 130 .

<effect>
As described above, in the ninth embodiment, model learning images and model learning depth maps are classified based on the scene discrimination results of model learning images, and a learning model is generated for each scene type. do. In this way, a learning model is generated for each scene type, and a learning model that matches the scene type of the captured image is used when calculating the position and orientation described in the first embodiment. Posture can be calculated.

<Modification>
In the ninth embodiment, a scene discrimination learning model is used to discriminate a scene captured in a model learning image. However, anything can be used as long as it can determine the type of scene captured in the model learning image. That is, as described in the first embodiment, the discrimination boundary of the feature space of the GLC features is calculated by SVN in advance for each scene type, and it is determined in which category the GLC features detected from the model learning image are located. The type of scene may be determined based on the results obtained. A color histogram may be created in advance for each scene type, and the model learning image may be classified into the scene type that best matches the color histogram.

In the ninth embodiment, the learning model is generated by classifying the learning data based on the scene captured in the model learning image. On the other hand, it is also possible to classify the learning data according to the type of object appearing in the model learning image and generate a learning model. In other words, the learning data can be classified for each object detection result such as "desk", "table", "car", and "traffic light". Note that the object detection method exemplified in the second embodiment can be used for object detection from the learning image. A learning model is generated for each learning data classified in this manner. Alternatively, the co-occurrence probabilities of objects may be calculated in advance, and the learning data may be classified based on these. The co-occurrence probability is the probability that objects are observed at the same time, such as the probability that ``table'', ``TV'', and ``bed'' are observed at the same time, and the probability that ``desk'', ``chair'', ``computer'', and ``display'' are observed at the same time. It is the probability of being observed at the same time. Using the probability that objects are observed at the same time, the scenes in which ``chabudai'', ``TV'', and ``bed'' are observed are Japanese houses, ``desks'', ``chairs'', ``desk'', ``chairs'', Scenes where "PC" and "Display" are observed can be classified into learning data for each scene such as an office.

It is also possible to classify the learning data using the location information from which the learning data was acquired. As described in the third embodiment, the position information is, for example, latitude and longitude coordinate values or an identification ID of a Wifi access point. Specifically, the latitude and longitude may be divided at predetermined intervals to classify the learning data. Also, the learning data may be classified for each identification ID of the Wifi access point observed when the learning data was acquired. Furthermore, from position information calculated from GPS, categories such as country, region, sea/mountain/road may be identified from map information (not shown), and learning data may be classified for each category. A learning model is generated for each learning data classified in this manner.

It is also possible to classify the learning data according to the date and time when the learning data was acquired, the season, the weather, and other situations that can change the appearance of the image. For example, the learning data may be classified for each shooting time. Alternatively, the photographing time may be divided into categories such as morning, noon, evening, and night, and the learning data may be classified for each category. The learning data may be classified according to the month and day when the image was taken, or by separating the seasons such as spring/summer/autumn/winter from the month and day. The learning data may be classified into weather categories such as sunny/cloudy/rainy/snow obtained from a website that distributes the weather via the I/F (H17) via the network. A learning model is generated for each learning data classified in this manner.

Learning data can also be classified based on the model learning depth map. For example, the learning models may be classified based on the average, maximum value, minimum value, median value, and variance value of the depth values of the depth maps for model learning. The learning model may be classified based on the degree of unevenness of the depth map for model learning. To determine the degree of unevenness, for example, a principal plane is calculated by plane fitting to a depth map for model learning, and the number of three-dimensional points calculated from the depth map at a predetermined distance from the principal plane can be used. Alternatively, the normal may be calculated for each pixel from the model image depth map, and the number of labeling results where the inner product of the normal to surrounding pixels is equal to or less than a predetermined distance may be used as the degree of unevenness.

In addition, the user can input the scenes, objects, positional information, and the like described so far through input means (not shown), and based on these, the learning data can be classified. Also, the learning data may be classified according to the purpose of use of the learning model. In other words, the user inputs the application type, such as the use of the learning model for estimating the position and orientation of an in-vehicle camera for automatic driving, or the use for the position and orientation of the camera for superimposing CG on a smartphone or tablet, and the input result ( You may classify|categorize the data for learning for every use classification). A learning model is generated for each learning data classified in this manner.

When the second imaging device 91 is installed in a device equipped with a car navigation system, such as an automobile, and learning data is acquired, the type of scene associated with the map information of the car navigation system (urban area, mountainous area, etc.) , seaside area, tunnel, highway) may be used as the scene discrimination result to classify the learning data. People and cars on the road, traffic lights, signs, their number and density, road conditions (number of lanes, road surface: asphalt and dirt) are acquired from the camera mounted on the car, and these are used as object information for learning. can also be classified. It acquires address information calculated by the car navigation system where the car is traveling, place name information recognized from the traffic signboard photographed by the camera mounted on the car, and sensor information obtained from GPS, WiFi, and various beacons. The learning data may be classified based on the obtained position information. Classify learning data based on the time information obtained from the car navigation system, whether lights are on (can be used to determine day/night), and whether wipers are operating (can be used to determine whether it is sunny or rainy). can also It is also possible to classify the learning data according to the type of vehicle and the position and orientation of the camera attached to the vehicle. Based on such classification results, a learning model can be generated for each classification.

Also, the learning data can be classified for each sequence for acquiring the learning data. In other words, starting up the information processing apparatus 4 and ending it are regarded as one sequence, and the data acquired during that period are classified as belonging to the same category. A learning model can also be generated in this way.

The classification methods described so far are only examples, and any classification method may be used as long as it is a classification method that can generate a learning model that can estimate geometric information with high accuracy. The above classification methods may be used individually, or any number may be combined.

In the ninth embodiment, the learning data classification unit 920 classifies the learning data immediately after the second imaging device 91 captures the image, and the learning data holding unit 930 holds the classified data. However, the second image pickup device 91 may capture and store model learning images and model learning depth maps in advance, and the learning data classification section 920 may classify the learning data later. In general, image recognition and training of a learning model have a large computational cost. Therefore, with such a configuration, learning data can be obtained in advance by hardware with small computational resources, and image recognition and learning of a learning model can be processed by hardware with large resources.

In addition, by adopting such a configuration, learning data individually photographed by the plurality of second imaging devices 91 are combined and used, and in addition to once acquired learning data, another learning data is added later. , etc. is also possible. Furthermore, as described in the eighth embodiment, the obtained second geometric information may be used as a depth map, and the image captured by the imaging device 11 may be combined with the obtained second geometric information as learning data. Further, the learning model generator 940 may additionally learn a learning model that has been learned once, using the learning data added by the method described above.

In the ninth embodiment, the learning data classification unit 920 classifies all the learning data captured by the second imaging device 91 . However, it is not necessary to classify all the learning data, and only some of the learning data may be classified. For example, the learning data may be classified only once every 60 times acquired by the second imaging device 91 and held in the learning data holding unit 930 . Further, if the learning data classification unit 920 has a low degree of similarity between the newly acquired learning data and the learning data already held in the learning data holding unit 930, the learning data holding unit 930 may hold the data. . Here, the degree of similarity is, for example, the difference between the average value, maximum value, minimum value, median value, and variance value of the brightness of the image. It is also possible to use the recognition likelihood when the scene discrimination learning model performs scene recognition on the model learning image. Specifically, the recognition likelihood value (the value of the matching degree of each scene) immediately before the scene discrimination learning model calculates the output of 0 or 1 indicating whether or not the scene is the relevant scene, and the recognition of the already stored learning data It is the distance from the likelihood value. In this way, data is collected so as to reduce the number of similar learning data, and learning data is collected such that the degree of separation of the similarity of the learning data is widened, thereby reducing the time required to generate a learning model. At the same time, the recognition accuracy of the learning model can be improved.

In the ninth embodiment, a method of first generating a learning model and then estimating the position and orientation using the learning model has been described. However, the generation of the learning model and the position and orientation estimation using the learning model can be performed by separate devices. For example, the learning model is generated by the first information processing device, and the generated learning model is uploaded to the cloud server via the network. It is also possible to use a method in which another second information processing device loads a learning model in a cloud server via a network and uses it to estimate the position and orientation. Furthermore, at the time of learning model generation, the first information processing device acquires learning data, uploads the acquired data to the server, and the image classification step S930 and the learning model generation step S960 are performed based on the second information on the cloud server. A system in which the processing device performs is also possible.

In the ninth embodiment, the acquisition of the learning data is ended by the user inputting the end command. As long as the learning data necessary for learning can be collected, any method can be used to determine whether acquisition of the learning data is completed. For example, the learning data classification unit 920 may determine that the processing is finished after a predetermined time has elapsed, or may determine that the processing is finished when a predetermined number of pieces of learning data have been acquired. Furthermore, it may be determined that the processing is completed when the number of learning data items for each classification category exceeds a predetermined number. Also, the learning model generation unit 940 may calculate the learning degree of the learning model, and determine that the learning is finished when the learning converges.

In the present embodiment, image classification means holding learning data for each folder in the learning data holding unit 930 based on the result of the learning model classification by the learning data classifying unit 920 . However, it is not necessary to classify by folder, and the learning data holding unit 930 may hold a list in which classification results are recorded for each learning data.

In Embodiments 1 to 5, the learning model group holding unit 130 is configured to hold an object information list, a position information list, and a situation information list in addition to at least two learning models. These lists may be generated by the learning data classification unit 920 as necessary. In other words, the process of detecting the object type from the model learning image and adding it to the object information list, and adding the acquired position information and situation information to the position information list and the situation information list are performed together. You can add the necessary information from to 5. Further, the process of copying and holding the learning images in the learning model group holding unit described in the ninth embodiment may be omitted if the learning images are not required when the learning model is selected.

The second imaging device 91 is not limited to a TOF sensor, and may be any device capable of acquiring an image and a depth map. Specifically, a depth camera that projects a pattern and estimates the depth may be used. Alternatively, a stereo camera configuration may be used in which two cameras are arranged side by side to calculate the depth by stereo matching and output it as a depth map. Furthermore, a device configured to combine a 3D LiDAR (Light Detection and Ranging) and a camera and output a depth map obtained by converting depth values acquired by the LiDAR into image coordinates may be used. Further, the image is not limited to the RGB image, and may be a gray image. Alternatively, the second imaging device 91 may acquire images and depth maps in advance, store them in a recording device (not shown), and the second input unit 910 may input the images and depth maps from these recording devices.

<Effects of each embodiment>
In the first embodiment, an evaluation value is given to each learning model among a plurality of learning models, and a learning model with a high evaluation value is selected. At this time, the degree of similarity between the input image and the learning image used in learning each learning model is calculated, and the evaluation value of each learning model is calculated such that the higher the degree of similarity, the higher the evaluation value. Then, the position and orientation of the imaging device are calculated using geometric information estimated using a learning model with a high evaluation value. In this way, by selecting a learning model whose input image and learning image are similar, the learning model can estimate geometric information with high accuracy, and can calculate the position and orientation of the imaging device with high accuracy. can.

Furthermore, by selecting and using a small-scale learning model trained for each individual scene, geometric information can be estimated even on a computer with a small memory capacity. Accordingly, the position and orientation of the imaging device can be calculated even in the mobile terminal.

Furthermore, by selecting and using a small-scale learning model trained for each individual scene, geometric information can be estimated with less execution time than when using a large-scale learning model. As a result, the position and orientation of the imaging device can be calculated in a short time.

In the second embodiment, the object information detected from the learning image used for learning the learning model is compared with the object information detected from the input image so that the more the same type of object appears, the higher the evaluation value. Calculate the evaluation value of the learning model. Then, the position and orientation of the imaging device are calculated using geometric information estimated using a learning model with a large evaluation value. As a result, it is possible to select a learning model in which the same type of object is shown in the input image and the learning image, and the learning model can estimate the geometric information with high accuracy, and the position and orientation of the imaging device can be determined with high accuracy. can be calculated.

In the third embodiment, the evaluation value of the learning model is calculated such that the more the position information of the input image and the learning image obtained by learning the learning model match, the higher the evaluation value. As a result, it is possible to select a learning model in which the positional information of the input image and the learning image match. The position and orientation can be calculated.

In the fourth embodiment, a higher evaluation value is given to a learning model as the situation information that can change the appearance of the input image and the learning image for which the learning model has been trained is more consistent. As a result, it is possible to select a learning model in which the shooting conditions of the input image and the learning image match. can be calculated.

In the fifth embodiment, the higher the degree of similarity between the input image and the learning image, the higher the degree of matching between the object type detected from the input image and the learning image, and the higher the degree of matching of the position information of the input image and the learning image, the higher the evaluation value. Calculate the evaluation value of the learning model as follows. More specifically, a learning model is selected in which the force image and the learning image are similar, the same type of object is captured in the input image and the learning image, and the photographed positions match. As a result, the learning model can estimate the geometric information with high accuracy, and can calculate the position and orientation of the imaging device with high accuracy.

In the sixth embodiment, the more similar the third geometric information estimated by the geometric information estimation unit 140 using the learning model and the second geometric information estimated from the input image by motion stereo, the higher the evaluation value. Calculate the evaluation value of the learning model as follows. As a result, it is possible to select a learning model that can output third geometric information similar to the second geometric information estimated by motion stereo, and to calculate the position and orientation of the imaging device with high accuracy. .

In addition, even when a learning model is trained using learning images of multiple scenes, or when information that characterizes the learning model is not stored, it is possible to select a learning model that can output geometric information with high accuracy. can. Therefore, the position and orientation of the imaging device can be calculated with high accuracy.

In the seventh embodiment, information characterizing a learning model, a composite image obtained by synthesizing CG of a virtual object, geometric information output by the learning model, and a three-dimensional shape restored based on the geometric information are presented. As a result, the user can visually confirm the suitability when each learning model is used in the scene shown in the input image, and whether or not the correct learning model has been selected. Additionally, the user can select a learning model based on the displayed information. Furthermore, when an inappropriate learning model is selected, the user can decide to redo the process and select an appropriate learning model again. Therefore, the position and orientation can be calculated with high accuracy.

In the eighth embodiment, even after the learning model is selected once, the evaluation value of the learning model is calculated again, and the learning model is selected again. By doing so, even if the scene shown in the input image changes due to, for example, the user moving while experiencing mixed reality, the learning model can be evaluated again. By selecting a learning model with a high re-evaluation result, the learning model can calculate geometric information with high accuracy in the scene captured in the input image at that time, and the position and orientation of the imaging device can be calculated with high accuracy.

In the ninth embodiment, the model learning image and the model learning depth map are classified based on the scene discrimination result of the model learning image, and a learning model is generated for each scene type. In this way, a learning model is generated for each scene type, and a learning model that matches the scene type of the captured image is used when calculating the position and orientation described in the first embodiment. Posture can be calculated.

<Definition>
The image input unit in the present invention may be anything as long as it inputs an image of the real space. For example, an image of a camera that captures a grayscale image may be input, or an image of a camera that inputs an RGB image may be input. You may input the image of the camera which can image depth information, a distance image, and three-dimensional point-group data. Also, a monocular camera may be used, or an image captured by two or more cameras or a camera equipped with a sensor may be input. Furthermore, an image captured by a camera may be directly input, or may be input via a network.

The learning model in the present invention may be anything as long as it outputs geometric information when a camera image is input. For example, it is a neural network or a CNN (Convolutional Neural Network) trained in advance so as to output geometric information when a camera image is input. Also, the geometric information estimated by the learning model is, for example, a depth map that is depth information estimated for each pixel of the input image. Note that the geometric information estimated by the learning model may be a learning model that calculates, as geometric information, saliency points for use in obtaining the position and orientation from the input image. The learning model may be a learning model that inputs two images of the previous frame and the current frame and estimates the six degrees of freedom of position and orientation between them as geometric information.

The learning model group holding unit may be anything as long as it holds at least two (plural) learning models. In addition to the learning model, an information list that characterizes the held learning model may also be held. The information that characterizes the learning model is information for calculating an evaluation value that indicates the suitability of the learning model to the scene shown in the input image. Specifically, the information list that characterizes the learning model includes the learning images used for learning the learning model, the object information list that appears in the learning images, the position information list when the learning images were captured, and the learning images that were captured. It is a situation information list that describes situations that can change the appearance of an image, such as the shooting date and time, the season, and the weather at that time. Also, the internal parameters of the camera that captured the learning image may be stored in accordance with each learning model.

The learning model selection unit may be anything as long as it gives an evaluation value to the learning model held by the learning model group holding unit. The evaluation value here is an index that indicates the degree of conformity of the learning model to the scene captured by the imaging device. Specifically, it is the degree of similarity between an input image and a learning image, and the degree of matching between object types detected from those images and position information at the time of image capturing.

The method of calculating the evaluation value is not limited to the above method, and may be calculated based on the degree of approximation between the geometric information output by the learning model and the geometric information measured from the input image. Specifically, the evaluation value of the learning model may be calculated by comparing the second geometric information calculated from the input image by the motion stereo method and the third geometric information output by the learning model.

Furthermore, the learning model selection unit may be configured to select a learning model based on the evaluation value. A list of information characterizing the learning models may be presented on a display, such as a mobile device, and the learning model input by the user may be selected.

The geometric information estimation unit may be anything as long as it inputs an input image to a learning model and calculates geometric information. Particularly in the present invention, the output obtained by the geometric information estimating unit by inputting the input image to the learning model selected by the learning model selecting unit is called first geometric information, and is used by the position/orientation obtaining unit to obtain the position/orientation. . Moreover, the geometric information estimation unit may estimate the third geometric information as an index for the learning model selection unit to calculate the evaluation value. At this time, the output obtained by inputting the input image to each learning model held by the learning model group holding unit is called third geometric information, and the learning model selection unit uses the third geometric information to calculate the evaluation value of the learning model. Furthermore, the geometric information estimation unit may be configured to calculate the second geometric information using motion stereo based on the input image.

The position and orientation acquisition unit may be anything as long as it calculates the position and orientation of the camera using the geometric information output by the learning model. For example, each pixel of the previous frame is projected onto the current frame using the geometric information output by the learning model, and the brightness difference between the pixel value of the projected pixel of the previous frame and the pixel value of the current frame is minimized. Position and attitude may be calculated. In addition, the geometric information estimating unit is not limited to calculating the position and orientation of the camera directly using the first geometric information that is the output of the learning model. may be calculated and used to calculate the position and orientation of the camera.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

1: information processing device, 11: imaging device, 12: display information generation unit, 13: display unit, 110: image input unit, 120: learning model selection unit, 130: learning model group holding unit, 140: geometric information estimation unit , 150: location information acquisition unit

Claims (15)

A captured image that is learned using a captured image captured by an imaging device and depth information of the captured image as teacher data, and that uses a plurality of learning models for estimating depth information corresponding to an input image as the teacher data. holding means for holding in correspondence with the imaging position of
A scene captured in the input image from the plurality of learning models based on an evaluation result obtained from the degree of matching between any one of the imaging positions held in association with the learning model and the imaging position at which the input image was captured. a selection means for selecting a learning model suitable for
estimating means for estimating first depth information using the input image and the selected learning model;
with
The estimating means further estimates second depth information from the input image by a motion stereo method, further estimates third depth information based on the input image and the learning model for each of the learning models, and selects The information processing apparatus, wherein the means calculates the evaluation result of the learning model such that the higher the degree of matching between the second depth information and the third depth information, the higher the evaluation result . 2. The information processing apparatus according to claim 1, further comprising acquisition means for acquiring the position and orientation of said imaging device based on said first depth information. Further comprising a sensor for measuring the amount of movement of the imaging device,
The estimating means is arranged so that the higher the degree of matching between at least one of the sensor information measured by the sensor and the depth information calculated based on the sensor information and the third depth information, the higher the evaluation result. 2. The information processing apparatus according to claim 1 , wherein the evaluation result of said learning model is calculated in the following manner. A captured image that is learned using a captured image captured by an imaging device and depth information of the captured image as teacher data, and that uses a plurality of learning models for estimating depth information corresponding to an input image as the teacher data. holding means for holding in correspondence with the imaging position of
A scene captured in the input image from the plurality of learning models based on an evaluation result obtained from the degree of matching between any one of the imaging positions held in association with the learning model and the imaging position at which the input image was captured. a selection means for selecting a learning model suitable for
estimating means for estimating first depth information using the input image and the selected learning model;
with
the estimation means further estimates third depth information based on the input image and the learning model for each learning model;
The selecting means calculates the evaluation result of the learning model such that the more the size or shape of the known object detected from the input image matches the third depth information, the higher the evaluation result. and information processing equipment. further comprising generating means for generating display information based on at least one of the input image, the first depth information, the information held by the holding means, the evaluation result, and the position and orientation. 3. The information processing apparatus according to claim 2, characterized by: 6. The information processing apparatus according to claim 5 , wherein said generating means generates said display information by synthesizing a CG image of a virtual object with said input image based on said first depth information. the estimation means further estimates third depth information based on the input image and the learning model for each learning model;
6. The information processing apparatus according to claim 5 , wherein said generating means generates said display information by synthesizing a CG image of a virtual object with said input image based on said third depth information. 6. The information processing apparatus according to claim 5 , wherein said generating means restores a three-dimensional shape of a scene in which said input image is captured based on said first depth information to generate said display information. . the estimation means further estimates third depth information based on the input image and the learning model for each learning model;
6. An information processing apparatus according to claim 5 , wherein said generating means restores a three-dimensional shape of a scene in which said input image is captured based on said third depth information to generate said display information. . 10. The information processing apparatus according to any one of claims 5 to 9 , further comprising a display unit that displays the display information. The estimation means updates the second depth information and the third depth information based on a second input image captured by the imaging device at a second time different from the time when the input image was captured. death,
The selecting means recalculates the evaluation result of the learning model so that the higher the degree of matching between the updated second depth information and the updated third depth information, the higher the evaluation result. 2. The information processing apparatus according to claim 1 . The estimation means estimates a plurality of second depth information and a plurality of third depth information based on a plurality of input images captured by the imaging device at a plurality of times,
The selecting means calculates the evaluation result of the learning model such that the higher the degree of matching between the plurality of second depth information and the plurality of third depth information, the higher the evaluation result. The information processing device according to claim 1 . a second image input means for inputting a third input image and fourth depth information acquired by the second imaging device;
learning data classification means for classifying the third input image and the fourth depth information for each type of scene appearing in the third input image or the fourth depth information;
learning data holding means for holding the third input image and the fourth depth information for each scene type based on the classification result of the learning data classification means;
learning the plurality of learning models for each scene type using the third input image and the fourth depth information held by the learning data holding means, and holding the learning model in the learning data holding means; a generating means;
13. The information processing apparatus according to any one of claims 1 to 12 , further comprising: A captured image that is learned using a captured image captured by an imaging device and depth information of the captured image as teacher data, and that uses a plurality of learning models for estimating depth information corresponding to an input image as the teacher data. A control method for an information processing apparatus comprising holding means for holding in correspondence with the imaging position of
A scene captured in the input image from the plurality of learning models based on an evaluation result obtained from the degree of matching between any one of the imaging positions held in association with the learning model and the imaging position at which the input image was captured. a selection step of selecting a learning model suitable for
an estimation step of estimating first depth information using the input image and the selected learning model;
has
In the estimating step, further estimating second depth information from the input image by a motion stereo method, further estimating third depth information based on the input image and the learning model for each learning model, and In the step, the evaluation result of the learning model is calculated such that the higher the degree of matching between the second depth information and the third depth information, the higher the evaluation result. . A program for causing a computer to function as the information processing apparatus according to any one of claims 1 to 13 .

Images