JP2019124539A - Information processing device, control method therefor, and program - Google Patents

Abstract

To calculate the position-posture of an imaging device with high accuracy even when it is moving at high speed.SOLUTION: The information processing device comprises: an input unit for inputting visual information captured by an imaging device at a first time of day; a holding unit for holding a learning model for predicting at least one of visual information and geometric information at a second time of day on and after the first time of day on the basis of the visual information inputted by the input unit; a prediction unit for predicting at least one of visual information and geometric information at the second time of day using the learning model; and a first calculation unit for calculating the position-posture of an imaging device at the second time of day on the basis of at least one of the visual information and geometric information at the second time of day predicted by the prediction unit.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus, a control method thereof, and a program.

  Measurement of the position and orientation of the imaging device based on image information has various purposes such as alignment between real space and virtual object in mixed reality / augmented reality, self-position estimation of robot or car, 3D modeling of object or space, etc. Used in

  In Non-Patent Document 1, geometric information (depth information), which is an index for calculating position and orientation from an image, is estimated using a learning model learned in advance, and position and orientation is calculated based on the estimated depth information. Methods are disclosed.

K. Tateno, F. Tombari, I. Laina and N. Navab, "CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction", In Proc. CVPR, 2017. Z. Zhang, "A flexible new technique for camera calibration," Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp. 1330-1334, 2000. X. Shi, Z. Chen, H. Wang, and D. Yeung, "Convolutional LSTM Network: A Machine Learning Approach for Precitation Nowcasting", In Proc. NIPS, 2015. I. Laina, C. Rupprecht, V. Belagiannis, F. Tombari, and N. Navab. "Deeper depth prediction with fully convolutional residual networks". In Proc. 3DV, 2016. K. He, G. Gkioxari, P. Dollar, and R. Girshick, "Mask R-CNN", In Proc. ICCV, 2017.

  In Non-Patent Document 1, based on an input image captured by an imaging device, geometric information (depth information) at the time of capturing the input image is estimated, and then the position and orientation of the imaging device are calculated. It takes a lot of time to estimate geometric information and calculate position and orientation. Non-Patent Document 1 assumes that the imaging device does not move significantly during the calculation. That is, there is a need for a solution for calculating the position and orientation at high speed and with high accuracy under conditions where the imaging device moves during estimation of geometric information and during position and orientation calculation.

In order to solve this problem, for example, the information processing apparatus of the present invention has the following configuration. That is,
An input unit that inputs visual information captured by the imaging device at a first time;
Holding means for holding a learning model for predicting at least one of visual information and geometric information at a second time after the first time based on the visual information input by the input means;
Prediction means for predicting at least one of visual information and geometric information at the second time using the learning model;
And a first calculation unit that calculates the position and orientation of the imaging device at the second time based on at least one of visual information and geometric information of the second time predicted by the prediction unit.

  According to the present invention, even if the imaging apparatus is moving, the position and orientation can be calculated with high accuracy at high speed.

BRIEF DESCRIPTION OF THE DRAWINGS The block diagram which shows the structural example of the operation control system in 1st Embodiment. FIG. 1 is a block diagram showing the arrangement of an information processing apparatus according to the first embodiment; FIG. 7 is a view showing a data structure of a holding unit in the first embodiment. FIG. 2 is a diagram showing a hardware configuration of the information processing apparatus in the first embodiment. 3 is a flowchart showing a processing procedure in the first embodiment. 10 is a flowchart showing a processing procedure of position and orientation calculation in the second embodiment. The flowchart which shows the process sequence in 3rd Embodiment. The flowchart which shows the processing procedure of object information calculation in a 3rd embodiment. A figure showing an example of GUI which presents display information in a 3rd embodiment.

  Hereinafter, embodiments according to the present invention will be described in detail with reference to the drawings. In addition, the structure shown in the following embodiment is only an example, and this invention is not limited to the illustrated structure.

First Embodiment
An information processing apparatus according to the present embodiment will be described by way of an example mounted on a mobile body such as a car. The information processing apparatus calculates the position and attitude (hereinafter referred to as position and attitude) of the car, and controls the car such as steering, acceleration, and braking of the car based on the calculation result, that is, performs automatic driving. Further, the image pickup apparatus in the present embodiment is a camera which is a single eye and picks up image data of three components of R, G and B per pixel, and in the vicinity of a windshield of a car and its optical axis direction It shall be installed so as to coincide with the straight direction of the car. The relative mounting position information of the imaging device to the car with respect to the center of gravity of the car is known and stored in advance in a non-volatile memory or the like. Therefore, the position and orientation of the automobile can be calculated by performing the geometric transformation based on the information stored and held in the position and orientation of the imaging device calculated based on the image captured by the imaging device. That is, determining the position and orientation of the imaging device is equivalent to determining the position and orientation of the vehicle. The calculation of the position and orientation of the imaging device will be described later. Then, for control of the vehicle, the vehicle is controlled so that the calculated position and orientation of the vehicle and the planned route to the destination of the vehicle calculated in advance by the car navigation system coincide with each other.

  For calculation of the position and orientation of the imaging device, geometric information estimated by an estimator based on machine learning based on an input image captured by the imaging device is used. The geometric information estimated by the estimator based on machine learning in the present embodiment is a depth map which is depth information estimated for each pixel of the input image. Specifically, first, each point of the depth map estimated by the estimator using the input image captured at a certain time t ′ other than the time t is input to the input image captured at a certain time t at a certain time t Project to the input image. The term “projection” here is used to calculate which pixel on the image at time t the three-dimensional point (representing the position of X, Y, Z in space) to be obtained from each depth value of the depth map at time t ′ is calculated. It is. Next, the position and orientation are calculated so that the difference between the brightness of the pixel of the input image at time t corresponding to each pixel of the projection source depth map and the brightness of the pixel of the input image at time t ′ to be projected is minimized. Do. These methods are described in detail in Non-Patent Document 1 and can be incorporated herein.

  At this time, the time at which the position and orientation are calculated is t + Δt. Here, Δt is a processing time taken until the CNN (Convolutional Neural Network) estimates the depth and further calculates the position and orientation based on them. When the vehicle is traveling, the vehicle moves for Δt time. However, at time t + Δt at which the position and orientation are calculated and controlled, the vehicle is controlled using the position and orientation at time t, and the control is delayed. Therefore, in the present embodiment, a method of calculating the position and orientation at time t + Δt based on an image predicted using a learning model capable of predicting a future image and a depth map by Δt time and the depth map will be described.

  The position and orientation of the imaging device in the present embodiment are six parameters including three parameters representing the position of the camera at world coordinates defined in the real space and three parameters representing the orientation of the camera. Further, a three-dimensional coordinate system defined on the camera whose optical axis of the camera is Z axis, the horizontal direction of the image is X axis, and the vertical direction is Y axis is called camera coordinate system.

  FIG. 1 is a schematic view of the configuration of an automobile 1 on which the information processing apparatus 10 according to the first embodiment is mounted. The automobile 1 is an imaging device 11 which is a camera, an accelerator / brake device which can change the speed of the automobile by changing the number of rotations of the tire, and a steering device which can change direction by changing the direction of the tire. A controller 12 is provided to perform the

  FIG. 2 is a functional block diagram of the information processing apparatus 10 in the present embodiment. The information processing apparatus 10 includes an input unit 110, a prediction unit 120, a holding unit 130, a calculation unit 140, and a control unit 150. The input unit 110 is connected to an imaging device 11 mounted on a car. The control unit 150 is connected to the control device 12 which is a steering, an accelerator, and a brake device of a vehicle. Note that FIG. 1 is an example of the device configuration, and the scope of application of the present invention is not limited.

  The input unit 110 inputs, as an input image, a two-dimensional image of a scene captured by the imaging device 11 in time series (for example, 60 frames per second), and outputs the image to the prediction unit 120 and the storage unit 130.

  The prediction unit 120 predicts an image after a preset time and its geometric information from the image input from the input unit 110 according to the learning model held by the holding unit 130. Hereinafter, the predicted image is referred to as a predicted image, and the predicted geometric information is referred to as predicted geometric information.

  The holding unit 130 holds a learning model. When the learning model inputs an image, it can predict the future image and depth map by Δt time. Note that this learning model internally holds a history of information used for prediction, and updates the internal state each time an image is input. That is, since the prediction is performed in consideration of the movement amount of each pixel in the latest image with respect to the past input, the present invention can be applied even when the vehicle speed changes. Details of this learning model will be described later. Further, the holding unit 130 associates an image of a scene photographed in advance with the position and orientation to make a key frame, and holds a key frame group as a map. Details of the data structure in the holding unit 130 and the learning model will be described later.

  The calculation unit 140 calculates the position and orientation of the imaging device 11 based on the predicted image predicted by the prediction unit 120, the predicted geometric information, and the map held by the holding unit 130. Then, the calculation unit 140 obtains the position and orientation of the vehicle based on the calculated position and orientation of the imaging device, and supplies the position and orientation to the control unit 150.

  The control unit 150 calculates the control value of the vehicle based on the position and orientation of the vehicle input from the calculation unit 140, and supplies the control value to the control device 12.

  The control device 12 controls the automobile 1 based on the control value calculated by the control unit 150.

  FIG. 3 is a diagram showing a data structure held by the holding unit 130. As shown in FIG. In the present embodiment, the holding unit 130 holds a learning model for the prediction unit 120 to predict a predicted image and geometric information. The learning model is, for example, a data file in which CNN classifiers are stored in binary format. Further, the holding unit 130 holds a map that is an index of position and orientation calculation. In the map, the image of the scene is associated with the position and orientation to form a key frame, and a plurality of key frames are held. Note that the position and orientation referred to here are six parameters of the position and orientation of the imaging device 11 that has captured the corresponding key frame.

  FIG. 4 is a diagram showing a hardware configuration of the information processing apparatus 10. As shown in FIG. The information processing apparatus 10 includes a CPU 311, a ROM 312, a RAM 313, an external memory 314, an input unit 315, a display unit 316, a communication I / F 317, and an I / O 318.

  The CPU 11 controls the entire information processing apparatus 10 by controlling various devices connected to the system bus 321. The ROM 312 stores a BIOS program and a boot program. A RAM 313 is a memory used as a main storage device of the CPU 311, and stores an operating system (OS) functioning as the information processing apparatus 10 and an application program. The external memory 314 is typically a large-capacity storage device such as a hard disk, and stores an OS, an application program for driving support described in the embodiment, and various data. An input unit 315 is a touch panel display and various input buttons, and performs processing related to input of information and the like. The display unit 316 outputs the calculation result of the information processing device 10 to the display device in accordance with an instruction from the CPU 311. The display device may be of any type such as a liquid crystal display device, a projector, or an LED indicator. The communication I / F (interface) 317 performs information communication via a network, and may be, for example, an Ethernet I / F, regardless of the type, such as USB, serial communication, or wireless communication. The I / O 18 is connected to 319 of the imaging device 11 and the control device 12.

  In the above configuration, when the apparatus is powered on, the CPU 311 loads the OS from the external memory 314 into the RAM 313 and executes it by executing the boot program stored in the ROM 312. Then, the CPU 311 loads an application program related to the operation of the automobile 1 from the external memory 314 into the RAM 313 under the control of the OS and executes the program to realize the functional configuration shown in FIG. The holding unit 130 in FIG. 2 is realized by the RAM 313. In the embodiment, the components in FIG. 2 are realized by software executed by the CPU 311. However, part of the software may be realized by hardware independent of the CPU 311.

  Next, the processing procedure of the information processing apparatus 10 in the present embodiment will be described according to the flowchart of FIG. The steps constituting this process are initialization S110, image capture S120, image input S130, geometric information prediction S140, position and orientation calculation S150, control value calculation S160, control S170, and system termination determination S180.

  In step S110, the CPU 311 initializes the system. That is, the OS and the program are read from the external memory 14 to make the information processing apparatus 10 operable. Further, the parameters of each device (such as the imaging device 11) connected to the information processing device 10 and the initial position and orientation of the imaging device 11 are read. The internal parameters (focal length fx (horizontal direction of image), fy (vertical direction of image), image center position cx (horizontal direction of image), cy (vertical direction of image), lens distortion parameter) of the imaging device 11 are It is assumed that it is calibrated in advance by the method of Zhang (see Non-Patent Document 2), and each control device of the automobile 1 is activated to be in an operable / controllable state.

  In step S120, the imaging device 11 captures a scene, and supplies the captured image to the input unit 110.

  In step S130, input unit 110 acquires an image captured by imaging device 11 as an input image. In the present embodiment, the input image is an RGB image in which one pixel is composed of three components of R, G, and B.

  In step S140, the prediction unit 120 estimates a prediction image and prediction geometric information from the input image with reference to the learning model held by the holding unit 130. Note that the estimator based on machine learning in the present embodiment is a CNN (Convolutional Neural Network). It is assumed that this CNN has been learned so as to be able to predict a predicted image of time t + Δt after Δt time and predicted geometric information when an image at a certain time t is input. The CNN network configuration and learning method will be described later.

  In step S <b> 150, the calculation unit 140 calculates the position and orientation of the imaging device 11 based on the predicted image and predicted geometric information predicted by the prediction unit 120 and the map held by the holding unit 130. Specifically, the calculation unit 140 projects the predicted geometric information onto the nearest key frame in the map by the above-described position and orientation calculation method. Then, the calculation unit 140 calculates the position and orientation of the imaging device 11 at time t + Δt so as to minimize the luminance difference between each pixel value of the predicted image and each pixel value of the key frame associated by projection. Do.

  In step S160, the control unit 150 minimizes the distance between the position and orientation obtained by the calculation of step S150 and the route information calculated in advance from the information of the destination of the car and the place of departure. Calculate a control value to control the steering of the car. The route information referred to here is a list storing three-dimensional positions (referred to as anchor points) representing positions on the route calculated in advance from the departure place and the destination by the car navigation system. Then, the control unit 150 selects an anchor point at which the square distance between the calculated position of the imaging device 11 and the anchor point is minimum, and further calculates a direction vector in which the distance between the anchor point and the imaging device 11 decreases. Supply to 12.

  In step S170, control device 12 controls steering in the direction of the direction vector in which vehicle 1 is calculated according to the provided information.

  Then, in step S180, the CPU 311 repeats the processing of step S120 and subsequent steps until the system is terminated (the instruction to cancel the driving support by the user or the arrival to the destination).

  Next, the network configuration and learning method of the CNN for estimating the predicted image and the predicted geometric information described above will be described. The CNN in the present embodiment is a network structure that can hold information of past input images, and a network structure that can estimate a depth map from images, such as ConvLSTM (Convolutional Long short-term memory). The configuration is based on Fully Convolutional Network. While estimating the prediction image which is a future image from the image input in the past by ConvLSTM, it enabled it to estimate prediction geometric information from a prediction image by FCN. As a specific configuration, first, the output layer of ConvLSTM is branched into two. One is used as a first output to output a predicted image. The other output layer is connected to the FCN as the input layer of the FCN, and the output of the FCN is output as the second output to output predicted geometric information. Then, the following learning procedure is performed so that prediction can be accurately performed.

  Specifically, first, ConvLSTM and FCN are separately learned, and then they are combined and learned. First, when ConvLSTM inputs an image at time t, learning is performed so that a predicted image at time t + Δt can be estimated. Among the learning images captured in advance by the on-vehicle camera every Δt time, predetermined M images before time t are sequentially input to ConvLSTM. At this time, only ConvLSTM is learned so that an error between the predicted image predicted by ConvLSTM when the image at time t is finished and the image of t + Δt is minimized. By learning in this manner, the ConvLSTM layer internally holds the image information input in the past when the imaging device 11 inputs an image, and the Δt time in consideration of the amount of change in the image information in the past Later images can be predicted. In particular, by learning using a learning image captured at different vehicle speeds, an image after Δt time can be predicted according to the vehicle speed even when images with different vehicle speeds are input at the time of estimation. Become. Next, to be able to predict geometric information from the image by the FCN layer, predicted geometric information estimated by inputting a learning image at time t captured in advance to the FCN, and a depth map captured by the depth camera at the same time t in advance The FCN is learned so as to minimize the error in the depth value of. By doing this, the FCN can estimate the depth map when the image is input. Finally, in order to reduce the inconsistency between the prediction results of ConvLSTM and FCN, ConvLSTM and FCN are connected and learned. The learning image at time t taken every Δt time taken in advance is input and the network is reduced so that the error between the first output of ConvLSTM and the image at time t + Δt, the output of FCN and the depth map at time t + Δt is reduced learn. The CNN learned as described above was used as a learning model.

  The details of ConvLSTM are disclosed in Non-Patent Document 3. The details of FCN for estimating a depth map are disclosed in Non-Patent Document 4, and these can be used.

<Effect>
In the first embodiment, the prediction unit predicts a predicted image and predicted geometric information after the time when the imaging device captures an input image, and the calculation unit calculates the position and orientation of the imaging device based on the prediction result. By calculating the position and orientation based on the prediction result as described above, it is possible to calculate the position and orientation at high speed without being influenced by the delay due to the estimation of geometric information and the calculation of the position and orientation taking time for processing. it can.

<Modification>
The calculation unit 140 in the first embodiment calculates the position and orientation based on the predicted geometric information, the predicted image, and the map. However, a configuration capable of calculating the position and orientation of the imaging device 11 after the time when the imaging device 11 captures an input image using at least one or both of the predicted geometric information and the predicted image predicted by the prediction unit 120 The method is not limited to the method described in the first embodiment.

  Specifically, the image feature is calculated from the predicted image and the image held in the key frame in the map, and the position and orientation are calculated using a five-point algorithm that calculates the position and orientation from the positional relationship between feature amounts having the same characteristics. You can also Here, the image feature is a point having a specific feature such as a corner in the image, and the feature amount is calculated from the image patch of the small area around the feature point for uniquely describing each image feature. Is an identifier that As the feature amount, a scale invariant feature transform (SIFT) feature amount or an accelerated-kaze (AKAZE) feature amount can be used. In such a configuration, the learning model only needs to predict the predicted image, and does not have to predict predicted geometric information. A method of constructing a learning model in such a configuration will be described later.

  When the key frame is configured to further hold the depth map, the depth map of the key frame is projected onto the depth map, which is predictive geometric information, and the distance between the nearest points of the three-dimensional points is minimum. The position and orientation may be calculated using an ICP (Iterative Closest Point) algorithm that calculates the position and orientation so that In such a configuration, the learning model only needs to predict prediction geometric information, and does not have to predict a prediction image. A method of constructing a learning model in such a configuration will also be described later.

  In the first embodiment, a learning model in which ConvLSTM and FCN are combined is used as the learning model. However, any configuration may be employed as long as information for calculating the position and orientation of the time after time t taken by the imaging device 11 can be predicted. Specifically, in the first embodiment, a neural network that considers local positional relationship of images such as ConvLSTM and FCN is used, but instead of them, LSTM or Dense Network is used to consider the entire image. Network configuration. Further, in the present embodiment, the output of ConvLSTM is divided into two, and a predicted image is estimated by one of the outputs, and a depth map is estimated by FCN with the other output as an input. However, different learning models may be prepared and predicted, such as a learning model that estimates only a predicted image and a learning model that calculates predicted geometric information. Furthermore, as described in the modification, only the ConvLSTM may be used when only the predicted image may be predicted, or when only the predicted geometric information may be predicted, the ConvLSTM as described in the first embodiment may be used. Alternatively, one output layer may be connected to the input layer of the FCN without dividing the output of the circuit into two.

  In the first embodiment, the configuration in which the imaging device 11 for capturing an image is an RGB camera has been described. However, the present invention is not limited to the RGB camera, and there is no particular limitation as long as it is a camera for capturing an image in real space, and for example, a camera for capturing a grayscale image may be used. Moreover, it may be a monocular camera, or may be a camera provided with two or more cameras and sensors. Furthermore, it may be an apparatus capable of acquiring geometric information such as depth information, distance images, and three-dimensional point cloud data. Specifically, a TOF (Time Of Flight) camera, LiDAR (Laser Imaging Detection and Ranging), a sonar sensor, etc. correspond. When using a camera other than these RGB cameras, the learning model is configured to be able to predict geometric information of a time earlier than the time of acquisition thereof, using an image or geometrical information acquired by the camera or measuring device used. Do. In such a configuration, learning models may be prepared in advance in accordance with various sensors. For example, in the case of predicting position / orientation by predicting predicted geometric information of time t + Δt from input geometric information (depth map) of time t acquired by the TOF camera, the learning model inputs the input geometric information of time t in advance. In this case, learning may be performed so that an error in depth value between predicted geometric information and input geometric information at time t + Δt is reduced.

  In the first embodiment, the vehicle is controlled based on the position and orientation calculated based on the prediction result. On the other hand, it can also be used in the case of shortening the processing time in the case of calculating a position and orientation based on an actual input image instead of a predicted image. Specifically, the initial value of the optimization calculation in the position and orientation calculation is set as the position and orientation calculated in the past from the predicted image. By setting the position and orientation obtained by prediction in advance as initial values, it is possible to reduce the number of loops in repetitive calculation of position and orientation calculation until convergence, and furthermore, according to actual measurement values instead of using uncertain prediction. The position and orientation can be calculated, and the position and orientation can be calculated at high speed and with high accuracy.

  In the first embodiment, the calculation unit 140 calculates the position and orientation of the imaging device 11 (as a result, the position and orientation of the automobile 1) using the predicted image and predicted geometric information predicted by the prediction unit 120. On the other hand, the predicted image and predicted geometric information may be used in combination with the calculated value of SLAM (Simultaneous Localization and Mapping) not using prediction. Specifically, first, using a plurality of input images before time t, a three-dimensional point representing a point group in a three-dimensional space in the scene is calculated by the method of Non-Patent Document 1. Next, the calculated three-dimensional point is projected on prediction geometric information. Next, geometric information obtained by integrating the projected three-dimensional point and the depth of the prediction geometric information by weighted sum and using high precision is used as prediction geometric information. In the present modification, the reliability described in Non-Patent Document 1 can be used as a weight. In the method of Non-Patent Document 1, three-dimensional points are calculated by motion stereo from input images at multiple times. The reliability of each point is the reciprocal of the variance of the depth value. In the above configuration, a point having a high degree of reliability of the three-dimensional point obtained by SLAM is used instead of each point of the depth map of the geometric information predicted by the prediction unit 120, or by integrating them Information can be made more accurate, and position and orientation can be calculated with high accuracy.

  In the first embodiment, based on the input image at time t captured by the imaging device 11, the calculating unit 140 estimates a predicted image and predicted geometric information at time t + Δt that is one time at which the predicting unit 120 is present. The position and orientation of the imaging device 11 after time t + Δt were calculated. However, if what the prediction unit 120 predicts is a predicted image or geometric information after time t captured by the imaging device 11, it may be configured to estimate predicted images or predictive geometric information at a plurality of earlier times. Specifically, the prediction unit 120 inputs a predicted image at time t + nΔτ to the learning model, and predicts a predicted image at time t + (n + 1) Δτ. By doing this, it is possible to predict predicted images and predicted geometric information at a time further after time t + Δt without changing the learning model, and it is possible to calculate the position and orientation of those times. By controlling the vehicle so that the distance between the calculated position and orientation of the future at a plurality of times calculated in this way and the calculated route is minimized, the error in the position and orientation from the latest route is even larger. Thus, the vehicle can be controlled so that the error at time t becomes small, the vehicle does not have to be suddenly braked, and the vehicle can be stably controlled.

  In the first embodiment, the vehicle is controlled based on the position and orientation calculated based on the prediction result. On the other hand, it is also possible to decide whether to use the prediction or not according to the prediction performance. Specifically, the luminance difference between each pixel of the predicted image at time t + Δt predicted by the calculation unit 140 based on the input image at time t and the input image captured by the imaging device 11 at a time near t + Δt actually If the average, maximum, minimum, and median are within a predetermined threshold, it is determined that the prediction performance is high, and the predicted image and predicted geometric information are used for position and orientation calculation. On the other hand, when it becomes more than the threshold value, the prediction performance is low, and the position and orientation are calculated based on the input image without using the prediction image and the prediction geometric information for position and orientation calculation. In addition, when the prediction performance is low, prediction can be stopped. With such a configuration, it is possible to avoid that the position and orientation estimation accuracy decreases when the prediction accuracy is low.

  In the modified example, the use of the prediction is determined based on the degree of coincidence between the predicted image and the input image. The control value may be changed based on the degree of coincidence between the predicted image and the input image. For example, when the prediction accuracy is high, the upper limit of the vehicle speed is increased, and when the prediction accuracy is low, the vehicle is controlled to decrease the upper limit of the vehicle speed. In this way, when the prediction performance is poor, the vehicle can be controlled safely by reducing the speed. In addition, when the predicted performance is poor, the driver may request control of the vehicle. Also, the control center may be requested to control the vehicle via the network.

  In the modification, the control value is changed based on the degree of coincidence between the predicted image and the input image. By the way, prediction parameters may be dynamically adjusted based on the degree of coincidence between the prediction image and the input image. The prediction parameter referred to here is a prediction time Δt at t + Δt representing a time to be predicted, and a predicted number N (maximum value of n) at which time in the prediction time t + nΔτ in the case of prediction of the plural times , Δτ representing the prediction interval. As a specific adjustment example of the prediction parameter, if the calculation unit 140 determines that the prediction accuracy is low, the adjustment is made to shorten the prediction time Δt, reduce the value of the prediction number N, and reduce the prediction interval Δτ. You may Generally, the longer the prediction period, the lower the prediction accuracy, but such adjustment makes it possible to secure the prediction accuracy of the prediction value of the prediction unit 120. In order to reflect the prediction time and the prediction intervals Δt and Δτ in the predictor, an input layer is added to the predictor to input one time information in addition to the input image. Further, in the case of such a network configuration, an input image at time t and an arbitrary value of Δt are input, and learning is performed so that the image at time t + Δt matches the prediction value of the predictor. Also, a region of interest (ROI) representing a region to be predicted may be used as a prediction parameter. In other words, the prediction may be performed only for a part of the input image. In this case, the input image and the predicted image are each divided into predetermined small areas, and the calculation unit 140 calculates the prediction accuracy for each small area. At this time, only a small area whose prediction accuracy is equal to or more than a predetermined threshold is used to predict the subsequent time.

  In the modification, the matching degree of the prediction image and the input image is used to adjust the prediction parameter. However, the prediction parameters are adjusted based on the input to the information processing apparatus 10, the predicted value of the prediction unit 120, the calculated value of the calculation unit 140, the control value of the control unit 150, or the state of the vehicle on which the information processing apparatus 1 is mounted. There is no restriction in particular. Specifically, the calculation unit 140 calculates a predicted image, a predicted position / posture at time t + Δt calculated based on predicted geometric information, and a calculated position / posture calculated based on an input image captured by the imaging device 11 around the time t + Δt. As the square error of the error is larger, the values of the prediction time Δt, the prediction interval Δτ, and the number of predictions N are made smaller.

  Also, the prediction parameters may be adjusted based on the state of the vehicle. Here, as an example, a configuration will be described in which the prediction parameter is adjusted with the braking distance of the vehicle measured in advance. In the case of a vehicle with a large braking distance, it takes time for the vehicle to complete the desired operation after the control, so shortening the time for performing the desired operation to perform control using earlier prediction Is desired. For this reason, the prediction time Δt is increased as the weight of the vehicle is increased, the value of the predicted number N is increased, and the prediction interval Δτ is increased. In addition to the braking distance, as the weight is larger, the contact area of the tire is smaller, the speed of the vehicle is higher, and the number of passengers of the vehicle is larger, the respective parameters may be increased. Also, the prediction range, which is the range of the image to be predicted, may be adjusted according to the speed and steering of the automobile. Specifically, when the speed of the vehicle is high, the prediction range is small, and when the speed is low, the prediction range is large. By adjusting the prediction range in this manner, when the prediction time interval or the prediction time becomes long, even when the number of predictions increases, the time relating to the prediction can be made constant time.

  In the first embodiment, the map held by the holding unit 130 is a key frame group in which the image of the scene photographed in advance and the position / posture at that time are associated. However, the map is not limited to the above configuration, and any map may be used as long as it can calculate the position and orientation between the predicted image predicted by the prediction unit 120 and the predicted geometric information. Specifically, a three-dimensional point group representing a point group in three-dimensional space acquired by LiDAR is projected onto an image captured by an omnidirectional camera and points that coincide are determined, and the brightness parameter of the image coincident with each three-dimensional point A four-dimensional point group having four parameters added with may be held as a map. With this configuration, it is possible to project the four-dimensional point group onto the predicted image and calculate the viewpoint position of the predicted image so that the difference in luminance between the predicted image and the projected point group is reduced. Alternatively, only a three-dimensional point group acquired by LiDAR may be held as a 3D map. In this case, the three-dimensional point group may be projected onto the depth map, which is predictive geometric information, and the position and orientation may be calculated such that the distance between the three-dimensional point group and the predicted depth map is minimized by the ICP algorithm. . With this configuration, the information processing apparatus can be applied to a configuration in which the map held by the holding unit 130 is changed.

  In the first embodiment, the holding unit 130 holds the map created in advance. However, when calculating the position and orientation, the calculation unit 140 may use a SLAM in which the position and orientation is estimated while creating a map based on the predicted position and orientation, instead of using a map created in advance. CNN-SLAM can be used as a method of generating a map. The CNN-SLAM is disclosed in detail in Non-Patent Document 1 and can be incorporated herein. However, in CNN-SLAM, although the input image which the imaging device 11 imaged was used, when adapting to this structure, a predicted image can also be used instead of an input image. With such a configuration, it is possible to perform self-position estimation and control of the vehicle while performing prediction even in a place where a map is not created in advance.

  In the first embodiment, the holding unit 130 holds the map. However, the holding unit 130 may not hold the map. Specifically, a predicted image predicted by the prediction unit 120 based on the input image at t ′ ′ one time before time t, the position and orientation calculated by the calculation unit 140, and the input image at time t The calculation unit 140 calculates the amount of change in position and orientation at time t + Δt with respect to time t ′ ′ using the predicted image at time t + Δt predicted for the above. The position / posture of the imaging device 11 can be calculated by performing the above-described process each time the input unit 110 inputs an image and integrating the position / posture change amount calculated each time.

  In the first embodiment, since the learning model is configured to internally hold the history of the input image, prediction according to the speed of the vehicle can be performed. However, a plurality of learning models corresponding to the vehicle speed may be prepared and switched and used according to the vehicle speed. That is, the holding unit 130 holds a plurality of learning models. The number of learning models to be held is preferably several minutes according to the vehicle speed, but for example 10km /, 20km / h, ... and 10km / h as a learning model, the learning model to be used is closest to the vehicle speed of the car 1 at that time It shall be selected. The vehicle speed is acquired from the control device 12. Such a configuration can also be made.

  In the first embodiment, the vehicle is controlled based on the position and orientation calculated by the calculation unit 140. However, the target to be controlled is not limited to a car, and there is no particular limitation as long as it is an apparatus that moves autonomously. For example, the information processing apparatus 10 according to the embodiment may be applied to an autonomous mobile robot such as an automated guided vehicle (AGV) that transports materials in a factory, a warehouse, or a shop, or an autonomous mobile robot (AMR).

  Moreover, you may use not only automatic driving | running | working but the structure which assists the operation which a person drive. Specifically, when the control value calculated at high speed by prediction and the operation of a person deviate, an alert can be alerted quickly by sounding an alert. Furthermore, in such a case, the vehicle may be controlled with a control value calculated as a person misoperated.

  Furthermore, the information processing apparatus may be used as an apparatus for calculating the position and orientation. Specifically, the method of the present invention is applied to alignment between real space and virtual object in a mixed reality system, that is, measurement of the position and orientation of the imaging device 11 in the real space for use in drawing a virtual object. You may In such a configuration, the control unit 150 and the control device 12 in FIG. 1 are not necessary, and a virtual object drawn based on the position and orientation calculated by the calculation unit 140 based on the predicted image and the predicted geometric information. CG images are superimposed and presented to the user through the display of the mobile terminal or head mounted display. With such a configuration, it is possible to reduce the delay until the presentation of the image due to the calculation time in the conventional position and orientation calculation and the drawing of CG.

Second Embodiment
In the first embodiment, the position and orientation of the imaging device 11 is calculated using a predicted image, predicted geometric information, and a map that is a calculation index of position and orientation created based on a past input image. It was to control. In the second embodiment, not only maps created based on past input images but also maps are created using predicted images and predicted geometric information, and the position and orientation of the imaging device are determined using the created maps. An example of calculation will be described.

  Even if there is no map created in advance, it is possible to calculate the position and orientation of the imaging device 11 by using Visual SLAM (Simultaneous Localization and Mapping) that performs self-position estimation while creating a map based on an input image. . In Visual SLAM, the input image and the position and orientation calculated based on the input image are associated and held as key frames. The conventional Visual SLAM creates a key frame only at a position which has passed in the past (a position at which an input image is captured). Thus, in the second embodiment, key frames are not generated only at points that have passed in the past, but are generated using predicted images and predicted geometric information to generate key frames in the vicinity of a point where a vehicle is predicted to move. Also, the created key frame is updated using the input image to make it more accurate. In the second embodiment, a method of generating and updating a key frame will be described, and a method of calculating and controlling a position and orientation of a vehicle using these will be described.

  The configuration of the apparatus in the second embodiment is the same as that of FIG. 1 showing the configuration of the information processing apparatus 1 described in the first embodiment, and therefore the description thereof will be omitted. In the second embodiment, the calculation unit 140 calculates a key frame at time t ′ after time t at which the imaging device 11 captures an input image.

  Further, the calculation unit 140 updates the key frame held by the holding unit 130. And the point which outputs and hold | maintains the key frame produced or updated to the holding | maintenance part 130 differs from 1st Embodiment. In the second embodiment, the RGB image of the scene captured by the imaging device 11 or the RGB image which is a predicted image predicted by the prediction unit 120, the depth map as geometric information, and the position and orientation of those viewpoints Let the data structure which associated with be a key frame. Furthermore, the key frame group is held in the holding unit 130 as a map.

  The entire processing procedure in the second embodiment is the same as that in FIG. 5 showing the processing procedure of the information processing apparatus 1 described in the first embodiment, and thus the description thereof is omitted. The difference from the first embodiment is that, in addition to the calculation unit 140 calculating the position and orientation in step S150, a key frame at time t 'after time t is calculated and updated.

  FIG. 6 is a flowchart showing details of the processing procedure of step S150 in the calculation unit 140 of the information processing apparatus 10 in the second embodiment.

  In step S2110, the calculation unit 140 calculates the position and orientation of the imaging device 11 using the predicted image from the prediction unit 120 and the key frame held by the holding unit 130. Specifically, first, the depth map of the key frame is projected onto the predicted image. Next, the position and orientation of t + Δt are calculated such that the luminance difference between each pixel value of the predicted image and each pixel value of the image held by the key frame associated by projection is minimized.

  In step S2120, calculation unit 140 calculates a key frame and outputs the key frame to holding unit 130. First, the calculation unit 140 determines whether to add a key frame. Specifically, calculation unit 140 selects a key frame that minimizes the square distance with the position and orientation calculated in step S2110, and the square distance with the key frame is equal to or greater than a predetermined distance and the sight line direction is predetermined. If it is separated by more than the angle of, it is judged that the key frame is added. If it is determined that the key frame is to be added, the calculation unit 140 uses the predicted image predicted by the prediction unit 120, the predicted geometric information, and the position and orientation calculated in step S2110 by the calculation unit 140 as the key frame in the holding unit 130. It outputs, and the holding unit 130 holds the same.

  In step S2130, calculation unit 140 updates the depth value of the depth map held by the key frame based on the predicted geometric information and the position and orientation calculated by calculation unit 140 in step S2110. First, the calculation unit 140 selects, from the map held by the holding unit 130, a key frame whose distance from the position and orientation calculated in step S2110 and the sight line direction is less than a predetermined threshold. Next, a depth map, which is predictive geometric information, is projected onto the selected key frame. Then, the depth map which is the geometric information held by the key frame and the depth map of the projected predicted geometric information are updated by time series filtering (smoothing). As time-series filtering, first, a correspondence is regarded that is regarded as the same three-dimensional point among three-dimensional points which can be calculated from two depth maps by the ICP algorithm. Next, weighted average positions of two three-dimensional points considered to be identical are calculated. The depth map of the key frame of the storage unit 130 is updated using the calculated weighted average position as the depth value of the relevant pixel of the depth map.

<Effect>
As described above, in the second embodiment, a key frame is created using a predicted image and predicted geometric information. Further, the created key frame is updated in time series using a predicted image and predicted geometric information. Then, the position and orientation are calculated using the created map. With such a configuration, it is possible to create, in the conventional Visual SLAM, a key frame that can be created only at the point where the imaging device 11 passes, in the vicinity of the point where it passes in the future. Then, by further updating the key frames created in advance in time series, it is possible to calculate a depth map as geometric information with high accuracy. By using the accurate map created as described above, the position and orientation can be calculated with high accuracy.

<Modification>
In the second embodiment, the calculation unit 140 updates the depth value of the depth map of the key frame of the map held by the holding unit 130 by time-series filtering each time the predicted image is predicted. However, the method of updating the map is not particularly limited as long as the map can be refined. Specifically, updating can be performed according to the method of Non-Patent Document 1 in which the depth map of a key frame is updated by time series filtering using depth values calculated by motion stereo method using an input image or a predicted image. Also, multiple key frames created in the past may be integrated and updated. The integration referred to here is to first calculate the position and orientation of the key frame using pose graph optimization so that there is no contradiction in the position and orientation of a plurality of key frames. Next, the obtained position and orientation are used to further smooth the depth maps of a plurality of key frames. In smoothing, the same method as the above-mentioned time-series filtering can be used. As described above, by updating the map using input images at plural times and predicted images, a highly accurate map can be generated, and the calculation accuracy of the position and orientation is improved.

  In the second embodiment, the position and orientation are calculated based on the map generated based on the predicted value calculated by the prediction unit 120. On the other hand, the control value of the vehicle may be changed based on the calculated map. Specifically, first, the calculation unit 140 calculates the three-dimensional shape of the scene from the predicted depth map of the key frame. At this time, when a three-dimensional structure exists in the traveling direction of the vehicle, the control device 12 calculates the control value so as to reduce the speed of the vehicle. Alternatively, a three-dimensional shape may be calculated from the predicted map, the degree of unevenness of the road surface may be calculated, and a control value for performing control to avoid a position where the unevenness is predicted to be large may be calculated. As described above, by controlling the vehicle using the map created based on the predicted value, it is possible to control in advance before measurement by the sensor, and it is possible to reduce sudden braking of the vehicle.

  Also in the second embodiment, as described in the first embodiment, when the map created in advance is held, the difference between the predicted key frame and the map created in advance is used. It is also possible to switch whether to calculate the position and orientation using a map created in advance. The difference here is a value that is calculated by the calculation unit 140 and represents the degree of coincidence between the predicted key frame and the map. Specifically, the map created in advance is projected onto the depth map of the predicted key frame, and the average value, median, maximum value, minimum value, and depth map of those differences are divided into small areas and the area map is calculated for each area It is possible to use an average value, a median value, a maximum value, and a minimum value of the difference in depth values. If the above difference value exceeds a predetermined threshold value, the position and orientation are calculated by the method of the second embodiment, assuming that the map and the scene have changed, in advance. On the other hand, if it is below the threshold value, the position and orientation are calculated using a map created in advance as described in the first embodiment. As described above, by determining whether or not to use the map created in advance based on the degree of coincidence between the predicted map and the map created in advance, for example, the time when the map is created in advance by construction and the landscape change Even in the case where the position and orientation can be calculated stably with high accuracy.

Third Embodiment
In the first embodiment, the method of controlling the vehicle by calculating the position and orientation of the imaging device 11 based on the map created in advance using the predicted image and the predicted geometric information has been described. In the second embodiment, a method of generating and updating a map using a predicted image and predicted geometric information has been described. In the third embodiment, a predicted image and predicted geometric information are used to calculate an object type which is object information to be shown in them, a position of the objects, and an amount of movement, and a method of controlling an automobile based on them. explain.

  In car control, not only self-position estimation but also prediction of the position / posture and movement amount of surrounding objects are important. As a specific example, a bicycle goes straight ahead and travels from the front right of the car while traveling, and further, a scene where there is a car parked in front of the bicycle is mentioned. At this time, it is predicted that the bicycle will meander in the left direction to avoid a parked car and collide. For this reason, the collision can be avoided by reducing the speed of the vehicle in advance. Another specific example is a scene where an unclear object pops out from the front of a parked car. At this time, the unclear object is a person and is expected to collide. Therefore, the collision can be avoided by lowering the speed of the vehicle in advance when the collision is predicted. In the third embodiment, a method of controlling the vehicle so as not to collide with the obstacle is described in this way by predicting the surrounding situation, the object type of the surrounding object, and the future position and movement amount of them. .

  The configuration of the apparatus according to the third embodiment is the same as that of FIG. 1 showing the configuration of the information processing apparatus 1 described in the first embodiment, and the description thereof will be omitted. In the third embodiment, the calculation unit 140 calculates an object type and their positions as object information at time t ′ after time t at which the imaging device 11 captures an input image. Also, collision probability information is calculated that represents the collision probability with each object. Further, the collision probability information is output to the control unit 150. Control unit 150 calculates a control value for controlling the vehicle based on the collision possibility information calculated by calculation unit 140, and outputs the calculated control value to control device 12.

  FIG. 7 is a flowchart of the information processing apparatus in the second embodiment. It differs from the first embodiment in that the object information calculation of step S310 is added instead of the position and orientation calculation of step S150 in the first embodiment.

  In step S310, the calculation unit 140 calculates an object type and its position and movement amount included in the predicted image predicted by the prediction unit 120 and the predicted geometric information. Also, based on them, as collision possibility information between a car and an object, a collision probability value for each car and object (a real number in the range of 0 to 1; assuming that the collision probability is higher as 1), collision prediction time , The predicted position of the object at the collision predicted time is calculated. The details of the process of step S310 will be described later.

  In step S160, the calculation unit 140 calculates the control value of the vehicle so that the control unit 150 avoids the collision based on the collision possibility information calculated in step S310. Specifically, when there is an object whose collision probability is equal to or more than a predetermined threshold, the brake for controlling the speed is controlled, and the steering is controlled so as to avoid the predicted position of the object at the collision predicted time. In the present embodiment, the prediction unit 120 predicts the image and the geometric information at a plurality of times (t + nΔt: n = 1,..., N).

  FIG. 8 is a flowchart showing details of object information calculation processing in step S310 of the information processing apparatus in the second embodiment. The present process comprises step S2110 of detecting what kind of object is from the predicted image, step S2120 of calculating the position of each object, and step S2130 of calculating the collision possibility between the object and the vehicle.

  In step S2110, calculation unit 140 detects an object type from the input image. In the present embodiment, in order to detect an object type from an image, a deep neural network is used to calculate an object candidate area from an image, estimate an object type of each candidate area, and calculate an object label for each pixel of the image. It was set as a structure using a certain Mask R-CNN. Note that Non-Patent Document 5 has a detailed description of Mask R-CNN, which can be used. However, the method is not limited to the above method as long as the method can detect the object type from the image.

  In step S2120, the calculation unit 140 obtains the correspondence between the predicted image at each time t + nΔt predicted by the prediction unit 120 and the input image representing the same point, whereby the object detected in step S2110 is a predicted image. Calculate which position is inside. Specifically, by calculating the optical flow between the input image and the predicted image, it is determined which pixel of the predicted image the pixel of the input image corresponds to. Next, the calculation unit 140 allocates the object type label of each pixel calculated in step S2110 to each pixel of the corresponding predicted image. Then, an object type label is similarly allocated to each pixel in the prediction geometric information that matches each pixel of the predicted image. An object type is allocated to each three-dimensional point of prediction geometric information by the above. A predicted position, which is three-dimensional information indicating where the object is located in the three-dimensional space at each predicted time t + nΔt, is derived.

  In step S2130, calculation unit 140 calculates the possibility of collision of an automobile at each time t + nΔt with an object based on the calculated predicted position of each object. The collision possibility is large when the three-dimensional point of the predicted geometric information to which the object type label is allocated in step S2120 is located in the predetermined area in front of the automobile 1, and the farther the object is from the predetermined area It is the smaller value. The specific calculation method of the value of collision possibility is described next. First, the minimum distance d between the three-dimensional point of each class and the braking distance space is calculated, where the space that can be obtained from the braking distance of the vehicle before the vehicle stops is the braking distance space. Next, the value (from 0 to 1) of the base −d root of the natural logarithm is used as the value of the collision possibility. Note that a memory for storing real numbers from 0 to 1 is allocated in advance for each object detected in step S2110, and the value of collision possibility is stored in the memory.

<Effect>
As described above, in the third embodiment, the predicted image, predicted geometric information are used to calculate the type of object which is the surrounding object information, and the predicted position of the object, and the possibility of future collision between the vehicle and each object Calculate And if the possibility of future collision is high, the vehicle is controlled so that the object and the vehicle do not collide. In this way, the object position can be calculated earlier before actual input image acquisition. In addition, it is possible to control the vehicle safely by obtaining the collision possibility between the object and the vehicle based on the predicted object position and controlling the vehicle to avoid the collision based on the collision possibility.

<Modification>
In the third embodiment, the optical flow between the predicted image predicted by the prediction unit 120 and the input image is calculated to calculate the predicted position of the object. However, the prediction unit 120 may use a learning model to calculate the movement amount of the object as geometric information. At this time, when the input image at time t is input, the learning model outputs, for each pixel, a two-dimensional (x, y) movement amount representing to which pixel each pixel in the image is reflected at time t + Δt. This learning model is learning so as to minimize an error between an output when a time-series image (t = 0,..., M) is input to the learning model and an optical flow of an image from t = m to t = m + 1. You should do it. As the learning model, for example, ConvLSTM as described in the first embodiment can be used. In addition, the learning model may be configured to output the movement amount (x, y, z) in the three-dimensional space for each pixel, instead of calculating the two-dimensional movement amount of each pixel of the image. Specifically, an output when a time-series image (t = 0,..., M) is input to a learning model and a three-dimensional vector representing a correspondence between pixels of a depth map from t = m to t = m + 1 The learning may be performed so as to minimize the error with the three-dimensional optical flow storing the three values. Note that, as a learning model at this time, for example, a configuration in which the ConvLSTM and the FCN described in the first embodiment can be used. As described above, when the prediction unit 120 predicts two- or three-dimensional optical flow as prediction geometric information using a learning model in advance, the calculation unit 140 does not have to calculate the optical flow, and the object can be calculated at high speed. The position can be calculated, and the collision possibility information can be calculated quickly.

  In the third embodiment, the object type is detected, and the collision possibility is calculated for each object type. However, if the collision possibility can be calculated, the object type may not be calculated explicitly. In this case, the distance between each three-dimensional point and the braking distance space may be calculated based on the predicted geometric information, and the collision probability may be calculated for each three-dimensional point group. Further, even when the prediction unit 120 predicts an optical flow as geometric information as in the modification described above, the calculation unit 140 may calculate the collision probability without calculating the object type. With such a configuration, collision avoidance can be performed even when the object type is not clearly known, and the vehicle can be safely controlled.

  In the third embodiment, the predicted image and predicted geometric information predicted by the prediction unit 120 are predicted based on the input image captured by the imaging device 11 at time t. However, it is also possible to integrate prediction results in time series using predicted images and predicted geometric information predicted in the past. Specifically, the depth map of the key frame in the map held by the holding unit 130 and the depth map that is predicted geometric information at time t + Δt predicted based on the input image at time t are integrated by time series filtering. At this time, it is also possible to calculate the possibility of collision based on the reliability, with the reciprocal of the variance of the position of each three-dimensional point group integrated by time series filtering as the reliability. Specifically, the product of the reliability of each three-dimensional point and the value of the collision possibility in the third embodiment is used as the reliability possibility collision value. Then, the control unit 150 controls the automobile so as to avoid a collision with the object when the reliability possibility collision value becomes equal to or more than a predetermined threshold value.

  In the third embodiment, the predicted position of the object is estimated based on the prediction of the prediction unit 120. On the other hand, the predicted position of the object can be calculated with higher accuracy by integrating the surrounding three-dimensional point group obtained by SLAM and the prediction result. Specifically, first, a three-dimensional point group of the scene is calculated by SLAM based on the past input image. In addition, as a method of SLAM, any method may be used as long as it can obtain a three-dimensional point group from an input image. For example, the method of Non-Patent Document 1 can be used. Next, object detection is performed from the input image, and a three-dimensional point group is segmented for each object. Thereafter, predictive geometric information at time t + Δt is matched with the three-dimensional point group segmented for each object by the ICP algorithm. As described above, it is calculated where each object calculated by SLAM is positioned at time t + Δt. With the above-described configuration, it is possible to calculate a global amount of movement of the object even when the prediction accuracy is low, so the position calculation performance of the object is improved.

  In the third embodiment, the calculation unit 140 predicts the position of the object by detecting the object in advance from the input image and projecting the object image on the predicted image. However, object detection is not limited to that performed from the input image, as long as it can be calculated where the object is in the predicted image or predicted geometric information. For example, object detection may be performed on a predicted image.

  In addition, object detection may be performed on predicted images at a plurality of times, and object type determination may be performed in consideration of a change in likelihood in object detection of the predicted images at each time. Specifically, first, object detection is performed on a predicted image at each time. Next, the time series change of the likelihood of the object is calculated. The likelihood here is a probability value from 0 to 1 representing the likelihood of each object type output from the object detector.

  In the position and orientation calculation, the position and orientation can be calculated with high accuracy by excluding a moving object or the like as an outlier from the position and orientation calculation calculation. Therefore, the movement amount of the object may be calculated from the prediction result, and the position and orientation may be calculated using only the non-moving object. Specifically, the optical flow between the prediction result and the input image is calculated to determine whether the object is moving or not. At this time, the position and orientation are calculated using only the three-dimensional point group on the object determined to be a stationary object. With such a configuration, conventionally, while the position and orientation are calculated, outlier removal is performed using RANSAC or M estimation, etc., while a three-dimensional point group on a moving object can be determined in advance. The position and orientation can be calculated with high accuracy.

  In the first to third embodiments, the user can not confirm information such as the prediction result of the prediction unit 120, the position and orientation calculation result using them, and the position of the object. Therefore, such information is presented as display information on a liquid crystal display or a HUD (Head-Up Display). FIG. 9 is a view showing an example of display of the display device in the present modification. In this modification, display information is presented on a liquid crystal screen. In addition, the process which concerns on a display shall be performed by the control part 150 (CPU311 shall be displayed, and display information shall be displayed on the display part 316. FIG.

  The GUI 100 is a window for presenting display information, G 120 is a window for presenting control information in the first embodiment and predicted position and collision possibility information of an object in the third embodiment, G 140 Is a window for displaying the predicted image, the predicted geometric information, and the position and orientation calculation result in the first embodiment. G121 and G123 are examples in which the predicted position of the object calculated by the calculation unit 140 is superimposed on the input image. At this time, the color of G121 and G123 or the thickness and shape of the frame may be changed based on the degree of coincidence between the actual measured value and the predicted value. Further, G122 and G124 are examples in which the collision possibility value with each object is displayed. Here, an example is given in which G121 is an oncoming vehicle and it is determined that the collision possibility is low. In addition, G123 is a bicycle, and it is predicted that the vehicle may jump out on the road to avoid a car parked at the tip of the bicycle, and an example in which the value of the collision possibility is increased is shown. G125 is an example where an alert is presented when the possibility of collision becomes larger than a predetermined threshold. Similarly, the color of G121 or G123, the thickness of the frame, or the shape may be changed according to the value of the collision possibility. Also, the value of the collision possibility may be visualized as a heat map. G126 is an example in which the control value is presented, and the current vehicle speed and steering angle are presented on the left side of the arrow. The control direction may be presented as an arrow on a display mounted in front of the vehicle in order to present the control direction to a person outside the vehicle. In addition, in response to the increase of the collision possibility, the change result of the control value is presented on the right side of the arrow. In addition, G127 presents the predicted position and orientation described in the first embodiment. G128 is the result of presenting the predicted position and orientation when the control value is changed in response to the increase in the collision possibility. G129 is a diagram visualizing the map created based on the prediction result described in the second embodiment. Specifically, key frames in the map created by prediction were projected and superimposed on the input image. G140 is an input image at the bottom, and the upper image presents a predicted image in which the future is predicted. G141 is the time of each predicted image, and G142 presents the predicted position and orientation at that time. In this modification, although the method of presenting a prediction image was illustrated, it is good also as composition of presenting a three-dimensional point cloud computed from a depth map as prediction geometric information. At this time, the size and color of the three-dimensional point group may be changed based on the degree of coincidence between the actual measurement value and the prediction value or the value of the collision possibility.

<Summary of effects>
In the first embodiment, the prediction unit predicts a predicted image and predicted geometric information after the time when the imaging device captures an input image, and the calculation unit calculates the position and orientation of the imaging device based on the prediction result. By calculating the position and orientation based on the prediction result as described above, it is possible to calculate the position and orientation at high speed without being influenced by the delay due to the estimation of geometric information and the calculation of the position and orientation taking time for processing. it can.

  In the second embodiment, a predicted image and predicted geometric information are used to create a key frame of a map serving as an index of position and orientation calculation near a point where the imaging device is predicted to pass in the future. Further, the created key frame is updated in time series using a predicted image and predicted geometric information. Then, the position and orientation are calculated using the created map. With this configuration, in the conventional Visual SLAM, a key frame that can only be created at a point through which the imaging device passes can be created at a point where the imaging device is predicted to pass in the future. Then, by further updating the key frames created in advance in time series, it is possible to calculate a depth map as geometric information with high accuracy. By using the accurate map created as described above, the position and orientation can be calculated with high accuracy.

  In the third embodiment, the predicted image and predicted geometric information are used to calculate the type of object which is the surrounding object information and the predicted position of the object, and to calculate the future collision possibility between the vehicle and each object. And if the possibility of future collision is high, the vehicle is controlled so that the object and the vehicle do not collide. In this way, the object position can be calculated earlier before actual input image acquisition. In addition, it is possible to control the vehicle safely by obtaining the collision possibility between the object and the vehicle based on the predicted object position and controlling the vehicle to avoid the collision based on the collision possibility.

<Summary of definition>
The type of the image input unit 110 in the present invention is not particularly limited as long as it is an image that is visual information obtained by imaging the real space. For example, an image of a camera for capturing a gray-scale image (monochrome image) may be input, or an image of a camera for inputting an RGB image may be input. You may input the image of the camera which can image depth information, a distance image, and three-dimensional point-group data. In addition, it may be a single-eye camera, or an image captured by a camera including two or more cameras or sensors may be input. Furthermore, an image captured by a camera may be directly input or may be input via a network.

  The prediction unit 120 predicts a predicted image or predicted geometric information after the time when the imaging device captures visual information. The visual information as used herein refers to an image or a depth map captured by an imaging device. Also, the predicted image is an image predicted by the prediction unit. The prediction geometric information is a depth map or an optical flow predicted by the prediction unit. The learning model used by the prediction unit for prediction is a CNN (Convolutional Neural Network). This CNN is trained so that when an image at a certain time t is input, one or both of a predicted image at time t + Δt and a predicted geometric information after Δt time can be predicted.

  The holding unit 130 holds the learning model described above. The holding unit 130 may further hold a map serving as an index of position and orientation calculation. A map may be configured to associate a scene image with a position and orientation to form a key frame, and to hold a plurality of key frames. An image of a scene is either an image captured by an imaging device or a predicted image predicted by a prediction unit, or both. In addition, the map may be configured to hold geometric information, and may further be configured to hold both the image of the scene and the geometric information.

  The calculation unit 140 calculates the position and orientation of the imaging device using at least one or both of the predicted image predicted by the prediction unit and the predicted geometric information. Further, the calculation unit may calculate a map serving as an index of position and orientation calculation using at least one or both of the predicted image predicted by the prediction unit and the predicted geometric information. Furthermore, the calculation unit may calculate the predicted image predicted by the prediction unit, the object type that is the object information included in the prediction geometric information, or the movement amount thereof. The calculation unit may further calculate collision possibility information which is a collision probability with an object.

  The control unit 150 sets the main information based on at least one or more of the input image, the predicted image predicted by the prediction unit, the predicted geometric information, the position and orientation calculated by the calculation unit, the map, the object information, and the collision possibility information. A control value for controlling a device on which the processing device is mounted is calculated.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

DESCRIPTION OF SYMBOLS 1 ... Car, 10 ... Information processing apparatus, 11 ... Imaging apparatus, 12 ... Control apparatus, 110 ... Input part, 120 ... Prediction part, 130 ... Holding part, 140 ... Calculation part, 150 ... Control part

Claims (24)

An input unit that inputs visual information captured by the imaging device at a first time;
Holding means for holding a learning model for predicting at least one of visual information and geometric information at a second time after the first time based on the visual information input by the input means;
Prediction means for predicting at least one of visual information and geometric information at the second time using the learning model;
First calculation means for calculating the position and orientation of the imaging device at the second time based on at least one of visual information and geometric information of the second time predicted by the prediction means;
An information processing apparatus comprising: An input unit that inputs visual information captured by the imaging device at a first time;
Holding means for holding a learning model for predicting at least one of visual information and geometric information at a second time after the first time based on the visual information input by the input means;
Prediction means for predicting at least one of visual information and geometric information at the second time using the learning model;
Object information included in at least one of visual information at the input first time and visual information or geometrical information of the predicted second time based on at least one of visual information and geometric information predicted by the prediction means Second calculating means for calculating
An information processing apparatus comprising:   The prediction means predicts visual information or geometric information of a third time after the second time based on at least one of visual information or geometric information of the second time predicted by the prediction means. The information processing apparatus according to claim 1 or 2, characterized in that   The imaging is performed based on at least one of visual information or geometric information of the predicted second time and the visual information acquired by the imaging device at a time near the second time. The information processing apparatus according to claim 1, wherein the position and orientation of the apparatus are calculated.   The second calculation means performs the imaging based on at least one of visual information or geometric information of the predicted second time and visual information acquired by the imaging apparatus at a time near the second time. The information processing apparatus according to claim 2, wherein the position and orientation of the apparatus are calculated.   The information processing apparatus according to claim 2, wherein the object information is an object type of an object included in at least one of the input visual information, the predicted visual information, or geometric information.   The object information is at least one of information indicating a position of an object included in at least one of the input visual information, the predicted visual information, and geometric information, or information indicating an amount of movement. The information processing apparatus according to 2 or 6. An input unit configured to input visual information captured by the imaging device;
Holding means for holding a learning model for predicting at least one of visual information or geometric information of a second time after the first time related to the visual information based on the visual information input by the input means When,
A prediction unit that predicts at least one of visual information and geometric information at the second time using the learning model, and the first calculation unit according to claim 1;
An information processing apparatus comprising: the second calculation unit according to claim 2;   The apparatus further comprises control means for calculating a control value for controlling a moving object equipped with the imaging device based on at least one of visual information and geometric information predicted by the prediction means, and performing control. An information processing apparatus according to any one of claims 1 or 8.   The control means is configured to match the degree of coincidence between at least one of the predicted visual information or geometric information at the second time and visual information further acquired at a time near the second time by the imaging device. The information processing apparatus according to claim 9, wherein a control value is calculated. The first calculation means calculates the position and orientation of the imaging device at a second time,
10. The information according to claim 9, wherein the control means calculates a control value for controlling the movable body such that the position and orientation of the imaging device match the position and orientation of the imaging device at the second time. Processing unit. The control value according to any one of claims 9 to 11, wherein the control means calculates a control value for controlling the movable body based on the object information calculated by the second calculation means. Information processing device. The prediction means is a prediction parameter based on at least one of the predicted visual information or geometric information at the second time, and the matching degree of the visual information further acquired by the imaging device at the time near the second time. The information processing apparatus according to any one of claims 1 to 12, further comprising: The said prediction means adjusts a prediction parameter based on the position and orientation of the said imaging device of the 2nd time which the said 1st calculation means calculated. The 1st aspect characterized by the above-mentioned. Information processing equipment. The information according to any one of claims 2, 5, 6, 7, and 8, wherein the prediction means adjusts a prediction parameter based on the object information calculated by the second calculation means. Processing unit. It further comprises measuring means for measuring a measurement value of at least one of the state of the mobile unit and the situation around the mobile unit,
The information processing apparatus according to any one of claims 9 to 12, wherein the prediction unit adjusts a prediction parameter based on the measurement value. 17. The prediction parameter according to any one of claims 13 to 16, wherein the prediction parameter is at least one of the time of the time predicted by the prediction means, the number of predictions, the prediction time interval, and the prediction range. Information processing device. The visual information captured by the imaging device, the geometric information of the second time predicted by the prediction means, the visual information of the second time predicted by the prediction means, and the position and orientation calculated by the first calculation means Generating means for generating display information based on at least one of them; display means for presenting the display information generated by the generating means;
The information processing apparatus according to any one of claims 1, 3, 4, 8 to 12, 14, and 16, further comprising: The visual information captured by the imaging device, the geometric information of the second time predicted by the prediction means, the visual information of the second time predicted by the prediction means, and the position and orientation calculated by the first calculation means Generation means for generating display information based on at least one of them;
Display means for presenting the display information generated by the generation means;
The information processing apparatus according to any one of claims 2, 5, 6, 7, 8 to 12, 15, further comprising:   20. The visual information according to any one of claims 1 to 19, wherein the visual information includes any one of a gray image, an RGB image, depth information, a distance image, and an image obtained by imaging three-dimensional point group data. Information processing equipment.     The information processing apparatus according to any one of claims 1 to 20, wherein the geometric information is a depth map of depth information estimated for each pixel of the visual information. An input step of inputting visual information captured by the imaging device at a first time;
Using the learning model for predicting at least one of visual information or geometric information of a second time after the first time based on visual information input by the input process, visualizing the second time Predicting at least one of information or geometric information;
A first calculation step of calculating the position and orientation of the imaging device at the second time based on at least one of visual information and geometric information of the second time predicted by the prediction step;
A control method of an information processing apparatus, comprising: An input step of inputting visual information captured by the imaging device at a first time;
Using the learning model for predicting at least one of visual information or geometric information of a second time after the first time based on visual information input by the input process, visualizing the second time Predicting at least one of information or geometric information;
Object information included in at least one of visual information at the input first time and visual information or geometrical information of the predicted second time based on at least one of visual information and geometric information predicted by the prediction process A second calculation step of calculating
A control method of an information processing apparatus, comprising:   The program for making the said computer perform each process of Claim 22 or 23, when read and performed by the computer.