JP7149707B2 - Information processing device, its control method and program, and operation control system - Google Patents

Description

The present invention relates to an information processing device, its control method and program, and an operation control system.

There are various technologies for measuring the position and orientation of imaging devices based on image information, such as self-position estimation of automobiles and robots, registration of real space and virtual objects in mixed reality/augmented reality, and 3D modeling of objects and spaces. used for the purpose.

Patent Literature 1 discloses a technique of estimating a self-position and orientation by accurately measuring landmarks using a stereo camera. Also, Non-Patent Document 1 discloses a technique of estimating the self-position by estimating the depth of a scene using only a monocular camera and using a model learned in advance by deep learning.

Japanese Unexamined Patent Application Publication No. 2009-20014

K. Tateno, F. Tombari, I. Laina and N. Navab, "CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction", IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2017. J. Engel, T. Schps, and D. Cremers. "LSD-SLAM: Large-Scale Direct Monocular SLAM", In European Conference on Computer Vision (ECCV), 2014.

However, considering its use as a module for estimating the self-position used for automatic driving and assisted driving of automobiles, it is possible to perform position and orientation estimation more stably than the methods of Patent Literature 1 and Non-Patent Literature 1. is required.

The present invention has been made in view of the above problems, and aims to provide a technique for more stably estimating the position and orientation.

In order to solve this problem, for example, the information processing apparatus of the present invention has the following configuration. i.e.
A first image obtained by the first imaging device and the second image obtained by the first imaging device and the second imaging device, which are arranged on a moving object so that at least a part of the imaging field of view overlaps. input means for inputting a second image obtained by the imaging device;
Using an image and distance information representing the three-dimensional shape of the scene in which the image was shot as training data, the learned system outputs distance information representing the three-dimensional shape of the scene in which the image was shot corresponding to the input image. Acquisition means for acquiring the trained model from the learning model holding means ;
a first estimating means for estimating distance information representing a three-dimensional shape of a first photographed scene from the first image and the second image;
second estimating means for estimating distance information representing a three-dimensional shape of a second shot scene corresponding to the first image using the learned model;
third estimating means for estimating distance information representing a three-dimensional shape of a third shot scene corresponding to the second image using the learned model;
position information of an imaging unit including the first imaging device and the second imaging device based on at least one of distance information representing the three-dimensional shape of the first, second, and third shot scenes ; a generating means for generating at least one of distance information representing a three-dimensional shape of a captured scene represented by an image captured by the imaging unit.

According to the present invention, it is possible to perform position and orientation estimation robustly against defects in the imaging device.

The figure which showed the structure of the motor vehicle in 1st Embodiment. The figure which shows the functional structural example of the information processing apparatus and operation processing part in 1st Embodiment. 2 is a diagram showing the hardware configuration of the information processing apparatus according to the first embodiment; FIG. 4 is a flowchart showing a processing procedure performed by the information processing system according to the first embodiment; 4 is a flowchart showing the procedure of measurement result processing according to the first embodiment; 6 is a flow chart showing a processing procedure of a modified example of the first embodiment; 9 is a flowchart showing the procedure of measurement result processing according to a modification of the first embodiment; 9 is a flowchart showing a processing procedure performed by an information processing system according to the second embodiment; FIG. 11 is a flowchart showing a procedure for determining whether or not there is a problem in the position and orientation by a measurement result processing unit according to the second embodiment; FIG. 10 is a flowchart showing a processing procedure of a modified example of the second embodiment; FIG. 11 is a flow chart showing a procedure for determining whether or not there is a defect in geometric information by a measurement result processing unit according to the second embodiment; FIG. The figure which shows the functional structural example of the information processing apparatus and operation processing part in 3rd Embodiment. 10 is a flow chart showing a processing procedure performed by an information processing system according to the third embodiment; 13 is a flow chart showing a procedure for solving a problem by a measurement result processing unit according to the third embodiment; The figure which shows the functional structural example of the information processing apparatus and operation processing part in 4th Embodiment. FIG. 11 is a flow chart showing a processing procedure for acquiring teacher data and performing learning in the fourth embodiment; FIG. The figure which showed the structure of the motor vehicle in 5th Embodiment. The figure which shows the functional structural example of the information processing apparatus and operation processing part in 5th Embodiment. 13 is a flowchart showing a processing procedure performed by an information processing system according to the fifth embodiment;

Hereinafter, information processing apparatuses according to embodiments of the present invention will be described in detail.

Prior to describing this embodiment, definitions of terms will be described.

Geometric information represents the three-dimensional shape of a captured scene, and is represented as a distance image in which a distance value is stored in each pixel of the image. It is not always necessary to form a distance image, and each pixel of the distance image may be represented as a three-dimensional point group such as a point cloud represented by X, Y, and Z coordinate values. Good to have.

The position and orientation are represented by a total of 6 degrees of freedom parameters, ie, a 3-degree-of-freedom parameter representing the position and a 3-degree-of-freedom parameter representing the orientation. In this embodiment, it mainly represents the position and orientation of the imaging device, and indirectly represents the position and orientation of the vehicle in which the device is mounted.

The learning model represents a model learned in advance so that a depth image corresponding to an input image can be estimated based on a plurality of images and corresponding depth images captured in the same field of view at the same time. The learning model is a model such as a CNN (Convolutional Neural Network) whose input and output are images. However, the learning model is not limited to this, and may be a model that takes an image as an input and outputs geometric information such as a three-dimensional point cloud.

Defective (or inappropriate) indicates that the estimated geometric information and position/orientation are in a state that is significantly different from the ideal geometric information and/or position/orientation. Factors that cause problems include noise generated in the image from the imaging device and shooting scene environment, geometric information estimation using a learning model, estimation in a scene where learning is insufficient, and calibration parameters. Various factors are conceivable, such as a difference from the actual situation and a failure of the imaging device itself.

An abnormality indicates a state different from a normal state, such as a failure of the imaging device or scratches or stains on the lens of the imaging device. In addition, a defect may also be called an abnormality in a broad sense.

[First embodiment]
In the first embodiment, an example in which an information processing device estimates the self-position of a vehicle body for automatic driving or driving assistance of a vehicle will be described. Information from a plurality of sensors is often used in order to stably determine the position and orientation of imaging devices attached to the vehicle body. Temporary geometric information of the system and temporary position and orientation information of the three systems are estimated, and stable geometric information and position and orientation information are obtained from the estimation results. FIG. 1 is a diagram showing the configuration of a driving control system for an automobile 10000 according to this embodiment. In this embodiment, automatic driving or driving assistance is performed by stably obtaining the position and orientation of the imaging unit 10 attached to the automobile 10000 and performing driving control according to the obtained position and orientation. Therefore, the information processing apparatus 1 is used to calculate the geometric information of the surroundings and the position and orientation of the imaging unit 10 . The calculated geometric information and position and orientation are supplied to the operation processing device 12 . Then, the operation processing device 12 controls the automobile 10000 by operating the actuator section 13 based on them. The provisional geometric information is a candidate for the finally determined geometric information, a material, and geometric information to be used as a reference.

<Configuration of information processing device>
FIG. 2 is mainly a block configuration diagram of the information processing device 1 and the operation processing device 12 in this embodiment.

The information processing apparatus 1 includes an image input unit 100, a learning model holding unit 101, a geometric information estimation unit 102, a position/orientation calculation unit 103, a measurement result processing unit 104, a display information generation unit 105, and controls for controlling the entire apparatus. It has a part 15 . The control unit 15 is composed of a CPU (processor), a memory (ROM and RAM) for storing programs executed by the CPU, and for use as a work area.

The operation processing device 12 also includes an ambient environment acquisition unit 106 , a destination acquisition unit 107 , and an operation control unit 108 that controls the operation processing device 12 as a whole. The operation control unit 108 is also composed of a CPU, a program executed by the CPU, and a memory (ROM and RAM) serving as a work area.

The image input unit 100 is connected to the imaging unit 10 attached to the vehicle body. The display information generation section 105 is connected to the display section 11 . The operation control unit 108 is connected to the actuator unit 13 that controls the torque, direction, etc. of the wheels of the vehicle. However, FIG. 2 is an example of the device configuration, and does not limit the scope of application of the present invention.

The imaging unit 10 is composed of two imaging devices for estimating geometric information in stereo based on triangulation. In other words, these two imaging devices are arranged such that their optical axis directions are parallel and their imaging field ranges mostly overlap each other. These imaging devices are video cameras capable of taking images of the surrounding environment of the vehicle in time series (for example, 60 frames per second), and taking images of three components of R, G, and B per pixel.

The image input unit 100 inputs image data of a two-dimensional image of a scene captured by each of the two imaging devices of the imaging unit 10 in chronological order. , the display information generation unit 105 and the operation processing device 12 . Hereinafter, two imaging devices will be referred to as the first imaging device and the second imaging device when specifying one of them, and the first imaging device when specifying the images obtained from each imaging device. Say an image and a second image.

The learning model holding unit 101 holds a learning model for estimating geometric information from one image. The geometric information estimation unit 102 estimates first geometric information from the first image and the second image (stereo image) by triangulation using calibration parameters held by a calibration parameter holding unit (not shown). Furthermore, the geometric information estimation unit 102 uses the learning model held in the learning model holding unit 101 to estimate second geometric information corresponding to the first image input from the image input unit 100 . The geometric information estimation unit 102 similarly uses the learning model to estimate third geometric information corresponding to the second image. The geometric information estimation unit 102 then supplies the estimated first, second, and third geometric information to the position/orientation calculation unit 103 .

The position/orientation calculation unit 103 calculates the first and second positions of the imaging unit 10 based on the image input by the image input unit 100 and the first, second, and third geometric information input from the geometric information estimation unit 102, respectively. , to calculate the third pose. The position/orientation calculation unit 103 then supplies the calculated first, second, and third positions/orientations to the measurement result processing unit 104 .

The measurement result processing unit 104 calculates a stable position and orientation based on the first, second, and third positions and orientations input from the position and orientation calculation unit 103 . A calculation method will be described later. Furthermore, the measurement result processing unit 104 updates the geometric information obtained in the previous frame based on the calculated position and orientation. That is, the measurement result processing unit 104 generates the final current position and orientation and the final current geometric information from the first, second, and third positions and orientations calculated by the position and orientation calculation unit 103 ( decide. Then, the measurement result processing unit 104 supplies the determined position/orientation and geometric information to the display information generation unit 105 and the operation processing device 12 .

The display information generation unit 105 generates display information based on the geometric information and the position and orientation input from the measurement result processing unit 104 . The display information generation unit 105 then outputs the generated display information to the display unit 11 . The display unit 11 is a display mounted in the vehicle, and displays display information input from the display information generation unit 105 .

The operation processing device 12 controls the actuator unit 13 and generates display information on the display unit 11 based on the image input by the image input unit 100 and the geometric information and the position and orientation received from the measurement result processing unit 104. output. The details are described below.

The surrounding environment acquisition unit 106 acquires information representing the surrounding environment of the vehicle body (hereinafter referred to as surrounding environment information) based on the image input by the image input unit 100 and the geometric information and position/orientation input by the measurement result processing unit 104. do. Then, the ambient environment acquisition unit 106 supplies the acquired ambient environment information to the operation control unit 108 .

The destination acquisition unit 107 acquires the destination of the vehicle from the user using a user interface (not shown). Then, the destination acquisition unit 107 supplies information indicating the acquired destination to the operation control unit 108 .

Based on the ambient environment input by the ambient environment acquisition unit 106 and the destination input by the destination acquisition unit 107, the operation control unit 108 calculates a control value for the actuator unit 13 for safely heading to the destination, and output. In addition, the operation control unit 108 also generates various types of information while traveling toward the destination and outputs the information to the display unit 11 . The actuator unit 13 controls/drives the movement of the vehicle according to the control values and signals received from the driving control unit 108 .

FIG. 3 is a diagram showing the main hardware configuration of the information processing device 1. As shown in FIG. The information processing apparatus 1 has a CPU 151, a ROM 152, a RAM 153, an external memory 154, an input section 155, a display control section 156, a communication I/F 157, an I/O 158, and a bus 160 connecting these. The ROM 152 stores a BIOS (Basic Input/Output System) program and a boot program. The RAM 13 is used as a main storage device for the CPU 151 . The external memory 154 is a storage device that holds an OS (operating system) executed by the information processing device 1, an application program for the information processing device 1 to function as a driving support device, and a learning model. is a writable large-capacity non-volatile memory such as a hard disk drive. An input unit 155 is a touch panel, a keyboard, a mouse, and a robot controller, and performs processing related to input of information and the like. The display control unit 156 generates various images such as menus to be displayed on the display unit 11 according to instructions from the CPU 151 and outputs the images to the display unit 11 . Note that the display unit 11 may be of any type, such as a liquid crystal display device, a projector, or an LED indicator. The communication interface 157 performs information communication via a network, and the communication interface may be Ethernet, USB, serial communication, wireless communication, or the like. The I/O 158 is an input/output unit (I/O) that connects the information processing device 1 and an external device, and in the case of the embodiment, the imaging unit 10 and the operation processing device 12 are connected.

In the above configuration, when the main power supply of the information processing apparatus 1 is turned on, the CPU 151 executes the boot program stored in the ROM 152 to load the OS from the external memory 154 to the RAM 153 and execute it. Under the control of the OS, the CPU 151 loads the driving assistance application program from the external memory 154 to the RAM 153 and executes it, so that the information processing device 1 functions as a driving assistance device. The learning model holding unit 101 shown in FIG. 2 is implemented by the external memory 154, and the other processing units are implemented by the CPU 151 that executes the above applications. Note that some of the various processing units shown in FIG. 2 may be realized by hardware independent of the CPU 151 .

The imaging unit 10 is composed of two imaging devices, as described above. The position and orientation of the imaging unit 10 in this embodiment are represented by parameters of six degrees of freedom in a coordinate system based on the first imaging device. The distance between the first imaging device and the second imaging device is fixed, and the position and orientation of one imaging device relative to the other is obtained as a calibration parameter by performing calibration in advance. It is held by the holding part. Therefore, by calculating the position and orientation of one of the imaging devices, it is possible to obtain the position and orientation of the other imaging device. Here, the position and orientation of the image pickup unit 10 are assumed to be a coordinate system based on the first image pickup device, but are not limited to this. The reference coordinate system of the imaging unit 10 may be set at a point different from that of the apparatus. In that case, it is assumed that relative position/orientation transformation parameters from the reference coordinate system to each imaging device are also known. Further, it is assumed that six degrees of freedom parameters for the relationship between the positions and orientations of the automobile 10000 and the imaging unit 10 are obtained in advance by calibration. At this time, the reference coordinate system may be fixed with respect to the automobile 10000 . Therefore, obtaining the position and orientation of the imaging unit 10 is equivalent to obtaining the position and orientation of the automobile 10000 . The geometric information determined by the measurement result processing unit 104 described above is also information from the viewpoint of the reference imaging device (the first imaging device in the embodiment).

<Processing>
Next, a processing procedure in this embodiment will be described. FIG. 4 is a flowchart showing a processing procedure performed by an information processing system including the information processing device 1 and the operation processing device 12 according to this embodiment. In the figure, steps S1000 to S1050 and S1090 represent processing of the information processing device 1 (control unit 15), and steps S1000 and S1060 to S1080 represent processing of the operation processing device 12 (operation control unit 108).

In step S1000, the control unit 15 initializes the system. That is, the control unit 15 loads the OS and the application program from the external memory 144 into the RAM 153 and executes them, thereby making the information processing device 1 operable as a device for assisting driving. Also, the control unit 15 reads the learning model held by the learning model holding unit 101 (external memory 154 ) into the RAM 153 . In addition, the control unit 15 starts up each device (such as the imaging unit 10 ) connected to the information processing apparatus 1 , reads parameters, and reads the initial positions and orientations of the two imaging devices included in the imaging unit 10 . The internal parameters (focal length, image center position, lens distortion, etc.) of the two imaging devices are calibrated in advance. Further, the operation control unit 108 of the operation processing device 12 receives the destination designated by the user from the destination acquisition unit 107 and sets it as the destination of the vehicle 10000 .

In step S1010, the control unit 15 controls the imaging unit 10 to capture a scene (image) by the first and second imaging devices. In step S1020, the control unit 15 controls the image input unit 100 to acquire a first image and a second image, which are scenes captured by the first imaging device and the second imaging device.

In step S1030, the control unit 15 controls the geometric information estimation unit 102 to estimate (generate) first, second, and third geometric information based on the first image and the second image. The geometric information estimation unit 102 generates first geometric information from the two images using the stereo method. Specifically, the geometric information estimating unit 102 obtains the correspondence between the first image and the second image, calculates the parallax image, and performs triangulation based on known calibration information between the two imaging devices. to estimate the range image. Also, the geometric information estimation unit 102 inputs the first image to the learning model to estimate the second geometric information. Similarly, the geometric information estimation unit 102 inputs the second image to the learning model to estimate the third geometric information.

In step S1040, the control unit 15 controls the position/orientation calculation unit 103 to calculate the position/orientation of the imaging unit 10 using the first, second, and third geometric information calculated in step S1030. The calculated positions and orientations are the first, second, and third positions and orientations obtained from the first, second, and third geometric information, respectively. Specifically, the position/orientation calculation unit 103 projects each pixel in the image of the previous frame onto the current frame based on the first geometric information. Next, the position/orientation calculation unit 103 calculates the position/orientation using the method of Engel et al. By obtaining , the first position and orientation are obtained. The position/posture calculation unit 103 obtains the second position/posture and the third position/posture by performing the same processing on the second and third geometric postures. After calculating the position and orientation, the position and orientation calculation unit 103 may further update the corresponding geometric information. Specifically, the position/orientation calculation unit 103 uses the first, second, and third geometric information as the geometric information at the current time t based on the obtained first, second, and third position/orientation, respectively. Geometric information is updated by time-series filtering from past time ti (described in Non-Patent Document 1).

In step S1050, the control unit 15 controls the measurement result processing unit 104 to calculate the final position and orientation of the imaging unit 10 based on the first, second, and third positions and orientations calculated in step S1040. do. Specific processing will be described later. Also, the control unit 15 updates the geometric information based on the determined position and orientation.

In step S1060, the operation control unit 108 controls the surrounding environment acquisition unit 106 to obtain a surrounding environment map based on the position and orientation calculated in step S1050, the updated geometric information, the first image, and the second image. get A surrounding environment map is a map showing where roads, cars, buildings, etc. exist in a three-dimensional space. The ambient environment acquisition unit 106 performs semantic segmentation from the geometric information, the first image, and the second image to identify roads, cars, buildings, and the like in the images. Then, the surrounding environment acquisition unit 106 updates the surrounding environment map using the geometric information and the semantic segmentation result according to the position and orientation calculated on the surrounding environment map obtained up to the previous frame.

In step S1070, the operation control unit 108 controls the display information generation unit 105, and based on the surrounding environment map obtained in step S1060, an image obtained by rendering the surrounding environment at the designated viewpoint is displayed as necessary for the user. Display information to which information is added is generated as an image. Then, the operation control unit 108 displays the generated image on the display of the display unit 11 so that the user can visually recognize it. Here, the information necessary for the user is, for example, the position and orientation of the automobile 10000 with respect to the surrounding environment obtained from the surrounding environment map and the position and orientation.

In step S1080, operation control unit 108 generates actuator control values for safely driving the vehicle toward the destination based on the position and orientation of imaging unit 10, the surrounding environment map, and the destination. Then, the driving control unit 108 controls the actuator unit 13 using the generated actuator control value to drive the vehicle.

In step S1090, the control unit 15 determines whether or not to terminate the system. Specifically, if the user has input an end command through an input unit (not shown), the process ends. If not, the process returns to step S1010 to repeat the above process.

<Processing for Obtaining Position and Orientation in Measurement Result Processing Unit>
FIG. 5 is a flow chart showing the process of step S1050, that is, the procedure of the measurement result processing unit 104 in this embodiment. Hereinafter, the processing contents of the measurement result processing unit 104 will be described with reference to the same figure.

In step S1100, the measurement result processing unit 104 initializes. In this initialization, the measurement result processing unit 104 reads the first, second, and third positions and orientations, reads the previous frame image required for subsequent processing, reads the geometric information of the surrounding environment map, and , and read parameters for solving optimization problems.

In step S1110, the measurement result processing unit 104 estimates the position and orientation of the current frame by linear interpolation from the positions and orientations of the image one frame before and the image two frames before, and uses this as an initial position and orientation. However, the initial position and orientation are not limited to this, and the initial position and orientation may be obtained from an acceleration sensor, a GPS sensor, or the like.

In step S1120, the measurement result processing unit 104 calculates the reliability of the first, second, and third positions and orientations. The reliability is a numerical representation of the reliability of each of the first, second, and third positions and orientations. A higher confidence indicates that the pose is likely to have a smaller error from the correct pose. As a method of obtaining the reliability, in step S1040, when the position and orientation are obtained by the method of Non-Patent Document 2, the minimized luminance difference is used to represent the reliability. In other words, a function is used in which the smaller the luminance difference referred to here, the higher the reliability.

However, the reliability is not limited to the luminance difference. For example, when obtaining the corresponding geometric information, the reliability of the geometric information obtained from the method of Non-Patent Document 1 and the degree of stereo matching are used as the reliability. may

In step S1130, the measurement result processing unit 104 obtains the final output position and orientation based on the first, second, and third positions and orientations, their corresponding reliability levels, and the initial position and orientation. Here, the first, second, and third positions and orientations are P1, P2, and P3, the initial position and orientation are PI, and the position and orientation output by the measurement result processing unit 104 are PO. Let c1, c2, and c3 be the degrees of reliability corresponding to the positions and orientations. A function f(Pi, Pj) is a function for obtaining the difference between the positions and orientations Pi and Pj. PO is obtained by solving an optimization problem that minimizes Equation E below.
E=k*f(PI, PO)+c1*f(P1, PO)+c2*f(P2, PO)+c3*f(P3, PO) (1)
where k is an adjustment parameter. However, the method of obtaining PO is not limited to this method, and the true state PO may be obtained using a Kalman filter with PI, P1, P2, and P3 as inputs.

In step S1140, measurement result processing section 104 updates the geometric information based on the position and orientation obtained in step S1130. Specifically, based on the determined position and orientation, the first, second, and third geometric information are assumed to be geometric information at the current time t, and the reliability of the geometric information from past time ti is taken as the weight. Geometric information is updated by time-series filtering (described in Non-Patent Document 1).

Although the above flowchart describes the process of performing step S1120 after step S1110, the flowchart does not necessarily follow this order. The process of step S1120 may be performed first, and then step S1110 may be performed. Through the above processing, the measurement result processing unit 104 obtains the position and orientation and the geometric information.

<effect>
As described above, in the first embodiment, a plurality of pieces of geometric information and a plurality of positions and orientations are estimated from a plurality of measurement systems. Since each measurement result is processed independently after the image is input, even if a large amount of noise occurs in one of the measurement results, it does not affect the other measurement results. Therefore, it is unlikely that all of the multiple measurement results will generate large amounts of noise. By increasing the weight of the measurement results with little error relative to the position and orientation predicted from the time-series information, some kind of noise will be mixed into the measurement results. Even in this case, the position and orientation can be stably estimated. Furthermore, by updating the geometric information based on the stable position and orientation, it is possible to obtain more accurate geometric information.

<Modification>
In the first embodiment, an example of performing processing according to the flowchart shown in FIG. 4 has been described, but the present invention is not limited to this. Here, a functional configuration example and a processing procedure of a modification of the present embodiment will be described. Since the functional configuration example of the modification is the same as that of the first embodiment shown in FIG. 2, only the geometric information estimation unit 102, the position/orientation calculation unit 103, and the measurement result processing unit 104, which are different, will be described.

The geometric information estimating unit 102 estimates the first, second, and third geometric information, and uses the estimated first, second, and third geometric information as the measurement result, as in the first embodiment described above. Output to the processing unit 104 . Alternatively, the geometric information estimation unit 102 temporarily supplies the first, second, and third geometric information to the position/posture calculation unit 103, and calculates the first, second, and third positions/postures corresponding to the respective geometric information. Corresponding geometric information may be updated according to the calculation result. Specifically, based on the obtained first, second, and third positions and orientations, each of the first, second, and third geometric information is defined as geometric information at the current time t, and the past time t−i The geometric information is updated by time-series filtering from (described in Non-Patent Document 1).

The measurement result processing unit 104 calculates accurate geometric information based on the first, second, and third geometric information updated by the geometric information estimating unit 102 . A calculation method will be described later. The measurement result processing unit 104 also outputs the calculated geometric information to the position/orientation calculation unit 103 . The measurement result processing unit 104 also updates the geometric information based on the position and orientation input by the position and orientation calculation unit 103, which will be described later. Furthermore, the measurement result processing unit 104 outputs the updated geometric information to the display information generation unit 105 and the operation processing device 12 . The measurement result processing unit 104 also outputs the position and orientation to the display information generation unit 105 and the operation processing device 12 .

The position and orientation calculation unit 103 calculates the position and orientation of the imaging unit 10 based on the image input by the image input unit 100 and the geometric information input by the measurement result processing unit 104 . Also, the calculated position and orientation are input to the measurement result processing unit 104 .

Next, the processing procedure of this modified example will be described. FIG. 6 is a flowchart showing a processing procedure performed by an information processing system including the information processing device 1 in this modification. Since many steps of this flowchart are the same as those of the flowchart of FIG. 4, steps S1041 and S1051, which are different, will be described.

In step S1041, the control unit 15 controls the measurement result processing unit 104 to calculate the geometric information of the shooting scene using the first, second, and third geometric information calculated in step S1030. Specifically, the first, second, and third pieces of geometric information each hold a degree of reliability for each measurement point or measurement area. Obtained by constructing geometric information. Details will be described later.

In step S1051, the control unit 15 controls the position/orientation calculation unit 103 to calculate the position/orientation of the imaging unit 10 based on the geometric information calculated in step S1041. Specifically, the position and orientation with respect to the surrounding environment map are calculated by aligning the geometric information calculated by the measurement result processing unit 114 with the geometric information in the surrounding environment map obtained in the previous frames. . Also, the position and orientation of the current frame are obtained by linear interpolation from the positions and orientations of the image one frame before and the image two frames before, and are used as the initial position and orientation for alignment. Moreover, the geometric information of the surrounding environment map is also updated by performing the alignment.

<Processing for obtaining geometric information in the measurement result processing unit>
Here, FIG. 7 is a flow chart showing the processing procedure of the measurement result processing unit 104 in step S1041 of FIG.

In step S1200, the measurement result processing unit 104 performs initialization processing. That is, the measurement result processing unit 104 reads the first, second, and third geometric information, reads the previous frame image required for subsequent processing, and the geometric information of the surrounding environment map, and performs optimization. Read parameters etc. when solving problems.

In step S1210, the measurement result processing unit 104 calculates reliability for each of the first, second, and third geometric information. The reliability is a numerical representation of the degree of reliability of each measurement point or each measurement area with respect to the first, second, and third geometric information. The higher the reliability, the higher the possibility that the three-dimensional position where the measurement point or measurement area exists is correct. As a method of obtaining the reliability, for the first geometric information, the degree of similarity is calculated for each minute region when obtaining the correspondence between the first image and the second image in stereo. The higher the degree of similarity, the higher the reliability of the region, and the reliability is obtained. Regarding the second and third geometric information, the reliability of the geometric information can be obtained by the method shown in Non-Patent Document 1.

In step S1220, the measurement result processing unit 104 obtains geometric information output by the measurement result processing unit 104 based on the first, second, and third geometric information and the reliability levels corresponding thereto. More specifically, by performing weighted ICP (Iterative Closest Point) using the reliability of each measurement point or each measurement area as a weight, the three pieces of geometric information are integrated to estimate more accurate geometric information. Also, when the first, second, and third geometric information are integrated by ICP, the measurement result processing unit 104 obtains the The position and orientation of the geometric information obtained from the Through the above processing, the measurement result processing unit 104 obtains geometric information.

In the first embodiment, the measurement result processing unit 104 obtains the position and orientation from a plurality of positions and orientations. you may ask.

Although only one image input unit 100 is shown in the example of FIG. 1, the present invention is not limited to this. The image input unit may receive images from two imaging devices. Similarly, a plurality of geometric information estimation units 102 and position/orientation calculation units 103 may be provided. Also, the learning model holding unit 101 may hold separate learning models for each of the two imaging devices.

Although the display information generated by the display information generation unit 105 in step S1070 is the position and orientation of the automobile 10000 with respect to the surrounding environment, the display information is not limited to this. For example, it may be an image captured by the imaging unit 10 or an image obtained by rendering the result of semantic segmentation obtained from the surrounding environment map on a two-dimensional image. Information such as a destination may also be used. This allows the user to visually confirm the surrounding conditions of the vehicle 10000 recognized by the information processing system.

In step S1090, an example has been described in which the user inputs the termination command as the condition for terminating the system. However, the system is not limited to this, and as a condition for terminating the system, the user who is the driver may change from the automatic driving mode, in which the control of the vehicle is entrusted to the system, to the driving mode in which a person drives the vehicle. Further, the system may be terminated when the driving processing device 12 determines that automatic driving or driving assistance cannot be performed due to inconsistency between the obtained geometric information, position and orientation, and the destination. good.

In the first embodiment, an example in which position/orientation estimation is applied to an application for automatic driving or driving assistance of a vehicle has been described. However, the application applicable to the information processing apparatus 1 described in the present embodiment is not limited to the application, and any application that uses a plurality of pieces of geometric information including the results output by the learning model or the position and orientation results may be used. For example, it may be used as a robot system that is attached to the tip of an industrial robot hand and measures the position and orientation of the robot hand. The robot system at this time may include a manipulator such as a robot arm, a grasping device such as a suction hand, and a control unit that controls the manipulator and the grasping device based on the position and orientation calculated by the measurement result processing unit 104. good. It may also be used for alignment between a real space and a virtual object in a mixed reality system. The mixed reality system at this time may include a head-mounted display equipped with an imaging device for acquiring the surrounding environment. A robot that cleans a room, a drone that flies in the air, a device that moves in water, etc. hold the learning model described above, estimate temporary geometric information based on the learning model, and generate it using the images of the two imaging devices. 3D temporary geometric information may be estimated. Furthermore, geometric information or position and orientation may be estimated and used to acquire the movement of the robot itself and the surrounding environment.

Further, the application of the information processing apparatus 1 is not limited to position and orientation estimation, and may be used for three-dimensional reconstruction. For example, it may be used as a measurement system for generating CAD models of industrial parts and buildings. The measurement system at this time may further include a three-dimensional model generation unit that generates a three-dimensional model from the geometric information updated by the measurement result processing unit 104 . Further, it may be used as a device for obtaining a range image with high accuracy from a camera that cannot obtain a range image, such as an RGB camera or a camera that obtains a grayscale image.

In the first embodiment, the in-vehicle information processing apparatus 1 has the learning model holding unit 101, the geometric information estimation unit 102, the position/orientation calculation unit 103, and the measurement result processing unit 104. FIG. However, the cloud server may have and execute some of the functions of the information processing apparatus 1 shown in this embodiment. For example, the cloud server may include the learning model holding unit 101 , the geometric information estimation unit 102 , the position/orientation calculation unit 103 , and the measurement result processing unit 104 . In this configuration, the information processing apparatus 1 first transfers an input image to a cloud server using a communication unit (not shown). Next, geometric information is estimated using the learning model held by the learning model holding unit 101 on the cloud server. Also, the geometric information estimation unit 102 estimates a plurality of pieces of geometric information. Then, the position and orientation calculation unit 103 calculates a plurality of positions and orientations, and the measurement result processing unit 104 obtains stable positions and orientations, and updates the geometric information. Then, the geometric information and the position and orientation estimated by the cloud server are transferred to the information processing device 1 using the communication unit. By adopting such a configuration, the information processing apparatus 1 can reduce the calculation load, so that the space can be saved with a small-sized computer.

In the present embodiment, the configuration in which the imaging device that captures an image is an RGB video camera has been described. However, it is not limited to the RGB video camera, and any camera that captures an image of the real space may be used. For example, a camera that captures a grayscale image, infrared image, ultraviolet image, distance image, A camera capable of capturing three-dimensional point cloud data may also be used. Also, although the stereo configuration has been described with two imaging devices, the present invention is not limited to this, and a camera having three or more cameras or sensors may be used. In that case, four or more pieces of geometric information and positions and orientations can be calculated, and the measurement result processing unit 104 performs calculations based on these pieces of measurement results.

[Second embodiment]
In the first embodiment, the measurement result processing unit 104 calculates more correct geometric information or position/orientation from a plurality of pieces of geometric information or position/orientation. On the other hand, in the second embodiment, the measurement result processing unit 104 performs determination on the geometric information or the position/orientation, and if there is a problem with the geometric information or the position/orientation, another geometric information or position/orientation is determined. switch to use In this way, automatic driving or driving assistance is performed using sensors and calculation results that can obtain optimum geometric information or position and orientation. Since the configuration of the vehicle in the second embodiment is the same as that shown in FIG. 1 for the first embodiment, the description thereof will be omitted.

<Configuration of information processing device>
A functional configuration example of the information processing apparatus 2 according to the present embodiment is substantially the same as that shown in FIG. 2 of the first embodiment. The difference lies in the processing contents of the geometric information estimation unit 102 , the position/orientation calculation unit 103 , and the measurement result processing unit 104 . Therefore, these will be described below. See the first embodiment for other components.

The geometric information estimation unit 102 has a function of estimating first, second, and third geometric information as in the first embodiment. Either one of the second and third geometric information is estimated and output to the position/orientation calculation unit 103 and the measurement result processing unit 104 . The parameter here indicates which of the first, second, and third geometric information is to be used.

The position and orientation calculation unit 103 calculates the position and orientation of the imaging unit 10 based on the image input by the image input unit 100 and the geometric information input by the geometric information estimation unit 102 . Further, the position/orientation calculation unit 103 updates the geometric information based on the calculated position/orientation. The position/orientation calculation unit 103 then supplies the calculated position/orientation and the updated geometric information to the measurement result processing unit 104 .

A measurement result processing unit 104 determines whether or not there is a defect in the position and orientation input from the position and orientation calculation unit 103 . If the measurement result processing unit 104 determines that there is no problem with the position and orientation, the measurement result processing unit 104 supplies the geometric information and the position and orientation to the display information generation unit 105 and the operation processing device 12 . Further, when there is a problem with the position and orientation, the measurement result processing unit 104 changes the parameters in the parameter holding unit (not shown), causes the geometric information estimation unit 102 to estimate different geometric information, The calculation unit 103 is also caused to calculate the position and orientation based on the newly estimated geometric information, and repeatedly determine whether or not there is a defect, and the information determined to have no defect is determined as an output target.

<Processing>
Next, a processing procedure in the second embodiment will be described. FIG. 8 is a flowchart showing a processing procedure performed by an information processing system including the information processing device 1 according to the second embodiment.

The flowchart in this embodiment is substantially the same as the flowchart in FIG. 4 in the first embodiment. Since the processing is similar to that of steps S1010, S1020, S1060, S1070, S1080, and S1090, a description thereof will be omitted, and steps S2030, S2040, and S2050, which are different, will be described.

In step S2030, the control unit 15 controls the geometric information estimating unit 102, based on the first image and the second image, one of the specified first, second, and third geometric information Estimate a piece of geometric information. Note that the method of obtaining each piece of geometric information is the same as the processing described in step S1030.

In step S2040, the control unit 15 controls the position/orientation calculation unit 103 to calculate the position/orientation of the imaging unit 10 using the geometric information calculated in step S2030. After calculating the position and orientation, the position and orientation calculation unit 103 may further update the corresponding geometric information.

In step S2050, the control unit 15 controls the measurement result processing unit 104 to determine whether or not there is a defect in the position and orientation calculated in step S2040. If there is no problem, the process proceeds to step S2060. If there is a problem, the process returns to step S2030 to start again from estimating different geometric information. A specific process for determining whether or not there is a problem will be described later.

<Processing for Determining Presence or Absence of Poor Position and Orientation in Measurement Result Processing Unit>
Here, FIG. 9 is a flowchart showing a procedure for the measurement result processing unit 104 to determine whether or not there is a defect in the position and orientation in step S2050 according to the second embodiment.

In step S2100, the measurement result processing unit 104 initializes. That is, reading of the position and orientation, reading of the previous frame image required for subsequent processing, reading of the geometric information of the surrounding environment map, and reading of the threshold parameters of reliability and similarity.

In step S2110, the measurement result processing unit 104 calculates the reliability of the position and orientation. Here, the reliability is calculated for any one of the first, second, and third positions and orientations. Since the method of obtaining the reliability has already been described in step S1120 in the first embodiment, a description thereof will be omitted.

In step S2120, the measurement result processing unit 104 confirms whether or not the reliability obtained in step S2110 is greater than the threshold Thp. If the reliability is greater than Thp, the process proceeds to step S2130. If the reliability is less than Thp, it is determined that there is a problem, and the process ends.

In step S2130, the measurement result processing unit 104 obtains the position and orientation of the current frame by linear interpolation from the positions and orientations of the image one frame before and the image two frames before, and sets this as the initial position and orientation. However, the initial position and orientation are not limited to this, and the initial position and orientation may be obtained from an acceleration sensor, a GPS sensor, or the like.

In step S2140, the measurement result processing unit 104 calculates the degree of similarity between the initial position/orientation obtained in step S2130 and the position/orientation calculated in step S2040. Here, the degree of similarity between two positions and orientations is higher when the two positions and orientations are more similar, and lower when they are farther apart and have different orientations. For example, the similarity S is obtained by the following equation (2).
S=k1*|ti−t|+k2*||qi−q|| (2)
Here, k1 and k2 are adjustment parameters, ti and t are three-dimensional vectors representing the initial position and posture, respectively, and qi and q are quaternions representing the initial position and posture, respectively. where |x| represents the norm of the real three-dimensional vector x, and ||y|| represents the norm of the quaternion y. Here, the formula (2) has been described as a similarity calculation method, but the method is not limited to this. Any calculation formula may be used to calculate the degree of similarity as long as it is calculated such that the more similar the positions and orientations, the higher the value, and the lower the value, the more different the positions and orientations are.

In step S2150, measurement result processing section 104 checks whether the similarity obtained in step S2140 is greater than threshold Tsp. If the degree of similarity is greater than Tsp, it is determined that there is no problem, and the process ends. If the degree of similarity is equal to or less than Tsp, it is determined that there is a problem, and the process ends.

Through the above processing, the measurement result processing unit 104 determines whether or not there is a defect in the position and orientation.

<effect>
As described above, in the second embodiment, one piece of geometric information is selected from a plurality of pieces of geometric information to estimate the geometric information and the position and orientation. , decides whether to use the pose or re-estimate another geometric information. Since it is only necessary to calculate any one of the geometric information and position/orientation during the calculation process, compared to the case of calculating a plurality of geometric information and/or a plurality of positions and orientations described in the first embodiment, processing can be performed simultaneously. Because of the low computational cost of the power, it can be processed even with relatively small computational resources. In addition, by determining whether or not there is a defect in the calculated position and orientation, the possibility of using an incorrect position and orientation is reduced, and safety in automatic driving and driving assistance is enhanced.

<Modification>
In the second embodiment, an example of performing processing according to the flowchart shown in FIG. 8 has been described, but the present invention is not limited to this. Here, a functional configuration example and a processing procedure of a modification of the present embodiment will be described. Since the functional configuration example of the modification is the same as that of the second embodiment, only the geometric information estimation unit 102, the position/orientation calculation unit 103, and the measurement result processing unit 104, which are different, will be described.

The geometric information estimation unit 102 estimates any one of the first, second, and third geometric information in the same manner as in the second embodiment, and outputs it to the measurement result processing unit 104 . Alternatively, any one of the first, second, and third geometric information may be temporarily supplied to the position and orientation calculation unit 103, the position and orientation corresponding to the geometric information may be calculated, and the corresponding geometric information may be updated.

The measurement result processing unit 104 determines whether or not the geometric information input from the geometric information estimating unit 102 has defects. If there is no defect in the geometric information, the geometric information is output to the position/orientation calculation unit 103 . If there is a defect in the geometric information, the parameters in the parameter holding unit (not shown) are changed to cause the geometric information estimating unit 102 to estimate other geometric information, and it is determined whether or not there is a defect in the geometric information. repeat. Also, the geometric information is updated based on the position and orientation input by the position and orientation calculation unit 103, which will be described later. Furthermore, the updated geometric information is output to the display information generation unit 105 and the operation processing device 12 . Also, the position and orientation are output to the display information generation unit 105 and the operation processing device 12 .

The position and orientation calculation unit 103 calculates the position and orientation of the imaging unit 10 based on the image input by the image input unit 100 and the geometric information input by the measurement result processing unit 104 . The position/orientation calculation unit 103 also supplies the calculated position/orientation to the measurement result processing unit 104 .

Next, a processing procedure of a modified example of this embodiment will be described. Here, FIG. 10 is a flow chart showing a processing procedure of a modification performed by an information processing system including the information processing apparatus 1 according to the second embodiment. Since many steps of this flowchart are the same as those of the flowchart of FIG. 8, steps S2041 and S2051, which are different, will be described.

In step S2041, the measurement result processing unit 104 determines whether or not there is a defect in the geometric information calculated in step S2030. Details of the determination method will be described later. If there is no defect in the geometric information, the process proceeds to step S2051. If there is a problem with the geometric information, the process returns to step S2030 to cause the geometric information estimation unit 102 to estimate another geometric information.

In step S2051, the position/orientation calculation unit 103 calculates the position/orientation of the imaging unit 10 based on the geometric information calculated in step S2041. Specifically, the position and orientation with respect to the surrounding environment map are calculated by aligning the geometric information calculated by the measurement result processing unit 104 with the geometric information in the surrounding environment map obtained in the previous frames. . Also, the position and orientation of the current frame are obtained by linear interpolation from the positions and orientations of the image one frame before and the image two frames before, and are used as the initial position and orientation for alignment. Moreover, the geometric information of the surrounding environment map is also updated by performing the alignment.

<Processing for Determining Presence or Absence of Defect in Geometric Information in Measurement Result Processing Unit>
Here, FIG. 11 is a flow chart showing the procedure for determining whether or not there is a defect in the geometric information in the measurement result processing unit 104 in S2041 in the modification of the second embodiment.

In step S2200, the measurement result processing unit 104 initializes. That is, it is the reading of geometric information, the reading of previous frame images required for subsequent processing, the geometric information of the surrounding environment map, and the reading of parameters for solving optimization problems.

In step S2210, measurement result processing section 104 calculates the reliability of the geometric information obtained in step S2030. The measurement result processing unit 104 obtains the reliability of the input geometric information according to which one of the first, second, and third geometric information is input. Since the method of obtaining the reliability has already been explained in the processing step S1210 of the modified example of the first embodiment, the explanation thereof will be omitted.

In step S2220, the measurement result processing unit 104 aligns the geometric information with reliability obtained in step S2210 with the geometric information in the surrounding environment map so far by weighted ICP.

In step S2230, measurement result processing section 104 determines whether the alignment error has converged in weighted ICP. If it converges, the processing is terminated assuming that there is no defect in the geometric information. If it does not converge, it is determined that there is a defect in the geometric information, and the process ends. Through the above processing, the measurement result processing unit 104 determines whether or not there is a defect in the geometric information.

As described above, in the second embodiment, the measurement result processing unit 104 determines whether or not there is a defect in the position and orientation. You may determine the presence or absence of

In the second embodiment, in the geometric information estimating unit 102 and step 2030, the parameter holding unit holds a parameter specifying any one of the first, second, and third pieces of geometric information. It is not limited. Reliabilities of the first, second, and third geometric information may be stored as parameters, and the geometric information with the highest reliability may be selected and estimated. Also, the reliability may be calculated from the information obtained up to the previous frame, instead of being stored in the parameter storage unit.

In the second embodiment and its modification, the method of determining whether there is a defect in the position/orientation or geometric information in the measurement result processing unit 104 has been described with reference to the flowcharts of FIGS. 9 and 11 . However, the method of determining the presence or absence of defects is not limited to these. For example, if a signal indicating an error state is output when an abnormality occurs in each of the first imaging device and the second imaging device, the measurement result processing unit determines whether one of the imaging devices is in an error state. Upon receiving a signal indicating that an error is present, it may be determined that there is a problem, and parameters in a parameter holding unit (not shown) may be changed so as to estimate geometric information and position/orientation based on an imaging device that is not in an error state. .

In the flowcharts of FIGS. 8 and 10 showing the flow of processing in the second embodiment, the explanation is such that until it is determined that there is no problem with the position/orientation or the geometric information, the geometric information is estimated, and the operation control is not reached. I did, but it's not limited to this. While the operation control is always performed, if updated with the latest information, the parameters of the operation control may be updated. In other words, a separate thread is allocated for operation control and control is continuously performed, and when it is determined that there is no problem with the position and orientation and geometric information, the position and orientation and the surrounding environment map are updated based on that information. It may also be reflected in operation control. As a result, even if there is a problem with the geometric information or the position and orientation, the operation can be continued without stopping the automatic driving and driving assist functions.

In the flowcharts of FIGS. 8 and 10 showing the flow of processing in the second embodiment, the estimation of the geometric information is repeated until it is determined that there is no defect in the position and orientation or the geometric information. If there is a problem with both the geometric information and the position and orientation, there is a possibility that you will get stuck in an infinite loop. If the determination that there is a problem continues three times in succession, the image capturing may be restarted. Alternatively, the display information generation unit 105 may generate and display an image that informs of the measurement failure. Alternatively, the display information generation unit 105 may display a message prompting a change to a driving mode in which a person drives.

In the second embodiment, steps S2010 and S2020 have been described assuming that the two imaging devices capture and input the first image and the second image, but the present invention is not limited to this. For example, normally, only the first imaging device may be used to input and use the first image, and the second geometric information may be estimated in subsequent processing. Furthermore, if it is determined that there is a problem with the second geometric information, the process returns to step S2010 to capture an image, and the scene is captured using only the second imaging device or both the first imaging device and the second imaging device. You may make it image. As a result, since only one imaging device is normally used, power consumption and calculation costs can be suppressed.

In the second embodiment, in step S2070, the display information generation unit 105 may generate display information for notifying the presence or absence of a problem, and display the information on the display unit 11. FIG. For example, if there is no problem, it may be displayed which geometric information or position/orientation is used for operation control. It may be displayed which imaging device the image was captured by. Also, if there is a problem, it may be displayed which one of the first, second, and third geometric information or the position/orientation has the problem. Alternatively, a display may be made to inform an abnormality of the first imaging device or the second imaging device. Alternatively, if the problem continues for a certain period of time, an indication may be displayed to prompt inspection or maintenance of the part considered to be the cause. This allows the user to visually confirm the state of the information processing system.

Although the case of estimating any one piece of geometric information has been described in the second embodiment, the present invention is not limited to this. For example, similarly to the first embodiment, the first, second, and third geometric information or the first, second, and third positions and orientations may be estimated, and the presence or absence of defects in each may be checked. . If there are multiple pieces of geometric information or position/orientation without defects, one of them may be selected at random to proceed to the next step, or like steps S1050 and S1041 in the first embodiment, Geometric information or position/orientation may be obtained from a plurality of pieces of geometric information or position/orientation without defects. Alternatively, defective geometric information or position/orientation may be obtained by setting the reliability in Equation 1 or the weight for ICP to be lower than those without defects. By doing so, it is possible to perform estimation that is more robust than the estimation result in the first embodiment by explicitly excluding defective data.

[Third embodiment]
In the second embodiment described above, the measurement result processing unit determines whether or not there is a defect in the geometric information or the position and orientation, and calculates the geometric information or the position and orientation using an estimation result with no defect. On the other hand, in the third embodiment, the measurement result processing unit not only judges the geometric information or the position and orientation, but also eliminates the cause of the problem and avoids the problem when there is a problem. , try to solve the problem. In this way, even if there is a problem, the geometric information or the position and orientation are estimated robustly to perform automatic driving or driving assistance. Since the configuration of the vehicle in the third embodiment is the same as that shown in FIG. 1 for the first embodiment, it will be omitted.

<Configuration of information processing device>
FIG. 12 is a diagram showing a device configuration example of the information processing device 1 and the operation processing device 12 in the third embodiment. The information processing apparatus 1 includes an image input unit 300, a learning model holding unit 301, a geometric information estimation unit 302, a position/orientation calculation unit 303, a measurement result processing unit 304, a display information generation unit 305, a calibration calculation unit 309, and the entire apparatus. A control unit 35 is provided for controlling the . The operation processing device 12 also includes an ambient environment acquisition unit 306, a destination acquisition unit 307, and an operation control unit 308 that controls the entire device. Image input unit 300, learning model storage unit 301, geometric information estimation unit 302, position/orientation calculation unit 303, display information generation unit 305, ambient environment acquisition unit 306, destination acquisition unit 307, operation control unit 308, and control unit Reference numeral 35 denotes the image input unit 100, the learning model holding unit 101, the geometric information estimation unit 102, the position/orientation calculation unit 103, the display information generation unit 105, the ambient environment acquisition unit 106, the destination acquisition unit 107, It corresponds to the operation control unit 108 and the control unit 15 . Descriptions of similar functions are omitted, and the measurement result processing unit 304 and the calibration calculation unit 309, which are different, will be described.

The measurement result processing unit 304 detects a defect or abnormality based on the first, second, and third positions and orientations calculated by the position/orientation calculation unit 303 and the first, second, and third geometric information. Take action to resolve it. Then, the position and orientation and geometric information are calculated and output to the display information generation unit 305 and the operation processing device 12 . Details will be described later.

Upon receiving an instruction to perform calibration from the measurement result processing unit 304, the calibration calculation unit 309 performs calibration of the first imaging device and the second imaging device, and obtains calibration parameters. The calibration parameters in the calibration parameter holding unit (not shown) are updated with the obtained calibration parameters.

<Processing>
Next, a processing procedure in the third embodiment will be described. FIG. 13 is a flowchart showing a processing procedure performed by an information processing system including the information processing device 3 according to this embodiment.

The flowchart in the third embodiment is substantially the same as the flowchart in FIG. 4 in the first embodiment, and includes steps S3000, S3010, S3020, S3030, S3040, S3060, S3070, and S3080. , S3090 are the same processes as steps S1000, S1010, S1020, S1030, S1040, S1060, S1070, S1080, and S1090, respectively, so the description is omitted, and steps S3050, Step S3051 will be explained.

In step S3050, the measurement result processing unit 304 determines the presence/absence of defects in the first, second, and third positions and orientations and the first, second, and third geometric information calculated in step S3040. If there is no problem, the process proceeds to step S3060. If there is a problem, the process advances to step S3051. The specific processing for determining whether or not there is a defect has already been described using the flowcharts of FIGS. 9 and 11 in the second embodiment, so a description thereof will be omitted.

In step S3051, the measurement result processing unit 304 solves the problem based on the first, second, and third position/orientation and the first, second, and third geometric information. (details will be described later). Then, the measurement result processing unit 304 returns the processing to step S3010.

As for the driving control while the problem is being resolved, the driving mode is switched to a driving mode in which a person drives the vehicle, and automatic driving or driving assistance is not performed. However, it is not limited to this, and if the time to resolve the problem is relatively short and there is no problem with the estimation results so far, we will Operation control may be performed. For example, when the vehicle is traveling on a straight road with no obstacles in the surroundings and it can be estimated that this state will continue, automatic driving or driving assistance may be performed.

<Processing by the measurement result processing unit to eliminate the problem>
Here, FIG. 14 is a flow chart showing a procedure for the measurement result processing unit 304 to solve the problem in step S3051 in the third embodiment.

In step S3100, the measurement result processing unit 304 obtains information indicating which of the first, second, and third positions and orientations and the first, second, and third pieces of geometric information have defects. get.

In step S3110, measurement result processing section 304 classifies the input information. As notation of the classification method, hereinafter, when there is a problem with the first position/orientation or geometric information, it is described as first: x, and when there is no problem, it is described as first: o. For example, when notated as 1st: O, 2nd: X, and 3rd: X, there is no problem in the first position/orientation and geometric information, but there is a problem in the second position/orientation or geometric information, and the third position/orientation or geometric information is Indicates that there is a problem with the position, orientation, or geometric information of the . If the defect is classified as 1st: O, 2nd: X, or 3rd: X, the measurement result processing unit 304 advances the process to step S3111. In the case of 1st: ◯, 2nd: ◯, 3rd: × or 1st: ◯, 2nd: ×, 3rd: ◯, the measurement result processing unit 304 advances the process to step S3112. Moreover, in the case of the first: ×, the second: ○, and the third: ○, the measurement result processing unit 304 advances the process to step S3113. Also, in the case of first: ×, second: ○, third: × or first: ×, second: ×, third: ○, the measurement result processing unit 304 advances the process to step S3114. Then, in the case of the first: x, the second: x, and the third: x, the measurement result processing unit 304 advances the process to step S3115.

In step S<b>3111 , the measurement result processing unit 304 changes the learning model held by the learning model holding unit 301 to improve the estimation of the second and third geometric information in the geometric information estimation unit 302 . The learning model to be changed is preferably a model that has been trained in a scene similar to the shooting scene. As a change method, a feature amount may be extracted from the image of the captured scene, and a learning model may be determined based on the extracted feature amount. A learning model may be selected from positionally modified experience.

In step S3112, the measurement result processing unit 304 unifies the learning models held by the learning model holding unit 301 used when estimating the second and third pieces of geometric information. Improve the estimation of either the second or third geometric information. The learning model to be unified is the learning model used for estimating defect-free geometric information. However, the present invention is not limited to this, and instead of unifying one learning model with the other learning model, both learning models may be changed in the same manner as in step S3111.

In step S3113, the measurement result processing unit 304 calculates calibration parameters between the first imaging device and the second imaging device at the time when the calibration calculation unit 309 captures images. Calibration parameters are calculated by obtaining a transformation from one to the other based on the second pose and the third pose. However, the calculation method is not limited to this, and may be calculated based on the first image and the second image, or may be calculated based on the second geometric information and the third geometric information. . Then, the measurement result processing unit 304 advances the process to step S3116.

In step S3116, the measurement result processing unit 304 compares the calibration parameter calculated in step S3113 with the calibration parameter held in the calibration parameter holding unit (not shown) to obtain the error. If the calibration error is large and the area where the scenes captured by the first image and the second image overlap is less than half of the area in the image, the measurement result processing unit 304 advances the process to step S3117. Although there is a calibration error, if the error is small, the measurement result processing unit 304 advances the process to step S3118. If the calibration parameter calculated with almost zero calibration error and the calibration parameter held by the calibration parameter holding unit can be considered to be almost the same, the measurement result processing unit 304 advances the process to step S3119.

In step S3117, the display information generation unit 305 generates an image for instructing the user to re-install at least one of the first imaging device and the second imaging device in an appropriate position and orientation. , the display unit 11 is used to display. Specifically, the position and orientation of each of the first imaging device and the second imaging device in the vehicle are obtained, and the user adjusts the position and orientation of the imaging device so that they fall within the preset ideal position and orientation range of the imaging device. move the At this time, the display information generation unit 305 determines how much the ideal position and orientation of the imaging device match the current position and orientation of the imaging device, and how to move the imaging device to approach the ideal position and orientation. An image indicating whether or not is generated, and is displayed using the display unit 31 . After the user finishes installing the imaging apparatus, the calibration calculation unit 309 calculates the current calibration parameters, and updates the calibration parameters held in the calibration parameter holding unit (not shown). In this way, the user improves the estimation of the first geometric information in the geometric information estimation unit 302 by repositioning the imaging device in an appropriate position and orientation. In addition, in order to stably obtain the position and orientation of the imaging device in the vehicle, a learning model learned from learning data measured in the vehicle is prepared in advance, and the learning model held by the learning model holding unit 301 is changed to this learning model. may

In step S3118, the measurement result processing unit 304 uses the calibration parameters calculated in step S3113 to update the calibration parameters held by the calibration parameter holding unit (not shown). This improves the estimation of the first geometric information in the geometric information estimator 302 .

In step S3119, in order for the geometric information estimating unit 302 to estimate the first geometric information, the feature point detection method used when obtaining the correspondence between the first image and the second image is changed. This improves the estimation of the first geometric information in the geometric information estimator 302 . There are various feature amounts such as SIFT, ORB, and AKAZE as methods for detecting feature points, but the method is not particularly limited. Also, although the method for detecting feature points has been described here as changing, the present invention is not limited to this. For example, geometric information estimation by stereo may be improved by changing the similarity index for determining the stereo correspondence among SAD, SSD, NCC, ZNCC, etc., or by changing the search range, threshold value, etc. good too.

In step S<b>3114 , the measurement result processing unit 304 restarts the defective imaging device of the imaging unit 10 to improve the imaging device itself. In addition, the display information generation unit 305 generates the first image and the second image so that the user can check whether there is any defect in the first image and the second image captured by the first imaging device and the second imaging device, respectively. An image for displaying the two images side by side is generated and displayed using the display unit 11 . Here, an example of improving the imaging device by restarting the imaging device has been described, but the present invention is not limited to this. For example, if the imaging device is broken and cannot capture an image, the display information generation unit 305 may display a prompt to replace the imaging device in order to replace the imaging device. Further, when there is an abnormality in the lens of the image pickup apparatus, the display information generation unit 305 may perform display prompting replacement or cleaning of the lens.

In step S<b>3115 , the display information generation unit 305 generates an image indicating that the automatic driving or the driving assist function cannot be performed and that the driving mode is switched to the driving mode for human driving, and displays the image using the display unit 11 . do. Furthermore, while the person is driving, the system is restored by executing improvement measures such as steps S3111, S3112, S3114, S3117, S3118, and S3119. Through the above processing, the measurement result processing unit 304 solves the problem.

<effect>
As described above, according to the third embodiment, if there is a defect in the estimated geometric information or position/orientation, a countermeasure is taken to resolve the defect, thereby making the system safer and more stable. system can be operated continuously. In addition, by displaying and informing the user of the status of the problem, the user can use the functions of automatic driving and driving assistance with peace of mind.

<Modification>
In the third embodiment, an example of performing processing according to the flowchart shown in FIG. 13 has been described, but the present invention is not limited to this. In the third embodiment, the position and orientation are calculated in step S3040, and then the presence or absence of a defect is determined in step S3050. Similar to the example shown in the modified example of the embodiment, the geometric information calculated in step S3030 may be used to determine whether or not there is a problem, take measures, and then calculate the position and orientation.

In the third embodiment, in step S3050, the example of proceeding to either step S3060 or step S3051 depending on whether there is a problem has been described. However, it is not limited to this. For example, even if there is a problem, if there is no problem in any of the first, second, and third geometric information and the position and orientation, the processing is divided into two, and the problem is dealt with in step S3051. At the same time, the process of step S3060 may be performed using geometric information and position/orientation without any defects. As a result, even if there is a problem, you can continue to use the automatic driving or driving assist function.

In the third embodiment, in step S3051, an example has been described in which if there is a problem, it is dealt with immediately. However, it is not limited to this. For example, when an obstacle crosses in front of the first imaging device, it is determined that there is a problem with the first and second geometric information. There is also For this reason, it is possible to take measures when a defect occurs continuously over several frames. This can prevent an unnecessary increase in computational load.

[Fourth embodiment]
The geometric information estimation unit in the first, second, and third embodiments has described an example of estimating geometric information using an existing learning model. On the other hand, in the fourth embodiment, teacher data for learning a learning model is acquired, learning is performed, and a learning model is generated. In this way, automatic driving or driving assistance is performed by generating a new learning model that enables estimation of geometric information even for an unknown scene in which geometric information cannot be estimated by existing learning models. Since the configuration of the vehicle in the fourth embodiment is the same as that shown in FIG. 1 for the first embodiment, it will be omitted.
<Configuration of information processing device>
FIG. 15 is a diagram showing a device configuration example of the information processing device 1 and the operation processing device 12 in this embodiment. The information processing apparatus 1 includes an image input unit 400 , a learning model storage unit 401 , a geometric information estimation unit 402 , a position/orientation calculation unit 403 , a measurement result processing unit 404 , a display information generation unit 405 , a learning unit 409 , and a control unit 25 . I have it. The operation processing device 12 also includes an ambient environment acquisition unit 406 , a destination acquisition unit 407 and an operation control unit 408 . Image input unit 400, learning model storage unit 401, geometric information estimation unit 402, position/orientation calculation unit 403, measurement result processing unit 404, display information generation unit 405, ambient environment acquisition unit 406, destination acquisition unit 407, operation control unit 408, the actuator unit 43, and the control unit 45 are the image input unit 300, the learning model holding unit 301, the geometric information estimation unit 302, the position/orientation calculation unit 303, the measurement result processing unit 304, and the display information in the third embodiment. It corresponds to the generation unit 305 , the ambient environment acquisition unit 306 , the destination acquisition unit 307 , the operation control unit 308 , the actuator unit 33 and the control unit 35 . A description of similar functions will be omitted, and the learning model holding unit 401, the measurement result processing unit 404, and the learning unit 409, which are different, will be described.

A learning model holding unit 401 holds a model for estimating geometric information from an image. Also, the learning model holding unit 401 holds a learning model supplied from a learning unit 409 which will be described later. This allows the geometric information estimation unit 402 to select an appropriate learning model according to the situation.

In addition to the functions of the measurement result processing unit 304 in the third embodiment, the measurement result processing unit 404 creates a learning model based on the first image, the second image, and the geometric information corresponding to these images. Generate teacher data used for learning. Then, the measurement result processing unit 404 outputs the generated teacher data to the learning unit 409 .

The learning unit 409 performs learning based on the teacher data input from the measurement result processing unit 404 and generates a learning model. The generated learning model is output to the learning model holding unit 401 .

<Processing>
Next, a processing procedure in the fourth embodiment will be described. The processing procedure in the fourth embodiment is similar to the flowchart described in the third embodiment with reference to FIG. 13 except that some processing is added, so description thereof will be omitted. In the fourth embodiment, the processing procedure for acquiring teacher data and learning is performed in parallel after step S3060 in FIG. 13 in the third embodiment. Here, FIG. 16 is a flow chart showing a processing procedure for acquiring this teacher data and performing learning.

In step S4100, the measurement result processing unit 404 acquires the first image and the second image and holds them in the acquisition list. However, it is not limited to this, and only one of the images may be acquired and held.

In step S4110, the measurement result processing unit 404 acquires the position and orientation, and stores them in the acquisition list in association with the images already stored in the acquisition list. However, the position and orientation here is the position and orientation finally calculated by the measurement result processing unit 404, and is the position and orientation used for operation control.

In step S4120, the measurement result processing unit 404 determines whether or not geometric information corresponding to the images held in the acquisition list can be acquired. If it is determined that acquisition is possible, the measurement result processing unit 404 advances the process to step S4130. If acquisition is not possible, the measurement result processing unit 404 ends the process while retaining the acquisition list. In this case, when the process of acquiring teacher data is restarted in the next frame or after several frames, further additions will be made to the acquired acquisition list. Here, being able to obtain geometric information means that geometric information updated with respect to an image can be obtained.

In step S4130, measurement result processing section 404 acquires geometric information corresponding to the image based on the image and position/orientation held in the acquisition list.

In step S4140, the measurement result processing unit 404 sets the image and the geometric information associated in step S4130, and outputs the set to the learning unit 409 as teacher data. A learning unit 409 holds teacher data input from the measurement result processing unit 404 .

In step S4150, learning unit 409 determines whether or not to start learning.If learning is to be started, the process proceeds to step S4160. If learning is not to be started, the processing is terminated. In this case, when the process of acquiring the teacher data is started again, the teacher data is additionally held. Also, whether or not to start learning is determined by whether or not the amount of retained teacher data is sufficient.

In step S4160, learning unit 409 performs learning using the retained teacher data group to generate a learning model. Specifically, the learning unit 409 is given training data as a pair of an image and geometric information, and estimates CNN weights that output geometric information corresponding to the image when an image is given as an input. learn by doing. However, the learning method is not limited to this, and models other than CNN may be used.

In step S4170, learning model holding unit 401 additionally holds the learning model generated by learning unit 409 in step S4160. By additionally holding, when the geometric information estimation unit 402 estimates geometric information, it is possible to select a learning model from among a plurality of learning models and estimate geometric information. However, it is not limited to this, and the learning model holding unit 401 may update the learning model to a new learning model. Through the above processing, teacher data is acquired and learning is performed.

<effect>
As described above, according to the fourth embodiment, by generating a learning model, geometric information can be obtained using the generated learning model for a scene for which geometric information cannot be estimated by the existing learning model. It is possible to estimate a stable position and orientation, and perform safer automatic driving or driving assistance. In addition, by automatically generating teacher data while this information processing device is processing, it is possible to avoid the trouble of separately creating and labeling a huge amount of teacher data necessary for learning a learning model. can be omitted.

<Modification>
In the fourth embodiment, the example in which the measurement result processing unit 404 acquires teacher data after step S3060 in FIG. 13 in the third embodiment has been described, but the present invention is not limited to this. For example, images may be held in the acquisition list before, after, or in parallel with step S3111 of FIG. 14 in the third embodiment. In other words, only when there is a problem with the second and third geometric information for estimating geometric information using the learning model and the second and third positions and orientations corresponding to them, the image is acquired from the acquisition list can be saved to As a result, it is possible to acquire a group of images of scenes in which geometric information cannot be estimated by existing learning models. By performing learning with teacher data generated based on these images, it is possible to expand the number of scenes for which geometric information can be estimated more efficiently. can be done.

In the fourth embodiment, the example in which the measurement result processing unit 404 performs learning after step S3060 in FIG. 13 in the third embodiment has been described, but the present invention is not limited to this. For example, images, positions and orientations, and geometric information required for learning are all stored as data, and a separate information processing device for offline learning is used to generate teacher data and perform learning. may In this case, the learning unit 409 may transfer its function to an information processing apparatus other than the information processing apparatus 1 . As a result, there is no need to have computational resources for learning inside the vehicle, and the cost of the information processing device 41 can be reduced.

In the fourth embodiment, the example in which the learning unit 409 performs learning using only the teacher data generated by the measurement result processing unit 404 has been described, but the present invention is not limited to this. For example, learning may be performed by adding part or all of the teacher data group used when learning the existing learning model. Further, additional learning may be performed based on an existing learning model.

In the fourth embodiment, in step S4120, it was explained that the geometric information can be obtained when the geometric information updated for the image is obtained, but the present invention is not limited to this. For example, the condition for obtaining geometric information may be that the ratio of geometric information obtained to the image is equal to or greater than a threshold. Alternatively, when the vehicle leaves the area within a certain range corresponding to the image, the geometric information may be acquired after the geometric information is updated.

In the fourth embodiment, the acquisition of the image in step S4100 and the acquisition of the geometric information in step S4130 are both measured by the own vehicle, and an example of generating teacher data using them has been described. , but not limited to this. For example, images or geometric information of the required area may be downloaded from a server and used to generate training data.

In the fourth embodiment, the geometric information estimation unit 402 selects an appropriate learning model from the plurality of learning models held by the learning model holding unit 401 according to the situation, but a specific example will be described. In the learning model, a partial image of teacher data corresponding to the learning model is linked in advance as a sample scene image. A geometric information estimation unit 402 evaluates the degree of similarity between the input image and the sample scene image, and estimates geometric information using a learning model linked to the sample scene image with a high evaluation value. Also, a plurality of sample scene images may be associated with one learning model.

In the fourth embodiment, the geometric information estimating unit 402 selects an appropriate learning model according to the situation from among the plurality of learning models held by the learning model holding unit 401, but the present invention is not limited to this. . For example, geometric information may be generated for each of a plurality of learning models, fourth and fifth geometric information may be further generated, and subsequent processing may be performed.

[Fifth Embodiment]
In the first, second, third, and fourth embodiments, the method of estimating geometric information based on triangulation has been described as a stereo estimation method using two imaging devices. On the other hand, in the fifth embodiment, a projection unit that projects a predetermined pattern toward the space imaged by the imaging unit is provided, and geometric information is estimated by active stereo using a single imaging device and the projection unit. FIG. 17 shows the configuration of the automobile in this embodiment. In the fifth embodiment, in order to obtain the position and orientation of the imaging unit 50 attached to the automobile 50000, geometric information obtained by active stereo by the projection unit 54 and the imaging unit 50 and the learning described in the first embodiment are used. Stable geometric information and position/orientation are obtained from the two systems of geometric information estimated using the model and the corresponding position/orientation.
<Configuration of information processing device>
FIG. 18 is a diagram showing a functional configuration example of the information processing device 5 and the operation processing device 52 in the fifth embodiment. The information processing apparatus 5 includes an image input unit 500, a learning model holding unit 501, a geometric information estimation unit 502, a position/orientation calculation unit 503, a measurement result processing unit 504, a display information generation unit 505, a projection control unit 509, and the entire apparatus. is provided with a control unit 55 for controlling . The operation processing device 52 also includes an ambient environment acquisition unit 506, a destination acquisition unit 507, and an operation control unit 508 that controls the entire device. Image input unit 500, learning model storage unit 501, geometric information estimation unit 502, position/orientation calculation unit 503, measurement result processing unit 504, display information generation unit 505, ambient environment acquisition unit 506, destination acquisition unit 507, operation control unit 508, the imaging unit 50, the display unit 51, the operation processing device 52, the actuator unit 53, and the control unit 55 are the same as the image input unit 100, the learning model holding unit 101, the geometric information estimation unit 102, the position/orientation unit 102 in the first embodiment. calculation unit 103, measurement result processing unit 104, display information generation unit 105, ambient environment acquisition unit 106, destination acquisition unit 107, operation control unit 108, imaging unit 10, display unit 11, operation processing unit 12, actuator unit 13, and corresponds to the control unit 15 . The description of the similar functions is omitted, and the projection control unit 509, the image input unit 500, and the geometric information estimation unit 502, which are different, will be described. Note that the position/orientation calculation unit 503 and the measurement result processing unit 504 are the same as the position/orientation calculation unit 103 and the measurement result processing unit 104 in the first embodiment. While the third and third systems of geometric information and position/orientation were handled, the only difference in this embodiment is that the first and second two systems of geometric information and position/orientation are handled, so a description thereof will be omitted. . The imaging unit 50 is composed of one imaging device, and the position and orientation relationship between the imaging device of the imaging unit 50 and the projection unit 54 is obtained in advance by calibration, and calibration parameters are stored in a calibration parameter holding unit (not shown). shall be retained.

A projection control unit 509 controls projection of the pattern light toward the imaging space of the imaging unit, and performs projection using the projection unit 54 . The projection control unit 509 also outputs information about the projected pattern to the geometric information estimation unit 502 .

The image input unit 500 inputs the image data of the two-dimensional image of the subject space on which the pattern is projected in time series by the imaging device of the imaging unit 50, and after processing, the learning model holding unit 501, the geometric information The information is output to the estimation unit 502 , the position/orientation calculation unit 503 , the display information generation unit 505 , and the operation processing device 52 . The process performed by the image input unit 500 is to separate the captured image into a first image affected by the projection pattern and a second image not affected by the projection pattern. Specifically, the pattern projected by the projection unit 54 is assumed to be infrared light, and the imaging unit 50 is capable of imaging two channels of visible light and infrared light. Then, the image input unit 500 processes the image of the infrared light channel as the first image and the image of the visible light channel as the second image, and outputs the first image and the second image.

The geometric information estimation unit 502 estimates the first geometric information by active stereo based on the first image input by the image input unit 500 and the pattern information input by the projection control unit 509 . Also, the geometric information estimation unit 502 estimates second geometric information from the second image input from the image input unit 500 using the learning model held by the learning model holding unit 501 . The estimated first and second geometric information are output to the position and orientation calculation unit 103 .

<Processing>
Next, a processing procedure in the fifth embodiment will be described. FIG. 19 is a flow chart showing a processing procedure performed by an information processing system including the information processing device 5 according to the fifth embodiment.

The flowchart in the fifth embodiment is substantially the same as the flowchart in FIG. 4 in the first embodiment. Since the processing is the same as that of S1050, step S1060, step S1070, step S1080, and step S1090, description thereof is omitted. Steps S5000, S5010, S5020, and S5030, which are different, will be described below.

At step S5000, the controller 55 initializes the system. In addition to the processing of step S1000 in the first embodiment, the control unit 55 also activates the projection unit 54 and reads the setting of the pattern projected by the projection control unit 509 .

In step S<b>5010 , the control unit 55 controls the projection control unit 509 to project a pattern onto the scene from the projection unit 54 , cause the imaging device of the imaging unit 50 to capture the scene, and transmit the captured image to the image input unit 500 . output.

In step S5020, the control unit 55 controls the image input unit 500 to acquire the image captured in step S5010 and separate it into a first image and a second image.

In step S5030, the control unit 55 controls the geometric information estimation unit 502 to estimate first and second geometric information based on the first image and the second image, respectively. The first geometric information is obtained by active stereo. Specifically, the geometric information estimating unit 502 obtains the correspondence relationship between the first image and the projection pattern based on the information of the first image and the projection pattern input by the projection control unit 509 . Then, the geometric information estimating unit 502 performs triangulation based on calibration parameters representing the position and orientation relationship between the imaging device of the imaging unit 50 and the projection unit 54 from a calibration parameter holding unit (not shown) to estimate a range image. Also, the geometric information estimation unit 502 estimates the second geometric information from the second image using the learning model, as in the first embodiment.

<effect>
As described above, the fifth embodiment has a function of estimating geometric information using active stereo by pattern projection. Therefore, even in an unknown scene with few features, geometric information can be stably obtained, which increases safety in automatic driving and driving assistance.

<Modification>
In the fifth embodiment, an example in which the image input unit 500 separates into two channels of infrared light and visible light has been described, but the method of separating into the first image and the second image is not limited to this. For example, if the projection unit 54 projects a random dot pattern of infrared light and the imaging unit 50 can capture a monochrome image of infrared light, the captured image is the first image, Gaussian blur or filter The second image may be an image that has been blurred by processing so that the effect of the random dot pattern is not visible. Alternatively, the projection unit 54 and the imaging unit 50 may be controlled at high speed, and an image captured with a projection pattern may be used as the first image, and an image captured without a projection pattern may be used as the second image.

In the fifth embodiment, the example in which the image input unit 500 separates the first image and the second image has been described, but the present invention is not limited to this. For example, with respect to the process of separating images, the geometric information estimation unit 502, the position/orientation estimation unit 503, the measurement result processing unit 504, and the like perform separation processing first, and the image input unit 500 inputs one image. may

In the fifth embodiment, the pattern projected by the projection control unit 509 is a random dot pattern, but the projection pattern is not limited to this. For example, the projection pattern may be a grid pattern, or if projection and imaging can be performed at high speed, a phase shift pattern or gray code pattern may be used to estimate geometric information from a plurality of projection pattern captured images.

In the fifth embodiment, the calculation result processing in step S5050 is described as being performed after position and orientation calculation, but the present invention is not limited to this. As in the modification of the first embodiment and the modification of the second embodiment, after estimating geometric information, calculation result processing may be performed to obtain one piece of geometric information, and then position and orientation calculation may be performed.

Although it has been described that the processing procedure in the fifth embodiment is substantially the same as that in the first embodiment, it is not limited to this. By performing similar processing to the second embodiment, the first geometric information and the second geometric information may be appropriately switched. This allows processing even with relatively small computing resources.

Although it has been described that the processing procedure in the fifth embodiment is substantially the same as that in the first embodiment, it is not limited to this. By performing the same processing as in the third embodiment, even if there is a problem, by taking measures to eliminate it, the system can be operated continuously more safely and stably. be able to.

Although it has been described that the processing procedure in the fifth embodiment is substantially the same as that in the first embodiment, it is not limited to this. By performing the same processing as in the fourth embodiment, a learning model is generated, and stable position and orientation estimation is performed even for a scene in which geometric information could not be estimated by the existing learning model, Autonomous driving or driving assistance can be performed more safely.

In the fifth embodiment, the example in which the projection unit 54 has only the role of projecting the pattern onto the scene to be imaged has been described, but the present invention is not limited to this. For example, the projection unit 54 may also serve as the function of the display unit 51 and display to the user by projection. Also, the projection unit 54 may be used as a front light for the automobile 50000 . Alternatively, a front light (not shown) may be used to have the function of the projection unit 54 as well.

In the fifth embodiment, the pattern projected by the projection unit 54 may be modulated so that the pattern can be uniquely identified even when there are other light sources and other projection patterns. As a result, even when a plurality of vehicles having the same function exist nearby, the geometric information can be stably estimated without mutual interference between projected patterns.

In the fifth embodiment, the example in which the projection unit 54 is one projection device and the imaging unit 50 is one imaging device has been described. However, it is not limited to this. For example, a plurality of projection devices and a plurality of imaging devices may be provided. In that case, three or more pieces of geometric information and positions and orientations can be obtained. Moreover, you may process using a part out of many. As a result, since estimation is performed using a large amount of geometric information and position/orientation, it is possible to obtain geometric information and position/orientation with higher accuracy and robustness.

(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

1: information processing device, 10: imaging unit, 11: display unit, 12: operation processing unit, 13: actuator unit, 100: image input unit, 101: learning model holding unit, 102: geometric information estimation unit, 103: position Posture calculation unit 104: measurement result processing unit

Claims (16)

A first image obtained by the first imaging device and the second image obtained by the first imaging device and the second imaging device, which are arranged on a moving object so that at least a part of the imaging field of view overlaps. input means for inputting a second image obtained by the imaging device;
Using an image and distance information representing the three-dimensional shape of the scene in which the image was shot as training data, the learned system outputs distance information representing the three-dimensional shape of the scene in which the image was shot corresponding to the input image. Acquisition means for acquiring the trained model from the learning model holding means ;
a first estimating means for estimating distance information representing a three-dimensional shape of a first photographed scene from the first image and the second image;
second estimating means for estimating distance information representing a three-dimensional shape of a second shot scene corresponding to the first image using the learned model;
third estimating means for estimating distance information representing a three-dimensional shape of a third shot scene corresponding to the second image using the learned model;
position information of an imaging unit including the first imaging device and the second imaging device based on at least one of distance information representing the three-dimensional shape of the first, second, and third shot scenes ; generating means for generating at least one of distance information representing a three-dimensional shape of a captured scene represented by an image captured by the imaging unit;
An information processing device comprising: The generating means is
a first calculation means for calculating first position information of the imaging unit based on distance information representing the three-dimensional shape of the first captured scene ;
a second calculation means for calculating second position information of the imaging unit based on distance information representing the three-dimensional shape of the second captured scene ;
a third calculation means for calculating third position information of the imaging unit based on distance information representing the three-dimensional shape of the third captured scene ;
2. The apparatus according to claim 1, further comprising: processing means for performing processing for outputting position information of said imaging unit based on at least one of said first, second and third position information. Information processing equipment. The generating means is
Depth information represented by an image captured by an imaging device serving as a reference in the imaging unit is generated based on distance information representing the three-dimensional shape of the first, second, and third captured scenes. The information processing device according to claim 1 . The generating means is
output means for outputting one of the distance information representing the three-dimensional shapes of the first, second, and third photographed scenes as depth information of the imaging unit;
a judgment means for judging whether the distance information representing the three-dimensional shape of the photographed scene output by the output means is appropriate or inappropriate;
and setting means for setting distance information representing a three-dimensional shape of another photographed scene as an output target of the output means when the judgment means judges that it is inappropriate. Item 1. The information processing apparatus according to item 1. The generating means calculates the position information of the imaging unit based on the generated distance information representing the three-dimensional shape of the photographed scene, and updates the distance information representing the three-dimensional shape of the photographed scene. 5. The information processing apparatus according to any one of claims 1 to 4. 6. The information processing apparatus according to claim 1, further comprising display information generating means for generating display information for a user based on the information generated by said generating means. 6. The information processing apparatus according to any one of claims 1 to 5, further comprising detecting means for detecting an abnormality related to the imaging unit based on the position information. further comprising calibration calculation means for performing calibration between the first and second imaging devices of the imaging unit based on distance information representing the three-dimensional shape of the second and third captured scenes ;
3. The calibration calculation means performs calibration when the detection means detects that an abnormality has occurred in the calibration between the first imaging device and the second imaging device. 8. The information processing device according to 7. When the detecting means detects that the distance information representing the three-dimensional shape of the first photographed scene is abnormal, the generating means reproduces the three-dimensional shape of the second and third photographed scenes. 8. The information processing apparatus according to claim 7, wherein the position information is generated based on at least one of the represented distance information. When the detecting means detects that the distance information representing the three-dimensional shape of the first photographed scene is abnormal, the generating means reproduces the three-dimensional shape of the second and third photographed scenes. 8. The information processing apparatus according to claim 7, wherein distance information representing a three-dimensional shape of a photographed scene is generated based on at least one of the represented distance information. 11. The apparatus according to any one of claims 7 to 10, further comprising learning means for learning a model to be newly acquired by said acquiring means based on distance information representing the three-dimensional shape of said photographed scene. The information processing device according to . When the detecting means detects that the distance information representing the three-dimensional shape of the second and third photographed scenes is abnormal, the learning means acquires additional teacher data for use in learning. 12. The information processing apparatus according to claim 11, characterized by: Furthermore, display information generating means for generating display information based on the information generated by the generating means;
13. The information processing apparatus according to claim 1, further comprising display means for displaying said display information. A first image obtained by the first imaging device and the second image obtained by the first imaging device and the second imaging device, which are arranged on a moving object so that at least a part of the imaging field of view overlaps. an input step of inputting a second image obtained by the imaging device;
Using an image and distance information representing the three-dimensional shape of the scene in which the image was shot as training data, the learned system outputs distance information representing the three-dimensional shape of the scene in which the image was shot corresponding to the input image. an acquiring step of acquiring the learned model from the learning model holding means ;
a first estimation step of estimating distance information representing a three-dimensional shape of a first captured scene from the first image and the second image;
a second estimation step of estimating distance information representing a three-dimensional shape of a second captured scene corresponding to the first image using the learned model;
a third estimation step of estimating distance information representing a three-dimensional shape of a third captured scene corresponding to the second image using the learned model;
position information of an imaging unit including the first imaging device and the second imaging device based on at least one of distance information representing the three-dimensional shape of the first, second, and third shot scenes ; a generation step of generating at least one of distance information representing a three-dimensional shape of a captured scene represented by an image captured by the imaging unit;
A control method for an information processing device, comprising: A program that, when read and executed by a computer, causes the computer to function as the information processing apparatus according to any one of claims 1 to 13. an information processing apparatus according to any one of claims 1 to 13;
an operation control means for controlling the operation of the moving body based on the information generated by the generation means in the information processing device;
and an actuator section operated by the operation control means.

Images