JP5012615B2 - Information processing apparatus, image processing method, and computer program - Google Patents

Description

  The present invention relates to an information processing apparatus, an image processing method, and a computer program. More specifically, in generating a three-dimensional map (3D map) using a camera-captured image, it is possible to calculate a feature point position with high accuracy and generate a highly accurate three-dimensional map by selecting an optimal image frame. The present invention relates to an information processing apparatus, an image processing method, and a computer program.

  Processing for analyzing a photographed image of a camera and obtaining a three-dimensional position of an object included in the photographed image is used in various fields. For example, an agent (moving body) such as a robot equipped with a camera analyzes the captured image of the camera and observes the moving environment, and performs processing of moving while grasping the environment around the agent according to the observation situation, It is used for environment map construction processing for creating a map of the surrounding environment (environment map) based on Non-Patent Document 1 discloses a method of tracking feature point positions in all frames and calculating the positions of feature points and camera positions by batch processing after data of all frames is obtained.

  An example of a three-dimensional map (3D map) generation processing sequence will be described with reference to FIG. First, in step S11, an image is taken by a camera. For example, a user or a robot holding a camera continuously takes surrounding images while moving.

  In step S12, sparse three-dimensional information including position information of feature points included in the image is constructed by analyzing the acquired image. In this process, a process such as SLAM (simultaneous localization and mapping) or SFM (Structure from Motion) is applied. SLAM is a process for detecting the position of a feature point in an image input from the camera and the position and orientation of the camera. SFM is, for example, processing for analyzing correspondence between feature points (Landmarks) included in an image using images taken from a plurality of different positions.

  Furthermore, in step S13, a three-dimensional map, which is dense three-dimensional information, which is detailed three-dimensional information, is generated using the camera trajectory information obtained in step S12 and the feature point information in the image.

  Accurate analysis of the feature points in the image in the SFM or SLAM processing in step S12 increases the accuracy of the final three-dimensional information. In step S12, a process of acquiring feature point information and camera trajectory information included in the image frame is performed. An example of this detailed processing is shown in FIG.

  First, in step S <b> 21, a plurality of captured images are input from the camera, and frame matching processing for calculating the position of the camera is performed using matching feature points between frames. The input image is, for example, a plurality of image frames of a moving image captured by a moving camera, that is, a plurality of images captured from different positions, and the same object is captured in a plurality of frames. In step S21, corresponding feature points are detected from a plurality of image frames, and the position of the camera is calculated using these pieces of information.

  In step S22, a frame stitch process (Frame Stitch) is executed. This process estimates the three-dimensional position of each image feature point (Landmark) using the information detected in step S21, and joins image frames based on a plurality of feature point positions in a plurality of image data. This is a process of generating data reflecting the three-dimensional position of each feature point.

  In step S23, bundle adjustment processing (Bundle Adjustment) is executed. This bundle adjustment process is a process for converging the three-dimensional positions of corresponding feature points included in a plurality of images taken from different positions into one position. This is a process of calculating the most probable three-dimensional position of the feature point using the camera position information of each captured image and the information of the corresponding feature point included in the image captured at each camera position. For this processing, basically, image frames taken from two or more different positions are required.

  For example, as shown in FIG. 3, the captured images 11 to 13 from three different positions are used, and the feature points 21 to 23 detected from the captured images are used to obtain the three-dimensional positions of the feature points. Among the feature points included in a plurality of images, the position of the corresponding feature point is the position of the feature point 21 where a line (bundle) connecting the feature point 21 to the shooting point of each image is a feature point 21 shown in FIG. Should cross.

  However, the position of the camera, the position information in the image of the feature points, etc. are not necessarily calculated accurately, and errors due to various factors are included. Therefore, it is necessary to remove such errors. Specifically, correction is performed so that a line connecting the three-dimensional position of one corresponding feature point and the camera position intersects at the one feature point. That is, it is necessary to correct the calculated camera position and feature point position. This processing is executed as bundle adjustment processing (Bundle Adjustment), and it becomes possible to generate more accurate three-dimensional information using the feature point position information and camera position information corrected by this adjustment processing.

In such a process, in order to generate accurate three-dimensional information,
Obtaining a sufficient number of corresponding feature points from multiple captured images;
The corresponding feature points are included in multiple images with different shooting directions,
Accurate feature point position information with few errors can be calculated from the corresponding feature point information,
For example, selection of an image frame to be analyzed is an important factor. An expert of 3D image analysis can intuitively determine an image satisfying such a condition and select an appropriate image. However, a normal user has many image frames taken by himself / herself. It is difficult to select an appropriate image from.
“Shape and motion from image stream under orthography: a factorization method”, C.I. Tomasi and T. Kanade, International Journal of Computer Vision, Volume 9, Number 2, pp. 137-154, (1992).

  The present invention has been made in view of the above-described problems, for example. For example, an image frame suitable for generating three-dimensional information from a large number of image frames constituting a moving image photographed by a video camera is provided. An information processing apparatus, an image processing method, and a computer program that generate high-precision three-dimensional information by selecting according to a preset algorithm and performing processing such as calculation of a feature point position using the selected image The purpose is to provide.

  Another object of the present invention is to provide an information processing apparatus, an image processing method, and a computer program for performing image evaluation processing applicable to image selection for realizing generation of accurate three-dimensional information. To do.

The first aspect of the present invention is:
An information processing apparatus that calculates a three-dimensional position of a pixel included in an image,
An image selection unit for selecting a plurality of images as input images from an input image;
An evaluation value calculation unit for calculating an evaluation value for determining whether or not the selected image is an image suitable for generating three-dimensional data;
Based on the evaluation value calculated by the evaluation value calculation unit, an evaluation value determination unit that determines whether or not the selected image is an image suitable for generating three-dimensional data,
The evaluation value calculation unit
(A1) Estimated number of feature points of the three-dimensional position included in the selected image (b1) Intersection of rays connecting the estimated three-dimensional position of the corresponding feature point included in the selected image and the feature point position of each image frame Horn,
(C1) Camera model fitting error of reference image At least one of the above (a1) to (c1) is calculated as an evaluation value,
The evaluation value determination unit
Comparing at least one of the above values (a1) to (c1) with a preset threshold value, it is configured to determine whether or not the selected image is an image suitable for generating three-dimensional data,
(A2) the estimated number of feature points of the three-dimensional position included in the selected image is equal to or greater than a specified threshold (Tha);
(B2) the intersection angle between rays connecting the estimated three-dimensional position of the corresponding feature point included in the selected image and the feature point position of each image frame is equal to or greater than a specified threshold (Thb);
(C2) The camera model fitting error of the reference image is not more than a specified threshold value (Thc),
An information processing apparatus is characterized in that, when at least one of the conditions (a2) to (c2) is satisfied, the selected image is determined to be an image suitable for generating three-dimensional data.

  Furthermore, in an embodiment of the information processing apparatus of the present invention, the evaluation value calculation unit calculates all of (a1) to (c1) as evaluation values, and the evaluation value determination unit includes the (a2) to (a2) to When all the conditions of (c2) are satisfied, it is determined that the selected image is an image suitable for generating three-dimensional data.

  Furthermore, in an embodiment of the information processing apparatus of the present invention, the evaluation value calculation unit calculates a plurality of calculated intersection angles between light rays by applying a plurality of corresponding feature points and image frames. The median value of the intersection angle is calculated, and the evaluation value determination unit is suitable for generating the three-dimensional data when the selected median is satisfied that the median is equal to or greater than a predetermined threshold (Thb). It is determined that the image is an image.

  Furthermore, in an embodiment of the information processing apparatus of the present invention, the information processing apparatus generates an environment recognition result including a three-dimensional position of a feature point included in the image and position and orientation information of the camera by analyzing the selected image. And the evaluation value calculation unit calculates the evaluation value by applying information included in the environment recognition result.

  Furthermore, in an embodiment of the information processing apparatus of the present invention, the information processing apparatus includes a reference image selection unit that selects a reference image for verifying the environment recognition result, and the reference image selection unit includes a reference The evaluation value is a configuration in which an image candidate and an image satisfying a condition that a distance within one image coordinate system of corresponding feature points commonly included in the selected image is equal to or greater than a predetermined threshold are selected as reference images. The calculation unit calculates the evaluation value by applying the reference image selected by the reference image selection unit.

  Furthermore, in an embodiment of the information processing apparatus of the present invention, the information processing apparatus includes a feature point tracking unit that obtains a feature point locus included in an image by analyzing the selected image, and the evaluation value calculation unit includes: The evaluation value is calculated by applying the feature point trajectory.

  Furthermore, in one embodiment of the information processing apparatus of the present invention, the input image selection unit sets and updates a variable [T] corresponding to the selected frame interval and a variable [F] corresponding to the first selected frame. , A configuration for selecting different images, and when the evaluation value determining unit determines that the selected image is not an image suitable for generating three-dimensional data, the variable [T] or [F] At least one of the variables is updated to select a new image.

  Furthermore, in one embodiment of the information processing apparatus of the present invention, the input image selection unit is configured to change the variable [T] or the variable [T] or according to a predetermined variable update algorithm according to the evaluation mode of the selection image in the evaluation value determination unit. Updating at least one of the variables of [F] is performed to select a new image.

  Furthermore, in an embodiment of the information processing apparatus of the present invention, the information processing apparatus further includes an SFM (Structure) to which the selected image determined by the evaluation value determination unit as an image suitable for generating three-dimensional data is applied. It has a configuration in which three-dimensional data generation is performed by a process to which an extended Kalman filter (EKF) using information generated by the SFM process as initial information is applied.

Furthermore, the second aspect of the present invention provides
In the information processing apparatus, an image processing method for performing an image selection process to be applied to calculate a three-dimensional position of a pixel included in an image,
An image selection step in which an image selection unit selects a plurality of images as selected images from the input image;
An evaluation value calculating step for calculating an evaluation value for determining whether the selected image is an image suitable for generating the three-dimensional data;
The evaluation value determination unit includes an evaluation value determination step of determining whether the selected image is an image suitable for generating three-dimensional data based on the evaluation value calculated in the evaluation value calculation step;
The evaluation value calculating step includes:
(A1) Estimated number of feature points of the three-dimensional position included in the selected image (b1) Intersection of rays connecting the estimated three-dimensional position of the corresponding feature point included in the selected image and the feature point position of each image frame Horn,
(C1) Camera model fitting error of reference image is a step of calculating at least one of the above (a1) to (c1) as an evaluation value,
The evaluation value determination step includes
Comparing at least one of the above values (a1) to (c1) with a preset threshold value to determine whether the selected image is an image suitable for generating three-dimensional data;
(A2) the estimated number of feature points of the three-dimensional position included in the selected image is equal to or greater than a specified threshold (Tha);
(B2) the intersection angle between rays connecting the estimated three-dimensional position of the corresponding feature point included in the selected image and the feature point position of each image frame is equal to or greater than a specified threshold (Thb);
(C2) The camera model fitting error of the reference image is not more than a specified threshold value (Thc),
An image processing method comprising: a step of determining that the selected image is an image suitable for generating three-dimensional data when at least one of the above conditions (a2) to (c2) is satisfied is there.

  Furthermore, in one embodiment of the image processing method of the present invention, the evaluation value calculating step is a step of calculating all of (a1) to (c1) as evaluation values, and the evaluation value determining step is the above ( It is a step of determining that the selected image is an image suitable for generating three-dimensional data when all of the conditions a2) to (c2) are satisfied.

  Furthermore, in an embodiment of the image processing method of the present invention, the evaluation value calculating step includes a plurality of calculated feature points and image frames applied to the intersection angle between the light rays in the evaluation value. Calculating a median value of intersection angles, and the evaluation value determining step is performed when the selected image is suitable for generating three-dimensional data when the median satisfies the specified threshold value (Thb) or more. It is a step of determining that the image is an image.

  Furthermore, in an embodiment of the image processing method of the present invention, the image processing method is further characterized in that the environment recognition unit analyzes the three-dimensional position of the feature point included in the image and the position and orientation information of the camera by analyzing the selected image. An environment recognition step of generating an environment recognition result including the evaluation value calculation step, wherein the evaluation value calculation step is a step of calculating the evaluation value by applying information included in the environment recognition result.

  Furthermore, in one embodiment of the image processing method of the present invention, the image processing method further includes a reference image selection step in which a reference image selection unit selects a reference image for verifying the environment recognition result, The reference image selection step selects, as a reference image, an image that satisfies a condition that a distance between a reference image candidate and a corresponding feature point included in the selected image in one image coordinate system is equal to or greater than a predetermined threshold value. The evaluation value calculating step is a step of calculating the evaluation value by applying the reference image selected in the reference image selecting step.

  Furthermore, in an embodiment of the image processing method of the present invention, the image processing method further includes a feature point tracking step in which the feature point tracking unit obtains a feature point trajectory included in the image by analyzing the selected image. The evaluation value calculating step is a step of calculating the evaluation value by applying the feature point locus.

  Furthermore, in an embodiment of the image processing method of the present invention, the input image selection step sets and updates a variable [T] corresponding to the selected frame interval and a variable [F] corresponding to the first selected frame. A step of selecting different images, and when the evaluation value determining step determines that the selected image is not an image suitable for generating three-dimensional data, the variable [T] or [F] It is a step of executing a new image selection by updating at least one of the variables.

  Furthermore, in an embodiment of the image processing method of the present invention, the input image selection step includes the variable [T] or the variable [T] or the variable [T] according to a predetermined variable update algorithm according to the evaluation mode of the selected image in the evaluation value determination step. It is a step of executing a new image selection by updating at least one of the variables of [F].

  Furthermore, in an embodiment of the image processing method of the present invention, the image processing method is further configured such that the three-dimensional data generation unit determines that the image is suitable for generating three-dimensional data in the evaluation value determination step. It has a step of executing three-dimensional data generation by an SFM (Structure from Motion) process to which an image is applied and a process to which an extended Kalman filter (EKF) using information generated by the SFM process as initial information is applied. .

Furthermore, the third aspect of the present invention provides
In the information processing apparatus, a computer program that executes an image selection process applied to calculate a three-dimensional position of a pixel included in an image,
An image selection step for causing the image selection unit to select a plurality of images as selected images from the input image;
An evaluation value calculation step for causing the evaluation value calculation unit to calculate an evaluation value for determining whether or not the selected image is an image suitable for generating three-dimensional data;
An evaluation value determination unit that causes the evaluation value determination unit to determine whether or not the selected image is an image suitable for generating three-dimensional data based on the evaluation value calculated in the evaluation value calculation step;
The evaluation value calculating step includes:
(A1) Estimated number of feature points of the three-dimensional position included in the selected image (b1) Intersection of rays connecting the estimated three-dimensional position of the corresponding feature point included in the selected image and the feature point position of each image frame Horn,
(C1) Camera model fitting error of reference image is a step of calculating at least one of the above (a1) to (c1) as an evaluation value,
The evaluation value determination step includes
Comparing at least one of the above values (a1) to (c1) with a preset threshold value to determine whether the selected image is an image suitable for generating three-dimensional data;
(A2) the estimated number of feature points of the three-dimensional position included in the selected image is equal to or greater than a specified threshold (Tha);
(B2) the intersection angle between rays connecting the estimated three-dimensional position of the corresponding feature point included in the selected image and the feature point position of each image frame is equal to or greater than a specified threshold (Thb);
(C2) The camera model fitting error of the reference image is not more than a specified threshold value (Thc),
A computer program characterized in that, when at least one of the above conditions (a2) to (c2) is satisfied, it is a step of determining that the selected image is an image suitable for generating three-dimensional data. is there.

  The computer program of the present invention is, for example, a computer program that can be provided by a storage medium or a communication medium provided in a computer-readable format to a general-purpose computer system that can execute various program codes. . By providing such a program in a computer-readable format, processing corresponding to the program is realized on the computer system.

  Other objects, features, and advantages of the present invention will become apparent from a more detailed description based on embodiments of the present invention described later and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

  According to the configuration of the embodiment of the present invention, in order to determine whether or not the selected image selected from the input image is an image suitable for generating three-dimensional data, (a) 3 included in the selected image The estimated number of feature points of the three-dimensional position, (b) the intersection angle of rays connecting the estimated three-dimensional position of the corresponding feature point included in the selected image and the feature point position of each image frame, and (c) the camera model of the reference image When at least one of the fitting error and (a) to (c) is calculated as an evaluation value, and these evaluation values satisfy a specified condition, the selected image is an image suitable for generating three-dimensional data. Since the determination is made and the three-dimensional data generation process such as SFM or EKF SLAM is executed, it is possible to realize generation of highly accurate three-dimensional data.

  Details of an information processing apparatus, an image processing method, and a computer program according to embodiments of the present invention will be described below with reference to the drawings.

  The outline of the present invention will be described with reference to FIG. The information processing apparatus 120 of the present invention inputs, for example, a photographed image of the camera 102 held by the moving user 101, for example, an image constituting a moving image, performs analysis of the input image, and includes various images included in the photographed image. Three-dimensional image information 103 composed of objects is generated.

  Similar to the processing described above with reference to FIG. 1, the information processing apparatus 120 analyzes a sparse three-dimensional map 131 that includes position information of feature points included in an image by analyzing an acquired image, using SLAM (simultaneous). Dense three-dimensional information that is generated by applying processing such as localization and mapping (SFM) and structure from motion (SFM), and further using detailed information such as camera trajectory information and feature point information in the image. A map 132 is generated.

  As described above, SLAM is a process for detecting the position of the feature point in the image input from the camera and the position and orientation of the camera together. SFM is, for example, processing for analyzing correspondence between feature points (Landmarks) included in an image using images taken from a plurality of different positions.

  The sparse three-dimensional map 131 has three-dimensional position information of feature points. The conventional example of this information generation processing sequence has been described with reference to the flowchart of FIG. 2, but in the information processing apparatus 120 of the present invention, the three-dimensional positions of feature points are processed by a process different from the flow shown in FIG. Get the.

  The acquisition sequence of the three-dimensional position of the feature point and the position and orientation information of the camera according to an embodiment of the present invention will be described with reference to the flowchart shown in FIG.

The process of acquiring the feature point 3D position information and the camera position and orientation information is executed in the following processing sequence.
Step S101: Initial information acquisition process (SFM)
Step S102: Camera position and orientation and feature point three-dimensional position information acquisition processing (EKF SLAM) to which the extended Kalman filter is applied
Step S103: Bundle adjustment processing (Bundle Adjustment)

The basic flow of the acquisition process of the three-dimensional position information of the feature point and the position and orientation information of the camera according to the embodiment of the present invention is as follows:
"Initial information acquisition process (SFM)"
→ "Camera position / posture and feature point 3D position information acquisition processing (EKF SLAM) applying extended Kalman filter"
→ "Bundle adjustment process (Bundle Adjustment)"
This is the processing sequence.

  The initial information acquisition process (SFM) in step S101 is performed, for example, by an SFM (Structure from Motion) process that analyzes the correspondence of feature points (Landmarks) included in the image using images taken from a plurality of different positions. However, this processing is executed by applying only the input images of the first few frames input from the camera, for example. Information obtained by this process is used as initialization information for the camera position and orientation and feature point three-dimensional position information acquisition process (EKF SLAM) to which the extended Kalman filter is executed in step S102.

  After initialization information to be applied to the EKF SLAM is obtained, the EKF SLAM is executed on the subsequent input image without executing the SFM, and the feature point position information and the position of the camera that captured each image are obtained. Get posture information.

  A specific processing example will be described with reference to FIG. Assume that the position information of the subject feature points included in the image frame acquired by the camera shown in FIG. In FIG. 6A, image frames acquired by the camera are shown at regular intervals.

  The information processing apparatus according to the present invention uses an image selection processing algorithm described later from the first few frames (input frames up to frame T1 shown in FIG. 6) instead of all the image frames to be analyzed for the three-dimensional position information of feature points. The image frame selected in step S101 is set as a processing target frame of the initial information acquisition process (SFM) in step S101 shown in FIG. By the initial information acquisition process (SFM) for the plurality of selected image frames, the trajectory information of the three-dimensional map and the camera frame is used as initialization information (Initialized Data) of “EKF SLAM” executed in the next step S102. Used.

  The trajectory information of the three-dimensional map and camera frame obtained by a plurality of preceding selected image frames up to the frame T1 becomes initial information. Using this initial information, “EKF SLAM” is initialized, and “EKF SLAM” is initialized. Process. That is, the feature point position information acquired from the preceding image frame is set as the state information for the initial image frame, and the process of applying the extended Kalman filter (EKF) to the subsequent image frame is used to obtain the three-dimensional position information of the feature points in the subsequent frame. get.

  The initial information acquisition process (SFM) in step S101 is an SFM (Structure from Motion) process for analyzing the correspondence between feature points (Landmarks) included in an image using images taken from a plurality of different positions as described above. The processing result is used as initialization information (Initialized Data) of “EKF SLAM” executed in step S102. Therefore, the accuracy of the SFM process greatly affects the accuracy of the finally generated three-dimensional data.

In SFM, for example,
Obtaining a sufficient number of corresponding feature points from multiple captured images;
The corresponding feature points are included in multiple images with different shooting directions,
Accurate feature point position information with few errors can be calculated from the corresponding feature point information,
For example, selection of an image frame to be analyzed is an important factor. An expert of 3D image analysis can intuitively determine an image satisfying such a condition and select an appropriate image. However, a normal user has many image frames taken by himself / herself. It is difficult to select an appropriate image from.

  The information processing apparatus of the present invention automatically executes appropriate image selection in the SFM and evaluates the processing result of the SFM to generate appropriate information as initialization information of “EKF SLAM”. By executing EKF SLAM, it is possible to generate highly accurate three-dimensional data. The details of the image selection process and the evaluation process will be described with reference to FIG.

  FIG. 7 shows a part of the components of the information processing apparatus of the present invention, that is, a configuration for performing an image frame selection process and an image evaluation process for image selection that are optimal for application to the generation of three-dimensional information such as SFM. Is shown. As illustrated in FIG. 7, the information processing apparatus includes an input image selection unit 301 that inputs video data 313 captured by the camera 312, an environment recognition unit 302, a reference image selection unit 304, a feature point tracking unit 303, an evaluation value, and the like. A calculation unit 305 and an evaluation value determination unit 306 are included.

Note that the configuration of the present invention for executing the processing shown in FIG. 7 has the following restrictions.
The camera 312 that captures the video data (image frame) 313 can be approximated to a pinhole camera model. The internal parameters of the camera 312 are known. The camera internal parameters do not change while the video is being captured. The video data 313 recorded in 312 is prepared in advance (offline processing is performed on the captured video data 313 instead of real-time processing at the time of shooting processing).
The time interval between frames of the video data 313 is the same.
The video data 313 is continuous in time (does not move to another scene suddenly)
Also, in this example, the number of video frames that can be handled by calculation is N (for example, N = 5, N needs to satisfy N> 1) due to memory and calculation amount constraints.

  The still environment 311 means that the moving object is not included in the image captured by the camera. However, even if the moving object is included, the calculation of the three-dimensional position for other stationary objects is not performed. It is possible and it is not essential that all objects photographed by the camera 312 are stationary. It is only necessary that the calculation target of the three-dimensional position is stationary. The camera 312 moves to shoot images and shoot images from various angles.

  Further, in the embodiment described below, it is assumed that camera photographing data is accumulated in the storage unit and offline processing for processing the image accumulated in the storage unit is premised. However, a predetermined processing capability such as a processor having a high processing speed is provided. In addition, an online processing configuration in which image selection and three-dimensional data generation are performed in parallel with camera shooting is also possible.

Hereinafter, with reference to FIG. 7 and the following, details of the image selection processing that enables generation of highly accurate three-dimensional data will be described. With the configuration shown in FIG. 7, an image selection process and an image evaluation process applied to the generation of three-dimensional data such as SFM are executed. This processing sequence will be described with reference to the flowchart shown in FIG. First, in step S301, the input image selection unit 301 selects an input frame from the video data 313. As an example of the selection method, variable [T] and variable [F] are used in this example.
Variable [T] and variable [F] are the following variables.
T: Variable corresponding to the selected frame interval,
F: Variable corresponding to the first selected frame,

  As shown in FIG. 9, N frames are selected and adopted at intervals of T frames from the Fth frame counting from the first frame in the future direction. As described above, N may be determined according to memory / computation constraints or the like, and N> 1 may be satisfied. For example, N = 5.

  As shown in FIG. 9, the selected frame is each frame of [F, F + T, F + 2T, F + 3T,..., F + (N−1) × T]. Note that the variable [T] corresponding to the selected frame interval has an initial value of 1, and the variable [F] corresponding to the first selected frame has an initial value of 0 (the number of the first frame, 1 for the first frame). Set and perform image evaluation in that setting, calculate an evaluation value to determine whether it is appropriate for generating three-dimensional data, and if the evaluation value is less than the regulation, change the variable, A new set of image frames will be set and evaluated again. The variable T and the variable F may be set or updated sequentially by the user, or may be configured to be automatically updated by determining an update mode according to the evaluation result. This process will be described in detail later.

  In step S302 of the flow shown in FIG. 8, the environment recognition unit 302 selects the image selected by the input image selection unit 301, for example, the frames [F, F + T, F + 2T, F + 3T,. −1) × T] is input, and the structure of the still environment 311 and the position and orientation of each video frame of the camera 312 are calculated. The process executed by the environment recognition unit 302 can be executed as a conventionally known process. For example, reference [1] [Yi Ma, Stefano Soatto, Jana Kosecha and S. Shankar Sastry, “An Invitation to 3-D Vision”, Springer (2006). The method described in Chapter 11 can be applied.

The result generated by the processing of the environment recognition unit 302 in step S302 is temporarily held in the storage unit in the information processing apparatus as the environment recognition result 314. The environment recognition result 314 holds the following information. However, the feature amount is not particularly specified as long as the feature amount can be compared (tracked) between feature points. It is desirable that the feature quantity is robust to scale and rotation.
-Camera position and posture for N frames-3D position of each feature point-Feature amount of each feature point (used for feature point tracking)
・ Image position (trajectory) for N frames for each feature point

  Next, in step S303, feature point tracking processing by the feature point tracking unit 303 is executed. In step S301, the image selected by the input image selection unit 301, that is, for example, each frame of frames [F, F + T, F + 2T, F + 3T,..., F + (N−1) × T] shown in FIG. The trajectory of the feature point is obtained by tracking the common feature point. This process is a process of tracking corresponding feature points in the environment recognition result, and a known process, specifically, the method described in Chapter 11 of the above-mentioned document [1] can be used.

  Next, in step S304, the reference image selection unit 304 selects a reference image to be used for verifying the environment recognition result, separately from the image used in environment recognition. The reference image is a process of selecting one image frame constituting the video data 313. As a selection method, for example, simply [N × of the final processing frame [F + (N−1) × T frame] is used. T] Processing such as selecting an image before the frame is possible. However, this method is not suitable as a reference image when encountered when the camera has not moved for a long time. There is a risk of selecting an image having no change in the shooting direction of the feature point position as the reference image.

  Therefore, in this processing example, using the feature point tracking result obtained as the processing result of the feature point tracking unit 303 in step S303, the position of the feature point of the reference image in the image is the feature point used by the environment recognition unit 302. It is verified how far away from the trajectory, and a frame that is sufficiently far away is selected as a reference image.

  The detailed sequence of the reference image selection process executed in step S304 will be described with reference to the flowchart shown in FIG. First, in step S351, one arbitrary frame is selected as a reference image candidate from the video frame 313 to be processed. In step S352, the feature point tracking unit 303 tracks the corresponding feature point in the environment recognition result for one frame as the reference image candidate selected in step S351. As described above, a known process is applied to the feature point tracking process. Specifically, it is possible to use the method described in Chapter 11 of the aforementioned document [1].

  Next, in step S353, the distance between the feature point position group tracked in step S352 and the feature point position group calculated in the environment recognition result 314 is calculated. As the calculation formula, the following formula (Formula 1) is applied.

However, in the above formula,
dist is the distance between the feature point position group tracked by the feature point tracking unit 303 and the feature point position group calculated by the environment recognition unit 302;
i is a feature point number,
t is the frame number,

Is a function that outputs the median for i,

Is a function that outputs the minimum value for t,
_mit is the position of the i-th feature point used in the environment recognition unit 302 in the frame t,
m_ref _i indicates a position in the reference image corresponding to the i-th feature point.

  FIG. 11 shows a processing example when the distance is calculated by applying the above formula (Formula 1) to one reference image candidate 321 and the image frame 322 selected by the input image selection unit 301. . Corresponding feature points 331, 341, and 342 exist in one reference image candidate 321 and the image frame 322 selected by the input image selection unit 301, respectively, and the locus of the feature points of the image frame 322 selected by the input image selection unit 301 11 is obtained as a feature point locus 351 as shown in the lower part of FIG. 11, and the feature point distance 353 from the position of the feature point 352 on the reference image is calculated using this locus information.

  Next, in step S354, the distance between the feature points of the feature point group of the selected frame calculated using the above formula (Formula 1) in step S353 and the feature point of the reference image candidate is equal to or greater than a predetermined threshold value, Further, it is determined whether or not the number of feature points is equal to or greater than a predetermined threshold value. If it is equal to or greater than the threshold value, it is determined that the feature image is suitable as a reference image, and the reference image selection process is terminated. Returns to step S351, selects another image as a reference image candidate, and executes the processing from step S352 onward. In this way, the reference image selection process is executed in step S304 of the flow shown in FIG.

  Next, in step S305 of the flow shown in FIG. 8, the evaluation value calculation unit 305 evaluates the image frame selected in step S301. That is, it is evaluated whether or not the selected frame is appropriate for generating a three-dimensional image. Specifically, for example, it is appropriate as an image frame to be applied to the SFM (Structure from Motion) process executed as the process of step S101 shown in the flow of FIG. 5, and further, the SFM process result using the image frame is obtained. Evaluation for determining whether or not it is appropriate as initial information applied to the camera position and orientation and feature point three-dimensional position information acquisition processing (EKF SLAM) to which the extended Kalman filter executed as the processing in step S102 of FIG. 5 is applied Calculate the value.

In this example,
(A) Number of feature points used in environment recognition unit 302 (b) Median value of crossing angles of light rays (see FIG. 12) (c) Camera model fitting error of reference image These three kinds of values are used as evaluation values Use.
Hereinafter, each of the values (a) to (c) will be described.

(A) Number of feature points used in environment recognition unit The number of feature points used in environment recognition unit 302 is the number of feature points whose 3D position is estimated in environment recognition unit 302.
In order to generate highly accurate three-dimensional data, the greater the number of feature points, the higher the accuracy of the three-dimensional data. Therefore, the greater the number of feature points, the higher the evaluation value.

(B) Median value of crossing angle between light rays Next, a median value of crossing angle between light rays will be described. First, the definition of “intersection angle between rays” will be described with reference to FIG. Assume that there are camera positions / postures of two image frames 371 and 372 and one feature point 381 that can be seen from the two frames 371 and 372 as shown in FIG. As shown in FIG. 12, the “intersection angle between the light rays” is an angle φ at which the light rays connecting the feature points 381 from the two image frames 371 and 372 photographed at the two camera positions intersect.

The larger the intersection angle [φ], the greater the parallax between frames. In order to calculate the three-dimensional position of the feature point with high accuracy, it is an important factor that a feature point corresponding to a frame having a larger parallax exists.
As a formula for obtaining the crossing angle [φ], the following formula (Formula 2) is used.

However, in the above formula,
w is the position of the feature point,
c _p is the camera position of the p-th frame (the focus position),
‖ ノ ル is the norm,
Is the inner product of vectors,
Represents
The function acos is an inverse cosine function.

Next, using the intersection angle [φ] calculated by the above equation (Equation 2), “the median value of the intersection angles of the light beams” is obtained. First, [ _n C ₂ ] pairs are prepared in which N frames to 2 frames form a pair. For example, it is assumed that there are a p frame and a q frame, and the i-th feature point is captured in each frame. For each set, the “median value of the intersection angle between rays” of the i-th feature point is obtained. Then, the above processing is performed on all feature points, and the median value of “intersection angle between rays” is set as “median value of intersection angle between rays”.

  When expressed as an equation, the median [angle] of the crossing angle between the rays is calculated by the following equation (Equation 3).

However, in the above formula,
Is not adopted if there is no i-th feature point in at least one frame of a frame pair.
As described above, in order to calculate the three-dimensional position of the feature point with high accuracy, it is an important element that there is a feature point corresponding to a frame having a larger parallax. Therefore, the evaluation value increases as the median [angle] of the intersection angle between the light beams increases.

(C) Camera Model Fitting Error of Reference Image Next, the camera model fitting error of the reference image will be described.
Both the three-dimensional position of the feature point obtained by the processing by the environment recognition unit 302 and the in-image feature point position obtained from the reference image (the three-dimensional position of the feature point / the in-image feature point position in the reference image) are correct. Then, according to the pinhole camera model, as shown in FIG. 13, all the light rays passing through the reference image frame 390 (the line connecting the feature point positions on the reference image frame 390 and the feature points 391a to 391f) intersect at one point. To do. This intersection is the focal point 392 of the reference image frame 390.

  The pinhole camera model will be described with reference to FIGS. In the pinhole camera model, the position of the feature point in the image frame can be calculated by the following formula (Formula 4).

  The meaning of the above formula will be described with reference to FIGS. The above formula is obtained by calculating the pixel position 412 on the camera image plane of the point (m) of the object 411 included in the captured image 410 of the camera, that is, the position expressed by the camera coordinate system, and the three-dimensional object 400 in the world coordinate system. It is an expression showing the correspondence with the position (M) 401.

  A pixel position 412 on the camera image plane is expressed by a camera coordinate system. The camera coordinate system is a coordinate system in which the focal point of the camera is the origin C, the image plane is a two-dimensional plane of Xc and Yc, and the depth is Zc, and the origin C is moved by the movement of the camera.

  On the other hand, the three-dimensional position (M) 401 of the object 400 is indicated by a world coordinate system composed of three XYZ axes having an origin O that does not move by the movement of the camera. An expression indicating the correspondence between the positions of the objects in the different coordinate systems is defined as the above-described pinhole camera model.

The values included in this equation are as shown in FIG.
λ: normalization parameter A: camera internal parameter,
Cw: camera position,
Rw: camera rotation matrix,
Means.
further,
Is a position on the image plane of the camera expressed in a homogeneous coordinate system.
λ is a normalization parameter,
Is a value for satisfying the third term.

The camera internal parameter A includes the following values.
f: Focal length θ: Image axis orthogonality (ideal value is 90 °)
ku: Scale of the vertical axis (conversion from a 3D position scale to a 2D image scale)
kv: horizontal scale (conversion from 3D position scale to 2D image scale)
(U0, v0): Image center position

  Thus, the feature point in the world coordinate system is expressed by the position [M]. The camera is represented by a position [Cw] and a posture (rotation matrix) Rw. The focal position of the camera, the image center, etc. are expressed by camera internal parameters [A]. From these parameters, the relational expression of the respective positions projected from the “feature point in the world coordinate system” onto the “camera image plane” can be expressed by the above-described expression (Expression 4).

  Further, when the position / orientation of the camera is expressed by a rotation matrix Rw and a translation vector, the following equation (Equation 5) is obtained. However, the relationship between the translation vector and the camera position is expressed by Expression (Formula 6).

A specific example of the calculation process of the “camera model fitting error of the reference image” in one embodiment of the present invention will be described with reference to FIG.
The calculation process of “camera model fitting error of reference image” is performed by the following two processes.
(Process 1) Processing for obtaining the camera position / posture of the reference image based on the pinhole camera model,
(Process 2) A process of re-projecting each feature point on the image plane based on the obtained camera position and orientation, and calculating an error between the re-projected point and the tracking point (observation point);

First, referring to FIG.
(Process 1) Processing for obtaining the camera position / posture of the reference image based on the pinhole camera model,
Will be described. FIG. 16A is a diagram similar to FIG. 13 described above. The position / orientation of the camera that captured the reference image frame 390 is obtained.
A formula for obtaining the position / posture of the camera is the following formula (Formula 7).

However,
w _i is the three-dimensional position of the i-th feature point,
x _i is a value calculated by the following formula (Formula 8). Here, _mi is the position of the feature point in the reference image, and the matrix A is an internal parameter described in the above-described equation (Equation 4).

Also,
I ₃ is a 3 × 3 identity matrix,
0 is a zero vector of 3 rows and 1 column,
Is an n-by-1 zero vector (depending on the size of the left-hand side matrix),
r ₁ , r ₂ , r ₃ are column vectors R of the rotation matrix shown in the following equation representing the orientation of the reference image frame,

t is the translation vector of the reference image frame.
A, R, and t are values corresponding to the values A, R, and t in the “pinhole camera model” described above with reference to FIGS. 14 and 15.

There are various ways of obtaining the above-described mathematical formula (Formula 7) for obtaining the position and orientation of the camera. As an example of the solution, the mathematical formula (Formula 7) is

When expressed as the above formula,
A solution can be obtained by setting that the eigenvector corresponding to the minimum eigenvalue of the matrix B ^T · B is q.
However, r ₁ , r ₂ , and r ₃ are column vectors of the rotation matrix and need to be corrected after calculating the scale of each component of q so that the norm is 1. Through the above processing, the camera position / posture of the reference image can be obtained by applying the above formula (Formula 7).

Next, referring to FIG. 16B and FIG.
(Process 2) A process of re-projecting each feature point on the image plane based on the obtained camera position and orientation, and calculating an error between the re-projected point and the tracking point (observation point);
This process will be described.

First, each feature point is re-projected on the image plane 500 based on the estimated camera position and orientation by (Process 1) described above. A reprojection point 501 is set as shown in FIGS. Further, the feature point three-dimensional position tracking point (observation point) 502 of the feature point acquired by the processing of the feature point tracking unit 303 is set on the image plane 500. The distance between these two points is calculated as a reprojection error [ε _i ]. i is an identification number of a feature point, and reprojection error [ε _i ] for each feature point is calculated.

In order to measure the error of the camera position / posture calculated by applying the above formula (formula 7), a reprojection error [ε _i ] is calculated. The reprojection error [ε _i ] is calculated by applying the following equations (Equation 9) and (Equation 10).

However,
_mi is the position in the image of the feature point obtained by tracking,

In the formula (Formula 9),
Is a reprojection point and is obtained using the above equation (Equation 10).

  The above equation (Equation 10) is an equation based on the pinhole camera model described above with reference to FIGS. 14 and 15, and parameters in the equation are described with reference to FIGS. 14 and 15. It corresponds to the parameter.

The median value of a plurality of reprojection errors [ε _i ] corresponding to the feature points (i = 1, 2,...) Obtained by the above equation (Equation 9) is defined as “camera model fitting error of reference image”. .
When the camera model fitting error of the reference image is expressed as [reproj_error], it is expressed by the following expression (Expression 11).

  The reason why the median is used as the camera model fitting error of the reference image is to eliminate a free value due to a feature point tracking error, that is, an error measurement value.

  The smaller the “camera model fitting error of the reference image” indicated by the above equation (Equation 11), the higher the evaluation value.

As described above, the evaluation value calculation unit 305 in step S305 of the flowchart shown in FIG.
(A) Number of feature points used in environment recognition unit 302 (b) Median value of crossing angles of light rays (see FIG. 12) (c) Camera model fitting error of reference image These three kinds of values are used as evaluation values calculate.

  That is, is the image frame selected in step S301 appropriate for generating three-dimensional data? Specifically, for example, it is appropriate as an image frame to be applied to SFM (Structure from Motion) processing executed as the processing in step S101 in FIG. 5, and the SFM processing result using the image frame is shown in FIG. As an evaluation value for determining whether or not it is appropriate as initial information applied to the camera position and orientation and feature point three-dimensional position information acquisition process (EKF SLAM) to which the extended Kalman filter is executed as the process of S102, Three types of evaluation values a) to (c) are calculated.

In addition, each evaluation value of said (a)-(c) becomes the following settings.
(A) The larger the number of feature points used in the environment recognition unit 302, the higher the evaluation value,
(B) The higher the median of the crossing angle between rays (see FIG. 12), the higher the evaluation value,
(C) The smaller the camera model fitting error of the reference image, the higher the evaluation value,
This is the setting.

  Next, in step S306 in the flowchart shown in FIG. 8, the image frame selected in step S301 by the evaluation value determination unit 306 based on the above three types of evaluation values is converted into a three-dimensional data generation process, for example, SFM ( It is determined whether or not it is appropriate as an image frame to be applied to Structure from Motion) processing or EKF SLAM processing.

The evaluation value determination process in step S306 is executed as follows, for example.
(A) Appropriate if the number of feature points used in the environment recognition unit 302 is equal to or greater than a preset threshold [Tha], and inappropriate if less than the threshold [Tha],
(B) Appropriate if the median of the angle of intersection between rays (see FIG. 12) is greater than or equal to a preset threshold [Thb], inappropriate if less than the threshold [Thb],
(C) Appropriate if the camera model fitting error of the reference image is less than or equal to a preset threshold [Thc], inappropriate if greater than the threshold [Thc],
Such a determination is performed. Note that the threshold depends on the angle of view of the camera, the number of feature points in the environment, the required accuracy, and the like, and may be changed as appropriate according to user settings, for example.

  If all of the above (a) to (c) are appropriate, it is determined that the selected image frame has an appropriate evaluation value, and the image frame selected in step S301 is applied to the generation of three-dimensional data. Specifically, for example, it is determined that the image frame is suitable for the SFM (Structure from Motion) process executed as the process of step S101 in FIG. 5 and the subsequent EKF SLAM process, and the process ends.

  If any of the above (a) to (c) is determined to be inappropriate, the image frame selected in step S301 generates three-dimensional data, specifically, SFM (Structure from Motion) processing or EKF SLAM. It is determined that the image frame is not appropriate as the image frame to be applied to the process, and the process returns to step S301. Further, a different image frame selection process is executed, and the process after step S302 is executed.

  In the above processing example, if it is determined in step S306 in the flowchart shown in FIG. 8 that all the evaluation values (a) to (c) described above are appropriate based on the determination based on the specified threshold value, the processing in step S306 is performed. The determination is determined as Yes, and in this case, the image frame selected in step S301 has been described as the image frame to be applied to the three-dimensional data generation process. For example, at least 2 in (a) to (c) If one or one is determined to be appropriate, the processing configuration is such that the determination in step S306 is determined as Yes and the image frame selected in step S301 is a frame that is applied to the generation of three-dimensional data. It is good also as a setting. That is, the criterion based on the threshold value and the evaluation value is an element that can be changed depending on the situation.

The information processing apparatus of the present invention has the configuration shown in FIG. 7, and generates three-dimensional data using an image frame selected according to the processing described with reference to the flowchart shown in FIG.
Specifically, for example, SFM (Structure from Motion) processing in step S101 shown in the flowchart of FIG. 5 is executed, and the processing result is used as initialization information (Initialized Data) of “EKF SLAM” in step S102.

The image selected according to the flow shown in FIG.
(A) The number of feature points is equal to or greater than a preset threshold [Tha],
(B) The median of the crossing angle between the rays (see FIG. 12) is equal to or greater than a preset threshold [Thb],
(C) The camera model fitting error of the reference image is equal to or less than a preset threshold [Thc],
It will be composed of image frames that satisfy these conditions, and SFM processing using these image frames enables highly accurate acquisition of three-dimensional position information of feature points, and the subsequent “EKF SLAM” It is possible to generate highly accurate three-dimensional data.

  That is, the three-dimensional data generation unit of the information processing apparatus executes SFM (Structure from Motion) processing to which the selected image determined to be an image suitable for generation of three-dimensional data by the evaluation value determination unit, and further performs SFM processing. The three-dimensional data generation is executed by a process to which an extended Kalman filter (EKF) using the generated information as the initial information is applied.

In the flow shown in FIG. 8, if the determination in step S306 is No, the process returns to step S301, and a new image frame is selected. In this image selection process, FIG. As described with reference to the above, settings for selecting and adopting N frames at T frame intervals from the Fth frame counting from the first frame in the future direction [F, F + T, F + 2T, F + 3T,. (N-1) × T]
Initial value of variable [T] corresponding to the selected frame interval = 1,
Initial value of variable [F] corresponding to the first selected frame = 0,
This initial value is first selected, and when a satisfactory evaluation value is not obtained and the process returns to step S306, the variable is changed and a new set of image frames is selected.

  In this case, the process of changing the variable [T] to 1, 2,... And the process of changing the variable [F] to 0, 1, 2,. It is possible to change the optimum variables [T] and [F] according to the value status. The variable change processing sequence will be described with reference to the flowchart shown in FIG.

The flowchart shown in FIG.
(1) Evaluation value determination process (2) Image selection variable setting process The process is constituted by these two processing parts. (1) In the evaluation value determination process, the evaluation value determination step in step S306 in the flowchart of FIG.
(A) Number of feature points used in environment recognition unit 302 (b) Median value of intersection angles of light rays (see FIG. 12) (c) Camera model fitting error of reference image Determination processing for each of these evaluation values It corresponds to the flow described separately. (2) The image selection variable setting process is performed according to the processing result of the (1) evaluation value determination process.
Variable [T] corresponding to the selected frame interval,
Variable [F] corresponding to the first selected frame,
This is a process for determining how to change these variables. The (2) image selection variable setting process shown in FIG. 18 is executed by the input image selection unit 301 having the configuration shown in FIG.

Each step of the flow shown in FIG. 18 will be described. Steps S501 to S503 correspond to the evaluation value determination process in step S306 in the flowchart of FIG. 8 described above, respectively.
(A) The number of feature points is equal to or greater than a preset threshold [Tha],
(B) The median of the crossing angle between the rays (see FIG. 12) is equal to or greater than a preset threshold [Thb],
(C) The camera model fitting error of the reference image is equal to or less than a preset threshold [Thc],
This is the step of making these determinations. If all the determinations are Yes, the process proceeds to step S504, and the set of image frames selected at that time, that is, the set of image frames selected in step S301 shown in FIG. 8, is shown in step S101 shown in the flowchart of FIG. It is selected as a frame to be applied to SFM (Structure from Motion) processing.

If any of the determinations in steps S501 to S503 is No, the process proceeds to (2) image selection variable setting process in FIG. 18, and the variables described above with reference to FIG.
Variable [T] corresponding to the selected frame interval,
Variable [F] corresponding to the first selected frame,
Processing for newly setting these variables is performed.

In step S501,
If it is determined that the number of feature points is not equal to or greater than the preset threshold value [Tha], the process proceeds to step S511, and it is determined whether or not the setting value of the variable [T] corresponding to the selected frame interval is a setting greater than 1. To do. If the selected frame interval [T] = 1, the process proceeds to the route (1) as shown in FIG. 18, and in step S514, the variable update is performed with the variable [F] corresponding to the first selected frame as [F + 1]. Do.

Hereinafter, the variable update process of each of the routes (1) to (5) illustrated in FIG. 18 will be described.
Route (1)
This is a state where the number of feature points is small even though the frame interval [T] is a minimum value of [1].
This state is
"I'm shooting an environment with no features",
“Camera movement is too fast and there is no overlap area between frames”,
These states are considered.
In this case, in step S514, a variable update is performed in which the variable [F] corresponding to the first selected frame is set to [F + 1], the process proceeds to step S517, and the next frame [F + 1] of the current selected frame [F] is executed. ] To select an image frame. In this case, the frame interval [T] is not updated.

  If it is determined in step S511 that the set value of the variable [T] corresponding to the selected frame interval is greater than 1, that is, if the selected frame interval [T]> 1, the process proceeds to step S512 as shown in FIG. Then, the selected frame interval [T] is set to [T-1], and in step S513, the same determination process as in step S502, that is, (b) the median of the crossing angles of the rays (see FIG. 12) is previously set. It is determined whether or not the set threshold value [Thb] is exceeded. This determination process is executed by the evaluation value determination unit 306.

  If it is determined in step S513 that (b) the median of the crossing angle between the rays (see FIG. 12) is not equal to or greater than the preset threshold [Thb], the process proceeds to route (2) as shown in FIG. In S514, the variable [F] corresponding to the first selected frame is updated to [F + 1], the process proceeds to step S517, and a new image is selected according to the updated variables [F] and [T].

The process of route (2) will be described.
In the case of this route (2), the number of feature points is small and the crossing angle of light rays is small.
This state is
"I'm shooting an environment without features"
This state is assumed.
In this case, the frame interval [T] is decremented by one [T-1], and an image frame is selected from the frame [F + 1] next to the currently selected frame [F].

  On the other hand, if it is determined in step S513 that (b) the median of the crossing angle between the rays (see FIG. 12) is equal to or greater than a preset threshold [Thb], the route (3) is routed as shown in FIG. The process proceeds directly to step S517 to select a new image according to the updated variable [T].

The process of route (3) will be described.
In the case of this route (3), the number of feature points is small and the crossing angle of light rays is large.
This state is
“Camera movement is too fast for the sampling frame [T] interval and there are few overlapping areas between frames”
It is assumed that this is the case.
In this case, an image frame is selected by decrementing the frame interval [T] by one [T-1]. The variable [F] corresponding to the first selected frame is not changed.

  Route (4) shown in FIG. 18 is a case where the determination in step S501 is Yes and the determination in step S502 is No. In this case, the process proceeds to step S515, the frame interval [T] is incremented by one [T + 1], the process proceeds to step S517, and a new image is selected. The variable [F] is not changed.

The process of route (4) will be described.
In the case of this route (4), the number of feature points is sufficient, but the crossing angle of light rays is small.
This state is
“Camera movement is slow relative to the sampling frame [T] interval”
It is assumed that this is the case.
In this case, the frame interval [T] is incremented by one [T + 1], and a new image frame is selected without changing the variable [F].

  The route (5) illustrated in FIG. 18 is a case where the determination in step S501 is Yes, the determination in step S502 is Yes, and the determination in step S503 is No. In this case, the process proceeds to step S516, and the variable [F] corresponding to the first selected frame is incremented by one [F + 1], and the process proceeds to step S517 to select a new image. The variable [T] is not changed.

The process of route (5) will be described.
In the case of this route (5), the number of feature points is sufficient and the crossing angle of rays is sufficiently large, but the pinhole camera error is large.
This state is
3D position estimation fell into a singular solution,
Tracking failure These are possible.
In this case, the frame interval [T] is not changed, and an image frame is selected from the frame [F + 1] next to the currently selected frame [F].

In the case of route (6) shown in FIG.
(A) The number of feature points is equal to or greater than a preset threshold [Tha],
(B) The median of the crossing angle between the rays (see FIG. 12) is equal to or greater than a preset threshold [Thb],
(C) The camera model fitting error of the reference image is equal to or less than a preset threshold [Thc],
This is a case where all these conditions are satisfied, and since all the conditions are satisfied, the image frame selected at this point may be determined as the processing frame.

According to the flow shown in FIG.
Variable [T] corresponding to the selected frame interval,
Variable [F] corresponding to the first selected frame,
By updating these variables and executing image selection, it is possible to efficiently select an image frame having a high evaluation value.

  As described above with reference to FIG. 7, the information processing apparatus according to the present invention prepares in advance video data 313 in which the static environment 311 is recorded by the camera 312, and performs offline processing in SFM or EKF SLAM. Although it has been described that the selection process of the image frame to be applied is executed and SFM or EKF SLAM is executed, if the processing capability of the information processing apparatus is sufficiently high and the apparatus can execute the processing at high speed, it is not offline, It may be configured to perform online processing for executing image selection, SFM, and EKF SLAM following the shooting processing.

  For example, among the image frames taken by the camera, only the first few frames are stored as video data in the storage unit, and the image selection process shown in FIG. 8 is executed using the image data stored in the storage unit, Furthermore, SFM and EKF SLAM are executed at a high speed (for example, a speed faster than 1/30 seconds) using a selected image having a high evaluation value. With such a processing configuration, it is possible to perform processing following the shooting speed of the camera.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present invention. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims should be taken into consideration.

  The series of processing described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run. For example, the program can be recorded in advance on a recording medium. In addition to being installed on a computer from a recording medium, the program can be received via a network such as a LAN (Local Area Network) or the Internet, and installed on a recording medium such as a built-in hard disk.

  Note that the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

  As described above, according to the configuration of the embodiment of the present invention, in order to determine whether the selected image selected from the input image is an image suitable for generating three-dimensional data, (a) The estimated number of feature points of the three-dimensional position included in the selected image, (b) the intersection angle of rays connecting the estimated three-dimensional position of the corresponding feature point included in the selected image and the feature point position of each image frame; c) Camera model fitting error of reference image, and at least one of these (a) to (c) is calculated as an evaluation value, and when these evaluation values satisfy a prescribed condition, the selected image generates three-dimensional data. Since it is determined that the image is suitable for the three-dimensional data generation processing such as SFM or EKF SLAM, it is possible to realize generation of highly accurate three-dimensional data.

It is a figure which shows the flowchart explaining an example of the production | generation process sequence of a three-dimensional map (3D map). It is a figure which shows the flowchart explaining an example of the process sequence which acquires the feature point information and camera locus information which are contained in an image frame. It is a figure explaining an example of a bundle adjustment process (Bundle Adjustment). It is a figure explaining the outline | summary of the process of this invention. It is a figure which shows the flowchart explaining the sequence of the process according to one Example of this invention. It is a figure explaining the sequence of the three-dimensional information generation process according to one Example of this invention. It is a figure explaining the structure and image selection process for performing the image selection of the information processing apparatus according to one Example of this invention. It is a figure which shows the flowchart explaining the sequence of the image selection process according to one Example of this invention. It is a figure explaining the aspect used for the image selection process according to one Example of this invention, and the variable used for an image selection process. It is a figure which shows the flowchart explaining the detailed sequence of the reference image selection process performed in step S304 in the flow shown in FIG. It is a figure explaining the distance between the tracking feature point position group using one reference image candidate, the image frame which the input image selection part selected, and the feature point position group calculated as an environment recognition result. It is a figure explaining the crossing angle of light rays. It is a figure explaining that all the light rays (a feature point position on a reference image frame and a line connecting feature points) passing through a reference image frame intersect at one point. It is a figure explaining the meaning of the formula which shows the correspondence of the pinhole camera model, ie, the position expressed by the camera coordinate system, and the three-dimensional position of the object in the world coordinate system. It is a figure explaining the meaning of the formula which shows the correspondence of the pinhole camera model, ie, the position expressed by the camera coordinate system, and the three-dimensional position of the object in the world coordinate system. It is a figure explaining the calculation process specific example of the "camera model fitting error of a reference image" in one Example of this invention. It is a figure explaining the calculation process specific example of the "camera model fitting error of a reference image" in one Example of this invention. It is a figure which shows the flowchart explaining the variable update process sequence in the image selection which the information processing apparatus which concerns on one Example of this invention performs.

Explanation of symbols

11-13 Photographed image 21-23 Feature point 101 User 102 Camera 103 3D image information 120 Information processing device 131 Sparse 3D map 132 Dense 3D map 301 Input image selection unit 302 Environment recognition unit 303 Feature point tracking unit 304 Reference image selection unit 305 Evaluation value calculation unit 306 Evaluation value determination unit 311 Still environment 312 Camera 313 Video data 314 Environment recognition result 321 Reference image candidate 322 Image frame 331, 341, 342 Feature point 351 Feature point trajectory 352 Feature point 353 Distance 371 , 372 Image frame 381 Feature point 390 Reference image frame 391 Feature point 392 Focus 500 Image plane 501 Reprojection point 502 Three-dimensional position tracking point (observation point)

Claims (15)

An information processing apparatus that calculates a three-dimensional position of a pixel included in an image,
An image selection unit for selecting a plurality of images as input images from an input image;
An evaluation value calculation unit for calculating an evaluation value for determining whether or not the selected image is an image suitable for generating three-dimensional data;
Based on the evaluation value calculated by the evaluation value calculation unit, an evaluation value determination unit that determines whether or not the selected image is an image suitable for generating three-dimensional data ;
An environment recognition unit that generates an environment recognition result including the three-dimensional position of the feature point included in the image and the position and orientation information of the camera by analyzing the selected image;
A reference image selection unit that selects a reference image for verifying the environment recognition result, wherein a distance between a reference image candidate and a corresponding feature point included in the selected image in one image coordinate system is equal to or greater than a predetermined threshold value A reference image selection unit that selects, as a reference image, an image that satisfies the condition of
The evaluation value calculation unit
Applying the information included in the environment recognition result and the reference image selected by the reference image selection unit,
(A1) Estimated number of feature points of the three-dimensional position included in the selected image (b1) Intersection of rays connecting the estimated three-dimensional position of the corresponding feature point included in the selected image and the feature point position of each image frame Horn,
(C1) Camera model fitting error of reference image At least one of the above (a1) to (c1) is calculated as an evaluation value,
The evaluation value determination unit
Comparing at least one of the above values (a1) to (c1) with a preset threshold value, it is configured to determine whether or not the selected image is an image suitable for generating three-dimensional data,
(A2) the estimated number of feature points of the three-dimensional position included in the selected image is equal to or greater than a specified threshold (Tha);
(B2) the intersection angle between rays connecting the estimated three-dimensional position of the corresponding feature point included in the selected image and the feature point position of each image frame is equal to or greater than a specified threshold (Thb);
(C2) The camera model fitting error of the reference image is not more than a specified threshold value (Thc),
An information processing apparatus that determines that the selected image is an image suitable for generating three-dimensional data when at least one of the conditions (a2) to (c2) is satisfied. The evaluation value calculation unit
Calculate all of (a1) to (c1) as evaluation values,
The evaluation value determination unit
2. The information processing apparatus according to claim 1, wherein when all of the conditions (a2) to (c2) are satisfied, the selected image is determined to be an image suitable for generating three-dimensional data. . The evaluation value calculation unit
Among the evaluation values, for the intersection angle between the light rays, it is a configuration for calculating a median value of a plurality of intersection angles calculated by applying a plurality of corresponding feature points and image frames,
The evaluation value determination unit
3. The method according to claim 1, wherein the selected image is determined to be an image suitable for generating three-dimensional data when the median satisfies a threshold value (Thb) or more. Information processing device. The information processing apparatus includes:
A feature point tracking unit for obtaining a feature point locus included in the image by analysis of the selected image;
The information processing apparatus according to claim 1, wherein the evaluation value calculation unit calculates the evaluation value by applying the feature point trajectory. The input image selection unit
The variable [T] corresponding to the selection frame interval and the variable [F] corresponding to the first selection frame are set and updated to perform different image selection processing.
When the evaluation value determination unit determines that the selected image is not an image suitable for generating three-dimensional data, the evaluation value determination unit updates at least one of the variables [T] and [F] to newly The information processing apparatus according to claim 1, wherein the image selection is performed. The input image selection unit
According to the evaluation mode of the selected image in the evaluation value determination unit, according to a predetermined variable update algorithm, at least one of the variables [T] and [F] is updated to perform a new image selection. The information processing apparatus according to claim 5 , wherein: The information processing apparatus further includes:
The SFM (Structure from Motion) process using the selected image determined by the evaluation value determination unit as an image suitable for generating three-dimensional data, and an extended Kalman filter (EKF) using the generation information generated by the SFM process as initial information 7. The information processing apparatus according to claim 1 , wherein the information processing apparatus has a configuration that executes three-dimensional data generation by a process to which () is applied. In the information processing apparatus, an image processing method for performing an image selection process to be applied to calculate a three-dimensional position of a pixel included in an image,
An image selection step in which an image selection unit selects a plurality of images as selected images from the input image;
An evaluation value calculating step for calculating an evaluation value for determining whether the selected image is an image suitable for generating the three-dimensional data;
An evaluation value determining unit that determines whether or not the selected image is an image suitable for generating three-dimensional data based on the evaluation value calculated in the evaluation value calculating step ;
An environment recognition step in which an environment recognition unit generates an environment recognition result including the three-dimensional position of the feature point included in the image and the position and orientation information of the camera by analyzing the selected image;
A reference image selection unit is a reference image selection unit that selects a reference image for verifying the environment recognition result, and within a one-image coordinate system of reference image candidates and corresponding feature points that are commonly included in the selected image A reference image selection step of selecting, as a reference image, an image that satisfies a condition that the distance is equal to or greater than a prescribed threshold,
The evaluation value calculating step includes:
Applying the information included in the environment recognition result and the reference image selected by the reference image selection unit,
The evaluation value calculating step includes:
(A1) Estimated number of feature points of the three-dimensional position included in the selected image (b1) Intersection of rays connecting the estimated three-dimensional position of the corresponding feature point included in the selected image and the feature point position of each image frame Horn,
(C1) Camera model fitting error of reference image is a step of calculating at least one of the above (a1) to (c1) as an evaluation value,
The evaluation value determination step includes
Comparing at least one of the above values (a1) to (c1) with a preset threshold value to determine whether the selected image is an image suitable for generating three-dimensional data;
(A2) the estimated number of feature points of the three-dimensional position included in the selected image is equal to or greater than a specified threshold (Tha);
(B2) the intersection angle between rays connecting the estimated three-dimensional position of the corresponding feature point included in the selected image and the feature point position of each image frame is equal to or greater than a specified threshold (Thb);
(C2) The camera model fitting error of the reference image is not more than a specified threshold value (Thc),
An image processing method, comprising: a step of determining that the selected image is an image suitable for generating three-dimensional data when at least one of the conditions (a2) to (c2) is satisfied. The evaluation value calculating step includes:
Calculating all of (a1) to (c1) as evaluation values,
The evaluation value determination step includes
When satisfying all the conditions of the (a2) ~ (c2), according to claim 8, wherein the selected image is determining that an appropriate image for the generation of three-dimensional data Image processing method. The evaluation value calculating step includes:
Among the evaluation values, for the crossing angle between the light rays, it is a step of calculating a median value of a plurality of crossing angles calculated by applying a plurality of corresponding feature points and image frames,
The evaluation value determination step includes
When satisfied that the median is defined threshold (Thb) above, claim 8 or 9, wherein the selected image is determining that an appropriate image for the generation of three-dimensional data An image processing method described in 1. The image processing method further includes:
A feature point tracking unit has a feature point tracking step of obtaining a feature point trajectory included in the image by analyzing the selected image;
The image processing method according to claim 8, wherein the evaluation value calculating step is a step of calculating the evaluation value by applying the feature point locus. The input image selection step includes:
A step of setting and updating a variable [T] corresponding to the selection frame interval and a variable [F] corresponding to the first selection frame, and performing a selection process of different images,
In the evaluation value determination step, when it is determined that the selected image is not an image suitable for generating three-dimensional data, at least one of the variables [T] and [F] is updated and a new one is executed. The image processing method according to claim 8, wherein the image processing method is a step of executing correct image selection. The input image selection step includes:
According to the evaluation mode of the selected image in the evaluation value determination step, a new image selection is performed by updating at least one of the variables [T] and [F] according to a predetermined variable update algorithm. The image processing method according to claim 12 , wherein the image processing method includes: The image processing method further includes:
The SFM (Structure from Motion) process using the selected image determined by the three-dimensional data generation unit as an image suitable for the generation of the three-dimensional data in the evaluation value determination step, and the information generated by the SFM process as initial information The image processing method according to claim 8, further comprising a step of executing three-dimensional data generation by a process to which an extended Kalman filter (EKF) used as a step is applied. In the information processing apparatus, a computer program that executes an image selection process applied to calculate a three-dimensional position of a pixel included in an image,
An image selection step for causing the image selection unit to select a plurality of images as selected images from the input image;
An evaluation value calculation step for causing the evaluation value calculation unit to calculate an evaluation value for determining whether or not the selected image is an image suitable for generating three-dimensional data;
An evaluation value determination step for causing the evaluation value determination unit to determine whether or not the selected image is an image suitable for generating three-dimensional data based on the evaluation value calculated in the evaluation value calculation step ;
An environment recognition step in which an environment recognition unit generates an environment recognition result including the three-dimensional position of the feature point included in the image and the position and orientation information of the camera by analyzing the selected image;
A reference image selection unit is a reference image selection unit that selects a reference image for verifying the environment recognition result, and within a one-image coordinate system of reference image candidates and corresponding feature points that are commonly included in the selected image A reference image selection step of selecting an image that satisfies a condition that the distance of the distance is equal to or greater than a predetermined threshold as a reference image;
The evaluation value calculating step includes:
Applying the information included in the environment recognition result and the reference image selected by the reference image selection unit,
(A1) Estimated number of feature points of the three-dimensional position included in the selected image (b1) Intersection of rays connecting the estimated three-dimensional position of the corresponding feature point included in the selected image and the feature point position of each image frame Horn,
(C1) Camera model fitting error of reference image is a step of calculating at least one of the above (a1) to (c1) as an evaluation value,
The evaluation value determination step includes
Comparing at least one of the above values (a1) to (c1) with a preset threshold value to determine whether the selected image is an image suitable for generating three-dimensional data;
(A2) the estimated number of feature points of the three-dimensional position included in the selected image is equal to or greater than a specified threshold (Tha);
(B2) the intersection angle between rays connecting the estimated three-dimensional position of the corresponding feature point included in the selected image and the feature point position of each image frame is equal to or greater than a specified threshold (Thb);
(C2) The camera model fitting error of the reference image is not more than a specified threshold value (Thc),
A computer program characterized by being a step of determining that the selected image is an image suitable for generating three-dimensional data when at least one of the above conditions (a2) to (c2) is satisfied.