JP2009237845A - Information processor, information processing method, and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain a configuration for efficiently calculating three-dimensional positions of feature points included in a photographic image based on an image obtained by a camera. <P>SOLUTION: In a configuration of acquiring the three-dimensional position of feature points based on the image obtained by a camera, the three-dimensional position information of feature points in subsequent frames are acquired by:acquiring the three-dimensional position information of the feature points based on corresponding feature point analysis using only a plurality of preceding image frames in the image frame of the camera photographic image as initial information, and adopting an extended Kalman filter (EKF) corresponding to the subsequent image frame while setting the initial information as state information corresponding the initial image frame. Thus, it is required to execute the feature point extraction processing with frame matching or the like only to the preceding image frame. Therefore, it is possible to efficiently acquire the three-dimensional position information of the feature points and generate the three-dimensional image data. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an information processing apparatus, an information processing method, and a computer program. More specifically, the present invention relates to an information processing apparatus, an information processing method, and a computer program for performing processing for calculating a three-dimensional position of a feature point applied to generation of a three-dimensional map (3D map) based on a photographed image by a camera.

  Processing for analyzing a photographed image of a camera and obtaining a three-dimensional position of an object included in the photographed image is used in various fields. For example, an agent (moving body) such as a robot equipped with a camera analyzes the captured image of the camera, observes the moving environment, and moves while grasping the environment around the agent according to the observation situation, It is used for environment map construction processing for creating a map of the surrounding environment (environment map) based on Non-Patent Document 1 discloses a method of tracking feature point positions in all frames and calculating the positions of feature points and camera positions by batch processing after data of all frames is obtained.

  An example of a three-dimensional map (3D map) generation processing sequence will be described with reference to FIG. First, in step S11, an image is taken by a camera. For example, a user or a robot holding a camera continuously takes surrounding images while moving.

  In step S12, sparse three-dimensional information including position information of feature points included in the image is constructed by analyzing the acquired image. In this process, a process such as SLAM (simultaneous localization and mapping) or SFM (Structure from Motion) is applied. SLAM is a process for detecting the position of a feature point in an image input from the camera and the position and orientation of the camera. SFM is, for example, processing for analyzing correspondence between feature points (Landmarks) included in an image using images taken from a plurality of different positions.

  Furthermore, in step S13, a three-dimensional map, which is dense three-dimensional information, which is detailed three-dimensional information, is generated using the camera trajectory information obtained in step S12 and the feature point information in the image.

  Accurate analysis of the feature points in the image in the SFM or SLAM processing in step S12 increases the accuracy of the final three-dimensional information. In step S12, a process of acquiring feature point information and camera trajectory information included in the image frame is performed. An example of this detailed processing is shown in FIG.

  First, in step S <b> 21, a plurality of captured images are input from the camera, and frame matching processing for calculating the position of the camera is performed using matching feature points between frames. The input image is, for example, a plurality of image frames of a moving image captured by a moving camera, that is, a plurality of images captured from different positions, and the same object is captured in a plurality of frames. In step S21, corresponding feature points are detected from a plurality of image frames, and the position of the camera is calculated using these pieces of information.

  In step S22, a frame stitch process (Frame Stitch) is executed. This process estimates the three-dimensional position of each image feature point (Landmark) using the information detected in step S21, and joins image frames based on a plurality of feature point positions in a plurality of image data. This is a process of generating data reflecting the three-dimensional position of each feature point.

  In step S23, bundle adjustment processing (Bundle Adjustment) is executed. This bundle adjustment process is a process for converging the three-dimensional positions of corresponding feature points included in a plurality of images taken from different positions into one position. This is a process of calculating the most probable three-dimensional position of the feature point using the camera position information of each captured image and the information of the corresponding feature point included in the image captured at each camera position. For this processing, basically, image frames taken from two or more different positions are required.

  For example, as shown in FIG. 3, the captured images 11 to 13 from three different positions are used, and the feature points 21 to 23 detected from the captured images are used to obtain the three-dimensional positions of the feature points. Among the feature points included in a plurality of images, the position of the corresponding feature point is the position of the feature point 21 where a line (bundle) connecting the feature point 21 to the shooting point of each image is a feature point 21 shown in FIG. Should cross.

  However, the position of the camera, the position information in the image of the feature points, etc. are not necessarily calculated accurately, and errors due to various factors are included. Therefore, it is necessary to remove such errors. Specifically, correction is performed so that a line connecting the three-dimensional position of one corresponding feature point and the camera position intersects at the one feature point. That is, it is necessary to correct the calculated camera position and feature point position. This processing is executed as bundle adjustment processing (Bundle Adjustment), and it becomes possible to generate more accurate three-dimensional information using the feature point position information and camera position information corrected by this adjustment processing.

However, in such a conventional process, if a sufficient number of corresponding feature points cannot be obtained, the frame matching process in step S21 becomes impossible, and it may be difficult to calculate the camera position. Frame matching cannot be omitted, and if frame matching fails, processing must be restarted. In addition, the camera must always move, and if the camera orbits around an object, it is assumed that there is a plurality of one object due to a feature point mapping error. There was a problem that errors such as recognition were likely to occur.
“Shape and motion from image stream under orthography: a factorization method”, C.I. Tomasi and T. Kanade, International Journal of Computer Vision, Volume 9, Number 2, pp. 137-154, (1992).

  According to the present invention, in a configuration for calculating a three-dimensional position of a feature point applied to generation of a three-dimensional map (3D map) based on a photographed image by a camera, only a frame matching process using only a part of the photographed image is performed. An object of the present invention is to provide an information processing apparatus, an information processing method, and a computer program that enable efficient three-dimensional position information generation of feature points without performing frame matching processing in subsequent processing.

The first aspect of the present invention is:
An information processing apparatus that calculates a three-dimensional position of a feature point included in an image,
An initial information generation unit that inputs a plurality of preceding image frames in an image frame of a camera-captured image and acquires three-dimensional position information of feature points by analyzing corresponding feature points of each image frame;
The feature point position information acquired from the preceding image frame by the initial information generating unit is set as state information for the initial image frame, and 3 of feature points in the succeeding frame is processed by applying an extended Kalman filter (EKF) to the succeeding image frame. A feature point position information generation unit for acquiring dimension position information;
There is an information processing apparatus characterized by having.

  Furthermore, in an embodiment of the information processing apparatus of the present invention, the information processing apparatus further inputs the feature point position information generated by the feature point position information generation unit and corrects the three-dimensional position information of the feature points. It has the bundle adjustment process part which performs.

  Furthermore, in an embodiment of the information processing apparatus according to the present invention, the initial information generation unit analyzes the correspondence between feature points included in each image frame using images taken from a plurality of different positions. Motion) processing is executed.

  Furthermore, in one embodiment of the information processing apparatus according to the present invention, the initial information generation unit is configured to calculate position and orientation information of a camera that captured the image frame by analyzing the preceding image frame, and the feature point position The information generation unit sets the feature point position information and the camera position and orientation information calculated by the initial information generation unit from the preceding image frame as state information corresponding to the initial image frame, and an extended Kalman filter (EKF) for the subsequent image frame. ) To obtain the three-dimensional position information of the feature points in the subsequent frame and the camera position and orientation information obtained by photographing the subsequent frame.

  Furthermore, in an embodiment of the information processing apparatus of the present invention, the feature point position information generation unit sets multidimensional normal distribution data including feature point position information and camera position and orientation information as state information, and uses an extended Kalman filter (EKF). ) To obtain the three-dimensional position information of the feature points in the subsequent frame and the camera position and orientation information obtained by photographing the subsequent frame.

  Furthermore, in an embodiment of the information processing device of the present invention, the information processing device further extracts feature points that have not been extracted by the initial information generation unit and the feature point position information generation unit as additional feature points. A feature point extraction unit that executes a process of calculating a three-dimensional position of the additional feature point extracted in the step, and the bundle adjustment processing unit includes the feature point position information generated by the feature point position information generation unit and the feature point. The feature is that the feature point position information of the additional feature point extracted by the extraction unit is input and the correction process of the three-dimensional position information of the feature point is executed.

  Furthermore, in an embodiment of the information processing apparatus of the present invention, the feature point extraction unit includes feature points extracted from an image frame, and feature points extracted by the initial information generation unit and the feature point position information generation unit. And a process of selecting only new feature points that do not overlap as additional feature points.

  The information processing apparatus further includes a 3D map generation unit that generates 3D image data using the feature point position information generated by the feature point position information generation unit.

Furthermore, the second aspect of the present invention provides
An information processing method for calculating a three-dimensional position of a feature point included in an image in an information processing device,
An initial information generating step for inputting a plurality of preceding image frames in an image frame of a camera-captured image and acquiring three-dimensional position information of the feature points by corresponding feature point analysis of each image frame;
The feature point position information generation unit sets the feature point position information acquired from the preceding image frame by the initial information generation unit as state information for the initial image frame, and applies an extended Kalman filter (EKF) to the subsequent image frame. A feature point position information generation step for acquiring three-dimensional position information of the feature points in the subsequent frame;
There is an information processing method characterized by comprising:

  Furthermore, in an embodiment of the information processing method according to the present invention, the information processing method further includes: a feature point position information generated by the bundle adjustment processing unit by inputting the feature point position information generated in the feature point position information generation step; It has the bundle adjustment process step which performs the correction process of dimension position information, It is characterized by the above-mentioned.

  Furthermore, in an embodiment of the information processing method of the present invention, the initial information generation step uses an SFM (Structure from) that analyzes the correspondence between feature points included in each image frame using images taken from a plurality of different positions. It is a step for executing a (Motion) process.

  Furthermore, in an embodiment of the information processing method of the present invention, the initial information generation step is a step of calculating position and orientation information of a camera that has captured the image frame by analyzing the preceding image frame, and the feature point position The information generation step sets the feature point position information and the camera position and orientation information calculated from the preceding image frame by the initial information generation unit as state information corresponding to the initial image frame, and an extended Kalman filter (EKF) for the subsequent image frame. ) To obtain the three-dimensional position information of the feature points in the subsequent frame and the camera position and orientation information obtained by photographing the subsequent frame.

  Furthermore, in an embodiment of the information processing method of the present invention, the feature point position information generation step sets multidimensional normal distribution data including feature point position information and camera position and orientation information as state information, and sets an extended Kalman filter (EKF). ) To obtain the three-dimensional position information of the feature points in the subsequent frame and the camera position and orientation information obtained by photographing the subsequent frame.

  Furthermore, in an embodiment of the information processing method of the present invention, the information processing method further includes a feature point extracted by the feature point extraction unit by the initial information generation unit and the feature point position information generation unit. A feature point extraction step for executing a process of calculating a three-dimensional position of the additional feature point extracted and extracted as the additional feature point, wherein the bundle adjustment processing step includes the feature point generated by the feature point position information generation unit The correction processing of the feature point three-dimensional position information is executed by inputting the position information and the feature point position information of the additional feature point extracted by the feature point extraction unit.

  Furthermore, in an embodiment of the information processing method of the present invention, the feature point extraction step includes a feature point extracted from an image frame, and a feature point extracted by the initial information generation unit and the feature point position information generation unit. And a process of selecting only new feature points that do not overlap as additional feature points.

  Furthermore, in an embodiment of the information processing method of the present invention, the information processing method further includes: a 3D map generation unit configured to generate 3D image data using the feature point position information generated by the feature point position information generation unit; It has a 3D map generation step to generate.

Furthermore, the third aspect of the present invention provides
In the information processing device, a computer program for calculating a three-dimensional position of a feature point included in an image,
An initial information generating step of inputting a plurality of preceding image frames in the image frames of the camera-captured image to the initial information generating unit and acquiring the three-dimensional position information of the feature points by analyzing the corresponding feature points of each image frame;
In the feature point position information generation unit, the feature point position information acquired from the preceding image frame by the initial information generation unit is set as state information for the initial image frame, and an extended Kalman filter (EKF) for the subsequent image frame is applied. A feature point position information generation step for acquiring three-dimensional position information of the feature points in the subsequent frame;
There is a computer program characterized by comprising:

  The computer program of the present invention is, for example, a computer program that can be provided by a storage medium or a communication medium provided in a computer-readable format to a general-purpose computer system that can execute various program codes. . By providing such a program in a computer-readable format, processing corresponding to the program is realized on the computer system.

  Other objects, features, and advantages of the present invention will become apparent from a more detailed description based on embodiments of the present invention described later and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

  According to the configuration of one embodiment of the present invention, in a configuration for acquiring a three-dimensional position of a feature point based on an image acquired by a camera, corresponding feature point analysis using only a plurality of preceding image frames in an image frame of a camera-captured image. In the subsequent frame, a process of acquiring the three-dimensional position information of the feature point as initial information and a process of setting this initial information as state information for the initial image frame and applying an extended Kalman filter (EKF) to the subsequent image frame Since the feature point extraction process with frame matching, for example, only needs to be performed on the preceding image frame, the feature point three-dimensional position information is efficiently acquired. Acquisition and generation of three-dimensional image data.

  Details of an information processing apparatus, an information processing method, and a computer program according to embodiments of the present invention will be described below with reference to the drawings.

  The outline of the present invention will be described with reference to FIG. The information processing apparatus 120 of the present invention inputs, for example, a photographed image of the camera 102 held by the moving user 101, for example, an image constituting a moving image, performs analysis of the input image, and includes various images included in the photographed image. Three-dimensional image information 103 composed of objects is generated.

  Similar to the processing described above with reference to FIG. 1, the information processing apparatus 120 analyzes a sparse three-dimensional map 131 that includes position information of feature points included in an image by analyzing an acquired image, using SLAM (simultaneous). Dense three-dimensional information that is generated by applying processing such as localization and mapping (SFM) and structure from motion (SFM), and further using detailed information such as camera trajectory information and feature point information in the image. A map 132 is generated.

  As described above, SLAM is a process for detecting the position of the feature point in the image input from the camera and the position and orientation of the camera together. SFM is, for example, processing for analyzing correspondence between feature points (Landmarks) included in an image using images taken from a plurality of different positions.

  The sparse three-dimensional map 131 has three-dimensional position information of feature points. The conventional example of the information generation processing sequence has been described with reference to the flowchart of FIG. 2, but the information processing apparatus 120 of the present invention performs the processing of the three-dimensional position of the feature points by processing different from the flow shown in FIG. 2. Acquire.

  The acquisition sequence of the three-dimensional position of the feature point and the position and orientation information of the camera according to an embodiment of the present invention will be described with reference to the flowchart shown in FIG.

The process of acquiring the feature point 3D position information and the camera position and orientation information is executed in the following processing sequence.
Step S101: Initial information acquisition process (SFM)
Step S102: Camera position and orientation and feature point three-dimensional position information acquisition processing (EKF SLAM) to which the extended Kalman filter is applied
Step S103: Bundle adjustment processing (Bundle Adjustment)

The basic flow of the acquisition process of the three-dimensional position information of the feature point and the position and orientation information of the camera of one embodiment of the present invention is as follows
"Initial information acquisition process (SFM)"
→ "Camera position / posture and feature point 3D position information acquisition processing (EKF SLAM) applying extended Kalman filter"
→ "Bundle adjustment process (Bundle Adjustment)"
This is the processing sequence.

  The initial information acquisition process (SFM) in step S101 is performed, for example, by an SFM (Structure from Motion) process that analyzes the correspondence of feature points (Landmarks) included in the image using images taken from a plurality of different positions. However, this processing is executed by applying only the input images of the first few frames input from the camera, for example. Information obtained by this process is used as initialization information for the camera position and orientation and feature point three-dimensional position information acquisition process (EKF SLAM) to which the extended Kalman filter is executed in step S102.

  After initialization information to be applied to the EKF SLAM is obtained, the EKF SLAM is executed on the subsequent input image without executing the SFM, and the feature point position information and the position of the camera that captured each image are obtained. Get posture information.

  A specific processing example will be described with reference to FIG. Assume that the position information of the feature points of the subject included in the image frame acquired by the camera shown in FIG. In FIG. 6A, image frames acquired by the camera are shown at regular intervals.

  In the information processing apparatus of the present invention, not only all the image frames to be analyzed for the three-dimensional position information of feature points, but only the first few frames (input frames up to frame T1 shown in FIG. 6) are shown in FIG. The processing target frame of the initial information acquisition process (SFM) in step S101 shown in FIG. By the initial information acquisition process (SFM) for the plurality of preceding image frames, the trajectory information of the three-dimensional map and the camera frame is used as initialization information (Initialized Data) of “EKF SLAM” executed in the next step S102. Used.

  The three-dimensional map obtained from a plurality of preceding image frames up to frame T1 and the trajectory information of the camera frame become initial information. Using this initial information, “EKF SLAM” is initialized, and “EKF SLAM” processing is performed. Do. That is, the feature point position information acquired from the preceding image frame is set as the state information for the initial image frame, and the process of applying the extended Kalman filter (EKF) to the subsequent image frame is used to obtain the three-dimensional position information of the feature points in the subsequent frame. get.

The flowchart shown in FIG. 7 is a flowchart showing a detailed sequence of the flow shown in FIG.
Step S101 [initial information acquisition process (SFM)] shown in FIG. 5 is the process of step S204 of FIG.
Step S102 [camera position and orientation and feature point three-dimensional position information acquisition process (EKF SLAM) applying the extended Kalman filter] shown in FIG. 5 is the process of step S207 of FIG.
Step S103 [bundle adjustment processing] shown in FIG. 5 is the same as the processing in step S209 in FIG.
Correspond to each.

  The processing sequence of the present invention will be described with reference to the flowchart of FIG. First, in step S201, a flag is set. This flag is a status flag indicating whether or not the initial information acquisition process (SFM) is completed. If the initial information acquisition process (SFM) is not completed, the flag is set to [0]. If completed, the flag is set to [1]. Is set.

  Initially, in step S201, the status flag is set to [0], and an image is input in step S202. For example, it is one image frame constituting a moving image taken by the camera 102 shown in FIG.

  Next, the value of the status flag is confirmed in step S203. If the flag is not 1, the initial information acquisition process (SFM) has not been completed, and the process proceeds to step S204. In step S204, an initial information acquisition process (SFM) is executed. This process is an SFM (Structure from Motion) process that analyzes correspondence between feature points (Landmarks) included in an image using images taken from a plurality of different positions. Similar to the conventionally known SFM processing, the feature point three-dimensional position and the position and orientation information of the image frame including the feature point (camera position and orientation information) are calculated by the feature point association processing and bundle adjustment processing by frame matching. To do.

The initial information acquisition process (SFM) in step S204 is executed only for a preset number of frames. The input information and output information of the initial information acquisition process (SFM) in step S204 are as follows.
Input: Image data with a preset number of frames,
Output: Position and orientation of input image frame and feature point position information Note that the position and orientation of the input image frame is information corresponding to the position and orientation of the camera, and is information corresponding to the camera position and orientation information or corresponding one-to-one.
The output information [position and orientation of the input image frame and feature point position information] generated in the initial information acquisition process (SFM) in step S204 is the camera position and orientation and feature point three-dimensional position information acquisition using the extended Kalman filter in step S207. Used for processing (EKF SLAM). The output information [position and orientation of input image frame and feature point position information] generated in the initial information acquisition process (SFM) in step S204 is stored in the storage unit 152 together with the processing target image frame.

  After the initial information acquisition process (SFM) in step S204, in step S205, it is determined whether or not SFM has been completed by reaching the preset number of frames. If not, the process returns to step S202. The next image is input and the initial information acquisition process (SFM) in step S204 is continued.

  If it is determined in step S205 that the preset number of frames has been reached and the initial information acquisition process (SFM) has been completed, the process proceeds to step S206, and the status flag is set to [1], that is, the initial information acquisition process (SFM). Is set to a value indicating completion, and the process returns to step S202 to input the next image.

  Next, in step S203, it is confirmed that the value of the status flag is [1], and the process proceeds to step S207. In step S207, the process of step S102 in the flow of FIG. 5, that is, the camera position and orientation and feature point three-dimensional position information acquisition process (EKF SLAM) to which the extended Kalman filter is applied is executed.

  In processing using an extended Kalman filter (EKF), for example, information such as the camera and the position of a feature point acquired from a preceding image frame is held as a state variable, and the corresponding feature point is observed from the subsequent frame. Based on the information, information such as the position of the camera and feature points as state variables is updated to obtain the camera trajectory and the three-dimensional position of the feature points. The state variable is held as multidimensional normal distribution data including a plurality of state values. That is, it is held as multidimensional normal distribution data indicating the probability of the state of each variable as a normal distribution.

  In order to start processing using the extended Kalman filter (EKF), it is necessary to previously hold state variables indicating the positions of the cameras and feature points as initial values. In the process of the present invention, the data that becomes the initial value is acquired by the initial information acquisition process (SFM) in step S204 of FIG. 7 (the process of step S101 shown in FIG. 5).

The state variables used in the camera position and orientation and feature point three-dimensional position information acquisition process (EKF SLAM) to which the extended Kalman filter is applied will be described with reference to FIG. In FIG.
(A) Example of shooting process by camera,
(B) a state variable,
(C) State variable update processing,
These are shown.

  For example, as shown in FIG. 8B, the state variable [x] is determined by the position information P1 to Pn of a plurality of feature points detected from the camera position, posture, speed, each speed, and the camera's captured image. Composed. In this way, it is configured as multidimensional normal distribution data composed of a plurality of state values. The example of the state variable shown here is merely an example, and other detailed information may be included, or a data structure including only a part of these may be used.

The state variable is sequentially updated based on a new input image. As shown in FIG. 8C, update information [x _{t + 1} ] is generated as follows based on the feature point information detected from the new processing frame.
_{x t + 1 ← f (x} t, a t, s t) + u
However,
a _t : motion model,
_st : observation model,
u: Noise.
In this embodiment, the motion model is processed according to the “constant velocity motion model” and the observation model is processed according to the “pinhole camera model”.

Information obtained by the data conversion processing unit A151 in the initial information acquisition process (SFM) in step S204 at the start of the camera position and orientation and feature point three-dimensional position information acquisition process (EKF SLAM) to which the extended Kalman filter is applied in step S207. To generate state variables for application to EKF SLAM. As described above, the input information and output information of the initial information acquisition process (SFM) in step S204 are as follows.
Input: Image data with a preset number of frames,
Output: Position and orientation of input image frame and feature point position information

  The data conversion processing unit A151 receives the output information [position and orientation and feature point position information of the input image frame] of the initial information acquisition process (SFM), and the camera position and orientation and feature point 3 to which the extended Kalman filter of step S207 is applied. Data for use in the dimension position information acquisition process (EKF SLAM), that is, an initial frame state (= state variable) applied in the EKF SLAM is generated.

The input value and output value of the data conversion processing unit A151 are as follows.
Input values: position and orientation of the input image frame (camera position and orientation) and feature point position information obtained in the initial information acquisition process (SFM) in step S204,
Output value: State of initial frame applied in EKF SLAM (= state variable)

  For example, the data conversion processing unit A151 sets the camera position and orientation and the feature point position corresponding to the initial frame used in the EKF SLAM as an average of the state (multidimensional normal distribution) of the initial frame. When the camera speed / angular velocity is required as in the variable shown in FIG. 8, for example, the camera position and orientation calculated corresponding to the past image frame acquired as a result of SFM, and the camera moves at a constant speed. The camera speed / acceleration is predicted using the hypothesis that the image is to be reflected and reflected in the average of the initial frame states.

  Note that the state variable is multidimensional normal distribution data as described above, but there are various methods for setting the covariance matrix of the state variable. Obtaining the covariance matrix directly is difficult because of various error factors, and is generally determined by empirical rules. In this embodiment, a value preset by the user is used. For example, the covariance value is 0, the autocovariance value for the camera is 0, and the empirical value σ set by the user is used for the autocovariance value for the feature point position.

  As described above, the data conversion unit A151 obtains the position and orientation (camera position and orientation) and the feature point position information of the image frames of 0 to T frames set in advance in the initial information acquisition process (SFM) in step S204. Using this, a state variable corresponding to the initial frame used in the EKF SLAM is calculated, and in step S207, using this state variable, a camera position and orientation and feature point three-dimensional position information acquisition process (EKF) using the extended Kalman filter is used. SLAM), and update processing of the state variable according to the input of the subsequent image frame is performed.

Input information and output information of the camera position and orientation and feature point three-dimensional position information acquisition process (EKF SLAM) to which the extended Kalman filter of step S207 is applied are as follows.
Input: State of feature points and initial frame, that is, state variables (see FIG. 8) consisting of multi-dimensional normal distribution data of camera position and orientation information and feature point position information,
Output: State variables corresponding to each image frame, feature point tracking data,
The feature point tracking data can be calculated from the feature point position information included in the state variable corresponding to each image frame.
The output information [state variable and feature point tracking data corresponding to each image frame] generated in the camera position and orientation and feature point three-dimensional position information acquisition process (EKF SLAM) to which the extended Kalman filter of step S207 is applied is processed. The image is stored in the storage unit 152 together with the image frame, and is used for bundle adjustment processing (Bundle Adjustment) in step S209.

  In step S207, state variables corresponding to each processed image frame (for example, state variable x (multidimensional normal distribution) shown in FIG. 8B) and feature point tracking data are acquired and associated with each frame. As shown in FIG. That is, the storage unit 152 stores state variables (multidimensional normal distribution) corresponding to the image position processed in the camera position and orientation and feature point three-dimensional position information acquisition processing (EKF SLAM) to which the extended Kalman filter is applied in step S207. Feature point tracking data is recorded.

  In step S208, it is determined whether there is an unprocessed input image. If there is, the process returns to step S202, the next image is input, and the camera position and orientation and feature point three-dimensional position to which the extended Kalman filter of step S207 is applied. The information acquisition process (EKF SLAM) is continuously executed.

  If it is determined in step S208 that there is no unprocessed input image, the process advances to step S209 to execute bundle adjustment processing (Bundle Adjustment).

  The bundle adjustment process (Bundle Adjustment) in step S209 is the output information generated in the camera position and orientation and feature point three-dimensional position information acquisition process (EKF SLAM) to which the extended Kalman filter is applied in step S207 [corresponding to each image frame. This is executed using state variables (multidimensional normal distribution) and feature point tracking data].

  The data conversion unit B153 illustrated in FIG. 7 uses the information generated and stored in the storage unit 152 in the camera position and orientation and feature point three-dimensional position information acquisition process (EKF SLAM) to which the extended Kalman filter of step S207 is applied, Conversion processing to data used in the bundle adjustment processing (bundle adjustment) in step S209 is performed.

The input / output of the data conversion unit B153 is as follows.
Input: State variable (multidimensional normal distribution) corresponding to each image frame and feature point tracking data (stored data in storage unit 152)
Output: camera position and feature point position corresponding to each image frame

  The data conversion unit B 153 obtains the camera position and orientation information of each image frame using the state variable (multidimensional normal distribution) corresponding to each image frame that is stored data in the storage unit 152, and the feature point position is the latest frame. Is obtained from the state (multidimensional normal distribution). As described above, the state variable is multi-dimensional normal distribution data indicating the state probability of each variable as a normal distribution, and the value corresponding to the average of the normal distribution data has the highest probability. For the posture and the feature point position, an average value of each normal distribution data of the state variable is adopted. Note that the feature point tracking data uses data stored in the storage unit 152.

The bundle adjustment process (bundle adjustment) in step S209 is performed by inputting the data generated by the data converter B153, that is, the camera position and orientation and the feature point position corresponding to each image frame.
Input / output data of the bundle adjustment processing (Bundle Adjustment) in step S209 is as follows.
Input: Camera position and orientation and feature point position of each image frame, and feature point tracking data Output: Camera position and orientation and feature point position of each image frame

  As described above with reference to FIG. 3, the bundle adjustment process is performed by processing each image frame by converging the three-dimensional positions of corresponding feature points included in a plurality of images captured from different positions into one position. This is a process for obtaining the camera position and orientation and the feature point position. That is, as for the position of the corresponding feature point, a line connecting the feature point 21 from the shooting point of each image should intersect at the position of one feature point 21 like the feature point 21 shown in FIG. The position information and the position information of the feature points in the image are not necessarily calculated accurately, and errors due to various factors are included. Therefore, it is necessary to remove such errors. Specifically, the calculated camera position and feature point position are corrected so that a line connecting the three-dimensional position of one corresponding feature point and the camera position intersects at the one feature point. This processing is executed as bundle adjustment processing (Bundle Adjustment).

  A three-dimensional map is generated by using the camera posture position information and the feature point position information obtained by this correction process as final outputs. By this processing, a highly accurate three-dimensional map can be generated.

[Example of processing for adding feature points]
In the processing described with reference to the flowchart of FIG. 7, only the information on the feature point obtained as a result of the camera position and orientation and feature point three-dimensional position information acquisition processing (EKF SLAM) to which the extended Kalman filter of step SS207 is applied is used. The bundle adjustment processing (Bundle Adjustment) in step S209 is performed.

  Although it is possible to generate a three-dimensional map using the three-dimensional position information of feature points obtained by executing such processing, the number of feature points is relative to the number of camera frames to be processed. If the number is too small, the result of the generated three-dimensional map may be deteriorated. In order to generate a three-dimensional map with higher accuracy, it is advantageous that the number of available feature points is large. That is, if it is possible to add feature points before performing bundle adjustment processing (bundle adjustment), information on the added feature points is also used and bundle adjustment processing (bundle adjustment) is used to perform many. It is preferable that the three-dimensional position information of the feature points is output. Hereinafter, a processing example for executing this feature point addition processing will be described.

  The processing sequence of this embodiment is processing according to the flowchart shown in FIG. The process flow shown in FIG. 9 is that the feature point addition process of step S300 is added before the process of step S209 shown in FIG. 7 described above. Other processing is the same as the flow shown in FIG.

  The feature point addition process in step S300 is executed after the camera position and orientation and feature point three-dimensional position information acquisition process (EKF SLAM) to which the extended Kalman filter is applied in step S207 is completed.

  At this point, the storage unit stores state variables corresponding to each image frame generated by the EKF SLAM process in step S207 and feature point tracking data. In the feature point addition process in step S300, the storage unit stores The stored data and processed image frame data are input to perform feature point addition processing.

The input / output data of the feature point addition process in step S300 is as follows.
Input: Output of data conversion unit 153 [camera position and orientation and feature point position and feature point tracking data of each image frame] and all processed image frame data Output: three-dimensional position of feature point and within each image Feature point position,
It is.

  In the bundle adjustment process (bundle adjustment) in step S209, the bundle adjustment process (both the feature point information added in the feature point addition process in step S300) is used for the feature point information extracted in the EKF SLAM process in step S207. (Bundle Adjustment) will be performed. That is, it is possible to output three-dimensional position information of more feature points than the flow shown in FIG. 7 described above.

  A detailed sequence of the feature point addition process in step S300 will be described with reference to the flowchart of FIG. In step S301, the processing target image frames are sequentially input one by one. Note that the image frame to be processed is an image that has been subjected to the initial information acquisition process (SFM) in step S204 and the EKF-SLAM process in step S207. The processing after step S302 is sequentially executed for each processing target image frame.

  In step S302, a feature point extraction process from the image frame to be processed is executed. An existing processing method can be applied to the feature point extraction process, and for example, a feature point extraction process using a Harris Corner Detector is applied.

  A feature point extraction process using a Harris corner detector will be described with reference to FIG. The data processing unit of the information processing apparatus generates a plurality of Harris corner images 180 to 182 and a Laplacian image 190 to 192 from the acquired image 170 photographed by the camera as shown in FIG.

  A Harris corner image is image data generated by applying a Harris Corner Detector to an acquired image. From these Harris corner images 180 to 182, for example, a pixel point (maxima point) 185 having a higher value than the surrounding eight pixels is extracted as a detection point. Further, a LoG (Laplacian of Gaussian) filter is applied to the acquired image 170 to generate a multi-level (resolution) Laplacian image 190-192. The LoG (Laplacian of Gaussian) filter is a kind of second-order differential filter used for image edge enhancement, and is performed until information from the retina is relayed by the outer knee in the human visual system. It is used as an approximate model of the processing.

  The feature point extraction processing corresponds to the position of the detection point obtained from the Harris corner images 180 to 182 as to whether there is a change in position due to a resolution change within a predetermined level range of the Laplacian images 190 to 192 that are LoG filter output images. A point is examined, and a point having no change is defined as a feature point. Thereby, it is possible to realize matching between feature points that is robust to an enlargement / reduction operation of an image. Details of these feature point extraction processes are described in, for example, Japanese Patent Application Laid-Open No. 2004-326693 (Japanese Patent Application No. 2003-124225).

  Next, in step S303, duplicate feature points are removed. That is, feature points that overlap with the feature points extracted in the initial information acquisition process (SFM) in step S204 and the EKF-SLAM process in step S207 are removed from the feature points extracted in step S302. The feature point information extracted in the initial information acquisition process (SFM) in step S204 and the EKF-SLAM process in step S207 is stored in the storage unit 152. In step S302, Only the feature points that do not overlap are compared and registered in the storage unit 152 with the acquired feature points.

  Specifically, the position of the feature point registered in the storage unit 152 in the image frame with respect to the processed image frame is calculated by the following formula (Formula 1).

  The meaning of the above formula will be described with reference to FIGS. The above expression is obtained by calculating the pixel position 212 on the camera image plane of the point (m) of the object 211 included in the captured image 210 of the camera, that is, the position represented by the camera coordinate system, and the three-dimensional shape of the object 200 in the world coordinate system. It is an expression showing the correspondence with the position (M) 201.

  A pixel position 212 on the camera image plane is expressed by a camera coordinate system. The camera coordinate system is a coordinate system in which the focal point of the camera is the origin C, the image plane is a two-dimensional plane of Xc and Yc, and the depth is Zc, and the origin C is moved by the movement of the camera.

  On the other hand, the three-dimensional position (M) 201 of the object 200 is indicated by a world coordinate system composed of three XYZ axes having an origin O that does not move due to the movement of the camera. An expression indicating the correspondence between the positions of the objects in the different coordinate systems is defined as the above-described pinhole camera model.

Λ, A, Cw, and Rw included in this equation are as shown in FIG.
λ: normalization parameter A: camera internal parameter,
Cw: camera position,
Rw: camera rotation matrix,
Means.

The camera internal parameter A includes the following values.
f: Focal length θ: Image axis orthogonality (ideal value is 90 °)
ku: Scale of the vertical axis (conversion from a 3D position scale to a 2D image scale)
kv: horizontal scale (conversion from 3D position scale to 2D image scale)
(U0, v0): Image center position

If the feature point extracted by the feature point extraction process in step S302 is close to the position in the image of the feature point registered in the storage unit 152, it is deleted as a duplicate feature point. The determination of whether or not they are close is deleted if the difference is smaller than the threshold compared with a preset threshold. Specifically, for example, the determination is performed using the following expressions (Expression 2) and (Expression 3).

Using the dist and Xz in (Equation 2) and (Equation 3) above,
“Xz is positive and dist is above a certain threshold”
Are determined to be overlapping feature points. However, m _database is a feature point registered in the storage unit 152, and is the same as the left-side value of the above formula (Formula 1). Further, m _{Harris_corner} corresponds to the point extracted by the feature point extraction and processing in step S302. Further, each component on the left side of the formula (formula 3) is the same as the formula (formula 1) described above. The equation (Equation 3) is set to ignore the feature points on the back of the camera.

  Next, in step S304, the first feature point tracking process, that is, “feature point tracking # 1 (feature point position unknown)” is executed. This process is a feature point optical flow detection process by general template window matching. Details of optical flow detection processing of feature points by template window matching are described in, for example, [Digital Image Processing Editorial Committee, “Digital Image Processing”, Foundation for Image Information Education (2004) pp. 243].

  A feature point newly extracted in the feature point extraction processing in step S302 is selected from the image frame to be processed, and the periphery of the feature point of the frame is cut out as a window to obtain a template image. The window size is set by the user (for example, a 21 × 21 template for a 360 × 240 image). Next, template matching is performed on the frame in the future direction to obtain corresponding points. The search area is set around the corresponding point in the immediately preceding frame and the search is executed.

  For example, a normalized correlation value is used as a matching score used as an index of whether or not matching is performed. The normalized correlation value is described in the above-mentioned document [Digital Image Processing Editorial Committee, “Digital Image Processing”, Foundation for Image Information Education (2004) pp. 204]. In this process, if the matching score is small (if the correlation value is small), the search is stopped. The obtained corresponding point is the obtained tracking result. In this way, the tracking process of the feature point newly extracted in the feature point extraction process of step S302 from the image frame to be processed is executed.

  Next, in step S305, a feature point three-dimensional position estimation process is executed. This process is a process for obtaining the three-dimensional position of the feature point using the camera position of each image frame and the feature point tracking result obtained by the above-described process.

The position M of the feature point is obtained using the formula (Formula 1) described above, that is, the formula of the pinhole camera model. However, the camera position is c, the camera posture (rotation matrix) is R, the camera internal parameter is A, and the position of the feature point in the image is m. Next, the equation (Equation 1) is modified to create the following equation (Equation 4) relating to the position M of the feature point.

Each variable in the above formula (formula 4) is the same as the formula (formula 1) described above. D represents the distance between the camera position c and the feature point M. A subscript i represents a frame number. A vector x written in lower case is defined by the following equation (Equation 5).

In addition, since the said Formula (Formula 4) is formed only in a positive value, the following formula | equation (Formula 6) is formed with respect to the measured value obtained from data converter B153 shown in FIG.

However,
The symbol [^] means the measured value,
[Ε] is the error between the left and right sides caused by the measurement error,
[X ^] is a value defined by the following equation (Equation 7).

The input frame is not for one frame, but for n frames (1 frame = camera position / posture c, R and feature points) tracked in the process of step S304, that is, “feature point tracking # 1 (feature point position unknown)”. Since the position in the image is m), the above formula (formula 6) is expanded to correspond to a plurality of frames, and the following formula (formula 8) is obtained.

However, M is defined by the following formula.

The above formula (Formula 8) is prepared for n frames, and the following formula (Formula 10) is compiled into one formula.

In addition, each matrix and vector of the above formula (formula 10) is expressed by the following formula (formula 11).
Define as

If the vector X is calculated such that the norm (vector length) of ε on the left side of the above equation (Equation 11) is minimum (error minimum), the desired vector Y (three-dimensional position M of the feature points, A distance di) between the camera position of the frame data and the feature point is obtained. The expression of the square of the norm of ε is the following expression (Expression 12).

When the vector Y in the above equation (Equation 12) is regarded as a variable and the others are all constants, the above equation (Equation 12) is a quadratic function related to the vector Y. The following equation (Equation 13) which is a partial differential equation with respect to the vector Y of:

Y where the partial differential value of the above formula (formula 13) is zero. The vector Y is represented by the following formula (Formula 14).

The vector obtained by the above formula (Formula 14), that is,

From the vector [Y] defined by the above equation, the feature point position [M], that is,
The feature point position [M] is extracted as a result.

  In the next step S306, the second feature point tracking process, that is, “feature point tracking # 2 (feature point position known)” is executed. Tracking is performed more robustly using the feature point position information in the feature point optical flow detection method based on the general template window matching described above.

The feature point periphery of the image frame is cut out, and a template image having a preset window size, for example, a 21 × 21 template image is acquired for a 360 × 240 image. Up to this point, the process is the same as the first tracking process in step S304. In step S306, template matching is performed on the frame in the future direction to obtain corresponding points. The search center at this time is the feature point position obtained by the feature point three-dimensional position estimation process in step S305, that is,

  Calculation is performed using the feature point position M. For the calculation, the above-described formula (Formula 1) is used. A search center is determined by this calculation, and a search is performed in a window in which the center is set to the search center.

  As the matching score, which is a determination index for the presence / absence of matching, a normalized correlation value can be used as in “feature point tracking # 1” in step S304 described above. If the matching score is small (the correlation value is small), it is determined that there is no feature point in the frame. The image frame data is processed for all frames in the future direction, the feature points are tracked, and the corresponding feature points in each image frame are detected. According to this process, a feature point that has once deviated from the camera frame can be tracked again when it enters the camera frame again.

  In step S307, the three-dimensional position of the corresponding feature point information detected in each image frame by the processing in step S306 is calculated, and the calculated three-dimensional position information of the nuclear feature point is associated with each frame and added to the storage unit 152. Record as point information.

  In step S308, it is determined whether or not the processing of all feature points in the processing target frame has been completed. If there are unprocessed feature points, the processing from step S304 is executed on the unprocessed feature points. If it is determined in step S308 that processing of all feature points in the processing target frame has been completed, the process proceeds to step S309, where it is determined whether processing of all image frames has been completed. Then, the processing from step S301 is executed for an unprocessed image frame. If it is determined in step S309 that all image frames have been processed, the processing ends.

  In this way, the feature point addition process in step S300 shown in FIG. 9 is executed, the initial information acquisition process (SFM) in step S204, and the camera position and orientation and feature point three-dimensional position information acquisition using the extended Kalman filter in step S207. The feature points that are not extracted in the processing (EKF SLAM) are additionally registered in the storage unit 152.

  In the bundle adjustment process (bundle adjustment) in step S209, the bundle adjustment process (both the feature point information added in the feature point addition process in step S300) is used for the feature point information extracted in the EKF SLAM process in step S207. (Bundle Adjustment) will be performed. That is, it is possible to output three-dimensional position information of more feature points than the flow shown in FIG. 7 described above.

  In order to generate a more accurate three-dimensional map, it is advantageous that the number of available feature points is larger, and the three-dimensional map using the feature points added according to the present embodiment is data with higher accuracy.

  When bundle adjustment is performed in a state where the feature points described with reference to the flow of FIG. 7 are not added ”and“ bundle adjustment after adding feature points ”described with reference to the flow of FIG. FIG. 14 shows an example of data of each feature point in the case of

FIG. 14 is based on the photographed image of the wall with the poster shown in FIG.
(B1) Feature point information and camera trajectory without the feature point addition process,
(B2) Feature point information and camera locus when feature point addition processing is performed,
These data are shown.
As shown in (b2), the number of feature points increases, and a more detailed 3D map can be generated by using such many feature points. FIG. 15 shows an example of a 3D map generated using the three-dimensional position information of feature points acquired by performing the feature point addition process.

  FIG. 16 shows a configuration example of the information processing apparatus of the present invention. As described above with reference to FIG. 4, the information processing apparatus 120 inputs, for example, a captured image of the camera 102 held by the moving user 101, for example, an image constituting a moving image, and analyzes the input image Thus, the three-dimensional image information 103 about the object included in the captured image is generated.

  As illustrated in FIG. 16, the information processing apparatus 120 includes an image input unit 501, an initial information generation unit (SFM) 502, a data conversion unit A 503, a feature point position information generation unit (EKF-SLAM) 504, a data conversion unit B 505, A bundle adjustment processing unit 506, a storage unit 507, a 3D map generation unit 508, and a feature point extraction unit 511 are included.

  As described with reference to FIG. 4, the image input unit 501 inputs a captured image of the camera 102 held by the moving user 101, for example, and outputs it to the initial information acquisition unit 502 and the feature point position information acquisition unit 504. . Note that the image frame input from the image input unit 501 is stored in the storage unit 507 directly or through another processing unit.

  In the input image of the image input unit 501, a predetermined initial frame set in advance is output to the initial information acquisition unit (SFM) 502 and then output to the feature point position information acquisition unit 504.

The initial information generation unit (SFM) 502 executes the process of step S204 in the flowcharts of FIGS. That is, SFM (Structure from Motion) processing for analyzing the correspondence of feature points (Landmarks) included in the image using images taken from a plurality of different positions is executed. Similar to the conventionally known SFM processing, the feature point three-dimensional position and the position and orientation information of the image frame including the feature point (camera position and orientation information) are calculated by the feature point association processing and bundle adjustment processing by frame matching. To do. The input information and output information of the initial information generation unit (SFM) 502 are as follows.
Input: Image data with a preset number of frames,
Output: Position and orientation of input image frame and feature point position information This output information is output to the data conversion unit A503 and stored in the storage unit 507 in association with the processed image frame.

The data conversion unit A503 inputs the position and orientation (camera position and orientation) of the input image frame and the feature point position information obtained in the initial information acquisition process (SFM) of the initial information generation unit (SFM) 502, and
Output value: State of initial frame applied in EKF SLAM (= state variable)
Is generated. As described above with reference to FIG. 8B, the state variable includes the camera position, posture, speed, each speed, and position information of a plurality of feature points detected from the captured image of the camera. It is comprised by P1-Pn.
The state information (= state variable) of the initial frame generated by the data conversion unit A503 is input to the feature point position information generation unit (EKF SLAM) 504.

  The feature point position information generation unit (EKF SLAM) 504 executes the process of step S207 in the flowcharts of FIGS. That is, the camera position and orientation and feature point three-dimensional position information acquisition process (EKF SLAM) to which the extended Kalman filter is applied is executed.

  As described above, in the process to which the extended Kalman filter (EKF) is applied, for example, as described above with reference to FIG. 8B, the camera position, posture, speed, each speed, and further, from the captured image of the camera. A state variable constituted by position information P1 to Pn of a plurality of detected feature points is sequentially updated to obtain a state variable corresponding to each processed image frame. In order to start the processing using the extended Kalman filter (EKF), it is necessary to hold in advance a state variable indicating the position of the camera or the feature point as an initial value. Receiving from the data converter A503, processing is started.

Input information and output information in the feature point position information generation unit (EKF SLAM) 504 are as follows.
Input: State of feature points and initial frame, that is, state variables (see FIG. 8) consisting of multi-dimensional normal distribution data of camera position and orientation information and feature point position information,
Output: State variables corresponding to each image frame, feature point tracking data,
The feature point tracking data can be calculated from the feature point position information included in the state variable corresponding to each image frame.
The output information [state variable and feature point tracking data corresponding to each image frame] generated in the processing (EKF SLAM) of the feature point position information generation unit (EKF SLAM) 504 is stored together with the processing target image frame. Stored in

The data conversion unit B505 takes out the generated data of the feature point position information generation unit (EKF SLAM) 504 stored in the storage unit 507, and performs conversion processing to data used by the bundle adjustment processing unit (Bundle Adjustment) 506. . The input / output data of the data conversion unit B505 is as follows.
Input: State variables (multidimensional normal distribution) corresponding to each image frame and feature point tracking data (stored data in the storage unit 507)
Output: camera position and feature point position corresponding to each image frame

  The data conversion unit B505 uses the state variables (multidimensional normal distribution) corresponding to each image frame that is stored data in the storage unit 507 to obtain the camera position and orientation information of each image frame, and the feature point position is the latest frame. Is obtained from the state (multidimensional normal distribution). As described above, the state variable is multi-dimensional normal distribution data indicating the state probability of each variable as a normal distribution, and the value corresponding to the average of the normal distribution data has the highest probability. For the posture and the feature point position, an average value of each normal distribution data of the state variable is adopted.

The data generated by the data conversion unit B505 is provided to a bundle adjustment processing unit (bundle adjustment) 506, and the bundle adjustment processing unit (bundle adjustment) 506 performs bundle adjustment processing using this input information.
The input / output data of the bundle adjustment processing unit (bundle adjustment) 506 is as follows.
Input: Camera position and orientation and feature point position of each image frame, and feature point tracking data Output: Camera position and orientation and feature point position of each image frame

  As described above with reference to FIG. 3, the bundle adjustment process is performed by processing each image frame by converging the three-dimensional positions of corresponding feature points included in a plurality of images captured from different positions into one position. This is a process for obtaining the camera position and orientation and the feature point position. The camera posture position information and the feature point position information obtained by this processing are stored in the storage unit 507 and further provided to the 3D map generation unit 508. The 3D map generation unit 508 generates a 3D map using these feature point information and camera trajectory information. For example, a 3D map as shown in FIG. 15 is generated.

  The feature point extraction unit 511 is configured to execute the feature point addition processing described with reference to FIGS. 9 and 10. The feature point extraction unit 511 executes processing according to the flow shown in FIG. SFM) 502 and feature point position information generation unit (EKF-SLAM) 504 detect feature points that are not extracted, calculate the three-dimensional position information and position information in each image frame, and store them in storage unit 507. Register additional.

  When this process is executed, the bundle adjustment processing unit (bundle adjustment) 506 includes the feature points extracted by the initial information generation unit (SFM) 502 and the feature point position information generation unit (EKF-SLAM) 504, and further Then, bundle adjustment processing (Bundle Adjustment) is executed using both of the feature points newly extracted by the feature point extraction unit 511, and three-dimensional position information of more feature points is output. As described above with reference to FIG. By this process, a more accurate three-dimensional map can be generated.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present invention. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims should be taken into consideration.

  The series of processing described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run. For example, the program can be recorded in advance on a recording medium. In addition to being installed on a computer from a recording medium, the program can be received via a network such as a LAN (Local Area Network) or the Internet, and installed on a recording medium such as a built-in hard disk.

  Note that the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

  As described above, according to the configuration of one embodiment of the present invention, in the configuration for acquiring the three-dimensional position of the feature point based on the image acquired by the camera, only a plurality of preceding image frames in the image frame of the camera-captured image. Processing to obtain 3D position information of feature points as initial information by corresponding feature point analysis using, and set this initial information as state information for the initial image frame and apply an extended Kalman filter (EKF) to the subsequent image frame As a result of the above processing, the configuration is such that the three-dimensional position information of the feature points in the subsequent frame is acquired. For example, the feature point extraction processing involving frame matching or the like only needs to be performed on the previous image frame, which is efficient. Acquisition of three-dimensional position information of feature points and generation of three-dimensional image data are realized.

It is a figure which shows the flowchart explaining an example of the production | generation process sequence of a three-dimensional map (3D map). It is a figure which shows the flowchart explaining an example of the process sequence which acquires the feature point information and camera locus information which are contained in an image frame. It is a figure explaining an example of a bundle adjustment process (Bundle Adjustment). It is a figure explaining the outline | summary of the process of this invention. It is a figure which shows the flowchart explaining the sequence of the process according to one Example of this invention. It is a figure explaining the specific sequence of the process according to one Example of this invention. It is a figure which shows the flowchart explaining the detailed sequence of the process according to one Example of this invention. It is a figure explaining the state variable applied in the process which applied the extended Kalman filter (EKF). It is a figure which shows the flowchart explaining the detailed sequence of the process according to one Example of this invention. It is a figure which shows the flowchart explaining the detailed sequence of the feature point extraction process according to one Example of this invention. It is a figure explaining the feature point extraction process using a Harris corner detector (Harris Corner Detector). It is a figure explaining the meaning of the formula which shows the correspondence of the pinhole camera model, ie, the position expressed by the camera coordinate system, and the three-dimensional position of the object in the world coordinate system. It is a figure explaining the meaning of the formula which shows the correspondence of the pinhole camera model, ie, the position expressed by the camera coordinate system, and the three-dimensional position of the object in the world coordinate system. It is a figure explaining the process example when not performing the feature point addition process and when performing. It is a figure explaining the process example at the time of performing a feature point addition process. It is a figure explaining the structural example of the information processing apparatus which concerns on one Example of this invention.

Explanation of symbols

11-13 Photographed image 21-23 Feature point 101 User 102 Camera 103 Three-dimensional image information 120 Information processing device 131 Sparse three-dimensional map 132 Dense three-dimensional map 170 Acquisition image 180-182 Harris corner image 190-192 Laplacian image 501 Image input unit 502 Initial information generation unit (SFM)
503 Data conversion part A
504 Feature point position information generator (EKF-SLAM)
505 Data converter B
506 Bundle adjustment processing unit 507 Storage unit 508 3D map generation unit 511 Feature point extraction unit

Claims (17)

An information processing apparatus that calculates a three-dimensional position of a feature point included in an image,
An initial information generation unit that inputs a plurality of preceding image frames in an image frame of a camera-captured image and acquires three-dimensional position information of feature points by analyzing corresponding feature points of each image frame;
The feature point position information acquired from the preceding image frame by the initial information generating unit is set as state information for the initial image frame, and 3 of feature points in the succeeding frame is processed by applying an extended Kalman filter (EKF) to the succeeding image frame. A feature point position information generation unit for acquiring dimension position information;
An information processing apparatus comprising: The information processing apparatus further includes:
2. The information processing according to claim 1, further comprising a bundle adjustment processing unit that inputs the feature point position information generated by the feature point position information generation unit and executes a process of correcting the three-dimensional position information of the feature point. apparatus. The initial information generator is
The information according to claim 1, wherein an SFM (Structure from Motion) process for analyzing a correspondence between feature points included in each image frame is performed using images taken from a plurality of different positions. Processing equipment. The initial information generator is
By analyzing the preceding image frame, it is configured to calculate position and orientation information of the camera that captured the image frame,
The feature point position information generation unit
Processing in which the initial information generation unit sets feature point position information and camera position and orientation information calculated from the preceding image frame as state information corresponding to the initial image frame, and applies an extended Kalman filter (EKF) to the subsequent image frame The information processing apparatus according to claim 1, wherein three-dimensional position information of a feature point in a subsequent frame and camera position and orientation information obtained by capturing the subsequent frame are acquired. The feature point position information generation unit
Capturing 3D position information and subsequent frames of feature points in subsequent frames by setting multidimensional normal distribution data including feature point position information and camera position and orientation information as state information and applying an extended Kalman filter (EKF) 5. The information processing apparatus according to claim 4, wherein the camera position / orientation information obtained is acquired. The information processing apparatus further includes:
A feature point extraction unit that executes a process of calculating a three-dimensional position of an additional feature point extracted by extracting a feature point that has not been extracted by the initial information generation unit and the feature point position information generation unit as an additional feature point; Have
The bundle adjustment processing unit
The feature point position information generated by the feature point position information generation unit and the feature point position information of the additional feature point extracted by the feature point extraction unit are input, and correction processing of the three-dimensional position information of the feature point is executed. The information processing apparatus according to claim 2, wherein the information processing apparatus is provided. The feature point extraction unit includes:
The feature point extracted from the image frame and the feature point extracted by the initial information generation unit and the feature point position information generation unit are determined to overlap, and only new feature points that do not overlap are selected as additional feature points. The information processing apparatus according to claim 6, wherein processing is executed. The information processing apparatus further includes:
The information processing apparatus according to claim 1, further comprising: a 3D map generation unit that generates 3D image data using the feature point position information generated by the feature point position information generation unit. An information processing method for calculating a three-dimensional position of a feature point included in an image in an information processing device,
An initial information generating step for inputting a plurality of preceding image frames in an image frame of a camera-captured image and acquiring three-dimensional position information of the feature points by corresponding feature point analysis of each image frame;
The feature point position information generation unit sets the feature point position information acquired from the preceding image frame by the initial information generation unit as state information for the initial image frame, and applies an extended Kalman filter (EKF) to the subsequent image frame. A feature point position information generation step for acquiring three-dimensional position information of the feature points in the subsequent frame;
An information processing method characterized by comprising: The information processing method further includes:
The bundle adjustment processing unit includes a bundle adjustment processing step of inputting the feature point position information generated in the feature point position information generation step and executing a correction process of the three-dimensional position information of the feature point. 9. The information processing method according to 9. The initial information generating step includes
The information according to claim 9, wherein the information is a step of performing SFM (Structure from Motion) processing for analyzing correspondence between feature points included in each image frame using images taken from a plurality of different positions. Processing method. The initial information generating step includes
Calculating the position and orientation information of the camera that captured the image frame by analyzing the preceding image frame;
The feature point position information generation step includes:
Processing in which the initial information generation unit sets feature point position information and camera position and orientation information calculated from the preceding image frame as state information corresponding to the initial image frame, and applies an extended Kalman filter (EKF) to the subsequent image frame The information processing method according to claim 9, further comprising: acquiring three-dimensional position information of feature points in the subsequent frame and camera position and orientation information obtained by capturing the subsequent frame. The feature point position information generation step includes:
Capturing 3D position information and subsequent frames of feature points in subsequent frames by setting multidimensional normal distribution data including feature point position information and camera position and orientation information as state information and applying an extended Kalman filter (EKF) The information processing method according to claim 12, wherein acquired camera position and orientation information is acquired. The information processing method further includes:
The feature point extraction unit executes a process of extracting a feature point that has not been extracted by the initial information generation unit and the feature point position information generation unit as an additional feature point and calculating a three-dimensional position of the extracted additional feature point A feature point extracting step,
The bundle adjustment processing step includes:
Inputting the feature point position information generated by the feature point position information generation unit and the feature point position information of the additional feature point extracted by the feature point extraction unit, and executing correction processing of the three-dimensional position information of the feature point The information processing method according to claim 10. The feature point extraction step includes:
The feature point extracted from the image frame and the feature point extracted by the initial information generation unit and the feature point position information generation unit are determined to overlap, and only new feature points that do not overlap are selected as additional feature points. The information processing method according to claim 14, wherein processing is executed. The information processing method further includes:
The information processing method according to claim 9, wherein the 3D map generation unit includes a 3D map generation step of generating 3D image data using the feature point position information generated by the feature point position information generation unit. . In the information processing device, a computer program for calculating a three-dimensional position of a feature point included in an image,
An initial information generating step of inputting a plurality of preceding image frames in the image frames of the camera-captured image to the initial information generating unit and acquiring the three-dimensional position information of the feature points by analyzing the corresponding feature points of each image frame;
In the feature point position information generation unit, the feature point position information acquired from the preceding image frame by the initial information generation unit is set as state information for the initial image frame, and an extended Kalman filter (EKF) for the subsequent image frame is applied. A feature point position information generation step for acquiring three-dimensional position information of the feature points in the subsequent frame;
A computer program characterized by comprising: