JP2019070944A - Information processing device, method and program - Google Patents

Abstract

To provide an information terminal device capable of estimating a relative positional relationship with an imaging object in a wide range from a picked-up image.SOLUTION: An information processing device comprises: an imaging part 1 imaging an object to obtain a picked-up image; a calculation part 2 calculating feature information from the picked-up image; a storage part 5 storing the feature information in association with position information; an estimation part 3 obtaining feature information to be determined to be similar with the calculated feature information by referring the storage part 5 and obtaining relative position information of a pair of the feature information as a conversion relationship from coordinate information included in the feature information determined to be similar with the calculated feature information to coordination information included in the calculated feature information; and a synthesis part 4 obtaining position information of the picked-up image by synthesizing the relative position information with the position information of the feature information determined to be similar with the calculated feature information and making the storage part 5 store the obtained position information in association with the feature information calculated from the picked-up image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information terminal device, method, and program capable of broadly estimating the relative positional relationship with an imaging target from a captured image.

  The technology for determining an object from an image can convert analog information described in a medium that can be easily distributed or presented into digital information, and can improve user convenience. As the said technique, the thing of the nonpatent literature 1 is open | released, for example. In Non-Patent Document 1, the feature point is detected from the image, the feature amount is calculated from the periphery of the feature point, and then the type and the relative positional relationship of the object are compared with the feature amount stored in advance. Identify

  On the other hand, as a technique for further improving the accuracy with regard to the determination based on the above-described feature point and feature amount, for example, the one disclosed in Patent Document 1 is disclosed. In Patent Document 1, a rotation angle around the optical axis of the photographing apparatus is determined for a photographing object in a captured image, and orientation information calculated by the feature point extraction unit based on the rotation angle and an orientation stored in advance. When the coordinate system with the information is aligned, the accuracy is improved by setting feature points whose orientation information falls within a predetermined shift angle range as feature points to be compared.

Unexamined-Japanese-Patent No. 2014-109841

D. G. Lowe, `` Object recognition from local scale-invariant Features, '' Proc. Of the International Conference on Computer Vision (ICCV), pp. 1150-1157, 1999.

  However, in the conventional techniques such as Non-Patent Document 1 and Patent Document 1 as described above, there is a problem that the recognizable angle and distance are limited.

  Specifically, there is a problem that when imaging is performed obliquely, the feature amount changes due to projection distortion, and the determination processing does not function at all if the feature amount registered in advance can not match. In addition, even when the imaging target is imaged from a distance, the partial region used for feature amount calculation is relatively wide, so the feature amount changes and it is determined that the feature amount registered in advance can not match There is a problem that it does not function at all. Furthermore, there is a problem that when the imaging target deviates from the imaging range, the conventional method can not be recognized at all.

  An object of the present invention is to provide an information terminal device, a method and a program capable of estimating a relative positional relationship with an imaging target from a captured image in a wide range in view of the problems of the prior art.

  In order to achieve the above object, the present invention is an information terminal device, and corresponds to an imaging unit for capturing an object to obtain a captured image, a calculation unit for calculating feature information from the captured image, feature information and position information The characteristic information determined to be similar to the calculated characteristic information is determined with reference to the storage unit to be stored and the storage unit, and the calculation is performed from the coordinate information included in the characteristic information determined to be similar An estimation unit for obtaining relative position information of the feature information as a conversion relationship to coordinate information included in the extracted feature information, and combining the position information of the feature information determined to be similar to the relative position information And a combining unit configured to obtain position information of the captured image and to store the obtained position information in the storage unit in association with feature information calculated from the captured image. The features. Further, the method is a method and a program corresponding to the information terminal device.

  According to the present invention, it is possible to estimate the positional relationship over a wide range by sequentially storing the feature information and the position information at the acquired place.

It is a functional block diagram of an information processor concerning one embodiment. It is a flowchart of operation | movement of the information processing apparatus which concerns on one Embodiment. It is a schematic example of the imaging range in each time by an imaging part. 5 is a schematic example of feature information and position information corresponding to the example of FIG. 3; It is a schematic example in the case of applying threshold value judgment of the degree of similarity in the example similar to FIG. It is a schematic example for demonstrating the positional information on every several path | route. It is a schematic example in the case of limiting to specific feature information in the example of FIG.

  FIG. 1 is a functional block diagram of an information processing apparatus according to an embodiment. As illustrated, the information processing apparatus 10 includes an imaging unit 1, a calculation unit 2, an estimation unit 3, a combining unit 4, a storage unit 5, and a processing unit 6. As the information processing apparatus 10, any information terminal including the imaging unit 1 can be used, and in addition to the portable terminal, a tablet type terminal, a desktop type or laptop type computer, and the like can be used. In addition, a part or all of the units other than the imaging unit 1 may be installed in the server, and information may be appropriately exchanged by communication. The outline functions of each part are as follows.

  The imaging unit 1 captures an image at each time t under a predetermined frame rate, and outputs the obtained captured image to the calculating unit 2. The calculation unit 2 calculates feature information from the captured image and outputs the feature information to the estimation unit 3. The estimation unit 3 includes a plurality of stored feature information obtained by referring to the storage unit 5 (referred to as reference feature information for clarification of the description of the determination process and the like in the estimation unit 3). In any case, it is determined whether the feature information of the captured image at the current time t obtained from the calculation unit 2 (referred to as query feature information for clarification of the description of the determination process and the like) corresponds. The processing is performed, and the reference feature information and the query feature information determined to correspond to each other are output to the storage unit 5, and the relative relationship as the spatial correspondence between the query feature information and the reference feature information in the matching result The position information is calculated and output to the combining unit 4.

  The combining unit 4 combines the relative position information obtained by the estimating unit 3 and the position information corresponding to the reference feature information determined as corresponding by the estimating unit 3 obtained with reference to the storage unit 5. By this, the position information of the captured image at the current time t is calculated. The calculated position information is used as the first output from the information processing apparatus 10 and is also output to the storage unit 5 and the processing unit 6.

  The storage unit 5 stores the feature information and the position information in association with each other, thereby providing information necessary for the estimation unit 3 and the combination unit 4 for reference. The storage unit 5 further performs update management of the information to be stored. Specifically, the query feature information of the captured image of the current time t obtained from the estimation unit 3 and the position information of the captured image of the current time t obtained from the combining unit 4 are associated with each other, A storage target in itself is additionally updated as feature information and position information (as a reference). Further, the storage unit 5 may keep the amount of data in an appropriate range by selection of the information to be stored, or may have a hierarchical structure with respect to feature information and position information stored in association with each other. The definition processing may be accelerated in the estimation unit 3 by defining the definition.

  In the storage unit 5, when storing the association, information of similarity correspondence between feature information (generally given as route information) such as the schematic example will be described with reference to FIG. 4 and FIG. 5. Information) can be stored, and the information on the correspondence can be provided to the combining unit 4 for reference.

  The processing unit 6 can perform any additional processing using the position information of the captured image at the current time t obtained from the combining unit 4. For example, an augmented reality display in which predetermined information is superimposed on the captured image at the current time t is performed, and the position and orientation at which the superimposition information is displayed is controlled by using position information at the time of the augmented reality display. You may do it. The processing unit 6 may be omitted in the information processing apparatus 10.

  FIG. 2 is a flowchart of the operation of the information processing apparatus 10 according to an embodiment. In step S1, the time t is set to the present time (initial time t = 0), and the process proceeds to step S2. In step S2, the imaging unit 1 performs imaging at the current time t, and the process proceeds to step S3. In step S3, the calculating unit 2, the estimating unit 3 and the combining unit 4 respectively perform the processing as outlined above on the captured image of the current time t obtained in step S2, so that the combining unit 4 performs the current time t. After the position information of the captured image is obtained, the process proceeds to step S4. In step S3, the processing unit 6 may perform additional processing such as augmented reality display based on the position information of the captured image at the current time t obtained by the combining unit 4.

  In step S4, the storage unit 5 newly stores feature information obtained by the calculation unit 2 and position information of the captured image obtained by the combining unit 4 with respect to the captured image of the current time t in step S3. Then, after the update process of the information stored in the storage unit 5 is performed, the process proceeds to step S5. In step S4, the storage unit 5 may further update the information stored therein based on the current time t or the like. For example, it is possible to delete stored information that satisfies a condition such as a predetermined time or more has elapsed since it has been obtained (that is, obtained in the past step S4 in the flow of FIG. 2 having a loop configuration). It is also good.

  If, for example, time t = 0 (initial time) and no past information is stored in storage unit 5, then steps S3 and S4 may be changed as follows as they correspond to initial registration. Good. That is, in step S3, the estimation unit 3 omits the process, and causes the feature information obtained by the calculation unit 2 to be output to the combining unit 4 as it is. In step S3, the combining unit 4 outputs the feature information output as it is to the storage unit 5 as the initial position. In step S4, the storage unit 5 stores the output feature information as corresponding to the initial position Pinit. Thus, the position information obtained by the combining unit 4 in step S3 at time t11 thereafter is obtained as the position information based on the initial position Pinit.

  In step S5, the current time t is updated to the next time t + 1, and then the process returns to step S2, and similarly for the subsequent times t + 1, t + 2, t + 3,. The processes up to the above are similarly continued by the loop configuration. By the continuation, the information processing apparatus 10 can calculate position information of a captured image in real time at each time t.

  The details of the processing of each part of the information processing apparatus 10 of FIG. 1 operating as shown in FIG. 2 will be described below.

  The imaging unit 1 captures an imaging target and outputs the captured image to the calculation unit 2. Here, it is assumed that the user or the like performs imaging by the imaging unit 1 so that a previously known target estimated by the estimation unit 3 is included. As hardware for realizing the imaging unit 1, it is possible to use a digital camera or the like that is standardly installed in a portable terminal.

  The calculation unit 2 receives the captured image from the imaging unit 1 and outputs the feature amount calculated from the captured image to the estimation unit 3 as feature information. Specifically, an existing method such as Scale-Invariant Feature Transform (SIFT) or Speeded Up Robust Features (SURF) can be used as the feature amount to be calculated.

  The estimation unit 3 searches the reference feature information for feature information similar to the query feature information by comparing the query feature information calculated by the calculation unit 2 with the reference feature information stored in the storage unit 5. At the same time, the relative positional relationship between the reference feature information determined to be similar in the search result and the query feature information is output as the position information.

  That is, the estimation unit 3 identifies to which of the reference feature information stored in the storage unit 5 the query feature information has a correspondence relationship, and the relative position relationship in the correspondence relationship of the captured image Estimate the quantity. Here, all or part of the past reference feature information stored in the storage unit 5 is to be subjected to the similarity determination. For example, as schematically illustrated as rectangular ranges R0 to R7 in FIG. 3, while the imaging unit 1 is moving, images are captured by capturing the ranges R0 to R7 at times t = 0 to 7, respectively. If at the latest time t = 7, all or part of the reference feature information obtained and stored in each captured image of the range R0 to R6 at the past time t = 0 to 6 until that time is t7 Become.

  That is, more generally, it is already calculated (as a query in step S3 of FIG. 2) at time 0 to t-1 in the past from the query feature information Ft of the current time t calculated by the calculation unit 2 A part or all of the reference feature information F0 to Ft-1 stored as a reference in S4 is input from the storage unit 5, and the estimation unit 3 searches for one similar to the query feature information Ft. Here, the reference feature information determined to be similar may be only one determined to be the most similar, or all of a plurality of pieces of reference feature information for which the degree of similarity is equal to or more than a certain level is adopted as a result of the similarity determination. May be

  Here, when comparing the pieces of feature information to each other in order to make a similarity determination, various known methods can be used. For example, each feature constituting the feature information by RANSAC (Random Sample Consensus) or the like. A method may be used to identify the most matching one as a whole by excluding outliers while attempting to individually match points and feature quantities individually. Alternatively, the similarity between histograms of BoVW (Bag of Visual words) may be calculated.

  Then, when the reference feature information Fa similar to the query feature information Ft is obtained as a search result, the estimation unit 3 further estimates information Pat of the relative positional relationship from the reference feature information Fa to the query feature information Ft.

  In one embodiment, the relative positional relationship to be estimated is represented by a known similarity transformation matrix, an affine transformation matrix, a projective transformation matrix, or the like from the correspondence between coordinates of a plurality of feature points constituting feature information. It can be determined as For example, in the case of a projective transformation matrix, it is obtained by numerical calculation as a relationship in which feature point coordinates (u, v) of four or more points on an image (image in which coordinates of feature points in feature information are arranged) are mutually converted The translational component and the rotational component included in the plane projection transformation matrix may be set as a relative positional relationship. The correspondence relationship between feature points may be obtained when the estimation unit 3 determines similarity.

  Further, in another embodiment, based on the correspondence relationship of similar feature points, a basic matrix or a basic matrix is estimated using known motion parallax or stereo parallax, and a translation is obtained by decomposing the matrix. The component and the rotational component may be set as the relative positional relationship between the feature information.

  In addition, when calculating | requiring relative positional relationship by a projective transformation matrix etc., it is desirable for the imaging target from which feature information is extracted to be on one plane. For example, it is desirable that the imaging target be installed on one wall of a building. When estimating a base matrix or a base matrix, it is not necessary to be on one plane like this.

  FIG. 4 is similar to FIG. 3 in the case where the imaging unit 1 captures an image R as capturing each of the ranges R0 to R7 at times t = 0 to 7 while moving while the imaging unit 1 moves, as shown in FIG. The typical example of the characteristic information judged and the relative position information computed from the judgment result concerned is shown. Note that FIG. 4 also shows a schematic example of the information ST0 to ST7 (including the information of the route) which is updated and stored in the storage unit 5 in the same setting.

  The example of FIG. 4 is an example corresponding to the case of obtaining one piece of reference feature information determined to be most similar by the estimation unit 3. At time t = 0, the characteristic information F0 is obtained and stored as it is in the storage unit 5, as shown as the information ST0. Here, the time t = 0 corresponds to immediately after the start of the flow of FIG. 2 and the information stored as the past result does not exist in the storage unit 5, and thus stored as it is as the information ST0 Be done.

  Further, in FIG. 4, at time t = 1, query feature information F1 is obtained, it is determined that it is similar to reference feature information F0, relative position information P01 is calculated, and updated as information ST1 and stored as a reference Ru. At time t = 2, query feature information F2 is obtained, it is determined that it is similar to reference feature information F0, relative position information P02 is calculated, and it is updated as information ST2 and stored as a reference. At time t = 3, query feature information F3 is obtained, it is determined that it is similar to the reference feature information F2, relative position information P23 is calculated, and updated as information ST3 and stored as a reference. At time t = 4, query feature information F4 is obtained, it is determined that it is similar to the reference feature information F1, relative position information P14 is calculated, and updated as information ST4 and stored as a reference. At time t = 5, query feature information F5 is obtained, it is determined that it is similar to the reference feature information F4, relative position information P45 is calculated, and updated as information ST5 and stored as a reference. At time t = 6, query feature information F6 is obtained, it is determined that it is similar to the reference feature information F5, relative position information P56 is calculated, and updated as information ST6 and stored as a reference. At time t = 7, query feature information F7 is obtained, it is determined that it is similar to the reference feature information F3, relative position information P37 is calculated, and updated as information ST7 and stored as a reference.

  FIG. 5 shows an example corresponding to the case where reference feature information not necessarily limited to one is obtained by threshold determination of similarity by the estimation unit 3 in the case similar to the example of FIG. 4, the information ST70 is obtained at time t = 7. Are shown as being That is, in addition to the example of FIG. 4, additionally, at time t = 6, the relative position is added because there is not only the reference feature information F5 but also the reference feature information F4 as the one determined to be similar to the query feature information F6. Information P46 has been calculated. Further, in addition to the example of FIG. 4, at time t = 7, not only the reference feature information F3 but also the reference feature information F2 is determined to be similar to the query feature information F7, so that relative positional information is additionally added. P27 is calculated.

  As described above, the path information between the feature information is defined by combining the individual similar correspondence relationships between the feature information, such as the schematic examples shown as the information ST0 to ST7 and the information ST70 in FIGS. 4 and 5. Information can be stored while being updated in the storage unit 5, and in one embodiment, the similar correspondence relationship as the route information can be provided as a reference for calculating position information in the combining unit 4.

  The route information is, for example, a route between feature information “F0 → F1 → F4 → F5 → F6” obtained by combining the individual correspondence relationships P01, P14, P45, and P56 in the case of the information ST7 in FIG. This is a path between feature information “F0 → F2 → F3 → F7” obtained by combining individual correspondences P02, P23, and P37. That is, if there is a correspondence Pij (Fi → Fj) and Pjk (Fj → Fk), these are combined to obtain a partial path "Fi → Fj → Fk" of the corresponding feature information, and such a partial Route information can be obtained by connecting various routes. The route can be defined as the feature information associated with the reference position in the information stored in the storage unit 5 as the starting point, and if the information ST7 in FIG. Since it is set as corresponding to the position, the route information as described above can be obtained.

  In addition, whenever new feature information Ft is obtained at each time t in the storage unit 5, the route information is also updated by the update in step S 4 of FIG. It is possible to always update and manage the route information to the latest state.

  The combining unit 4 compares the position information of the query feature information calculated by the calculating unit 2 with the position information of the reference feature information whose similarity is determined by the estimating unit 3 and the query feature information calculated by the calculating unit 2. The information is calculated by combining the information, and the query feature information and the calculated position information are associated with each other and output to the storage unit 5. Thus, in the storage unit 5, the reference feature information is used for the similarity determination process and the like in the estimation unit 3 of the future times t + 1, t + 2,... As the position information in association with the position information. The schematic example to be stored is as shown in FIG.

  Each embodiment of the method of combining the position information in the combining unit 4 can be as follows. In the first embodiment, the position information of the feature information Fb (query feature information Fb) for the feature information Fa can be calculated as the product of the position information on the shortest route of the feature information Fa and the feature information Fb. Here, route information stored in the storage unit 5 may be used. The shortest path may be determined as the path number in the case where the shortest path is reached via the feature information whose similarity is determined from the feature information Fa to the feature information Fb on the route information. For example, in the case of obtaining position information for the feature information F0 of the feature information F6 using the route information of the information ST70 of FIG. 5, the position information can be calculated as the product P01 * P14 * P46. Here, there are two routes to reach from F0 to F6, “F0 → F1 → F4 → F6” and “F0 → F1 → F4 → F5 → F6”, the former has 3 paths and the latter is the latter Since the number of paths is 4, the product P01 * P14 * P46 is calculated by the former which is the shortest path.

  In the second embodiment, even if the positional relationship between the feature information Fn corresponding to the reference position and the reference feature information Fs determined to be similar is stored as the position information Pns in the storage unit 5, the position information can be calculated. Good. Here, the storage of the position information Pns is performed when the reference feature information Fs determined to be similar at the current time t is stored for the first time in the past time s (s <t), that is, the position as query feature information Fs at the past time s. The information Pns may be stored in the storage unit 5 in step S4 of FIG. 2 when the information Pns is obtained.

  Then, the specific positional information is calculated as the product Pns * Pst after obtaining the relative positional relationship between the similar feature information Fs and the query feature information Ft calculated by the calculation unit 2 as the conversion matrix Pst. be able to.

  In the third embodiment, position information Pab may be obtained as a weighted average of position information calculated by multiplying the position information by each of a plurality of routes from feature information Fa to feature information Fb. Here, the product of the position information of each of the plurality of paths may be obtained according to the first or second embodiment. Here, the weighting factor is the sum of all the routes set for each path of the route using the distance of the route, the similarity in the estimation unit, the duplication of the storage unit 5 described later, etc. Just do it. The distance of the path of one pass “Fi → Fj” can be determined as the size of the translation component of the position information Pij, and the similarity of one path “Fi → Fj” should be determined as the similarity of the feature information Fi and Fj. Can. The weight to be obtained may be calculated when the estimation unit 3 first processes the data as query feature information, and may be stored in the storage unit 5.

  Furthermore, in the third embodiment, the product of the position information for each of a plurality of paths may be stored in the storage unit 5 and may be used for calculation when it is referred to in the future. For example, as shown in [1] of FIG. 6, a captured image is acquired for the range R0 to R3 at time 0 to 3 and feature information F0 to F3 and similar relationships among them as shown in [2] of FIG. If the path information ST30 is obtained as the path information ST30, a plurality of paths exist for the position information for the feature information F0 of the feature information F2, and P02 and P01 * P12 are obtained. When calculating position information for feature information F0 of feature information F3 by holding both in storage unit 5, among the plurality of routes, the route to feature information F2 has already been calculated, and is therefore reused. This can reduce the calculation load of position information.

  Note that the ranges R0 to R3 and the feature information F0 to F3 and the like in the example of FIG. 6 are independent of the examples of FIGS.

Furthermore, in the third embodiment, a path may be stored in the storage unit 5 to trace back the relative positional relationship between feature information. In the case of [2] in FIG. 6, P01 = P02 * (P12) ^-1 is associated with feature information F1 and held, and P03 = P02 * (P12) ^-1 * for feature information F3. When calculating P13, the position information held by the feature information F1 can be reused.

  In the above description of the first to third embodiments, assuming that position information Pij corresponding to one pass “Fi → Fj” is determined by a matrix corresponding to plane projective transformation etc., a combination of these is determined by a product It explained as a thing. Here, when the path is traced in the reverse direction, the inverse matrix may be multiplied. If position information P ij is estimated as a basic matrix or basic matrix and is obtained as a translational component and a rotational component obtained by decomposing the matrix, a composition can be obtained by adding the translational component and the rotational component. Good. Here, if the path is traced in the reverse direction, subtraction may be performed.

  The storage unit 5 accumulates and stores the feature information and the corresponding position information obtained from the combining unit 4 at each time t, and performs management update, so that the following time t + 1, t + 2,. This is provided as a reference to the estimation unit 3. Here, at the initial time t = 0, the storage unit 5 is set as a state in which nothing is stored, and the initial position Pinit is set at steps S3 and S4 at the initial time t = 0 as described in the flow of FIG. The feature information corresponding to may be stored (initially registered) for the first time. Also, by storing in advance in the storage unit 5 one or more pairs of the initial position Pinit to be stored and the corresponding feature information, steps S3 and S4 are initially registered even if the initial time t = 0. Instead, it may be processed as usual.

  Regarding processing for providing feature information from the storage unit 5 to the estimation unit 3, in one embodiment, all or some of the features stored at the current time t (that is, all obtained as query feature information in the past time) The information may be provided only once, and the estimation unit 3 may obtain the estimation result at one time, or in another embodiment, the feature information is provided in stages and the estimation unit 3 obtains the estimation results in stages. May be determined.

  When providing at once, it is possible to provide all of the information obtained as query feature information and stored as reference feature information in the past time to the estimation unit 3, or from among all of them, limited to unique feature information. It is also possible to provide only to the estimation unit 3.

  For example, when the current time is t = 7 and imaging is performed for the range R0 to R7 as in FIG. 3 described above at time t = 0 to 7, the bold frame is displayed in FIG. 7 corresponding to FIG. As schematically shown, instead of providing all the feature information of the past time t = 0 to 6 to the estimation unit 3, the feature information of the time t = 0, 3, 4, 6 corresponding to the unique feature information It can be provided limited to

  Specifically, for example, feature information having a feature information duplication ratio equal to or less than a threshold may be defined as unique feature information, and limited to this. Here, the overlapping ratio may be evaluated as a ratio of the number of feature points determined to be corresponding among feature points in feature information, or may be subjected to planar projection conversion corresponding to relative position information The image range corresponding to the feature information may be evaluated as the overlapping ratio with the other image range when converted to the coordinates (u, v) of the other image.

  Alternatively, k-means (k-means method) or the like is applied to space coordinate values of position information corresponding to feature information to obtain a plurality of centroid positions, and feature information having position information closest to the centroid position Only may be defined as unique. Alternatively, it may be simply decimated, for example, by setting as the specific information only one whose remainder when the obtained time t of the feature information Ft is divided by a predetermined divisor N is a predetermined value.

  When unique feature information is set, the obtained time t may be set as unique feature information in preference to a new one. In this case, in addition to the evaluation of the overlapping ratio as described above, the characteristic information may be set as the unique feature information by performing an evaluation based on the elapsed time from the obtained time t to the current time. Alternatively, when it is determined that the duplication ratio is large with the feature information Fold that already constitutes unique feature information, but there is feature information Fnew that is newer than that which is already unique feature information by a predetermined time or more Alternatively, the unique feature information may be updated by replacing the old feature information Fold with the new feature information Fnew.

  It should be noted that the determination of which of the pieces of feature information stored in itself becomes unique may be performed by the storage unit 5 at the time of updating of step S4 in FIG.

  In the case of providing in stages, for example, it is provided in two stages, and in the first stage, only the above unique feature information is provided and the estimation unit 3 determines the most similar one, and the second stage In this case, the feature information determined to be close to the feature information determined most similar in the first step is additionally provided, and the feature information not provided in the first step is additionally provided. What is determined to be most similar to the one provided additionally at the stage may be taken as the final similarity determination result in the estimation unit 3. Here, the determination of “close” feature information to be additionally provided in the second step may be determined by the same scale (proximity of position information or the ratio of overlapping) as in the case of limiting unique feature information. . The determination may be performed in three or more stages by providing a hierarchical structure in the feature information stored in the storage unit 5 in exactly the same manner. The management of the hierarchical structure may also be performed when the storage unit 5 updates the step S4 in FIG. At this time, the hierarchical structure or the like may be updated and managed with respect to the additional portion corresponding to the new feature information.

  As an example provided in two stages, for example, it is assumed that time t = 6 in FIG. 7, and in a situation where the feature information F6 is input as a query (a situation where F7 does not exist yet), the following occurs. That is, when the unique feature information F0, F3 and F4 is provided in the first stage and the result of the most similar judgment in the estimation unit 3 becomes the feature information F4, the feature information F5 close to the feature information F4 in the second stage Is further provided, and it is determined whether to adopt the previous estimation result F4 based on the similarity between the feature information F6 and the feature information F5. That is, if F4 is similar, F4 is adopted, and if F5 is similar, F5 is adopted. In this way, high-speed and high-precision can be achieved by limiting the similarity evaluation to only those close to the feature information F5 obtained from the first stage determination result.

  In the case of providing in stages based on the proximity of the position information, position information P2 of the query feature information is provisionally obtained using position information P1 of the reference feature information determined to be most similar in the first step. What is determined to be close to both of the position information P1 and P2 may be set as the second stage provision target. In the above example, it is assumed that the position information of the feature information F5 determined to be the most similar and the position information of the query feature information F6 provisionally calculated based on this (as calculated by the synthesizing unit 4 What is determined to be closer to both of the position information (temporarily calculated) and may be provided as the second stage.

  In the storage unit 5, regarding the feature information registered in association with the position information from the combining unit 4 at the past time t, the positions associated with each other based on the elapsed time from the past time t (that is, the registration time t) Information and feature information may be deleted. For example, among items not set as specific feature information, items for which a predetermined period or more has elapsed from the registration time t may be deleted.

  Also, in the storage unit 5, when a predetermined period or more has passed from a certain initial time t = 0, all the information already stored is deleted, and the deletion time is set as a new initial time t = 0, The information processing apparatus 10 may be operated.

  As described above, according to the present invention, the feature information and the position information are sequentially stored at the acquired place, so that the feature information not included in the feature information calculated from the imaging target in advance is gradually obtained. By incorporating the feature information calculated from the surrounding environment other than the imaging target becomes a clue for estimation, it is possible to obtain an effect of enabling wider recognition than ever before. Here, if the feature information to be obtained at a certain initial time t = 0 is initially registered, it is also possible to omit calculating and storing feature information from a predetermined imaging target in advance.

  As one of the preferable application examples of the present invention, in the case where the processing unit 6 performs the augmented reality display based on the marker by using a predetermined square marker or the like known in the augmented reality display field, the marker Even when the frame is out, the augmented reality display can be continued without requiring prior registration of feature information of objects other than the marker. In this case, when a marker is detected with a predetermined accuracy, augmented reality display is performed with only the marker as a detection target, and feature information is acquired for the entire captured image at a constant time interval to obtain an initial position Pinit. The position and orientation of the captured image at the current time is estimated by using the feature information registered as the initial position Pinit in the immediate past when the marker is no longer detected with a predetermined accuracy. Then, the method of the present invention may be applied until the marker is detected again with a predetermined accuracy.

  The information processing apparatus 10 of the present invention can be realized as a computer having a general configuration. That is, a CPU (central processing unit), a main storage device that provides a work area to the CPU, an auxiliary storage device that can be configured with a hard disk or SSD, etc., an input interface that receives user input such as a keyboard, mouse, touch panel etc. The information processing apparatus 10 can be configured by a general computer such as a communication interface for communicating by connecting to it, a display for displaying, a camera, and a bus for connecting these. Furthermore, each process of each part of the information processing apparatus 10 shown in FIG. 1 can be realized by a CPU that reads and executes a program for executing the process, but any part of the process can be realized by a separate dedicated circuit etc. (Including the GPU). The imaging unit 1 may be realized as an image acquisition program as an interface for acquiring a captured image acquired by a camera as hardware, or an imaging control program for controlling imaging processing of the camera as the hardware. Furthermore, it may be realized including.

  DESCRIPTION OF SYMBOLS 10 ... Information processing apparatus, 1 ... Imaging part, 2 ... Calculation part, 3 ... Estimation part, 4 ... Combining part, 5 ... Storage part, 6 ... Processing part

Claims (9)

An imaging unit that captures an object and obtains a captured image;
A calculation unit that calculates feature information from the captured image;
A storage unit that stores feature information and position information in association with each other;
With reference to the storage unit, feature information determined to be similar to the calculated feature information is determined, and coordinates included in the feature information calculated from coordinate information included in the feature information determined to be similar An estimation unit for obtaining relative position information of the feature information as a conversion relationship to information;
The position information of the captured image is obtained by combining the relative position information and the position information of the feature information determined to be similar, and the obtained position information corresponds to the feature information calculated from the captured image. An information processing apparatus including: a combining unit to be added and stored in the storage unit.   The information processing apparatus according to claim 1, wherein the storage unit provides only part of the feature information as a reference to the reference from the estimation unit.   3. The information processing apparatus according to claim 2, wherein the storage unit adopts, as the partial feature information to be provided for the reference, information that is determined to have a small overlapping ratio. In the storage unit, only a part of the feature information is referred to with respect to the reference from the estimation unit, and after the result of the feature information determined in the first round to be similar in the estimation unit is obtained,
In order to obtain a result of the feature information determined in the second step as being similar in the estimation unit by using, as a reference, a part of the feature information determined to overlap with the feature information determined to be similar in the first round The information processing apparatus according to any one of claims 1 to 3, characterized in that:   5. The storage device according to any one of claims 1 to 4, wherein the stored associated feature information and position information are deleted based on an elapsed time after being stored by the combining unit. Information processor as described.   In the storage unit, a plurality of feature information and position information can be obtained by providing a path between feature information items that are associated by relative position information when they are combined and stored by the combining unit. The information processing apparatus according to any one of claims 1 to 5, wherein the information processing apparatus stores the path information along with the route information between them.   The information processing apparatus according to claim 6, wherein the combining unit uses the path information when obtaining the position information of the captured image by combining. An imaging stage for imaging an object to obtain an imaged image;
Calculating the feature information from the captured image;
A storage step of storing feature information and position information in association with each other;
The characteristic information determined to be similar to the calculated characteristic information is determined with reference to the storage of the storage stage, and is included in the characteristic information calculated from coordinate information included in the characteristic information determined to be similar. Estimating the relative position information of the feature information as a conversion relationship to the coordinate information to be obtained;
The position information of the captured image is obtained by combining the relative position information and the position information of the feature information determined to be similar, and the obtained position information corresponds to the feature information calculated from the captured image. And updating the memory of the memory stage.   A program that causes a computer to function as the information acquisition device according to any one of claims 1 to 7.

Images