JP2013109773A - Feature matching method and article recognition system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a feature matching method realizing acceleration with a simple system.SOLUTION: A feature matching method which recognizes a target in two dimensional or three dimensional image data includes: detecting features each having a local maximum and/or minimum of a prescribed attribute in the image (10); excluding features along an edge and a linear outline from the detected features (12); and allocating the rest of features on a plain surface, selecting a part of features from among the allocated features using local information, and performing a feature matching on the selected features (14). At least one of the detection of features, the exclusion of features, the allocation of the rest of features, the selection of part of features, and the matching of features is performed on plural pieces of image data having different scales formed from the two dimensional or the three dimensional image data.

Description

  The present invention relates to a feature matching method for recognizing an object in two-dimensional or three-dimensional image data and a product recognition system using the same.

  Patent Document 1 performs a plurality of processes (bounding box generation, geometric normalization, wavelet decomposition, color cube decomposition, shape decomposition, generation of a low-resolution grayscale image) on one target region, A method for recognizing objects is disclosed.

US Pat. No. 7,016,532

  The technique disclosed in Patent Document 1 cannot perform stable feature matching for large-scale features such as lines, end faces, and regions. Moreover, even if parallel processing is performed, a reduction in processing speed due to performing a plurality of processes on all target regions is unavoidable.

  The present invention has been made in view of the above points, and an object thereof is to provide a feature matching method capable of speeding up with a simple system and a product recognition system using the feature matching method.

A feature matching method according to an aspect of the present invention includes:
Detect features that have a specified attribute extreme value (Local Maximum and / or Minimum) in one 2D or 3D image data (10),
Excluding features present along the edges and line contours from the detected features (12);
Assign the remaining features to a plane (14),
Select some features from the assigned features using local information (14),
Perform feature matching on the selected feature (14),
A feature matching method for recognizing objects in 2D or 3D image data,
Creating a plurality of image data having different scales from the one two-dimensional or three-dimensional image data,
At least one of detection of the feature, exclusion of the feature, assignment of the remaining feature, selection of the partial feature, and execution of the feature matching is performed on the plurality of different created image data. Called
It is characterized by that.
A product recognition system according to an aspect of the present invention includes:
A feature storage unit (134) configured to record features of a plurality of pre-registered products;
An image input unit (130) configured to photograph a product;
The product photographed by the image input unit by extracting features from the image obtained by photographing the product by the image input unit and comparing and comparing the extracted features with the features recorded in the feature storage unit. An automatic recognition unit (132) configured to automatically recognize
A settlement unit (132) that performs a settlement process using the recognition result of the automatic recognition unit,
Comprising
The automatic recognition unit uses the feature matching method according to one aspect of the present invention.

  According to the present invention, it is possible to provide a feature matching method capable of speeding up with a simple system and a product recognition system using the feature matching method.

It is a block diagram of the feature matching method concerning a 1st embodiment of the present invention. It is a figure which shows an original image. FIG. 5 shows a series of multi-scale images used to extract features. It is a figure which shows the feature extracted by the multiscale feature detection (multi-scale feature detection). It is a figure which shows matching with the feature of the image of the original image, and the image feature which moved the original image by 20 pixels in parallel. It is a figure which shows the matching with the characteristic of the image of the original image, and the image of 0.7 times the original image. It is a figure which shows the matching with the characteristic of the original image, and the characteristic of the image which rotated the original image 30 degree | times. It is a figure which shows the matching with the characteristic of the image which performed the shearing of 0.4 (shearing) so that it might correspond to the affine 3D deformation | transformation of the original image. It is a figure which shows the final matching result from a data set. It is a block diagram of a high-speed matching search method in the feature matching method according to the second embodiment of the present invention. It is a figure for demonstrating a brute force (Brute-Force) matching method. It is a figure which shows the example of the matching search of two multidimensional sets using exhaustive search (exhaustive search). It is a figure which shows the experimental statistical result of the time required for the matching search using the exhaustive search with respect to a lot of feature points (feature points). It is a figure which shows the procedure which decomposes | disassembles hierarchically the whole feature space into several subspaces (subspace). It is a figure which shows the subspace decomposed | disassembled hierarchically. It is a figure which shows the statistical result of the comparison experiment of the brute force matching method with respect to a small database, and a high-speed matching method. It is a figure which shows the statistical result of the comparison experiment of the brute force matching method with respect to a large database, and a high-speed matching method. It is a figure which shows the structure of the information search system as a 1st application. It is a figure which shows the operation | movement flowchart of the information search system as a 1st application. It is a figure which shows the structure of the modification of the information search system as a 1st application. It is a figure which shows the structure of the information search system as a 2nd application. It is a figure for demonstrating the modification of the information search system as a 2nd application. It is a figure which shows the structure of another modification of the information search system as a 2nd application. FIG. 18 is a diagram illustrating an operation flowchart of a mobile phone to which the configuration of FIG. 17 is applied. It is a figure which shows the structure of the information search system as a 3rd application. It is a figure which shows the structure of the goods recognition system as a 4th application. It is a figure for demonstrating the characteristic registered into a database previously. It is a figure which shows the flowchart of goods settlement in the goods recognition system as a 4th application. It is a figure which shows the flowchart of a feature extraction and recognition process. It is a figure for demonstrating the comparison object of the characteristic of the image from a camera, and the characteristic of the reference image registered beforehand. It is a figure which shows schematic structure of the search system as a 5th application. It is a block block diagram of the search system as a 5th application. It is a figure which shows the operation | movement flowchart of the search system as a 5th application. It is a figure which shows the detailed flowchart of a matching process with DB. It is a figure which shows the display screen of the display part of a digital camera in the case of displaying only one image candidate. It is a figure which shows the display screen in the case of displaying nine image candidates. It is a figure which shows the flowchart for demonstrating the preparation method of a characteristic database. It is a figure which shows the flowchart for demonstrating another example of the production method of a characteristic database. It is a figure which shows the flowchart for demonstrating another example of the production method of a characteristic database. It is a figure which shows the flowchart for demonstrating the other example of the production method of a characteristic database. It is a figure for demonstrating the operation | movement concept at the time of imaging | photography the station name display board of a station as a signboard. It is a figure which shows the example which displayed the photograph on the map. It is a figure which shows another example which displayed the photograph on the map. It is a figure which shows the example of a display of the photograph on a map in case there are many photographs. It is a figure which shows another example of a display of the photograph on the map in case there are many photographs. It is a block block diagram of the search system as a 6th application. It is a figure which shows the operation | movement flowchart of the search system as a 6th application. FIG. 6 is a diagram illustrating a detailed flowchart of printout shooting processing. It is a figure which shows the flowchart for demonstrating the preparation method of a characteristic database. It is a block block diagram of the mobile phone with a camera to which the search system as a 7th application was applied. It is a figure which shows the operation | movement flowchart of the search system as an 8th application. It is a figure for demonstrating the outline | summary characteristic used with the search system as a 9th application. It is a figure for demonstrating the detailed feature used with the search system as a 9th application. It is a figure for demonstrating the positional relationship of original image data, an outline template, and a detailed template. It is a figure which shows the operation | movement flowchart of the search system as a 9th application. It is a figure for demonstrating the detailed template which paid its attention to the center part of image data. It is a figure for demonstrating the detailed template distributed several places in the image. It is a figure for demonstrating the detailed template which set the attention area | region in the focus position at the time of original image data imaging | photography. It is a figure for demonstrating the detailed template produced with respect to the same area | region as an outline template. It is a figure which shows the operation | movement flowchart of the search system as a 10th application. It is a figure which shows the structure of the search system as an 11th application. It is a flowchart which shows recognition element identification (identify) processing.

The feature matching method of the present invention will be described below with reference to the drawings.
[First Embodiment]
The feature matching method according to the first embodiment of the present invention is also referred to as PBR (Point Based Recognition), and as shown in FIG. 1, feature detection (Feature Detection) 10, feature selection (Feature Adoption) 12, and feature recognition. (Feature Recognition) 14 consists of three parts. Note that the features are spatially and temporally dispersed. For example, when the recognition target of the present method is an image, the feature matching (Feature Matching) in a two-dimensional expanse. If the time spread is taken into account, the video can be recognized.

  In the feature detection 10, spatially stable features that are not affected by scale and arrangement are detected from input object data, for example, images. The feature selection 12 selects a robust and stable portion for performing robust recognition from the features detected by the feature detection 10. Feature recognition 14 uses the features extracted in feature selection 12 and additional restrictions to locate, index, and recognize objects that have been previously analyzed and stored in database 16. Recognize.

  Hereinafter, each of the feature detection 10, feature selection 12, and feature recognition 14 will be described in detail.

First, the feature detection 10 will be described.
Whether robust recognition is possible depends on both the selected feature and the matching technique. Good features make matching better and more robust. Therefore, reliability and stability can be increased by combining optimal feature selection and matching methods. In general, large-scale features such as lines, end faces, and regions give a wider range of information for matching operations, and matching is easier. However, large scale features also tend to cause large image distortion due to changes in viewpoint, geometry, and illumination. Therefore, large constraints and assumptions are necessary for matching in order to compensate for these distortions. Unfortunately, large scale features are often recovered approximately only from the image geometry, since the mathematical formulas needed to model these constraints are generally unknown.

  For image recognition, it is necessary to restore an exact two-dimensional match in image space. Small scale features, such as dots, have the advantage of being able to expect pixel-by-pixel accuracy even if the corresponding measurement results are small. Furthermore, the point feature is more discriminating power, strength at the time of occlusion (when part of the feature is hidden), and sufficient at affine transformation than large scale features (lines, surfaces, etc.) It is advantageous in that it is invariant. A disadvantage of feature points is that often only sparsely scattered point groups and measurement results can be used, and there are only local information, making matching difficult. However, on the other hand, if many feature points are reliably detected, many image coincidence points can be obtained without causing degradation of measurement quality caused by restrictions and various conditions required for other features. Can be restored. In fact, complete affine field conditions, often showing near feature points, with various methods using large-scale features and the state where the most reliable measurement results appear . Considering these factors, we have chosen to use points (= feature points) as features used for recognition.

  In general, the problem of feature detection is not small. For image recognition and matching, the detected features need to exhibit excellent reliability and stability in the recognition method, even if they do not physically match the real world structure at all. In other words, the feature detection method should be able to detect as many features as possible that are identifiable, reliable and reproducible under various affine conditions. By doing so, even if many of the feature quantities become unusable due to occlusion, a sufficient feature quantity can be secured for further image matching and parameter restoration.

  The feature detection 10 in the present embodiment uses a method for detecting a point feature with rich texture regions. In this method, three filters are used. First, a high pass filter is used to extract points with local maximum. R is a 3 × 3 window whose center is P, and F (P) is the output value of the high-pass filter F applied at that point.

if,
F (P) = max {P> P _i : R}> Threshold (1)
If so, the point P becomes a feature candidate and is kept for the next check. Note that this filter may extract a local minimum.

  The second filter is a distinctive feature filter. As is known, points that exist along the edge or line outline are not stable for matching purposes. This is because a so-called matching arbitrary effect works, and these points must be removed for the reliability of matching. It is also known that the image covariance matrix is a good indicator of the structure of a small area image. Summarizing the relationship between the covariance matrix and the image structure, a small eigenvalue represents a relatively uniform intensity within the region. A set of large and small eigenvalues represents a high texture pattern, and two large eigenvalues represent a texture or other pattern interspersed with linear features, black and white. Therefore, it is possible to design a filter that removes linear feature points using these characteristics.

  Let M be a 2 × 2 matrix calculated from the image derivatives as

また、λ_１とλ_２をＭの固有値であるとすると、線形なエッジの度合い（measure）は、
Ｒ＝det（Ｍ）−ｋ（trace（Ｍ））^２ （３）
である。但し、det（Ｍ）＝λ_１λ_２であり、trace（Ｍ）＝λ_１＋λ_２である。

If λ ₁ and λ ₂ are eigenvalues of M, the linear edge degree (measure) is
R = det (M) −k (trace (M)) ² (3)
It is. However, det (M) = λ ₁ λ ₂ and trace (M) = λ ₁ + λ ₂ .

So if the edge degree is R (P)> Threshold (4)
If so, the point P is treated as a linear edge point and removed from the feature candidate (list).

  The third filter is an interpolation filter that repeatedly refines the detected points to sub-pixel accuracy. An affine plane is first used to reconstruct local points into a continuous plane. As a result, until the optimal solution converges and subpixel accuracy is reached, the filter repeatedly narrows the points down to the rearranged plane.

  A novel aspect of this embodiment is that scale invariance is improved by employing a multi-resolution technique and extracting features from multiple resolution images, each.

  In order to realize the scale invariance of the affine transformation, a multi-resolution method is adopted in the feature detection processing. This is different from the conventional method of using a pyramid whose main purpose is to improve the processing speed. In other words, from the coarse search to the finer search, that is, to achieve useful affine scale invariance, the goal is to find useful features across different scales so that features at each level of the pyramid are independent. To process.

  2A to 2C show the result of applying this approach to a cluttered scene. FIG. 2A shows the original image, FIG. 2B shows a series of multi-scale images used to extract the features, and FIG. 2C shows the extracted features.

Next, the feature selection 12 will be described.
Once detected in the feature detection 10, the detected feature must be adopted as a robust and stable feature for robust recognition. As mentioned above, a weakness in using point features is that matching is difficult at an early stage, often because sparse point sets and mere local information can only be used. It is very important to use a suitable feature selection method in consideration of changes in viewpoint, geometric arrangement, lighting, and the like.

  In the approach, in the feature selection 12 in the present embodiment, individual feature points using local information called affine regions are selected (adopt). Three constraints are used to determine the local region. In other words, the three limits are intensity, scale, and orientation. The intensity constraint is the image gradient value G (x, y) calculated from the pixels in the region, and it represents the degree of texture of the feature.

二つの画像のマッチングにおいて、ベースラインが小さい場合には、線形誤差が小さいため強度による特徴選択で画像のマッチングは十分行える。シンプルな相関によるマッチングを使う事ができる。さらに、もしマッチした画像が大きな画像歪みを持っている場合には、アフィン変換による変形マッチングはその歪み補正に有効である。

In the matching of two images, when the baseline is small, the linear error is small, so that image matching can be sufficiently performed by feature selection based on intensity. Simple correlation matching can be used. Furthermore, if the matched image has a large image distortion, deformation matching by affine transformation is effective for correcting the distortion.

  However, in the context of large image baselines (the corresponding image has geometric deformation including scaling and 2D & 3D rotation), simple intensity selection alone is not sufficient. It is well known that simple intensity correlation is not scaling / rotation invariant. In such a case, all constraints that can be considered as a feature with a robust and stable advantage should be considered in order to select matching corresponding points. Magnification and local orientation constraints are built into the selection and matching process. First of all, a directionally continuous space is quantized and becomes discrete.

それらの、量子化された方向成分が、その幾何学的配置空間を構成する。画像を分解したモデルを使う事で、特徴の全ての局所的な方向成分を離散的な基底空間（the discrete base space）に対して割り当てることが出来る。このようにすることで、それらの局所的な方向成分をコンパクトな数式で表す事ができる。全ての考えうる性質（qualities）（強度、倍率、方向）に対して当てはまる表現式（consistent）を形成するために、強度と倍率の値が用いられて特徴マッチングの為のあらゆる局所的な方向成分の重み付けがなされる。さらにその上、量子化効果（誤差）を低減する為に、ガウシャンによる平滑化関数（Gaussian smoothing processing）が使われ、重み付けが強調される。

Those quantized direction components constitute the geometric arrangement space. By using a model that decomposes the image, all local directional components of the feature can be assigned to the discrete base space. By doing in this way, those local direction components can be expressed by a compact mathematical expression. All local directional components for feature matching using intensity and magnification values to form a consistent expression for all possible qualities (intensity, magnification, direction) Are weighted. Furthermore, in order to reduce the quantization effect (error), a Gaussian smoothing function (Gaussian smoothing processing) is used to emphasize weighting.

  A novel aspect of the present embodiment is to have the features in the direction normalized from the peripheral region of the features in the form shown by the following equation (8).

R is a weighting range defined by a Gaussian filter for creating a scale pyramid. At all the points P (x _i , y _i ) within the range, the quantized direction component is expressed by Equation (8).

ここで、Ｇ（ｘ_ｉ，ｙ_ｉ）は上記式（５）によって算出される勾配であり、Weight（ｘ_ｉ，ｙ_ｉ）は、処理された点（ｘ，ｙ）に中心がある以下の式（９）のようなガウシャン重み関数である。

Here, G (x _i , y _i ) is the gradient calculated by the above equation (5), and Weight (x _i , y _i ) is centered at the processed point (x, y) It is a Gaussian weighting function like Formula (9).

Weight (x _i , y _i ) = exp (− ((x _i −x) ² + (y _i −y) ² ) / σ ² ) (9)
The above-described selection method is effective when dealing with enlargement / reduction of an image and rotation outside the plane. However, it is sensitive to the in-plane orientation. In order to correct this deviation, the affine region is normalized in the same direction in the weighting calculation. In addition, bi-linear interpolation and Gaussian smoothing processing are applied in the processing window to cancel the rotational quantization error. Similarly, the input image is normalized to enhance robustness against changes in lighting conditions.

  The output of feature selection 12 is a compact vector representation that represents the area associated with each match (all constraints are incorporated, represented by an affine transformation, and achieve illumination invariance). It is.

  FIGS. 3A to 3D show the results of applying this method to a scene that has undergone different affine transformations. 3A is a scene obtained by translating the original image by 20 pixels, FIG. 3B is a scene obtained by multiplying the original image by 0.7, FIG. 3C is a scene obtained by rotating the original image by 30 degrees, and FIG. 3D is equivalent to the affine 3D transformation of the original image. The results are shown respectively applied to a scene that has been sheared by 0.4.

Next, the feature recognition 14 will be described.
The features detected by the feature detection 10 and selected by the feature selection 12 exhibit good characteristics with respect to geometry invariance. The matching process is performed based on the selected feature. SSD (Sum of Square Difference) is used to determine matching similarity. That is, the similarity (P) for each feature P of the matching image is calculated, and the SSD search finds the best matching point with the highest similarity. if,
Similarity (P) = {P, P _i }> Threshold (10)
If there is a relationship, P _i coincides with the point P.

  The pair evaluation method using RANSAC (Random Sample Consensus) is used as a reliability evaluation method for image recognition. In particular, when there are few matching points, recognition is performed from the affine transformation matrix calculated by this method. It is effective to calculate the posture of the hour and to evaluate the reliability of recognition based on this posture.

The experimental result shows that the feature portion satisfying the plurality of constraint conditions has a good image matching characteristic. However, in very cluttered scenes, mismatching (called outliers) may occur, especially because of the feature quantities in the background. In order to eliminate these mismatches, a RANSAC based approach is used to find pairs that satisfy basic geometric constraints. It is known that when the matched image features are the same object, the relationship is expressed by two-dimensional transformation (as planar projection transformation (homography)). In order to speed up the calculation, the feature recognition unit 14 approximates the planar projective transformation using a constraint by two-dimensional affine transformation to remove mismatching. Three points are sufficient for estimating the change in parameters. First, RANSAC sequential assignment, in order to estimate the initial transformation matrix _{M init,} using 3 feature points randomized.

推定された変換行列は、次に、マッチングする全ての特徴を用いて繰り返し絞り込まれていく。このミスマッチングはこれらのマッチング点が大きなマッチングずれを有する事を示している。

The estimated transformation matrix is then iteratively refined using all matching features. This mismatch indicates that these matching points have a large mismatch.

ここで、ｘ_ｉ ^ｔは推定されたアフィン変換式によってｘ_ｉからｘ_ｉ ^ｓに向かってアフィン変形された点を表す。すなわち、

Here, x _i ^t represents the point which has been affine deformation toward the x _i to x _i ^s by the affine transformation equation estimated. That is,

その特徴マッチングの最終アウトプットはミスマッチングかどうかの判定と推定された２次元パラメータ変換（アフィンパラメータ）が添付されたマッチング点のリストである。

The final output of the feature matching is a list of matching points attached with determination of whether or not mismatching and an estimated two-dimensional parameter transformation (affine parameter).

  FIG. 4 shows an example of a final matching result obtained by the feature recognition 14 from an object data set analyzed in advance and stored in the database 16.

[Second Embodiment]
In the present embodiment, a high-speed matching search method for further speeding up the feature recognition 14 will be described.

  This high-speed matching search method is called dBTree (Data Base Tree). The dBTree, which is an effective image matching search technique, can rapidly restore matching pairs from the multidimensional database 16 from which PBR feature points have been extracted as described in the first embodiment. Technically, this issue is the NP data query problem. In other words, when a point on an N-dimensional database and a query point q are given, finding the best match (nearest neighbor) between q and a point in the database is a problem. The fast matching search method according to the present embodiment is a tree-structured matching approach that forms a hierarchical representation of PBR features for effective data representation, matching, and indexing of a multidimensional feature space.

  Technically, the dBTree matching is composed of a dBTree Construction 18, a dBTree Search 20, and an Index Matching 22 as shown in FIG. In the dBTree construction 18, in order to realize a high-speed feature search and query, a hierarchical data representation on the PBR feature space (from the PBR feature obtained from the object data input as in the first embodiment above) ( Hereinafter, a dBTree expression) is created. The created dBTree expression is registered in the database 16. In the database 16, dBTree expressions for a large number of object data are registered in this way. In the dBTree search 20, the nearest neighbor point of the PBR feature obtained from the input object data as in the first embodiment is searched on the dBTree space configured in the database 16. In the index matching 22, a matching pair is extracted and corrected from the found nearest neighbors (NNs) and further PBR restriction conditions.

  Before describing the details of the dBTree approach in the present embodiment, the problem of matching search will be described.

  The goal of matching search is fast restoration of matching, which can be in a multidimensional database. Although the present embodiment focuses on a unique case such as PBR feature matching, the present dBTree search structure can be applied to any general data search application.

Two point groups P = {p _i , i = 1, 2,..., N} and Q = {q _j , j = 1, 2,..., M} are given, and p _i and q _i are k-dimensional vectors. In this case, the goal is to find all possible matches between the two point clouds P and Q. In other words, Matches = {p _i <=> q _j } has certain similarity.

Since PBR feature points have good invariant characteristics in feature matching, the Euclidean distance of invariant features is used for similarity matching. In other words, each feature p _i and similarity (p _i ) are calculated for the matched feature q _j . The matching search then finds the best match with the shortest Euclidean distance.

  Obviously, the matching performance and speed depend on the number of feature points of two point groups such as N and M.

  Probably the first thing that comes to mind when matching points in two datasets is the “brute-force” search method. As shown in FIG. 6, the brute force approach takes all points of set P and computes their similarity for each point of set Q. Obviously the matching speed of the exhaustive search is linearly proportional to the number of features in the point set, resulting in a total O (N × M) algorithm computation (Euclidean distance computation). As an example, for the matching of two typical PBR feature sets of 547 points and 547 points, the brute force matching took 3.79 seconds on a 1.7 GHz PC. FIG. 7 shows an example of matching of two multidimensional data sets (2955 points × 5729 points) using exhaustive search. As a result, it can be estimated that it takes 169.89 seconds.

  FIG. 8 shows the matching of a brute force search (of more than 50 experimental images) against a large number of feature points (total number N × M of input image feature number N and database feature number M). The experimental statistical results for the time required are shown.

Details of the dBTree approach in this embodiment will be described below.
First, the dBTree construction 18 will be described.

  The data structure that is the core of dBTree matching is a tree structure that forms an effective hierarchical representation (characterizing superfunctions). Unlike the representation of features that scan a row (in other words, all features are represented in a grid structure) using a brute force search, dbTree matching Then, k-dimensional data is expressed by a balanced-binary tree in which the entire space is divided into several subspaces by tree nodes. The root node of this tree represents the entire matching space, and its branch node represents a rectangular subspace with different character features in the lower space. Since the subspace is relatively small compared to the original space, the number of input features is also small, so any input feature can be accessed at high speed regardless of its position in the tree representation. By down the hierarchy until a subspace containing input features is found, the matching points can be identified in the subspace by scanning a small number of nodes.

  9A and 9B show a procedure for hierarchically decomposing the entire feature space 24 into several subspaces 26 to form a dBTree data structure. First, the input point group is divided (segmented) according to a predefined dividing method. Since median filtering is used here, the same number of points enter both subspaces 26 divided. Each node of the tree is defined by a value of one parameter dimension, and points are divided into the subspace 26 in the vertical and horizontal directions of the subspace 26, and half of the points of the parent node are entered. These child nodes are again split into the same number of two sets with different parameter dimensions. This process is repeated until the number of points contained in each leaf reaches the log (N) level.

Next, the dBTree search 20 will be described.
There are two steps to search for a query point in the tree (to search the nearest subspace 26 and to find the nearest node in it). First, the tree is searched across to find a subspace 26 containing query points. Since many of the subspaces 26 are relatively small, the subspace 26 can be found quickly and simply by log (N) comparison operations, and the probability that the space includes a matching point is high. Once the subspace 26 is found, a node level search of all nodes in the subspace 26 is performed and matching points are identified. This process is repeated until the closest node for the query point is found.

  In the experiment using the above search method, it was confirmed that the speed was improved in matching data sets with a small number of dimensions. Surprisingly, however, it has no effect on large data sets and is even slower than the brute force search approach. There are two aspects to this reason. First, the efficiency of classic tree searches is based on the fact that if the distance to the query point is extremely far, many tree branches may be pruned (ignored), and this is not good. The required search time can be greatly reduced. This is common for data sets with a small number of dimensions, but as the number of dimensions increases, there are too many branches to examine that are adjacent to the central branch. In order to remove the branches and find the best search path, a lot of calculations are performed, which makes the tree-type search difficult. Secondly, all subordinate nodes (depending on how many subordinate nodes) are examined by a cross-level search at the node level in the subspace 26, which is also labor intensive. In a multidimensional data set, each subspace 26 contains too many nodes that require laborious cross-search.

  In the present embodiment, two strategies (methods) are used to solve these problems and realize efficient matching even in a multidimensional data set. First, a tree pruning filter (pruning filter) is used to cut (reduce) the branches that need to be investigated. When the search for a certain number of nearby branches (ie, the search step) has been completed, the search is forcibly terminated. A distance filter can also be used for this purpose, but from a wide range of experiments, the search step filter finds better performance when looking for the correct matching pair and considering the computational cost. The search result based on the approximate value will be seen. From the experimental result, the increase in mismatching due to the approximation is 2% or less.

  The second strategy (method) is an improvement by introducing a node distance filter. Based on the matching constraints in many cases in the real world, correct matching is interspersed. Therefore, instead of searching for all feature nodes, a distance threshold is provided in the search range. The node search is performed with a circumferential pattern, and a node close to the target is searched first. As soon as the search limit is reached, the search is forcibly stopped and the nearest neighbors at that time are output.

Next, the index matching 22 will be described.
When the nearest neighbor is detected, it is determined in the next step whether the nearest neighbors are correct matching. Similar to the PBR matching described above, a matching threshold is provided in order to select a correct matching. For example, if the difference in similarity between the first nearest neighbor and the second nearest neighbor (distance to the first nearest neighbor / distance to the second nearest neighbor) is equal to or less than a predetermined threshold value, That point is determined to be correct matching.

  FIG. 10 and FIG. 11 show statistical results of comparison experiments between the brute force and dBTree matching methods (with more than 50 test images).

  Note that the difference in similarity between the first and second nearest neighbors is a parameter that expresses the accuracy in determining the similarity of the similarities at that point. Also, the number of matching points in the image itself is a parameter expressing the accuracy in determining the identity of the image. Further, the sum of the differences (residuals) in the affine transformation of the matching points in the image expressed by the above equation (13) is also a parameter expressing the accuracy in determining the identity as the image. A part of these may be used, or a conversion formula with each as a variable may be defined, and this may be used as the accuracy of identity determination in matching.

  By using the accuracy value, it is possible to output a plurality of images in a fixed order as a result of matching. For example, by using the number of matching points as accuracy and displaying the matching results in descending order of the number, the images are output in order from more reliable images.

  Hereinafter, an application using the feature matching method as described in the first and second embodiments will be described.

[First application]
FIG. 12 is a diagram illustrating a configuration of an information search system as the first application.

  This information retrieval system includes an information presentation device 100, a storage unit 102, a data set server 104, and an information server 106. The information presentation apparatus 100 is configured by platform hardware. The storage unit 102 is provided in the platform. The data set server 104 and the information server 106 are configured in a site accessible by the platform hardware.

  Here, the information presentation device 100 includes a photographing unit 108, a recognition and identification unit 110, an information designation unit 112, a presentation image generation unit 114, and an image display device 116. The recognition / identification unit 110, the information designation unit 112, and the presentation image generation unit 114 are realized by application software of an information presentation apparatus installed in the platform hardware.

  The photographing unit 108 and the image display device 116 may be provided with platform hardware as a physical configuration or connected to the outside. Therefore, the recognition and identification unit 110, the information specifying unit 112, and the presentation image generation unit 114 may be referred to as an information presentation device. However, in this application, it is considered as a device that performs from image shooting to final image presentation, and the shooting unit 108, the recognition and identification unit 110, the information designation unit 112, the presented image generation unit 114, and the image display device 116 is collectively referred to as an information presentation device.

  Here, the photographing unit 108 is a camera or the like having a predetermined photographing range. The recognition and identification unit 110 recognizes and identifies individual objects within the imaging range from the image captured by the imaging unit 108. The information specifying unit 112 acquires predetermined information (display content) from the information server 106 in accordance with the information on each object identified by the recognition and identification unit 110. Then, the information specifying unit 112 specifies the predetermined information as related information. The presented image generation unit 114 generates a presented image that associates the related information specified by the information specifying unit 112 with the image captured by the imaging unit 108. The image display device 116 is a display such as a liquid crystal that displays the presentation image generated by the presentation image generation unit 114.

  The storage unit 102 in the platform stores a data set 118 from the data set server 104 via a communication unit or storage medium (not shown). However, introduction (downloading or media exchange) and storage of the data set 118 can be performed regardless of whether the information presentation apparatus 100 is activated or not.

  In such a configuration, the information presentation apparatus 100 first acquires an image by the photographing unit 108 as shown in FIG. 13 (step S100). Next, the recognition and identification unit 110 extracts a predetermined object from the image acquired in step S100 (step S102). Subsequently, the recognition and identification unit 110 compares the image of the object extracted in step S102 (for example, an image in a rectangular frame) based on the characteristics in the data set 118 read from the in-platform storage unit 102. And identification. In this way, the recognition and identification unit 110 detects a matching object image. When the recognition and identification unit 110 detects a matching object image (step S104), the information specifying unit 112, which is the next step, detects the information to be acquired from the corresponding data in the data set 118 again. The location and / or acquisition method is read and executed (step S106). In general, information is acquired by accessing the information server 106 that exists on the outside of the network or the like by communication from the platform. Then, the presentation image generation unit 114 processes the information (not shown) acquired by the information specifying unit 112 so that the information can be displayed on the image display device 116 inside or outside the platform, and generates a presentation image. By sending the generated presentation image to the image display device 116 from the presentation image generation unit 114, information is displayed on the image display device 116 (step S108). Here, in some cases, it is also useful information presentation that a presentation image in which the information acquired above is superimposed on the original image acquired by the photographing unit 108 is generated and sent to the image display device 116. Therefore, the user can select which method is used for presentation.

  Further, as shown in FIG. 14, a position and orientation (orientation) calculation unit 120 may be provided between the recognition and identification unit 110 and the information specifying unit 112. The presented image generation unit 114 superimposes the related information specified by the information specifying unit 112 on the image captured by the imaging unit 108 at the position and orientation calculated by the position and orientation calculation unit 120. A presentation image to be displayed is generated.

  Although not shown in FIGS. 12 and 14, when the storage capacity of the platform is large, the following is also possible. That is, when the data set 118 is introduced from the data set server 104, the information server 106 and the data set server 104 communicate with each other, and information (display content) corresponding to the introduced data set server 104 is stored in the in-platform storage unit 102. Introduce or store in advance. If it does in this way, the operation efficiency of information presentation device 100 can be raised.

  Here, a case where the first application is a mobile phone with a camera will be described. A mobile phone is basically a device used by an individual. In recent years, it is possible to install application software on an Internet site (hereinafter abbreviated as “mobile site”) that can be accessed by a mobile phone (so-called downloadable installation). The information presenting apparatus 100 is basically based on such a cellular phone. That is, the application software of the information presenting apparatus 100 is installed in the storage unit 102 of the mobile phone. The data set 118 is appropriately stored in the storage unit 102 of the mobile phone via communication from the data set server 104 connected to a specific mobile site (not shown).

  In particular, examples of the range of use of the information presenting apparatus 100 in a mobile phone include the following usage methods. That is, a photograph existing in a magazine or a newspaper is designated as an object in advance, and a data set related thereto is prepared. A user photographs an object with a mobile phone from the paper of the publication, and reads information related to the object from the mobile site. In such cases, it is impossible to keep all the photographs, icons, illustrations, etc. of all publications as features. Therefore, it is realistic to provide features limited to a specific use range. That is, what is necessary is just to supply to a user by the way of summarizing “a data set for referring to a photograph published in a monthly issue as an object” of a specific magazine. In this way, it is easy for the user to use, and as described above, if the reference image is about 100 to several hundreds in one data set, it can be sufficiently stored in the storage unit 102 of the mobile phone and can be recognized and identified. Can be done within seconds. In addition, no special device or processing is required for the photo / illustration on the printed matter side used in the information presenting apparatus 100.

  According to the first application as described above, a user can easily introduce a plurality of data in a range to be used into the information presenting apparatus 100 in a batch, and the data set supply side is also prepared. Services that are easy to do and that are easy to provide commercially.

  In addition, when a function for calculating the position and orientation is also provided, information acquired from the information server 106 can be displayed on the original image with an appropriate position and orientation. That is, it leads to improving the information acquisition effect of the user.

[Second application]
Next, the second application will be described.
FIG. 15 is a diagram illustrating a configuration of an information search system as the second application. The basic configuration and operation of this information retrieval system are the same as those of the first application described above. The ability to handle features in units of sets in the information presenting apparatus 100 increases the convenience for the user as described above and makes the supply of data sets realistic.

  However, when the information presenting apparatus 100 is widely spread and the data set is supplied from an extremely wide variety of businesses, it is desirable to do the following. That is, a data set 118 that is highly used by the user of the information presenting apparatus 100 (hereinafter referred to as basic data 122) is not supplied as a separate data set 118, and any data set 118 is selected. It is desirable to make it available. For example, the objects associated with the index information of the data set 118 itself, the objects that are used most frequently, etc. are excluded from the data set 118, and these several features are included in the application software of the information presentation apparatus 100. It is effective to make only resident. In other words, in the second application, the data set 118 is assembled according to the purpose of use of the user and the publication or object to be associated, and supplied separately from the application software. However, characteristics or the like of an object that is particularly frequently used or highly necessary is resident or held in the application software itself as basic data 122.

  Here, the case where the mobile phone is a platform will be described again as an example. For example, it is most practical to download a normal data set 118 from a predetermined mobile site via communication. However, at this time, it is convenient for the mobile phone user that guidance and search are possible on the index site (page on the mobile site) of the data set 118. Even when accessing the site itself, in order to make it accessible by letting the information presenting apparatus 100 photograph the object dedicated to the site and passing the URL to the site to the viewing software, it is necessary to prepare the data set 118 in particular. do not need. That is, the feature corresponding to the object is present as basic data 122 in the application software. In this case, a specific illustration, logo, or the like may be used as an object, or a plain quadrilateral that can be obtained anywhere is set as an object.

  Alternatively, instead of making the basic data 122 resident or held in the application software itself, as shown in FIG. 16, in the supplied data set 118, the same data file (which is always the basic data 122) is included in any data set 118. In the figure, at least one set of features “A”) may be included.

  That is, as described above, when the information presenting apparatus 100 is actually operated, the user introduces an arbitrary data set 118. Here, at least one type of the basic data 122 is included in any data set 118, and an object that is frequently used or highly necessary can be dealt with without fail. For example, as shown in FIG. 16, a large number of data sets 118 (data sets (1) to (n)) are prepared, and one or a plurality of data sets 118 are introduced into the in-platform storage unit 102. And consider the case of saving. In this case, regardless of which data set 118 is selected, one or more types of basic data 122 are always included. Therefore, the user can shoot a basic object and perform a basic operation without special consideration. Again, the basic operation is “access to the index page of the data set”, “access to the support window for the information presentation device 100 supplier”, or “weather information” in a predetermined area. “Access to the site” and other actions that many users want. That is, the basic operation is defined as an operation that is frequently used by the user.

  In addition, as shown in FIG. 17, when the information presenting apparatus 100 is activated, it is connected to the data set server 104 so that the basic data 122 is always downloaded and taken into another data set 118 or can be referred to at the same time. Anyway.

  This configuration provides a method for introducing the basic data 122 that is effective when the data set 118 is supplied separately, and is downloaded from the data set server 104 via the network. That is, when the data set 118 is supplied to the information presenting apparatus 100 via the network with the configuration shown in FIG. 17, when the user selects the data set 118 and downloads it from the data set server 104, the data set 118 is displayed. In addition, the basic data 122 can be automatically downloaded at the same time. In addition, with the configuration shown in FIG. 17, when the basic data 122 is already stored in the storage unit 102 of the platform in which the information presenting apparatus 100 exists, the basic data 122 can be updated.

  Thus, the user can always use the basic data 122 with the information presenting apparatus 100 without special consideration.

  For example, in recent years, mobile phones with cameras that can use application software have become popular. Consider a case where application software having functions other than the photographing unit 108 and the image display device 116 of the information presenting device 100 is installed on the platform. In this case, as shown in FIG. 18, when using the application software, the data set 118 is browsed to a predetermined data set download site via communication of the mobile phone (step S110). Next, the data set server 104 is initially downloaded (step S112). Subsequently, the necessity of updating the basic data 122 is determined from the data set server 104 (step S114).

  That is, when the basic data 122 does not exist in the mobile phone, it is determined that it is necessary. Further, even if the basic data 122 already exists in the storage unit 102 in the mobile phone, the version of the basic data 122 is older than the version of the basic data 122 that the data set server 104 intends to provide. It is determined that updating is necessary.

  Subsequently, the basic data 122 is downloaded in the same manner as the data set 118 (step S116). Then, the downloaded basic data 122 is stored in the storage unit 102 in the mobile phone (step S118). The downloaded data set 118 is stored in the storage unit 102 in the mobile phone (step S120).

  As described above, when the basic data 122 already exists in the storage unit 102 in the mobile phone, the basic data 122 is downloaded and stored after determining the necessity of updating through version comparison.

  As described above, the necessity of the data set 118 is limited to a certain extent according to the needs of the user, and only the data set 118 is stored in the mobile phone, which is inconvenient for ensuring the object identification processing speed and the necessity of the user. Make sure not to multiply.

  The scope of use of the information presenting apparatus 100 includes access from a mobile phone to information related to or resulting from designs such as photographs and illustrations on publications such as newspapers or magazines, and information thereof. There is sophistication of information presentation by superimposing on an image acquired by a camera. Furthermore, it is possible to register not only the printed matter but also an object, a signboard in the town, and the like as features. In this case, it is possible to acquire additional information or latest information by recognizing them as objects from the mobile phone.

  Furthermore, as a usage form using a mobile phone, since products with packages such as CDs and DVDs have various jacket designs, they can be used as objects. For example, by distributing a data set related to the jacket group from a store or a separate record company to the user, each jacket can be recognized as a target by a mobile phone in a CD and / or DVD shop or a rental store. Become. For this reason, for example, a URL is associated with the target object, and, for example, voice distribution of a touch portion of music can be distributed to the mobile phone as information associated with the target object. In addition, as the information to be associated, an annotation corresponding to the jacket surface (individual annotation of the jacket photo) can be appropriately added.

  That is, when using a jacket design of a product having a package such as a CD or DVD as an object as a usage form using a mobile phone, the following may be performed. First, (1) at least a part of an appearance image of a recording medium on which music is fixed or its packaging is distributed in advance to a mobile phone as object data. Then, (2) predetermined music information (audio data or annotation information) related to the fixed music is distributed to the mobile phone that has accessed the address guided by the object.

  In this way, it is effective as a promotion on the record company side, and there are merits that the store side can save time for preparations such as trial listening.

  Note that the recognition and identification unit, information specifying unit, presentation image generation unit, position and orientation calculation unit described in each application described above are all realized by a CPU built in the information presentation device and a program operating on the CPU. It was described as to be. However, other modes such as providing a dedicated circuit can be realized.

  In addition, examples of the implementation mode of the storage unit in the platform include an external data pack and a removable storage medium (for example, a flash memory), but are not limited thereto.

  The second application may also include the position and orientation calculation unit 120 as in the first application, and present related information based on the calculated position and orientation.

  Further, as shown by broken lines in FIGS. 12 and 14 to 17, an exchangeable storage medium 124 may be used instead of the data set server 104 and / or the information server 106. In this case, the introduction of the data set 118 and the basic data 122 to the platform storage unit 102 means the development of data from the storage medium 124 to the internal memory.

[Third application]
For example, the configuration of the information search system as the first application shown in FIG. 12 can be modified as shown in FIG. That is, the recognition and identification unit 110 provided in the information presentation device 100 in the first application and the data set 118 provided in the storage unit 102 are provided on the server side as shown in FIG. Of course. In the case of an information search system having such a configuration, the storage medium 124 provided in the storage unit 102 is not provided because it is not necessary.

[Fourth application]
Next, the fourth application will be described.
FIG. 20 is a diagram illustrating a configuration of a product recognition system as the fourth application.

  This merchandise authentication system includes a bar code scanner 126 which is a reader for recognizing merchandise to which a bar code is attached, a weight scale 128 for weighing the merchandise, and a camera 130 for photographing the merchandise. It has. The control unit / cash storage box 132 for storing cash recognizes the product by the database 134 in which the product features for recognition are registered, and displays the type, unit price, and total price of the recognized product on the monitor 136. Note that the field of view 138 of the camera 130 matches the range of the weight scale 128.

  In such a product recognition system, the system provider takes an image of an object that needs to be recognized in advance, and registers the features extracted from the image in the database 134. For example, when used in a supermarket, vegetables such as tomatoes, apples, and peppers are photographed, and as shown in FIG. 21, features 140 are extracted, and identification indices such as recognition IDs and names corresponding thereto are attached. Stored in the database 134. Further, auxiliary information such as the average weight and average size of each object is also stored in the database 134 as necessary.

  FIG. 22 is a diagram showing a flow chart of product checkout in the product recognition system as the fourth application.

  The purchaser of the product takes the product as an object and places it in the field of view 138 of the camera 130 installed at the cash register, thereby photographing the product (step S122). The product photographing data is transferred from the camera 130 to the control unit / cash storage box 132 (step S124), and the control unit / cash storage box 132 performs feature extraction and product recognition with reference to the database 134. (Step S126).

  If the product is recognized, the control unit / cash storage box 132 calls the set price of the recognized product from the database 134 (step S128), displays the price on the monitor 136, and performs settlement (step S128). S130).

  When the purchaser purchases two types of tomatoes and peppers, first, the camera 130 captures the tomatoes, and the control unit / cash storage box 132 extracts the features in the captured data and compares them with the database 134. Done. Then, when a single commodity as a target is specified after collation, a coefficient corresponding to the price or weight if the pay-as-you-go system is read out from the database 134 and the price is output to the monitor 136. Next, for the bell pepper, the target product is identified and the price is output. Finally, the total price of the merchandise is calculated and sent to the monitor 136 for settlement.

  In addition, when a plurality of target candidates that exceed the similarity threshold after collation appear, (1) the candidate is displayed and selected on the monitor 136, and (2) the target is imaged again. The target is established (establish).

  In the above description, an example in which shooting by the camera 130 is performed for each product is shown. However, it is also possible to capture a plurality of types of target products at a time and perform collation.

In addition, if the purchaser performs these processes, an automatic cash register can be realized.
FIG. 23 is a flowchart of the feature extraction / recognition process in step S126.

  That is, a plurality of features are extracted from the image (product photographing data) input from the camera 130 (step S132), and the features of the object registered in advance from the database 134 are read out as comparison data (step S134). ). Then, as shown in FIG. 24, the feature of the image 142 from the camera 130 is compared with the feature of the reference image 144 registered in advance (step S136), and the identification of the target is judged (step S138). ). If it is determined that they are not identical (step S140), the feature of the next object registered in advance is read out from the database 134 as comparison data (step S142), and the process returns to step S136.

  On the other hand, if it is determined that they are the same (step S140), it is determined that the object currently being compared and the product in the input image are the same (step S144).

  As described above, in the product recognition system as the fourth application, the product can be recognized without attaching a recognition index such as a barcode or an RF tag to the product. In particular, it is particularly effective because it can automatically recognize agricultural products such as vegetables, meat, fish, etc. that require a lot of effort to install the recognition indicators, unlike those that can be easily attached with recognition indicators such as industrial products. It is.

  In addition, examples of objects that are difficult to attach such a recognition index include minerals, and can be applied to industrial uses such as automatic sorting.

[Fifth application]
Next, the fifth application will be described.
FIG. 25 is a diagram illustrating a configuration of a search system as the fifth application. As shown in the figure, the search system includes a digital camera 146, a storage 148, and a printer 150. Here, the storage 148 stores a plurality of image data, and the printer 150 prints out the image data stored in the storage 148.

  For example, the storage 148 is a memory built in or removable from the digital camera 146, and the printer 150 prints out image data in the memory that is the storage 148 in accordance with a printout instruction from the digital camera 146. Alternatively, the storage 148 is a device that can be connected to the digital camera 146 via a connection terminal, a cable, or a wireless / wired network, or can be loaded with a memory removed from the digital camera 146 and transfer image data. There may be. In that case, the printer 150 may be connected to or integrated with the storage 148 and execute a printout in accordance with a printout instruction from the digital camera 146.

  Note that the storage 148 also has a database function configured so that image data can be searched based on feature amounts. That is, the storage 148 constitutes a feature database that stores feature groups created from digital data of the original image.

The search system having such a configuration operates as follows.
(1) First, the digital camera 146 shoots a subject including an image of the search source printout 152 once printed out by the printer 150. Then, an area corresponding to the image of the search source printout 152 is extracted from the obtained photographed image data, and features of the extracted area are extracted.

  (2) Then, the digital camera 146 performs matching processing with the feature group stored in the storage 148 by the extracted feature.

  (3) As a result, the digital camera 146 reads the image data corresponding to the matched feature from the storage 148 as the original image data of the search source printout 152.

  (4) Thereby, the digital camera 146 can print out the read original image data with the printer 150 again.

  As the search source printout 152, an index print in which a plurality of reduced images are output together can be used in addition to the printout output in units of one sheet. This is because it is more cost-effective and convenient to select the necessary ones from the index print and print them.

  Further, the search source printout 152 may be printed out by a printer outside the system (not shown) as long as the original image data is in the feature database configured in the storage 148 in the system. Absent.

  Hereinafter, the search system as the fifth application will be described in detail with reference to a block configuration diagram shown in FIG. 26 and an operation flowchart shown in FIG. Note that the digital camera 146 has a captured data search mode in addition to the normal shooting mode, and the operation flowchart of FIG. 27 shows processing when the search mode is set.

  In other words, after setting the search mode, the user photographs the search source printout 152 desired to be printed out again with the photographing unit 154 of the digital camera 146 on a table or a wall surface ( Step S146).

  Next, the feature extraction unit 156 performs a process of extracting features (step S148). The feature may be a feature point in the image data, or a divided area in the image data according to a predetermined rule, that is, a relative density of a small area allocated by a predetermined grid, etc. May be used, or may be based on a Fourier transform value for each divided area. It is desirable that the information held by the feature points includes point arrangement information.

  Next, the matching unit 158 compares the features extracted by the feature extraction unit 156 with the feature database (feature template) of the photographed image data configured in the storage 148, and sequentially extracts those having high similarity. A matching process is executed (step S150).

  That is, in the matching process with the DB, as shown in FIG. 28, first, the similarity with the feature of each photographed image data is calculated (step S152), and sorted based on the similarity (step S154). Then, original image candidates are selected according to the similarity (step S156). In this selection, a threshold value may be set, or a higher order may be designated in descending order of similarity. In any case, there are two methods: a method of selecting one point with the highest degree of similarity and a method of selecting a plurality of points from those with a high degree of similarity.

  Thereafter, the selected original image candidate image data is read from the storage 148 on the display unit 160 and displayed as an image candidate to be extracted (step S158), and selection by the user is accepted (step S160).

  FIG. 29 shows a display screen of the display unit 160 when only one image candidate is displayed. In this display screen, a “previous” and “next” icons 164 representing buttons to be operated when instructing the display of other image candidates are displayed next to the display of the image candidates 162, and the image candidates 162 are displayed as desired images. A “decision” icon 166 representing a button to be operated when instructing as data is arranged. The “Previous” and “Next” icons 164 represent the left and right keys of the so-called cross key that the digital camera 146 normally has, and the “OK” icon 166 is an enter key arranged in the middle of the cross button. Represents something.

  If the cross key corresponding to the “previous” or “next” icon 164 is operated (step S162), the process returns to step S158 to update and display the image candidate 162. On the other hand, when the enter key corresponding to the “OK” icon 166 is operated (step S162), the matching unit 158 stores the original image data corresponding to the image candidate 162 stored in the storage 148. Then, the data is sent to the connected printer 150 and printed again (step S164). If the printer 150 is not directly connected by wire / wireless, a process of performing predetermined marking such as adding a flag to the original image data corresponding to the image candidate 162 stored in the storage 148 is performed. By executing this, it is possible to print out by the printer 150 that can access the storage 148.

  In addition, in the display of the image candidates in step S158, a plurality of candidates may be displayed at the same time. In this case, since the display unit 160 normally installed in the digital camera 146 is naturally a small size of several inches, display of about 4 or 9 points is easy to use. FIG. 30 shows an example in which nine image candidates 162 are displayed. In this case, the thick frame 168 indicating the selected image is moved according to the operation of the left key or the right key of the cross key corresponding to the “previous” or “next” icon 164. Although not specifically illustrated, a so-called page is displayed in which the display of the nine image candidates 162 is changed to the display of the previous and the next nine image candidates in accordance with the operation of the up key and the down key of the cross key. A switching display may be performed.

  It should be noted that the feature database of captured image data configured in the storage 148 as a comparison target used in step S150 needs to be created in advance based on the original image data in the storage 148. Further, the storage 148 may be a memory attached to the digital camera 146, or may be a database accessible via the communication unit 170 as indicated by a broken line in FIG.

Various methods are conceivable for creating this feature database.
For example, when the captured image data is stored in the memory area of the digital camera 146 when the original image is captured, the feature is calculated and the database is registered. That is, as shown in FIG. 31, the digital camera 146 performs shooting (step S166), and the captured image data is stored in the memory area of the digital camera 146 (step S168). Then, a feature is calculated from the stored photographed image data (step S170) and stored in association with the photographed image data (step S172). Therefore, if the storage 148 is a memory built in the digital camera 146, a database is constructed. When the storage 148 is separate from the digital camera 146, the captured image data and features stored in the memory area of the digital camera 146 are transferred to the storage 148 and a database is constructed.

  In addition, when the original image data stored in the storage 148 is printed out by the printer 150, the feature extraction processing is performed simultaneously with the print-out instruction and stored in the database is also a process-efficient method. That is, as shown in FIG. 32, when printing out the original image data stored in the storage 148, the original image data to be printed is usually selected by a user instruction (step S174), and the print conditions are set. In step S176, printing is executed (step S178). Normally, the printing process is ended here, but in this example, the feature is further calculated from the selected original image data (step S180), and the created feature is associated with the original image data. Save (step S182). Note that the accuracy of matching between the search source printout 152 and the features can be improved by reflecting the print conditions when creating the features. According to such a method, since the feature is created only for the original image data that may be subjected to the matching processing, unnecessary feature creation time and storage capacity can be omitted.

  Of course, it may be performed by batch processing. That is, as shown in FIG. 33, when there is a collective feature creation execution instruction from the user (step S184), the feature-uncreated original image data in the storage 148 is selected (step S186), and the selected features are not yet displayed. A batch feature creation process is executed on the created original image data (step S188). In this collective feature creation process, features are extracted from individual feature-uncreated original image data to create features (step S190), and the created features are associated with the corresponding original image data and stored in the storage 148. Yes (step S192).

  Furthermore, it may be processed individually by user input. That is, as shown in FIG. 34, the user selects one of the original image data in the storage 148 (step S194), and gives an instruction to create a feature for the selected original image data (step S196). Then, features are extracted from the selected original image data (step S198), and the features are associated with the selected original image data and stored in the storage 148 (step S200). Marking a photo to be printed out may be a feature creation instruction.

  Up to now, when trying to reprint image data that has been printed out, the user often searches with the accompanying information (file name, shooting date, etc.) of the image data as a reference. According to the search system as this application, it is possible to access a file (image data) of an original image simply by taking an image of a desired search source printout 152 with the digital camera 146, which is intuitive and user-friendly. It is possible to provide an easy-to-use search method.

  In addition, it is possible to search not only the original image data itself but also image data having a similar image configuration, and can provide a secondary but novel application. That is, street signs, posters, and the like are photographed in this so-called search mode, and the image data existing in the storage 148 such as a memory attached to the digital camera 146 or a database accessible via communication and the features thereof are similar or different. The same image data can be easily searched.

  As shown in FIG. 35, for example, if a station name display board of a station is photographed as a signboard, the position of the photographer can be recognized by recognizing the station name from the image data. Therefore, the map information around the recognized position, that is, around the station, the image information related to the position, the character information, and the like can be accessed via a memory attached to the digital camera 146 or a communication database. It is possible to search and provide from related information existing in the storage 148. The station name recognition method includes character recognition, pattern recognition, recognition estimation by searching for similar images, and the like, and can be implemented as a function of the matching unit 158.

  In addition, for example, by taking images of Tokyo Tower and searching for images in storage 148 such as a database that can be accessed via memory or communication attached to digital camera 146, not only Tokyo Tower but also tower-like buildings around the world. It is possible to search and extract photographs of objects. Further, based on the position information as supplementary information of the photo extracted and extracted, the location of each tower is informed, and the photo is superimposed and displayed at the corresponding location on the map as shown in FIGS. It is also possible to do. In this case, a map and a photograph are related information.

  Note that when displaying photographs on a map in an overlapping manner, many images may overlap and become difficult to see due to factors such as the scale of the map, the size of the photographs, and the number of photographs associated with the location. In that case, instead of changing the display size of the photo according to the map scale as shown in FIG. 38 or displaying the photo with a display size proportional to the number of photos as shown in FIG. 39 when the number is large. To display only one representative photo. In addition, only one photograph representing a set that overlaps and becomes difficult to see due to being too gathered may be displayed. This representative photograph can be selected from various viewpoints such as the one with the highest similarity in the set and the one most frequently viewed.

  The processing in steps S148 to S162 has been described as being performed in the digital camera 146. However, when the storage 148 exists separately from the digital camera 146, such processing is stored in the storage 148 as software. It is also possible to actually activate the digital camera 146 and the storage 148 in a divided manner.

[Sixth application]
Next, an outline of a search system as the sixth application will be described with reference to FIG.

  In other words, the search system includes a digital camera 146, a storage 148, a printer 150, and a personal computer (PC) 172. Here, the storage 148 is a storage device built in the PC 172 or accessible by the PC via communication. Further, the PC 172 is configured to be able to read out image data stored in the memory of the digital camera 146 by attaching a memory that is wirelessly or wiredly connected to the digital camera 146 or attached to the digital camera 146. .

The search system having such a configuration operates as follows.
(1) First, the digital camera 146 shoots a subject including an image of the search source printout 152 once printed out by the printer 150.

  (5) The PC 172 extracts an area corresponding to the image of the search source printout 152 from the obtained photographed image data, and extracts features of the extracted area.

  (6) Then, the PC 172 executes a matching process with the feature stored in the storage 148 by the extracted feature.

  (7) As a result, the PC 172 reads the image data corresponding to the matched feature as the original image data of the search source printout 152 from the storage 148.

  (8) Thereby, the PC 172 can print out the read original image data with the printer 150 again.

  Hereinafter, the search system as the sixth application will be described in detail with reference to a block configuration diagram shown in FIG. 40 and an operation flowchart shown in FIG. In these drawings, the same reference numerals are assigned to those corresponding to the fifth application.

  That is, this application stores the image data captured by the digital camera 146 in the storage 148 built in or connected to the PC 172 specified by the user, and the processing shown as the PC side in FIG. This is the case when it operates. The application software is launched in a state where the PC 172 and the digital camera 146 are connected by wire or wirelessly and the communication state is established. This may be a state where the function is activated by an operation of turning on a switch such as “search mode” set in the digital camera 146.

  When the application software operates in this way, a printout shooting process is executed on the digital camera 146 side (step S146). That is, as shown in FIG. 42, at least the search source print is taken by the photographing unit 154 of the digital camera 146 with the search source printout 152 that the user desires to print out again on the table or the wall surface. Photographing is performed so that the out 152 is not missing (step S202). As a result, the obtained captured image data is stored in the storage unit 174 that is a memory in the digital camera 146. The stored photographed image data is transferred to the connected PC 172 by wire or wireless (step S204).

  Then, in the PC 172, the feature extraction unit 176 realized by application software performs a process of extracting features from the transmitted captured image data (step S148). The feature amount extraction process may be performed on the digital camera 146 side. By doing so, the amount of communication from the digital camera 146 to the PC 172 can be reduced.

  Thereafter, the matching unit 178 realized by the application software compares the extracted features with the feature database of the photographed image data configured in the storage 148, and performs matching processing with a DB that sequentially extracts the features with high similarity. Is executed (step S150). That is, based on the calculated feature, the matching unit 178 on the PC 172 side compares the feature included in each image data of the storage 148 (or comprehensively databased), and selects the closest one. Selecting a plurality of closest feature candidates by setting is also effective for usability. This feature includes designation information of the original image data for which the feature is calculated, and the candidate image is called up accordingly.

  Thereafter, the image data of the selected original image candidate (or candidate image) is read from the storage 148 and displayed as an image candidate to be extracted on the display unit 180 which is the display of the PC 172 (step S158), and selected by the user Accept. At this time, the image data of the selected original image candidate (or candidate image) is transferred to the digital camera 146 from the PC 172 as it is or appropriately compressed, and displayed on the display unit 160 of the digital camera 146. It does not matter (step S206).

  Then, in accordance with the selection by operating the mouse or the like, the original image data corresponding to the image candidate stored in the storage 148 is sent to the connected printer 150 and printed out again (step S164). That is, the displayed original image candidate is determined by the user's judgment and passed to the printing process, so that the user can easily reprint the printed out image, which is the original purpose. At this time, it is not only simply printed, but depending on the user judgment among a plurality of candidate images selected, it means that “similar to the target original image, but similar images are collected”, and similar image data This also leads to the realization of a function for batch search.

  In this application, the feature database may be created when the captured image data is transferred from the digital camera 146 to the storage 148 via the PC 172. That is, as shown in FIG. 43, transfer of captured image data from the digital camera 146 to the PC 172 is started (step S208), and the transferred captured image data is stored in the storage 148 by the PC 172 (step S210). ), A feature is created from the captured image data (step S212). Then, the created feature is stored in the storage 148 in association with the captured image data (step S214).

  As described above, in the sixth application as well, as in the fifth application, a file (image data) of an original image is accessed simply by taking a desired search source printout 152 image with the digital camera 146. Therefore, it is possible to provide an intuitive and user-friendly search method.

  In addition, it is possible to search not only the original image data itself but also image data having a similar image configuration, and can provide a secondary but novel application. That is, street signs and posters are photographed in this so-called search mode, and exist in the storage 148 such as a memory attached to the digital camera 146 or an external database accessible via the communication unit 182 as indicated by a broken line in FIG. It is possible to easily search for similar or identical image data from the image data to be processed and its features. Furthermore, an Internet site associated with the data can be displayed on a display such as a PC 172 or a digital camera, or a specific application (such as voice / movie (movies)) can be activated.

  In the above description, the digital camera 146 is used. However, the present application is not limited thereto, and may be a scanner.

  Further, the search source printout 152 actually printed out is photographed by the digital camera 146. However, for example, even when the display displaying the image obtained by photographing the search source printout 152 is photographed by the digital camera 146, it is the same. It can be implemented.

[Seventh application]
Next, a search system as the seventh application will be described. This application is an example applied to application software 188 of a mobile phone 184 with a camera 186 as shown in FIG.

  That is, application software for mobile phones is currently available on most mobile phones, and a large amount of image data can be stored in an internal memory or an external memory card. Furthermore, in a specific mobile site (Internet site for mobile phones), a storage service for image files specifying a user is also provided. In such a situation, a very large amount of image data is accumulated and can be used in a variety of ways such as own activity records and business operations. It was troublesome. Actually, searching from a list of text by title, date, etc. of image data is the center, it is extremely troublesome when there are a lot of image data, and multiple words or long names etc. are input even when entering text Must be said to be inconvenient.

  According to the introduction of this search system, an “image input function” is activated by operating as an application of a camera-equipped mobile phone, and a desired “interesting region extraction” and “feature calculation” are performed. This feature (data) is sent to the corresponding server via the mobile line. The corresponding server may correspond to the above camera on a one-to-one basis, or may correspond to a one-to-many camera. The feature sent to the server is actually matched with the feature read from the database requested by the server by the “matching function” installed in the server, and image data with high similarity is extracted. This extracted image data can be returned from the server to the mobile phone of the transmission source, and the image data can be output from the mobile phone by an unspecified printer. However, if various information related to the image data is added to the image data extracted by the server here, it is possible to expand the function such as “send the information back to the mobile phone”. Then, the extracted image data is highly compressed and returned to the mobile phone, and the user confirms that the image data is the desired image data, and then stores it in the memory area of the mobile phone or displays it on the display 190 of the mobile phone. Needless to say, even just doing it is worth using.

[Eighth application]
Next, a search system as the eighth application will be described.

  This application is configured by a digital camera 146 having a communication function and a server connected by communication, and a function for image search is divided and installed in the digital camera 146 and the server. . The digital camera 146 having a communication function functions as a communication device with a photographing function, and of course includes a camera-equipped mobile phone.

  In this case, like the fifth application described above, the digital camera 146 has a photographing function and a function for calculating features from the image data. In the case of the fifth to seventh applications, the feature (or feature database) to be compared and referred to is originally created based on an image photographed and printed by the user or the digital camera 146. This is because the initial purpose is to photograph and search for a printout on which image data that has already been photographed is printed. On the other hand, this application extends this purpose, and generally features calculated based on street signs, posters, printed materials, and publication images are also captured in the database configured in the storage 148 arranged on the server. The point is greatly different.

  Needless to say, it is possible to perform extraction from an image on a database as well as printout.

  In addition, features extracted from the captured image can be added to the database.

  At the time of registration, the position information related to the image is recognized and registered manually, by a sensor such as GPS, or the character recognition described above. By doing this, it is possible to extract the position information to be added to the captured image by extracting a similar image by searching the database when the image is captured next in the same place.

  FIG. 45 is a diagram showing an operation flowchart of the search system as this application. In the figure, the same reference numerals are assigned to the components corresponding to the fifth application.

  In other words, in this application, for example, a poster such as a product advertisement existing on the street is photographed by the digital camera 146 (step S146). Then, a feature extraction process is executed from the photographed image data by the digital camera 146 (step S148). Then, the extracted features are sent to a predetermined server by the communication unit 170 built in or attached to the digital camera 146.

  The server refers to the feature database configured in the storage 148 accessible by the server, compares the features sent from the digital camera 146 (step S150), and extracts similar image candidates having similar features. (Step S216). The extracted image data of similar image candidates is subjected to a predetermined compression process as necessary to reduce the amount of communication, and then transmitted to the digital camera 146 and simply displayed on the display unit 160 of the digital camera 146. (Step S218). Thereby, the user can select the same as in the fifth application.

  Then, the image data of the extracted (further selected) image candidate is transmitted and output to the digital camera 146, or based on the specific information associated with the extracted (further selected) image candidate feature. The next stage of operation is performed (step S220). The next operation may be an explanation of the product or connection to a mail order site if it is the above product advertisement, or the site screen may be returned to the digital camera 146 as image data. In addition, when a street signboard is photographed, the surrounding information of the signboard is taken in as a feature, and the location of the radio base station location at the time of communication is compared to present the location and address as information to the user. It is also possible.

[9th application]
Next, a search system as the ninth application will be described.

  This application retrieves a plurality of image data from the storage 148 by matching using the first feature based on the image of the photographed search source printout 152, and from the plurality of image data as a result of the retrieval, A single feature or a plurality of pieces of image data are searched by feature matching using a second feature that is narrower than the first feature or in the same region and has a high resolution.

  The search system as this application has the same configuration as that of the fifth application described above. In particular, in this application, the entire feature database in which the summary feature as the first feature is registered in the storage 148, and the second feature database. And a detailed feature database in which detailed features are registered as features.

  Here, as shown in FIG. 46, the outline feature is obtained by extracting a feature with a relatively coarse resolution in an area including most (for example, about 90%) of the entire image data (100%). is there. In addition, as shown in FIG. 47, the detailed features are obtained by extracting features with a higher resolution than the outline feature resolution in the region including the central region portion (for example, about 25% of the center) of the image data. It is a thing. Note that the positional relationship between the original image data and the outline features and detailed features is as shown in FIG.

  FIG. 49 is an operation flowchart of the search system as this application. In the figure, the same reference numerals are assigned to the components corresponding to the fifth application.

  That is, in this application, first, similarly to the fifth application, the search source printout 152 that is desired to be printed out again is pasted on the table or the wall by the photographing unit 154 of the digital camera 146 set to the search mode. In this state, photographing is performed so that at least the search source printout 152 is not missing (step S146).

  Then, the feature extraction unit 156 performs an overall feature extraction process for extracting features from the entire image data captured by the imaging unit 154 (step S222), and the matching unit 158 stores the extracted overall features in the storage 148. Compared with the registered overall feature database configured as described above, matching processing is executed with the overall feature DB that sequentially extracts the features with the highest similarity (step S224).

  Thereafter, the feature extraction unit 156 further extracts, as detailed search target image data, the detailed search target region, in this example, the image data of the central region of the region of interest, from the obtained image data of the entire region of interest. (Step S226). Then, the feature extraction unit 156 performs a detailed feature extraction process for extracting features from the extracted detailed search target image data (step S228). Next, the matching unit 158 compares the extracted detailed feature data with the detailed feature database registered in the storage 148 and registered with the detailed feature database. Processing is executed (step S230). However, in this case, feature matching is not performed with all the detailed features registered in the detailed feature database, but the detailed features corresponding to a plurality of image data extracted by the matching processing with the overall feature DB in step S224 described above. Only feature matching is performed. Therefore, feature matching processing with detailed features that require processing time due to high resolution is minimized. In addition, as a reference to be extracted in the matching process with the entire feature DB in step S224, a method of setting a threshold value for similarity or a method of selecting a fixed number for the top 500 is performed.

  Thus, if image data with a high degree of similarity is extracted as an original image candidate in the matching process with the detailed feature DB, it is displayed on the display unit 160 as an image candidate to be extracted (step S158). When the user's selection is accepted and the user's desired image is determined (step S162), the matching unit 158 converts the original image data corresponding to the image candidate stored in the storage 148 to the connected printer. 150 and print out again (step S164).

  According to such an application, it is possible to improve both the quality (satisfaction) of the search result of the original image data and an appropriate search time.

  In addition, it is possible to obtain a search result in consideration of the region of interest of the photographer. In other words, since the photographer normally captures and captures the main subject in the center of the image area, good search results can be obtained by using the detailed features focused on the center of the image data as shown in FIG. Can do. Therefore, in a system that easily retrieves the original image data from the search source printout 152, which is a printed photo, and easily reprints it, the print photo is highly effective.

  Furthermore, it is highly effective as a means for quickly discriminating differences in details in a search for an original image population that is difficult to classify by keywords. In other words, it is possible to narrow down the search results in stages for a large-scale population.

  Also in this application, it is necessary to create an outline feature and a detailed feature for one original image data in advance and register them in the database, and the registration is performed as described in the fifth application. Just do it. However, it is not always necessary to create both features at the same time. For example, a detailed feature may be created when it becomes necessary at the stage of executing the secondary search.

  Further, the detailed features are not limited to those focused on the central portion of the image data as shown in FIGS.

  For example, as shown in FIG. 51, several detailed features may be set in the image. By disposing detailed features in this manner, failure due to print shooting conditions can be avoided. That is, it is possible to narrow down by dynamically changing the position and number.

  Also, as shown in FIG. 52, the attention area may be a detailed feature at the in-focus position at the time of photographing the original image. With such detailed features, a result reflecting the photographer's intention can be expected.

  Further, as shown in FIG. 53, detailed features are created for the same region as the summary features and registered in the database, and at the time of feature matching with actual detailed features, some of the regions, that is, FIGS. An area as shown in FIG. 52 may be used as the reference area 192 and another area may be used as the non-reference area 194.

  In addition, although this application was demonstrated corresponding to the said 5th application, of course, it is applicable similarly to the said 6th thru | or 8th application.

[10th application]
Next, a search system as the tenth application will be described.

  This application is an example in which a digital camera 146 having a communication function is used, and by capturing a pre-registered image, the image is recognized, and a predetermined operation (for example, audio output or a predetermined program) is performed according to the recognition result. This is applied to the case of starting (or displaying a predetermined URL). The digital camera 146 having a communication function functions as a communication device with a photographing function, and of course includes a camera-equipped mobile phone.

  When recognizing an image, the image data is registered as a database to be referred to (so-called dictionary data), but it is efficient and practical to compare the features of the image rather than comparing the images as they are. Use the extracted feature database. This database may be built-in or may exist on a server through communication.

  In this application, the arrangement relation of the feature points of the image is calculated as a combination of vector quantities, and a large number of sets are defined as features. At that time, the accuracy of this feature differs depending on the number of feature points that appear, and if the definition of the original image data is high, a large number of feature points can be detected. Is calculated. At this time, if features are calculated on the same image material based on image data with reduced definition, the number of feature points is relatively small, so the features themselves have a small capacity. A small capacity is inferior in matching accuracy, but has advantages such as a high matching speed and a high communication speed.

  This application focuses on this, and when registering image data as reference data (features), a feature is calculated from a plurality of different resolutions when registering one image material, and a database that is individualized for each resolution is created. Configure. A matching server corresponding to each database is connected to each database so that the databases can operate in parallel. That is, as shown in FIG. 54, a primary feature matching server and primary information DB 196-1, secondary feature quantity matching server and secondary information DB 196-2,. Information DB196-n is prepared. The secondary feature quantity matching server and the secondary information DB 196-2 to the n-th feature quantity matching server and the n-order information DB 196-n are higher than the primary feature quantity matching server and the primary information DB 196-1. A database of fine features or special categories.

  After preparing such a matching processing system, as shown in FIG. 54, an already registered design (object) is photographed from a digital camera 146 having a communication function (step S232), and the digital camera The feature is calculated from the arrangement relationship of the feature points by the application software built in 146 (step S148). And the matching process with each DB is performed by transmitting the characteristic to each matching server via communication (step S150). If a matching result is obtained by this matching processing, operation information (for example, a URL link) associated with the result is acquired (step S234), and the operation information is transmitted to the digital camera 146, for example, A designated operation such as 3D object acquisition and display is performed (step S236). Of course, the digital camera 146 may transmit the entire captured image or a part thereof to the matching server, and may execute step S148 on the matching server.

  Assuming that the camera resolution at this time is 2 million pixel class, even when searching with a matching server via communication, if the matching is performed with the data from the feature database of the resolution of 2 million pixel class, the false recognition rate Less is. However, since matching in the feature database of low resolution (for example, VGA class resolution) that operates simultaneously responds at high speed, the result is sent to the digital camera 146 first. Thus, it is advantageous in terms of speed and recognition accuracy to arrange matching servers in parallel for each resolution. In addition, the response from the follow-up high-resolution matching server may differ from the low-resolution matching server that has already appeared first. In such a case, the display based on the earlier result is performed first, The display is updated based on the follow-up result. For example, when trying to recognize a banknote or the like, even if the answer in the low resolution matching is “100 dollar bill”, the answer in the high resolution matching is “100 dollar bill is HD85866756A”. Thus, detailed or correct results due to higher definition can be obtained. In addition, it is also effective to display a plurality of candidates for the low resolution result and narrow down the result candidates as the high resolution result arrives.

  Further, as described above, in the high resolution matching server, the capacity of the feature itself is large and the feature amount of the XGA class is enlarged to about 40 kB, but is reduced to about 10 kB in advance by the low resolution matching. In the second and subsequent matching servers and databases, if only the difference from the lower resolution database is held, a smaller database configuration is realized, which leads to speeding up of recognition processing. In addition, when extraction with features (a method of assigning areas and comparing density values) is advanced, it is generally 10 kB or less, and multidimensional features combining both methods are also improved in recognition accuracy. It is confirmed that it is effective.

  In this way, the resolution of a part or the whole of the captured image plane is multi-staged, and the realization of the matching hierarchy is substantially faster than the case where a plurality of matching servers are distributed and processed in a cluster manner. It is effective in both recognition accuracy.

  In particular, this method is effective when the number of images registered in the database in advance is very large (1000 or more), and it is also effective when images with high similarity are included therein.

[11th application]
Next, a search system as the eleventh application will be described.

  As shown in FIG. 55, the search system as the eleventh application includes a mobile phone 184 with a camera 186 and a search unit. The mobile phone 184 with the camera 186 includes a camera 186 that inputs an image and a display 190 that outputs an image of a search result. The search unit searches for an image from the database using the hierarchically managed features based on the image input by the camera 186. Here, the search unit is realized by the application software 188 of the mobile phone 184 with the camera 186 and the matching processing unit 200 configured in the server 198 that can communicate with the mobile phone 184 with the camera 186.

  The server 198 further includes a feature management database (DB) 202 in which a plurality of features are registered and the layers are hierarchically managed. The template registered in the feature management DB 202 is created by the feature creation unit 204 from the target image 206 arranged on the paper surface 208 by the desktop publishing (DTP) 210.

  That is, in the search system as this application, the target image 206 is printed on the paper surface 208 by the DTP 210 in advance, and the feature creation unit 204 creates the feature of the target image 206. Then, the created feature is registered in the feature management DB 202 of the server 198. If there are many target images 206 to be registered, the creation and registration of such features are repeated.

  Then, when a user who desires to retrieve the target image 206 from the paper 208 using the camera 186 of the mobile phone 184, the application software 188 performs image feature extraction from the input image. Then, the application software 188 sends the extracted features to the matching processing unit 200 of the server 198. The matching processing unit 200 performs matching with the features registered in the feature management DB 202. If the matching result is acquired, the matching processing unit 200 sends the matching result information to the application software 188 of the mobile phone 184 with the camera 186. The application software 188 displays the result information on the display 190.

  As described above, in the eleventh application, a plurality of features are extracted from the input image, and a feature group composed of these features is compared with a feature group for each target object registered in advance (matching processing is performed). ) To identify the same object.

  The feature in the image here means that the difference from other pixels is a certain level or more, for example, contrast of light and darkness, color, distribution of surrounding pixels, differential component value, arrangement of features, etc. Can be mentioned. In the eleventh application, after extracting the feature, it is registered for each object. At the time of actual identification, the input image is searched to extract features and compared with pre-registered data.

  Hereinafter, with reference to FIG. 56, the flow of operation control of identification processing in the matching processing unit 200 in the eleventh application will be described. First, the feature of the recognition element of the target Z (for example, the target image 206) registered in advance is read from the feature management DB 202 in which the feature point group is recorded (step S238). Subsequently, the feature is input to the matching processing unit 200 that performs feature comparison (step S240). Then, the matching processing unit 200 compares and compares the feature with the feature of the input object (step S242). Thereafter, the identity of the object Z and the input object is determined (step S244). Then, it is determined whether or not the number of matching features is equal to or greater than a predetermined value (here, X) (step S246). When step S246 is branched to NO, the process returns to step S242. On the other hand, when step S246 is branched to YES, it is determined that the recognition element of the object Z currently being compared is the same as the input object (step S248).

  Thereafter, it is determined whether or not the comparison has been completed for all recognition elements (step S250). When step S250 is branched to NO, the feature in the feature group of the next recognition element is input as comparison data to the matching processing unit 200 (step S252), and the process returns to step S242.

  When step S250 is branched to YES, it is determined whether or not the number of matching features is equal to or greater than a predetermined value (here, Y) (step S254). Here, when this step S254 is branched to YES, it is determined that the input object and the object Z coincide with each other, and this is displayed on the display 190 to notify the user (step S256). On the other hand, when step S254 is branched to NO, it is determined that the input object does not match the object Z (step S258).

  At the time of actual identification, when a numerical value indicating the degree of similarity (difference between components of features) exceeds a preset threshold value, the feature is determined as a similar feature. Then, it is determined that an object having a plurality of matching features is the same as the object of the input image. At this time, the features in the input image are compared with the feature groups registered in advance as follows.

  First, the object is divided into a plurality of elements and registered. Thereby, at the time of comparison of objects, it is made to recognize with the judgment logic that the said object will not be recognized if a some element (for example, 3 pieces) is not recognized.

  Secondly, when a similar object is shown in the image when the object is recognized, for example, the company S that uses the object OBJ1 (feature; A, B, C) as its own logo mark; Assume that the company M uses the object OBJ2 (feature; E, F, G) as its own logo mark. Here, it is assumed that Company S and Company M are competing companies. In such a case, naturally, confusion between the two logo marks should be avoided as much as possible. In view of such circumstances, in the eleventh application, when the features A and E are simultaneously detected from the same screen, neither object is recognized. That is, the recognition determination is made strict.

  Thirdly, conventionally, even when the number of features is recognized, the sentence expression for transmitting the recognition result to the user is the same. For this reason, for example, when only a part of the features can be recognized, that is, when the matching degree between the input image and the comparison image is a matching degree including uncertainty, this fact cannot be transmitted to the user. On the other hand, in the eleventh application, when the number of recognition elements is small, the result display method (expression method) is changed to an expression including uncertainty.

The following effects can be obtained by the above-described devices.
First, it is possible to reduce the probability of erroneous recognition due to matching only a part of the object.

  Secondly, it is possible to tighten the criteria for particularly avoiding erroneous recognition of an object.

  Thirdly, even when the accuracy of the identity determination of an object is lower than a predetermined value, it is possible to notify the user of the match determination result after alerting the user.

  By the way, in the case of the objects OBJ1 (features; A, B, C) and OBJ2 (features: E, F, G) in which the features in the object are divided and registered, the following determination logic is used. Recognition by.

  First, the recognition of the object OBJ1 is not successful unless “A and B and C”.

  That is, when the object OBJ1 composed of the features A, B, and C as recognition elements is recognized, the object OBJ1 is successfully recognized in the state of recognition of one or two features of A, B, and C. Don't make it a form.

  As a modification, the features A, B, and C are weighted as evaluation points. For example, the weights are 1.0, 0.5, and 0.3, respectively. Here, if authentication is performed when the total evaluation score exceeds 1.5, when the features A and B are found as recognition elements, the total evaluation score is 1.5, so the object OBJ1 is recognized. On the other hand, when the features B and C are found, the object OBJ1 is not recognized.

  The evaluation points of these recognition elements can be managed together with the characteristics of the recognition elements.

  In addition, it is possible to change the priority of each element as a logical expression. In addition to “A and B and C”, for example, “A and (B or C)” or “A or (B and C)” Is possible. These examples are examples in which the feature A is an essential element for successful recognition.

  It should be noted that the above evaluation points and logical expression examples can be used in combination. In other words, the logical expression priority and the weighting of each element can be used in combination.

  Second, when “E and A” is extracted, it is never assumed that both the object OBJ1 and the object OBJ2 are recognized.

  For example, if company S, which uses the object OBJ1 as a logo, and company M, which uses the object OBJ2 as a logo, have a competitive relationship and want to avoid confusion between them as much as possible, When the object OBJ2, which is a logo of M company, is shown on the same screen, neither logo is recognized. In this case, the reason why the user cannot recognize the target image is not because the target image is not detected, but because the recognition element is detected from both (A, B, C) and (E, F, G). It is displayed that it is because it is.

  In this way, in the eleventh application, regarding the identification of logos of companies that are in a mutually competitive relationship, for example, only the object OBJ1 that is the logo of the S company or only the object OBJ2 that is the logo of the M company, The logo is recognized only when only one of them is in the captured image. Specifically, the object OBJ1 or the object only when only one of (A, B, C) or only one of (E, F, G) is detected in one image. The object OBJ2 is recognized. In other words, when any one of (A, B, C) and any one of (E, F, G) is detected in one image, both the object OBJ1 and the object OBJ2 are recognized. do not do.

Third, when only a part such as “A and B” is extracted, the presentation method of the result is changed (prevents the expression including uncertainty)
For example, regarding the recognition of the object OBJ1, when all the features A, B, and C of the recognition element are recognized, the recognition result is presented to the user with a strong expression “object OBJ1 has been recognized”. . In addition, when two recognition elements such as feature A and feature B, feature B and feature C, or feature A and feature C of the recognition element can be recognized, for example, “It seems to be the object OBJ1” The recognition result is presented to the user in a slightly weakened expression. Further, when there is one recognition element that can be recognized, the recognition result is presented to the user in an expression including uncertainty such as “the object OBJ1 may be recognized”.

  As a modification of the eleventh application, when the above-described weighted evaluation points are used, the above-described device in the expression method when presenting the recognition result based on the total evaluation points to the user can be considered. . In addition, it goes without saying that the idea of the above-described expression method when presenting the recognition result to the user can be applied in various scenes. For example, the present invention can be applied to recognition of a desired recognition element alone. In addition, for example, the above-described expression method when presenting the recognition result to the user can be applied depending on the number of matching features in the recognition element and the matching degree between the extracted feature and the registered feature.

  In the eleventh application, the feature creation unit 204 may of course operate on the server 198. The paper surface 208 means a display surface and is not necessarily limited to paper. For example, it may be another material such as metal or plastic, or may be a video display device such as a liquid crystal monitor or a plasma television. Of course, it is natural that the information displayed on them corresponds to that displayed in the visible light region for humans. However, as long as the information can be input to the camera 186, information that is invisible to humans may be used. In addition, since anything that can be acquired as an image is a target, for example, an image such as an X-ray image or a thermography may be used.

  In FIG. 55, an image including a target image input from the camera 186 is transmitted from the mobile phone 184 with the camera 186 to the matching processing unit 200 in the server 198. At this time, the image acquired by the camera 186 may be transmitted as it is as image data, or may be transmitted by reducing the image. Of course, features used for matching may be extracted from the image and transmitted. Of course, both the image and the feature may be transmitted. In other words, any form of data may be transmitted as long as the data can be derived from the image.

    DESCRIPTION OF SYMBOLS 10 ... Feature detection, 12 ... Feature selection, 14 ... Feature recognition, 16, 134 ... Database, 18 ... dBTree construction, 20 ... dBTree search, 22 ... Index matching, 24 ... Feature space, 26 ... Subspace, 100 ... Information presentation Device: 102 Storage unit 104 Data set server 106 Information server 108, 154 Imaging unit 110 Recognition / identification unit 112 Information specifying unit 114 Presentation image generation unit 116 Image display device 118 ... Data set, 120 ... Position and orientation calculation unit, 122 ... Basic data, 124 ... Storage media, 126 ... Bar code scanner, 128 ... Weigh scale, 130,186 ... Camera, 132 ... Control unit / cash storage box, 136: Monitor, 138: Field of view, 140: Feature, 1 42 ... Image, 144 ... Reference image, 146 ... Digital camera, 148 ... Storage, 150 ... Printer, 152 ... Search source printout, 156,176 ... Feature extraction unit, 158,178 ... Matching unit, 160,180 ... Display unit 162 ... Image candidates, 164 ... "Previous" and "Next" icons, 166 ... "Determination" icon, 168 ... Thick frame, 170, 182 ... Communication part, 172 ... Personal computer (PC), 174 ... Storage part, 184 ... mobile phone, 188 ... application software, 190 ... display, 192 ... reference area, 194 ... non-reference area, 196-1 ... primary feature matching server and primary information DB, 196-2 ... secondary feature matching server And secondary information DB, 196-n ... n-order feature matchons 198 ... server, 200 ... matching processing unit, 202 ... feature management database (DB), 204 ... feature creation unit, 206 ... target image, 208 ... paper surface, 210 ... desktop publishing (DTP).

Claims (11)

Detect features that have a specified attribute extreme value (Local Maximum and / or Minimum) in one 2D or 3D image data (10),
Excluding features present along the edges and line contours from the detected features (12);
Assign the remaining features to a plane (14),
Select some features from the assigned features using local information (14),
Perform feature matching on the selected feature (14),
A feature matching method for recognizing objects in 2D or 3D image data,
Creating a plurality of image data having different scales from the one two-dimensional or three-dimensional image data,
At least one of detection of the feature, exclusion of the feature, assignment of the remaining feature, selection of the partial feature, and execution of the feature matching is performed on the plurality of different created image data. Called
A feature matching method characterized by that.   2. The feature matching method according to claim 1, wherein the selection of some of the features uses a constraint based on a texture-ness of the feature.   The feature matching method according to claim 2, wherein the selection of the part of the features further uses a restriction by an orientation component.   4. The feature matching method according to claim 3, wherein the selection of some of the features further uses a constraint based on a scale.   The feature matching method according to claim 1, wherein the feature matching is performed using a RANSAC method.   The feature matching method according to claim 1, wherein the feature matching is performed using a dBTree method (18, 20, 22). further,
Calculating the accuracy of the above feature matching (22),
A plurality of recognition results are output based on the calculated accuracy (22),
The feature matching method according to claim 1, wherein:   The feature matching is performed by matching the two-dimensional or three-dimensional image data based on a combination condition of a plurality of image data registered in a database expressed by a logical expression (S242). Item 2. The feature matching method according to Item 1.   The feature detection is performed by applying a high-pass filter to a point of the two-dimensional or three-dimensional image data, and comparing an output value of the high-pass filter with a threshold value. The feature matching method described. A feature storage unit (134) configured to record features of a plurality of pre-registered products;
An image input unit (130) configured to photograph a product;
The product photographed by the image input unit by extracting features from the image obtained by photographing the product by the image input unit and comparing and comparing the extracted features with the features recorded in the feature storage unit. An automatic recognition unit (132) configured to automatically recognize
A settlement unit (132) that performs a settlement process using the recognition result of the automatic recognition unit,
Comprising
The product recognition system according to claim 1, wherein the automatic recognition unit uses the feature matching method according to claim 1. A specific information storage unit (134) configured to record specific information including at least one of weight and size of the plurality of pre-registered products,
The automatic recognition unit uses specific information recorded in the specific information storage unit in order to increase the recognition accuracy of the product.
The product recognition system according to claim 10.

Images