WO2008100248A2 - Procede de mise en correspondance de caracteristiques - Google Patents

Procede de mise en correspondance de caracteristiques Download PDF

Info

Publication number
WO2008100248A2
WO2008100248A2 PCT/US2007/003653 US2007003653W WO2008100248A2 WO 2008100248 A2 WO2008100248 A2 WO 2008100248A2 US 2007003653 W US2007003653 W US 2007003653W WO 2008100248 A2 WO2008100248 A2 WO 2008100248A2
Authority
WO
WIPO (PCT)
Prior art keywords
features
image
feature
image data
matching
Prior art date
Application number
PCT/US2007/003653
Other languages
English (en)
Other versions
WO2008100248A3 (fr
Inventor
Yuichiro Akatsuka
Takao Shibasaki
Yukihito Furuhashi
Kazuo Ono
Ulrich Neumann
Suya You
Original Assignee
Olympus Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Olympus Corporation filed Critical Olympus Corporation
Priority to JP2009549056A priority Critical patent/JP2010518507A/ja
Priority to CN2007800512883A priority patent/CN101627399B/zh
Priority to PCT/US2007/003653 priority patent/WO2008100248A2/fr
Priority to EP07750486A priority patent/EP2106599A4/fr
Publication of WO2008100248A2 publication Critical patent/WO2008100248A2/fr
Publication of WO2008100248A3 publication Critical patent/WO2008100248A3/fr
Priority to US12/539,786 priority patent/US20100092093A1/en
Priority to HK10101874.7A priority patent/HK1134158A1/xx

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/5838Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using colour
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/40Extraction of image or video features
    • G06V10/44Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/757Matching configurations of points or features
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00Indexing scheme relating to image or video recognition or understanding
    • G06V2201/11Technique with transformation invariance effect

Definitions

  • the present invention relates to a feature matching method for recognizing an object in two- dimensional or three-dimensional image data.
  • the present invention has been made in view of the circumstance described above. It is an object of the invention to provide ia feature matching method that enables a high speed processing with a simplified system.
  • a feature matching method for recognizing an object in two-dimensional or three- dimensional image data comprising: detecting features in each of which a predetermined attribute in the two-dimensional or three-dimensional image data takes a local maximum and/or minimum; excluding features existing along edges and line contours from the detected features; allocating the remaining features to a plane; selecting some features from the allocated features by using local information; and performing feature matching for the selected features.
  • FIG. 1 is a block diagram depicting a feature matching method according to a first embodiment of the present invention.
  • FIG. 2A is a view showing an original image.
  • FIG. 2B is a view showing an array of multi-scale images that are used for detecting features.
  • FIG. 2C is a view showing features detected by a multi-scale feature detection.
  • FIG. 3A is a view showing matching between features of an original image and features of an image obtained by moving the original image in parallel by 20 pixels .
  • FIG. 3B is a view showing matching between features of an original image and features of an image obtained by multiplying the original image by 0.7.
  • FIG. 3C is a view showing matching between features of an original image and features of an image obtained by rotating the original image by 30 degrees..
  • FIG. 3D is a view showing matching between features of an original image and features of an image obtained by carrying out sharing of 0.4 so that the original image is equivalent to an affine-3D transformation.
  • FIG. 4 is a view showing a final matching result from a dataset.
  • FIG. 5 is a block diagram depicting a high speed matching search technique in a feature matching method according to a second embodiment of the present invention.
  • FIG. 6 is a view for explaining a Brute-Force matching technique .
  • FIG. 7 is a view showing an example of a matching search of two multi-dimensional sets using an exhaustive search.
  • FIG. 8 is a view showing an experimental statistic result of a time required for a matching search using an exhaustive search with respect to a large amount of feature points.
  • FIG. 9A is a view showing procedures for hierarchically decomposing a whole feature space into some subspaces.
  • FIG. 9B is a view showing the hierarchically decomposed subspaces.
  • FIG. 10 is a view showing a statistic result of a comparative experiment between a Brute-Force matching technique and high speed matching technique with respect to a small database.
  • FIG. 11 is a view showing a statistic result of a comparative experiment between a Brute-Force matching • technique and high speed matching technique with respect to a large database.
  • FIG. 12 is a view showing the configuration of an information retrieval system of a first application.
  • FIG. 13 is a flowchart showing operation of the information retrieval system of the first application.
  • FIG. 14 is a view showing the configuration of a modified example of the information retrieval system of the first application.
  • FIG. 15 is a view showing the configuration of an information retrieval system of a second application.
  • FIG. 16 is a view showing the configuration of a modified example of the information retrieval system of the second application.
  • FIG. 17 is a view showing the configuration of another modified example of the information retrieval system of the second application.
  • FIG. 18 is a flowchart showing operation of a mobile phone employing the configuration of FIG. 17.
  • FIG. 19 is a view showing the configuration of an information retrieval system of a third application.
  • FIG. 20 is a view showing the configuration of a product recognition system of a fourth embodiment.
  • FIG. 21 is a view of features preliminarily registered in a database (DB) .
  • FIG. 22 is a flowchart of product settlement by the product recognition system of the fourth application.
  • FIG. 23 is a flowchart of an extraction and recognition process of features .
  • FIG. 24 is a view used to explain an object of comparison between features in an image from a camera and features in a reference image registered in advance.
  • FIG. 25 is a view of an overall configuration of a retrieval system of a fifth application.
  • FIG. 26 is a block diagram of the configuration of the retrieval system of the fifth application.
  • FIG. 27 is a flowchart showing operation of the retrieval system of the fifth application.
  • FIG. 28 is a detailed flowchart of a process for matching with the DB.
  • FIG. 29 is a view of a display screen of a display unit of a digital camera in the event of displaying only one image candidate
  • FIG. 30 is a view of a display screen in the event of displaying nine image candidates.
  • FIG. 31 is a flowchart used to explain an example of a feature DB creation method.
  • FIG. 32 is a flowchart used to explain another example of the feature DB creation method.
  • FIG. 33 is a flowchart used to explain another example of the feature DB creation method.
  • FIG. 34 is a flowchart used to explain yet another example of the feature DB creation method.
  • FIG. 35 is a view used to explain an operation concept in the case that a station name board of a station is photographed as a signboard.
  • FIG. 36 is a view of an example displaying a photograph on a map.
  • FIG. 37 is a view of another example displaying a photograph on a map.
  • FIG. 38 is a view of an example of a photograph display on a map in the case of a large number of photographs.
  • FIG. 39 is a view of another example of a photograph display on a map in the case of a large number of photographs.
  • FIG. 40 is a block diagram of the configuration of a retrieval system of a sixth application.
  • FIG. 41 is a flowchart showing operation of the retrieval system of the sixth application.
  • FIG. 42 is a detailed flowchart of an image acquisition process for imaging a printout.
  • FIG. 43 is a flowchart used to explain a feature DB creation method.
  • FIG. 44 is a block diagram of the configuration of a camera mobile phone employing a retrieval system of a seventh application.
  • FIG. 45 is a flowchart showing operation of a retrieval system of an eighth application.
  • FIG. 46 is a view used to explain general features used in a retrieval system of a ninth application.
  • FIG. 47 is a view used to explain detail features used in the retrieval system of the ninth application.
  • FIG. 48 is a view used to explain a positional relationship between original image data, the general features, and the detail features.
  • FIG. 49 is a flowchart showing operation of the retrieval system of the ninth application.
  • FIG. 50 is a view used to explain detail features with attention drawn to a central portion of image data .
  • FIG. 51 is a view used to explain detail features distributively disposed within an image.
  • FIG. 52 is a view used to explain detail features in which an attention region is placed in focus position in the event of imaging an original image.
  • FIG. 53 is a view used to explain detail features created in a region identical to that of general features .
  • FIG. 54 is a flowchart showing operation of a retrieval system of a tenth application.
  • FIG. 55 is a view showing the configuration of a retrieval system of an eleventh application.
  • FIG. 56 is a flowchart showing a recognition element identification process.
  • a feature matching method is also referred to as a PBR (Point Based Recognition) .
  • this method includes three portions: feature detection 10; feature adoption 12; and feature recognition 14.
  • the features are spatially and temporally dispersed. For example, in the case where an image is to be recognized by this method, feature matching in a two-dimensional expanse is carried out. Recognition of a moving picture can be carried out in consideration of time-based expanse.
  • the feature detection 10 detects spatially stable features, which do not depend on a scale or a layout, from inputted object data, for example, an image.
  • the feature adoption 12 adopts a robust and stable portion for making robust recognition from the features detected by the feature detection 10.
  • the feature recognition 14 uses the features extracted by the feature adoption 12 and additional constrains to locate, index, and . recognize objects pre-analyzed and stored in a database 16.
  • a point feature has advantages over the large-scale features (such as lines and faces) in distinctiveness, robustness to occlusions (when part of the features is hidden) , and good invariance to affine transformation.
  • the related disadvantages of point features are that often only a sparse set of points and measurements are available, and matching them is also difficult, because only local information is available.
  • the feature detection 10 in the present embodiment uses a method for finding the point features with rich- texture regions.
  • three filters are used.
  • a high-frequency passed filter is used to detect the points having local maximum responds.
  • R is a 3 x 3 windows centered are point P
  • the second filter is a distinctive feature filter.
  • the points lie along the edges or linear contours are not stable for matching. This is so- called matching arbitrary effect (effect that can be seen as if matching were successful), and these points must be removed for reliable matching.
  • the covariance matrix of image derivatives is a good indicator to measure the distributions of image structure over a small patch. Summarizing the relationship between the matrix and image structure, small eigenvalues of correspond to a relatively constant intensity within a region. A pair of large and small eigenvalues corresponds to a high texture pattern, and two large eigenvalues can represent linear features, salt-and-pepper textures, or other patterns. Therefore, it is possible to design the filter to remove those linear feature points.
  • M is a 2 x 2 matrix computed from image derivatives
  • ⁇ i and ⁇ £ are eigenvalues of M.
  • the measure of a linear edge response is
  • the third filter is an interpolation filter which iteratively refines the detected points to sub-pixel accuracy.
  • An affine plane is first used to fit the local points to reconstruct a continuous super-plane. Then the filter iteratively refines the points upon the reconstructed plane till an optimal fitting solution is converge and the final fitting is used to update the points to the sub-pixel accuracy.
  • a novel aspect of the present embodiment is that scale invariance is improved by employing a multi- resolution technique, thereby extracting features from each of a plurality of images having various resolutions .
  • affine scale invariance To achieve affine scale invariance, a multi- resolution strategy is employed in the above feature detection processing. Unlike the traditional pyramid usage in which the main goal is to accelerate the processing, i.e. coarse-to-fine search, it is a goal to detect all the possible features across different scales to achieve a effective affine scale invariance. So, the features in each level of the pyramid are processed dependently.
  • FIGS. 2A to 2C each show a result for this approach has. been applied to a cluttered scene.
  • FIG. 2A shows an original image
  • FIG. 2B shows an array of multi-scale images that are used for detecting features
  • FIG. 2C shows the detected features, respectively.
  • the feature adoption 12 in the present embodiment adopts each feature point using its local region information, called affine region.
  • affine region Three constraints are used to quality the local region, i.e., intensity, scale, and orientation.
  • the intensity constraint is the image gradient value G(x, y) calculated inside the region pixels, which indicate the texture-ness of the feature.
  • the intensity adoption is sufficient to match the images under small linear displacements.
  • a simply correlation matching strategy could be used.
  • an affine warping matching is effect to compensate for the distortion.
  • the simple intensity adoption is not sufficient. It is well known that the simple intensity correlation is not scale and rotation invariant. In this situation, the all the possible constraints should be considered in order to adopt the matching points as in a robust and stable multi-quality representation.
  • the scale and local orientation constraints are embedded into the adoption and matching processing. First, the continuous orientation space is quantized into discrete apace.
  • a novel aspect of the present embodiment is that features of the orientations normalized from the peripheral regions of the features are provided in the form as shown in formula (8) below.
  • R is a voting range that its size is defined by a Gaussian filter used for generating a scale pyramid. For any point P(XJ_, y ⁇ ) within the voting range, its contribution to a quantized orientation is represented by formula (8) below:
  • the final output of the feature adoption 12 is a compact vector representation for each matching point and associated region that embeds all the constraints, achieving affine geometry and illumination invariance .
  • FIGS. 3A to 3D each show a result of using this approach to a scene under different affine transformation.
  • FIG. 3A is a scene obtained by moving the original image in parallel by 20 pixels;
  • FIG. 3B is a scene obtained by multiplying the original image by 0.7;
  • FIG. 3C is a scene obtained by rotating the original image by 30 degrees;
  • FIG. 3D is a scene obtained by carrying out sharing of 0.4 so that the original image is equivalent to an affine-3D deformation, respectively.
  • the features detected by the feature detection 10 and adopted by the feature adoption 12 establish good characteristics for geometry invariance.
  • the matching is performed based on the adopted feature representations.
  • the SSD (Sum of Square Difference) is used for the similarity matching, i.e. for each features P, a similarity value Similarity (P) is computed against the matched image, and the SSD search is performed to find the best matched point with maximal similarity. If the following relationship is established,
  • a RANSAC based approach is used to make a search for a pair that fulfills the fundamental geometrical constraint. It is well known that the matched image features corresponding to a same object will fulfill a 2D parametric transformation (a homography) . To accelerate the computation, the feature recognition 14 uses the 2D affine constraint to approximate the homography for outlier removing, which requires only 3 points to estimate the parametric transformation. First, the RANSAC iteration is applied using randomly selected 3 features to estimate an initial transformation Mj_ n ⁇ f
  • the estimated parametric transform is then refined iteratively using all the matched features.
  • the matching outliers are indicated for those matching points that have large fitting residuals.
  • the final output of the feature matching is a list of matching points with outlier indicators and the estimated 2D parametric transformation (affine parameters) .
  • FIG. 4 shows an example of the final matching results obtained by this feature recognition 14 from an object dataset pre-analyzed and stored in the database 16.
  • the present embodiment describes a fast matching search for achieving further speed in the foregoing feature recognition 14.
  • This fast matching search is referred to as a Data Base Tree (dBTree) .
  • the dBTree is an effective image matching search technology that can rapidly recover possible matches to a high-dimensional database 16 from which PBR feature points as described in the foregoing first embodiment have been extracted.
  • the problem is a typical NP data query problem, i.e. given an N-dimension database points and a query point q, it is wanted that the closest matches (Nearest Neighbors) of q among the database is fined.
  • the fast matching search according to the present embodiment is a tree- structure matching approach that forms a hierarchical representation of the PBR features to achieve an effective data representation, matching, and indexing of high-dimensional feature spaces.
  • the dBTree matcher is composed of dBTree construction 18, dBTree search 20, and match indexing 22.
  • the dBTree construction 18 creates a hierarchical data representation over the PBR feature space (hereinafter, referred to as a dBTree representation) from the PBR features obtained from the object data input as described in the foregoing first embodiment.
  • the created dBTree representation is registered in the database 16.
  • the dBTree representation relevant to data on a number of objects is thus registered in the database 16.
  • the dBTree search 20 searches over the dBTree space configured in the database 16 to locate possible Nearest Neighbors (NNs) of given PBR features obtained from the input object data as described in the first embodiment.
  • the match indexing 22 uses the found NNs and additional PBR constrains to locate and index corrected matches .
  • the goal of the match search is to rapidly recover possible matches to a high-dimensional database.
  • Pi and qj are k-dimensional vectors, for example, 128-D vector for PBR feature
  • a Euclidean distance for the invariant features is used for the similarity matching, i.e. for each feature pj_, a similarity value Similarity (pj.) is computed against the matched features qj , and the matching search is performed to find the best matched point with minimal Euclidean distance.
  • FIG. 8 shows experimental statistic results (over 50 testing images) of the matching time of Brute-Force search with respect to the number of feature points (the total feature numbers NxM of input image features N and database features M) .
  • a central data structure in the dBTree matcher is a tree structure that forms an effective hierarchical representation of the feature distribution.
  • the dBTree matcher represents the k-dimension data in a balanced binary tree by hierarchically decomposing the whole space into several subspaces according to the splitting value of each tree-node.
  • the root-node of this tree represents the entire matching space, and the branch-nodes represent rectangular sub-spaces that contain the features havingdifferent characters of their enclosed spaces.
  • the tree representation should provide a fast way to access any input feature by feature's position. By traversing down the hierarchy until find the sub-spaces containing the input feature, an identifying operation of the matching points can be carried out merely by scanning trough few nodes in the sub-spaces .
  • FIGS. 9A and 9B each show procedures of hierarchically decomposing the whole feature space 24 into several subspaces 26 to build a dBTree data structure.
  • input point sets are partitioned (segmented) in accordance with a defined splitting measure.
  • the median filtering is used for the embodiment so that an equal number of points fall into each side of the split subspaces 26.
  • Each node in the tree is defined by a plane through one of the dimensions that partitions the set of points into left/right and up/down subspaces 26, each with half the points of the parent node.
  • These children nodes are again partitioned into equal halves, using planes through a different dimension.
  • the process is repeated until partitioning reaches log (N) levels, with each point in its own leaf.
  • search for closest subspace 26 There are two steps for search a query point over the tree: search for closest subspace 26 and search for closest node within the subspace 26.
  • the tree is traversed to find the subspace 26 containing the query point. Since the number of subspace 26 is relatively small, it is possible to rapidly locate the closest subspace 26 with only log (N) comparisons, and the space would have a high probability that contains that the matched points.
  • N log
  • two strategies are employed to overcome those problems and to achieve effective matching for high -dimensional dataset.
  • a tree-pruning-filter (branch cutting filter) is used to cut (reduce) the number of branches needs to be examined. After exploring a specific number of nearest branches (i.e. search-steps), the branch search is enforced stopped.
  • the distance filtering could also be used for this purpose, but extensive experiments have shown that using the search- steps filtering has demonstrated better performance in terms of corrected matches and computation cost. Although search results obtained from the strategy give approximate solutions are observed, experiments shows that the mismatching rate only increased less 2%.
  • the second strategy (method) is to improve the node search by introducing a node-distance-filter.
  • a distance threshold is used for limiting the node research range.
  • the node search is performed as a circular pattern so that nodes that are closer to the target will be searched first. Once the search boundary is reached, the search is enforced stopped and nearest neighbors (NNs) are outputted.
  • the next step is to decide if the NNs are accepted as correct matches.
  • a related matching cost threshold is used for selecting correct matching, i.e. if the similarly difference between the highest NN and second-highest NN (a distance up to the highest NN/a distance up to the second-highest NN) is less than a pre-defined threshold, the point is accepted as correct match.
  • FIGS. 10 and 11 each show a statistical result (over 50 testing images) of comparative experiment between the Brute-Force and dBTree matching methods. The difference in similarity between the highest
  • NN and the second-highest NN is obtained as a parameter that expresses preciseness in identity judgment of the similarity of that point.
  • the number per se of matching points in the image is also obtained as a parameter that expresses preciseness in identity judgment of the image.
  • a differential total sum (residual difference) in affine transformation of matching points in the image expressed by formula (13) above is also obtained as a parameter that expresses preciseness in identity judgment of the image. Part of these parameters may be utilized.
  • a transform formula defining each of these parameters as a variable is defined, whereby this formula may be defined as preciseness of identity judgment in matching.
  • the number of matching points is utilized as preciseness, and then, the matching results are displayed in descending order of the number of matching points, whereby images are outputted in sequential order from the most reliable image.
  • FIG. 12 is a view showing the configuration of an information retrieval system of a first application.
  • the information retrieval system is configured to include an information presentation apparatus 100, a storage unit 102, a dataset server 104, and an information server 106.
  • the information presentation apparatus 100 is configured by platform hardware.
  • the storage unit 102 is provided in the platform hardware.
  • the dataset server 104 and the information server 106 are configured in sites accessible by the platform hardware.
  • the information presentation apparatus 100 is configured to include an image acquisition unit 108, a recognition and identification unit 110, an information specification unit 112, a presentation image generation unit 114, and an image display unit 116.
  • the recognition and identification unit 110, the information specification unit 112, and the presentation image generation unit 114 are realized by application software of the information presentation unit installed in the platform hardware.
  • the image acquisition unit 108 and the image display unit 116 are provided as physical configurations in the platform hardware, or are connected to outside.
  • the recognition and identification unit 110, the information specification unit 112, and the presentation image generation unit 114 could be referred to as an information presentation apparatus.
  • the information presentation apparatus is defined to perform processes from the process of imaging or image capture to the process of final image presentation, such that the combination of the image acquisition unit 108, the recognition and identification unit 110, the information specification unit 112, the presentation image generation unit 114, and the image display unit 116 is herein referred to as the information presentation apparatus.
  • the image acquisition unit 108 is a camera or the 5 like having a predetermined image acquisition range.
  • the recognition and identification unit 110 recognizes and identifies respective objects within the image acquisition range from an image acquired by the image acquisition unit 108.
  • the 10 unit 112 obtains predetermined information (display contents) from the information server 106 in accordance with information of the respective objects identified by the recognition and identification unit 110. The information specification unit 112 then specifies the
  • the presentation image generation unit 114 generates a presentation 'image formed by correlation between the relevant information, which has been specified by the information specification unit 112, and the image
  • the image display unit 116 is, for example, a liquid crystal display that displays the presentation image generated by the presentation image generation unit 114.
  • the 25 contains a dataset 118 stored by the dataset server 104 via a communication unit or storage medium (not shown) . Admission (downloading or media replacement) and storing of the dataset 118 is possible regardless of pre-activation or post-activation of the information presentation apparatus 100.
  • the information presentation apparatus 100 configured as described above performs operation as follows. First, as shown in FIG. 13, an image is acquired by the image acquisition unit 108 (step SlOO) . Then, for the image acquired in step SlOO described above, the recognition and identification unit 110 extracts a predetermined object (step S102) .
  • the recognition and identification unit 110 executes comparison and identification of an image (image in a rectangular frame, for example) of the object, which has been extracted in step S102 described above, in accordance with features in the dataset 118 read from the storage unit 102 in the platform. In this manner, the recognition and identification unit 110 detects a matched object image. If the recognition and identification unit 110 has detected the matched object image (step S104) , then a location and/or acquiring method for information necessary to be obtained from corresponding data in the dataset 118 is again read and executed in the information specification unit 112 (step S106) . In an ordinary case, the information is obtained by accessing the information server 106, which externally exists in a network or the like, from the platform through communication.
  • the presentation image generation unit 114 processes the information (not shown) obtained in the information specification unit 112 so that the information can be displayed on the image display unit 116 provided in the platform or outside, thereby generating a presentation image.
  • the presentation image thus generated is transferred to the image display unit 116 from the presentation image generation unit 114, whereby the information is displayed on the image display unit 116 (step S108) .
  • the configuration can be such that a position and orientation calculation unit 120 is provided between the recognition and identification unit 110 and the information specification unit 112.
  • the presentation image generation unit 114 generates a presentation image in such a form that relevant information specified by the information specification unit 112 is superposed on an image acquired by the image acquisition unit 108 in a position and orientation calculated by the position and orientation calculation unit 120.
  • the first application using a camera mobile phone as a platform will be described herebelow.
  • mobile phones are devices that are used by individuals.
  • most models of mobile phones allow admission (that is, installation by downloading) of application software from an Internet site accessible from the mobile phones (which hereinbelow will be simply referred to as a "mobile-phone accessible site") .
  • the information presentation apparatus 100 is, basically, also assumed as a prerequisite to be a mobile phone of the aforementioned type.
  • Application software of the information presentation apparatus 100 is installed into the storage unit 102 of the mobile phone.
  • the dataset 118 is appropriately stored into the storage unit 102 of the mobile phone through communication from the dataset server 104 connected to a specific mobile-phone accessible site (not shown) .
  • a utilization range of the information presentation apparatus 100 in the mobile phones includes a utilization method described hereinbelow.
  • a case is assumed in which photographs existing in publications, such as magazines or newspapers, are preliminarily specified, and data sets relevant thereto are preliminarily prepared.
  • a mobile phone of a user acquires an image of an object from paper space of any of the publications and then to read information relevant to the object from a mobile-phone accessible site.
  • the data can be provided to a user in a summarized form, such as "a data set for referencing, as objects, photographs contained in an n-th month issue" of a specific magazine.
  • a summarized form such as "a data set for referencing, as objects, photographs contained in an n-th month issue" of a specific magazine.
  • the configuration further including the function of calculating the position and orientation, information obtained from the information server 106 becomes displayable with an appropriate position and orientation over an original image. Consequently, the configuration leads to enhancement of user information obtainment effects.
  • FIG. 15 is a view showing the configuration of an information retrieval system of the second application.
  • the basic configuration and operation of the information retrieval system is similar to those in the first application.
  • features can be handled in units of the set, whereby, as described above, the usability for the user is increased, and data set supply is made practical .
  • the information presentation apparatus 100 becomes pervasive and data sets also are supplied in wide variety from many businesses, the following arrangements are preferably made.
  • data enjoying high utilization frequency (which data hereinbelow will be referred to as "basic data” 122) is not supplied as a separate dataset 118, but preferably is provided usable even if any type of a dataset 118 is selected.
  • basic data data enjoying high utilization frequency
  • objects associated with index information of the dataset 118 itself or object and the like most frequently used is excluded from the dataset 118, but only the some number of features are stored to be resident in application software in the information presentation apparatus 100.
  • the dataset 118 is composed in a set corresponding to the utilization purpose of a user or a publication or object correlated thereto, and is supplied as a separate resource from the application software.
  • features or the like relevant to an object with an especially high utilization frequency or necessity is stored to reside or is retained as the basic data 122 in the application software itself.
  • any of the datasets 118 to be supplied includes at least one set of an identical data file ("feature A" in the drawing) that always becomes the basic data 122.
  • the user when actually operating the information presentation apparatus 100, the user admits an arbitrary dataset 118. At least one item of the basic data 122 is included in any of the datasets 118, so that it is always addressable for an object either with high utilization frequency or high necessity.
  • a case is contemplated in which, as shown in FIG. 16, a large number of datasets 118 (data sets (1) to (n) ) are prepared; and among them, one or multiple sets of datasets 118 are admitted and stored into the storage unit 102 in the platform.
  • any selected ' one of the datasets 118 always includes one or multiple types of basic data 122. Therefore, even without giving specific consideration, the user is able to cause a basic operation in which a basic object is imaged.
  • the basic operation is any one of operations, such as "access to an index page of dataset”, “access to a support center for a supplier of the information presentation apparatus 100", “access to a weather information site” for a predetermined district, and other operations desired many users. That is, the basic operation is defined to be an operation with high frequency of utilization by users.
  • the configuration can be such that in the event of activation of the information presentation apparatus 100, the dataset server 104 is connected, and the basic data 122 surely is downloaded and retained for another dataset 118, or is made referable simultaneously.
  • This configuration provides a method for admitting the basic data 122 useful in a configuration mode in which the dataset 118 is supplied as a separate resource, and especially, is downloaded through a network from the dataset server 104. More specifically, in the configuration shown in FIG. 17, in the event that a dataset 118 is to be supplied through a network to the information presentation apparatus 100, when the dataset 118 is to be selected by a user and is to be downloaded by the dataset server 104, also the basic data 122 can be concurrently automatically downloaded in addition to the dataset 118. Further, in the configuration shown in FIG. 17, in the case that the basic data 122 is already stored in the storage unit 102 of the platform having the information presentation apparatus 100, the basic data 122 can be updated.
  • the user is able to always use the basic data 122 with the information presentation apparatus
  • a camera mobile phone of this type is used as a platform, and application software having functions, except those of the image acquisition unit 108 and image display unit 116 of the information presentation apparatus 100, is installed on the platform.
  • a predetermined dataset download site is accessed through communication of the mobile phone (step SIlO) .
  • downloading is initially performed by the dataset server 104 (step S112) .
  • the update is necessary.
  • a version of the basic data 122 is older than a version of a basic data 122 intended to be supplied from the dataset server 104, it is determined that the update is necessary.
  • the basic data 122 is downloaded (step S116) .
  • the basic data 122 thus downloaded is stored into the storage unit 102 of the mobile phone (step S118) .
  • the dataset 118 downloaded is stored into the storage unit 102 of the mobile phone (step S120) .
  • the necessity of the update is determined through the version comparison, and then the basic data 122 is downloaded and stored.
  • the utilization range of the information presentation apparatus 100 includes, for example, access from the mobile phone to information relevant or attributed to a design of photograph or illustration of a publication, such as newspaper or magazine, as a object, and improvement of information presentation by superimposing the aforementioned information over an image acquired by the camera. Further, not only such the printout, but also any of, for example, physical objects and signboards existing in a town can be registered as an object into the features. In this case, such a physical object or signboard is recognized as an object by the mobile phone, thereby to make it possible to obtain additional information or latest information.
  • the design of a jacket thereof is variant, and thus the respective jacket designs can be used as a object.
  • the respective jackets can be recognized as an object by the mobile phone in, for example, a CD and/or DVD store or rental store.
  • a URL is correlated to the object, and audio distribution of, for example, a selected part of music can be implemented to the mobile phone as information correlated to the object through the URL.
  • an annotation (respective annotation of a photograph of the jacket) corresponding to the surface of the jacket can be appropriately added.
  • the arrangement can be made as follows. First, (1) at least a part of an exterior image of a recording medium containing music fixed thereto or a package thereof is preliminarily distributed to the mobile phone as object data. Then, (2) predetermined music information (such as audio data and annotation information) relevant to the fixed music is distributed to the mobile phone accessed to an address guided by the object.
  • predetermined music information such as audio data and annotation information
  • the arrangement thus made is effective for promotion on the side of the record company, and produces an advantage in that, for example, time and labor can be reduced for preparation for viewing and listening on the side of the store.
  • the recognition and identification unit, the information specification unit, the presentation image generation unit, and the position and orientation calculation unit are each implemented by a CPU, which is incorporated in the information presentation apparatus, and a program that operates on the CPU.
  • this can be in another mode in which, for example, leased lines are provided.
  • an external data pack and a detachable storage medium are usable, without being limited thereto.
  • the configuration can be formed to include the position and orientation calculation unit 120 so that relevant information is presented in accordance with calculated position and orientation.
  • replaceable storage media 124 can be used instead of the dataset server 104 and/or the information server 106.
  • the admission of data such as the dataset 118 and the basic data 122 to the storage unit 102 in the platform means expansion of data on internal memory from the replaceable storage media 124.
  • the configuration of the information retrieval system of the first application shown in FIG. 12 can be modified to a configuration shown in FIG. 19. More specifically, the recognition and identification unit 110 provided in the information presentation apparatus 100 and the dataset 118 provided in the storage unit 102 in the first application can, of course, be provided to the side of the server, as shown in FIG. 19. In the case that this configuration is used for the information retrieval system, the storage media 124 provided in the storage unit 102 is unnecessary, so that it is not provided. [Fourth Application]
  • FIG. 20 is a view showing the configuration of a product recognition system of the fourth application.
  • the product recognition system includes a barcode scanner 126 serving as a reader for recognizing products each having a barcode, a weight scale 128 for measuring the weights of respective products, and in addition, a camera 130 for acquiring images of products.
  • a control unit/cash storage box 132 for storing cash performs recognition of a product in accordance with a database 134 having registered product features for recognition, and displays the type, unit price, and total price of the recognized products on a monitor 136.
  • a view field 138 of the camera 130 matches with the range of the weight scale 128.
  • a system provider preliminarily acquires an image of an object that would need to be recognized, and registers a feature point extracted therefrom into the database 134.
  • a feature point extracted therefrom For example, for use in a supermarket, vegetables and the like such as tomato, apple, and green pepper are photographed, and feature points 140 thereof are extracted and stored, with identification indexes such as respectively corresponding recognition IDs and names, into the database 134 as shown in FIG. 21.
  • auxiliary information such as an average weight and average size, of the respective objects is preliminarily stored into the database 134.
  • FIG. 22 is a flowchart of product settlement by the product recognition system of the fourth application.
  • a purchaser of a product carries the product (object) and places it within the view field 138 of the camera 130 installed to a cash register, whereby an image of the product is acquired (step S122) .
  • Image data of the product is transferred from the camera 130 to the control unit/cash storage box 132 (step S124) .
  • the control unit/cash storage box 132 features are extracted, and the product is recognized with reference to the database 134 (step S126) .
  • the control unit/cash storage box 132 calls or retrieves a specified price of the recognized product from the database 134 (step S128), causes the price to be displayed on the monitor 136, and carries out the settlement (step S130).
  • a green pepper and tomato In the event that a purchaser purchases two items, a green pepper and tomato, at first, an image of the tomato is acquired by the camera 130. Then, in the control unit/cash storage box 132, features in the image data are extracted, arid matching with the database 134 is carried out. After matching, in the event that one object product is designated, a coefficient corresponding to the price thereof, or the weight thereof if a weight-base'd system is used, is read from the database 134 and is output to the monitor 136. Then, similarly, also for the green pepper, product identification and price display are carried out. Finally, a total price of the products are calculated and output to the monitor 136, thereby carrying out the settlement.
  • the following method is applied: (1) the candidates are displayed on the monitor 136 to be selected; or (2) re-acquiring of an image of an objects is carried out. Thereby, object establishment is carried out .
  • object establishment is carried out .
  • an image of each product is acquired one by one by the camera 130
  • an image including a plurality of object products can be acquired at one time for matching.
  • FIG. 23 is a flowchart of the feature extraction and recognition process in step S126 described above.
  • a plurality of features is extracted from an image (product image data) input from the camera 130 (step S132) . Then, preliminarily registered features of object are read as comparison data from the database 134 (step S134) . Then, as shown in FIG. 24, comparative matching between the features of an image 142 received from the camera 130 and the preliminarily registered features of a reference image 144 (step S136) is carried out, thereby to determine the identifiability of the object (step S138) . If the object is determined to be not identical (step S140), features of a next preliminarily registered object are read from the database 134 as comparison data (step S142) . Then, the operation returns to step S136.
  • step S140 the object currently in comparison and the product in the input image are determined to be identical to one another (step S144).
  • product recognition can be accomplished without affixing a recognition index such as barcode or RF tag to the product.
  • this is useful as automatic recognition is possible in recognizing agricultural products, such as vegetables, and other products, such as meat and fish, for which significant time and labor are necessary to affix recognition indexes, unlike those such as industrial products to which recognition indexes can easily be affixed by printing and the like.
  • objects to which such recognition indexes are less affixable include minerals, such that the system can be adapted for industrial use, such as automatic separation thereof.
  • FIG. 25 is a view of an overall configuration of a retrieval system of the fifth application.
  • the retrieval system includes a digital camera 146, a storage 148, and a printer 150.
  • the storage 148 stores multiple items of image data.
  • the printer 150 prints image data stored in the storage 148.
  • the storage 148 is a memory detachable from or built in the digital camera 146.
  • the printer 150 prints out image data stored in the memory, i.e., the storage 148, in accordance with a printout instruction received from the digital camera 146.
  • the storage 148 is connected to the digital camera 146 through connection terminals, cable, or wireless/wired network, or alternately, can be a device mounting a memory detached from the digital camera 146 and capable of transferring image data.
  • the printer 150 can be of the type that connected to or is integrally configured with the storage 148 and that executes printout operation in accordance with a printout instruction received from the digital camera 146.
  • the storage 148 further includes functionality of a database from which image data is retrievable in accordance with the feature value. Specifically, the storage 148 configures a feature database (DB) containing feature sets created from digital data of original images .
  • DB feature database
  • the digital camera 146 acquires an image of a photographic subject including a retrieval source printout 152 once printed out by the printer
  • the digital camera 146 reads image data corresponding to matched features from the storage 148 as original image data of the retrieval source printout 152.
  • the retrieval source printout 152 can use not only a printout having been output in units of one page, but also an index print having been output to collectively include a plurality of den ⁇ agnified images. This is because it is more advantageous in cost and usability to select necessary images from the index print and to copy them.
  • the retrieval source printout 152 can be a printout output from a printer (not shown) external of the system as long as it is an image of which original image data exists in the feature DB.
  • the retrieval system of the fifth application will be described in more detail with reference to a block diagram of configuration shown in FIG. 26 and an operational flowchart shown in FIG. 27.
  • the digital camera 146 has a retrieval mode for retrieving already- acquired image data in addition to the regular imaging mode.
  • the operational flowchart of FIG. 27 shows the process in the retrieval mode being set.
  • a user After having set the mode to the retrieval mode, a user operates an image acquisition unit 154 of the digital camera 146 to acquire image of a retrieval source printout 152 desired to be printed "out again in the state where it is pasted onto, for example, a table or a wall face (step S146) .
  • features are extracted by a feature extraction unit 156 (step S148) .
  • the features can be any one of the following types: one type uses feature points in the image data; another type uses relative densities of split areas in the image data in accordance with a predetermined rule, that is, small regions allocated with a predetermined grating; another type in accordance with Fourier transform values corresponding to respective split areas.
  • information contained in such feature points includes point distribution information.
  • a matching unit 158 performs a DB- matching process, in the manner that the features extracted by the feature extraction unit 156 are compared to the feature DB (feature sets) of already- acquired image data composed in the storage 148, and data with a relatively high similarity is sequentially extracted (step S150) .
  • the DB- matching process is carried out as follows. First, similarities with features of respective already- acquired image data are calculated (step S152), and features are sorted in accordance with the similarities (step S154). Then, original image candidates are selected in accordance with the similarities (step S156) . .
  • the selection can be done such that either threshold values are set or high order items are specified in the order of higher similarities. In either way, two methods are available, one for selecting one item with the highest similarity and the other for selecting multiple items in the order from those having relatively higher similarities.
  • FIG. 29 shows a display screen of the display unit 160 in the event of displaying only one image candidate.
  • the display screen has "PREVIOUS" and “NEXT” icons 164 and a "DETERMINE” icon 166 on a side of a display field of an image candidate 162.
  • PREVIOUS and NXT icons 164 represent a button that is operated to specify display of another image candidate.
  • the “DETERMINE” icon 166 represents a button that is operated to specify the image candidate 162 as desired image data.
  • the "PREVIOUS” and “NEXT” icons 164 respectively represent left and right keys of a so-called arrow key ordinarily provided in the digital camera 146, and the "DETERMINE” icon 166 represents an enter key provided in the center of the arrow key.
  • step S162 the process returns to step S158, at which the image candidate 162 is displayed.
  • the enter key which corresponds to the
  • step S162 the matching unit 158 sends to the connected printer 150 original image data that corresponds to the image candidate 162 stored in the storage 148, and the image data is again printed out (step S164) .
  • the storage 148 is not connected to the printer 150 through a wired/wireless network, the process of performing predetermined marking, such as additionally writing a flag, is carried out on the original image data corresponding to the image candidate 162 stored in the storage 148. Thereby, the data can be printed out by the printer 150 capable of accessing the storage 148.
  • step S158 of displaying the image candidate a plurality of candidates can be displayed at one time.
  • the display unit 160 ordinarily mounted to the digital camera 146 is, of course, of a small size of several inches, such that displaying of four or nine items is appropriate for use.
  • FIG. 30 is view of a display screen in the event of displaying nine image candidates 162.
  • a bold-line frame 168 indicating a selected image is moved in response to an operation of a left or right key of the arrow key, respectively, corresponding to the "PREVIOUS" or "NEXT" icon 164.
  • the arrangement may be such that the display of nine image candidates 162 is shifted, that is, so-called page shift is done, to a previous or next display of nine image candidates by operating an up or down key of the arrow key.
  • the feature DB of the already-acquired image data composed in the storage 148 as comparative objects used in step S150 has to be preliminarily created from original image data stored in the storage 148.
  • the storage 148 can be either a memory attached to the digital camera 146 or a database accessible through a communication unit 170 as shown by a broken line in FIG. 26.
  • One example is a method that carries out calculation of features and database registration when storing acquired image data in the original-image acquiring event into a memory area of the digital camera 146. More specifically, as shown in FIG. 31, the digital camera 146 performs .an image acquiring operation (step S166) , and the acquired image data thereof is stored into the memory area of the digital camera 146 (step S168). Then, features are calculated from the stored acquired image data (step S170) , and is stored in correlation with the acquired image data (step S172) .
  • the storage 148 is a built-in memory of the digital camera 146, a database is built therein.
  • the storage 148 is a separate device independent of the digital camera 146
  • the acquired image data and features stored into the memory area of the digital camera 146 are both transferred into the storage 148, and a database is built therein.
  • Another method is such that, when original image data stored in the storage 148 is printed out by the printer 150, printing-out is specified, and concurrently, feature extraction process is carried out, and the extracted features are stored in the database, therefore producing high processing efficiency. More specifically, as shown in FIG. 32, when printing out original image data stored in the storage 148, ordinarily, the original image data to be printed out is selected in response to a user specification (step S174); and printout conditions are set (step S176), whereby printing is executed (step S178).
  • the printing process is completed at this stage; however, in the present example, processing is further continued, thereby to calculate features from the selected original image data (step S180) and then to store features thereof in correlation with the original image data (step S180) .
  • the printout conditions are reflected in the operation, thereby making it possible to improve matching accuracy between the retrieval source printout 152 and the features.
  • features are created only for original image data that may be subjected to the matching process, consequently making it possible to save creation time and storage capacity for unnecessary feature value data. Further, of course batch processing can be performed. More specifically, as shown in FIG.
  • step S184 when a batch feature creation specification from a user is received (step S184), feature uncreated original image data in the storage 148 is selected (step S186), and a batch feature creation process is executed on the selected feature uncreated original image data (step S188) .
  • steps S190 features are extracted from the respective feature uncreated original image data to create features
  • step S192 created features are stored into the storage 148 in correlation with the corresponding original image data
  • the data can be discretely processed in accordance with the input of a user specification. More specifically, as shown in FIG. 34, one item of original image data in the storage 148 is selected by the user (step S194), and creation of features for the selected original image data is specified by the user (step S196) . Thereby, features are extracted from the selected original image data (step S198) , and the features are stored into the storage 148 in correlation with the selected original image data (step S200) .
  • the specification for feature creation can be given by marking of a photograph desired to be printed out.
  • image data similar in image configuration can be retrieved, thereby making it possible to provide novel secondary adaptabilities.
  • an image of a signboard or poster on the street is acquired in a so-called retrieval mode such as described above.
  • image data similar or identical to the acquired image data can easily be retrieved from image data and features thereof existing in the storage 148, such as database, accessible through, for example, the memory attached to the digital camera 146 and communication.
  • recognized relevant information such as peripheral portion of the recognized station, i.e., map information of the peripheral portion of the station, image information, and relevant character (letter) information
  • relevant information existing in the storage 148 such as database, accessible through, for example, the memory attached to the digital camera 146 and communication.
  • a method of recognizing such a station name there are available methods, such as those of character recognition, pattern recognition, recognition estimation based on retrieval of similar images, and these methods can be practiced by functions of the matching unit 43.
  • an example case is assumed in which an image of the Tokyo Tower is acquired.
  • images existing in the storage 148, such as database, accessible through, for example, the memory attached to the digital camera 146 and communication are retrieved, whereby photographs of not only the Tokyo Tower, but also photographs of tower-like buildings in various corners of the world can be retrieved and extracted.
  • the locations of the respective towers can be informed, or as shown in FIGS. 36 and 37, displaying can be performed by superimposing the photograph over the location on a map.
  • maps and photographs are relevant information.
  • Such a representative photograph is selectable from various viewpoints, such as highest similarity and most frequently viewed among those in the set.
  • the process of steps S148 to S162 is carried out within the digital camera 146, the process can be carried out in a different way as follows.
  • the storage 148 is provided as a separate resource independent of the digital camera 146, the process described above can be actually operated by being activated in the form of software in the storage 148 or by being separated into the digital camera 146 and the storage 148.
  • the retrieval system includes a digital camera 146, a storage 148, a printer 150, and a personal computer (PC) 172.
  • the storage 148 is a storage device built in the PC 172 or accessible by the PC 172 through communication.
  • the PC 172 is wired/wireless connected to the digital camera 146, or alternatively is , configured to permit a memory detached from the digital camera 146 to be attached, thereby being able to read image data stored in the memory of the digital camera 146.
  • the retrieval system thus configured performs operation as follows.
  • the digital camera 146 acquires an image of a photographic subject including a retrieval source printout 152 once printed out by the printer 150.
  • the PC 172 extracts a region corresponding to the image of the retrieval source printout 152 from the image data acquired, and then extracts features of the extracted region.
  • the PC 172 executes matching process of ' the extracted features with the features stored in the storage 148.
  • the PC 172 reads image data corresponding to matched features as original image data of the retrieval source printout 152 from the storage 148.
  • the PC 172 is able to again print out the read original image data by the printer 150.
  • the present application contemplates a case where image data acquired by the digital camera 146 is stored into the storage 148 built in or connected to the PC 172 designated by a user, and a process shown on the PC side in FIG. 41 operates in the PC 172 in the form of application software.
  • the application software is activated in the state that the PC 172 and the digital camera 146 are hard wired or wirelessly connected together thereby to establish a communication state.
  • the state may be such that functional activation is carried out through the operation of tuning on a switch such as a "retrieval mode" set for the digital camera 146.
  • an image acquisition process for acquiring an image of a printout is executed on the side of the digital camera 146 (step S146) . More specifically, as shown in FIG. 42, a user operates an image acquisition unit 154 of the digital camera 146 to acquire an image of a retrieval source printout 152 desired to be again printed out in the state where it is pasted onto, for example, a table or a wall face so that at least no omission of the retrieval source printout 152 occurs (step S202) . Thereby, acquired image data is stored into a storage unit 176 serving as a memory of the digital camera 146. Then, the acquired image data thus stored is transferred to the PC 172 hard wired or wirelessly connected (step S204) .
  • a feature extraction unit 176 realized by application software performs the process of extracting features from the transferred acquired image data (step S148) .
  • the feature extraction process can be performed on the digital camera 146 side. Thereby, the amount of communication from the digital camera 146 to the PC 172 can be reduced.
  • a matching unit 178 realized by application software performs a DB-matching process such that the extracted features are compared to the feature DB of already-acquired image data composed in the storage 148, and those with relatively high similarities are sequentially extracted (step S150) . More specifically, in accordance with the calculated features, the matching unit 178 on the PC 172 side performs comparison with the features stored in correlation to respective items of image data in the storage 148 (or, comprehensively stored in the form of a database), and most similar one is selected. It is also effective in usability to set such that a plurality of most similar feature candidates is selected.
  • the features include specification information of original image data from which the features have been calculated, and candidate images are called in accordance with the specification information.
  • image data of the selected original image candidates are read from the storage 148 and are displayed on a display unit 180 serving as a display of the PC 172 as image candidates to be extracted (step S158), whereby to receive a selection from the user.
  • the processing may be such that the selected original image candidates (or the candidate images) are transferred as they are or in appropriately compressed states from the PC 172 to the digital camera 146, and are displayed on the display unit 160 of the digital camera 146 (step S206) .
  • original image data corresponding to the image candidate stored in the storage 148 is sent to the connected printer 150 and is printed thereby (step S164).
  • the displayed original image candidate is determined through determination of the user and is passed to the printing process, thereby to enable the user to easily perform the preliminarily desired reprinting of already-printed image data.
  • the plurality of selected candidate images result in a state that "although different from the desired original image, similar images have been collected", depending on the user' s determination, thereby realizing the function of batch retrieval of similar image data.
  • the feature DB can be created in the event of transfer of the acquired image data from the digital camera 146 to the storage 148 through the PC 172. More specifically, with reference to FIG. 43, transfer of the acquired image data from the digital camera 146 to the PC 172 is started (step S208) . Then, by using the PC 172, the transferred acquired image data is stored into the storage 148 (step S210) , and the features are created from the acquired image data (step S212) . Then, the created features are stored into the storage 148 in correlation with the acquired image data (step S214) .
  • an image of a signboard or poster on the street is acquired in a so-called retrieval mode such as described above.
  • image data similar or identical to the acquired image data can easily be retrieved from image data and features thereof existing in the storage 148, such as an external database, accessible through, for example, the memory attached to the digital camera 146 and a communication unit 182 shown by the broken line in FIG. 40.
  • Internet sites associated to the data can be displayed on the displays of, for example, the PC 172 and digital camera, and specific applications (for audio and motion images (movies) , for example) can be operated. Description has been given with reference to the case where the digital camera 146 is used, the present application is not limited thereto, and a scanner can be used.
  • an image of the retrieval source printout 152 which has actually been printed out, is acquired by the digital camera 146
  • an image of a display displaying the acquired image of the retrieval source printout 152 can be acquired by the digital camera 146.
  • a retrieval system of a seventh application will be described herebelow.
  • the present application is an example of adaptation to application software 188 of a mobile phone 184 with a camera 186, as shown in FIG. 44.
  • Mobile phone application software is at present usable with most mobile phones, and a large number of items of image data are storable in a memory such as an internal memory or an external memory card. Further, in specific mobile phone sites (mobile phone dedicated Internet sites) , storage services for, for example, user-specified image files are provided. In these environments, a very large number of image data can be stored, thereby to make it possible to use them for various user's own activity recording and jobs. On the other hand, however, retrieval of desired image data is complicate and burdensome for hardware of the mobile phone having the interface relatively inferior in freedom degree. In most cases, actual retrieval is carried out from a list of texts representing, for example, the titles or date and time of image data. As such, it must be said that, in the case of large number of image data, the retrieval is complicate and burdensome; and even when keying-in a text, it is inconvenient to input a plurality of words or a long title, for example.
  • the system is operated as the application of the camera mobile phone, thereby to carry out the activation of "image input function", "segmentation of a region of interest", and "feature calculation.”
  • the features are transmitted to a corresponding server via a mobile phone line.
  • the corresponding server can be provide in a one to one or one to multiplicity relation with respect to the camera or cameras.
  • the features sent to the server are actually subjected to the process of matching by a "matching function" provided in the server with the features read from a database required by the server. Thereby, image data with high similarity is extracted.
  • the image data thus extracted is returned to the call-side mobile phone from the server, whereby the image data can be output by a printer unspecified from the mobile phone.
  • an extended function "the information is returned to the mobile phone" can be implemented.
  • the extracted image data is highly compressed and returned to the mobile phone, and after a user verifies that the data is a desired image data, the data is stored in the memory area of the mobile phone or is displayed on a display 190 of the mobile phone. Even only from this fact, it can of course be said that the system is useful.
  • the present application has a configuration including a digital camera 146 with a communication function and a server connected through communication, in which a function for image retrieval is sharedly provided to the digital camera 146 and the server.
  • the digital camera 146 with the communication function provides the function as an image-acquiring-function mounted communication device, and of course includes a camera mobile phone.
  • the digital camera 146 includes the image acquiring function and a calculation function for calculating the features from the image data.
  • the features (or the feature DB) to be compared and referred are originally created based on images acquired and printed out by users or the digital camera 146. This is attributed to the fact that the initial purpose is to image printouts of already-acquired image data and to carry out retrieval.
  • the present application is configured by extending the purpose and is significantly different in that features calculated based on images of, for example, on-the-street sign boards, posters, printouts, and publications are also stored into the database formed in the storage 148 of the server.
  • features extracted from an acquired image can be added to the database.
  • FIG. 45 is a flowchart showing operation of the retrieval system of the present application.
  • the same reference numerals designate the portions corresponding to those in the fifth application.
  • an image of a poster such as a product advertisement present on the street is acquired by the digital camera 146, for example (step S146) .
  • a feature extraction process is executed by the digital camera 146 from the acquired image data (step S148) .
  • the extracted features are sent to a predetermined server by the communication unit 170 built in or attached to the digital camera 146.
  • the feature DB formed in the storage 148 accessible by the server is looked up (accessed) , and features sent from the digital camera 146 are compared thereto (step S150) , thereby to extract similar image candidates having similar features (step S216) .
  • Image data of the extracted similar image candidates are, by necessity, subjected to a predetermined compression process to reduce the amount of communication, and then are sent to the digital camera 146, whereby the candidates can be simply displayed on the display unit 160 of the digital camera 146 (step S218) . Thereby, user selection can be performed similarly as in the fifth application. Then, image data of an image candidate extracted
  • next operation is carried out in accordance with specified information correlated to the features of the extracted (and selected) image candidate (step S220).
  • the next operation can be, for example, description of the product or connection to a mailorder site or returning of a screen of the site, as image data, to the digital camera 146.
  • peripheral information of the signboard is retrieved as features.
  • data of the location of a wireless communication base station during communication is compared, thereby to make it possible to present identifications of, for example, the location and address, as information to the user.
  • the present application retrieves multiple items of image data from a storage 148 by matching using first features in accordance with an acquired image of an acquired retrieval source printout 152.
  • the application retrieves a single or multiple items of image data from the multiple items of image data, obtained as a result of the retrieval by feature matching using second features of a region narrower than or identical to the first features and high in resolution.
  • the retrieval system of the present application has a configuration similar to that of the fifth application.
  • the storage 148 is configured to include a total feature DB containing general features registered as first features, and a detail feature DB containing detail features registered as second features.
  • the general features are obtained by extraction of a region containing most (about 90%, for example) of the totality (100%) of image data at a relatively coarse (low) resolution.
  • the detail features are obtained by extraction of a region containing a central region portion (about central 25%, for example) of the image data at a high resolution relative to the resolution of the general features.
  • FIG. 48 The positional relationship between the original image data and the general features and the detail features is shown in FIG. 48.
  • FIG. 49 is a flowchart showing operation of the retrieval system of the present application. In the diagram, the same reference numerals designate the portions corresponding to those in the fifth application.
  • an image acquisition unit 154 of a digital camera 146 set in a retrieval mode acquires an image of a retrieval source printout 152 desired to be printed out again in the state where it is pasted onto, for example, a table or a wall face so that at least no omission of the retrieval source printout 152 occurs (step S146) .
  • a total feature extraction process for extracting features from the totality of the image data acquired by the image acquisition unit 154 is performed by a feature extraction unit 156 (step S222) .
  • a matching process with the total feature DB which compares the extracted total features to the total feature DB composed in the storage 148 and containing registered general features and sequentially extracts data with a relatively high similarity, is executed by a matching unit 158 (step S224) .
  • a detail retrieval object region namely image data of the central region portion of the region of interest in the present example, is further extracted as detail retrieval object image data from the acquired image data of the total region of interest (step S226) .
  • a detail feature extraction process for extracting features from the extracted detail retrieval object image data is performed by the feature extraction unit 156 (step S228) .
  • a matching process with the detail feature DB which compares the extracted detail features to the detail feature DB formed in the storage 148 and having registered detail features and sequentially extracts data with higher similarity, is executed (step S230) .
  • feature matching with all detail features registered into the detail feature DB is not performed, but feature matching is executed only for detail features corresponding to multiple items of image data extracted by the matching process with the total feature DB in the step S224. Therefore, although the feature value matching process with the detail features takes a process time by nature as the resolution is high, the process can accomplished within a minimum necessary time. As a criterion for the extraction in the matching process with the total feature DB in step S224, such a method is employed that provides a threshold value for the similarity or that fixedly selects high order 500 items.
  • the candidates are displayed on the display unit 160 as image candidates for extraction (step S158) , thereby to receive a selection from the user. If an image desired by the user is determined (step S162), then' the matching unit 158 sends original image data corresponding to the image candidate stored in the storage 148 to the connected printer 150; and the data is again printed out (step S164) .
  • quality (satisfaction level) of the retrieval result of the original image data and an appropriate retrieval time period are compatible with one another.
  • the retrieval result incorporating the consideration of the attention region for the photographer can be obtained. More specifically, ordinarily, the photographer acquires an image of a main photographic subject by capturing it in the center of the imaging area. Therefore, as shown in FIG. 50, the detail features with attention drawn to the center of the image data are used to obtain a good retrieval result. Accordingly, in the system in which original image data is retrieved and extracted from retrieval source printout 152, which is the printed out photograph, and copying thereof is easily performed, the effectiveness is high in retrieval of the printed photograph.
  • the effectiveness as means for performing high speed determination of small differences is high. That is, the retrieval result can be narrowed down in a stepwise manner with respect to a large population.
  • the general features and the detail features have to be preliminarily created and registered into the database for one item of original image data. The registration can be performed as described in the fifth application.
  • both the features do not necessarily have to be created at the same time.
  • the method can be such that the detail features are created when necessary in execution of secondary retrieval. Further, the features are not limited to that as shown in, for example, FIG. 47 or 50, which draws attention to the central portion.
  • features can be set in several portions of the image. Failure due to a print-imaging condition can be prevented by thus distributively disposing features. Thereby, convergence can be implemented by dynamically varying, for example, the positions and the number of features.
  • the detail features may be such that an attention region can be placed in a focus position in the event of acquiring an original image. With such detail features, a result reflecting the intention of a photographer can be expected.
  • The. retrieval system of the present application is an example using a digital camera 146 including a communication function.
  • the application is adapted in the case where a preliminarily registered image is acquired to thereby recognize the image, and a predetermined operation (for example, activation of an audio output or predetermined program, or displaying of a predetermined URL) is executed in accordance with the recognition result.
  • a predetermined operation for example, activation of an audio output or predetermined program, or displaying of a predetermined URL
  • the digital camera 146 with the communication function functions as an imagirig-function mounted communication device, and includes a camera mobile phone -
  • DB feature value database
  • the database can be of a built-in type or a type existing in the server through communication.
  • an arrangement relationship of feature points of an image is calculated as a combination of vector quantities, and a multigroup thereof is defined to be the feature.
  • the feature is different in accuracy depending on the number of feature points, such that as the fineness of original image data is higher, a proportionally larger number of feature points are detectable.
  • the feature is calculated under a condition of a highest- possible fineness.
  • the number of feature points is relatively small, such that the feature itself has a small capacity.
  • advantages are produced in that, for example, the matching speed is high, and the communication speed is high .
  • the second feature matching server and second information DB 198-2 to the n-th feature matching server and n-th information DB 198-n are each a database having features with higher fineness or in a special category in comparison to the first feature matching server and first information DB 198-1.
  • an image of a design (object) already registered is acquired by the communication function mounted digital camera 146 (step S232).
  • feature is calculated from the arrangement relationship of the feature points by application software built in the digital camera 146 (step S148).
  • the feature is transmitted to the respective matching servers through communication, whereby matching process with the respective DBs is carried out (step S150) .
  • operation information such as a URL link
  • the operation information is transmitted to the digital camera 146, whereby a specified operation, such as displaying of 3D object acquirement, is performed (step S236) .
  • the digital camera 146 can transmit whole or part of acquired image to the matching servers, whereby step S148 can be executed in the matching servers.
  • the camera resolution is about two million pixels.
  • the digital camera 146 can transmit whole or part of acquired image to the matching servers, whereby step S148 can be executed in the matching servers.
  • the camera resolution is about two million pixels.
  • a feature DB having a resolution of about two million pixels an erroneous-recognition ratio is low.
  • matching in a concurrently operating feature DB with a low resolution VGA class resolution, for example
  • a case can occur in which a response (result) from the followingly operating high-resolution matching server is different from an already-output result of the low-resolution matching server.
  • displaying in accordance with the earlier result is first carried out, and then it is updated to a display in accordance with the following result.
  • a displaying manner is also effective in which a plurality of candidates are obtained from the low resolution result, and the resultant candidates are narrowed down to be accurate as a high resolution result arrives.
  • the capacity of the feature itself is large in the high resolution matching server.
  • a feature in an XGA class increases to about 40 kB; however, the capacity is reduced to about 10 kB by preliminary low resolution matching.
  • the second or higher matching server and database when only a difference from a lower low resolution database is retained, a smaller database configuration is realized. This leads to an increase in the speed of the recognition process. It has been verified that, when extraction with feature (method in which area allocation is carried out, and respective density values are compared) is advanced for features, the feature is generally 10 kB or lower, and also multidimensional features obtained by combining the two methods appropriately are useful to improve the recognition accuracy.
  • the method in which the resolution of some or entirety of the acquired image surface is divided into multiple resolutions to thereby realize substantial matching hierarchization is effective in both recognition speed and recognition accuracy in comparison with the case in which a plurality of matching servers are simply distributed in a clustered manner.
  • the above-described method is a method effective in the case that the number of images preliminarily registered into a database is very large (1000 or larger), and is effective in the case that images with high similarity are included therein.
  • the retrieval system of the eleventh application includes a mobile phone 184 with a camera 186 and a retrieval unit.
  • the mobile phone 184 with the camera 186 includes the camera 186 for inputting an image, and a display 190 for outputting the image of the retrieval result.
  • the retrieval unit retrieves an image from a database by using features hierarchically managed.
  • the retrieval unit is realized by application software 188 of the mobile phone 184 with the camera 186 and a matching process unit 200 configured in a server 198 communicable with the mobile phone 184 with the camera 186.
  • the server 198 further includes a feature management database (DB) 202 that contains a multiple items of features registered and that performs the hierarchical management thereof.
  • DB feature management database
  • Features to be registered into the feature management DB 202 is created by a feature creation unit 204 from an object image 206 arranged on a paper space 208 by using a desktop publishing (DTP) 210.
  • DTP desktop publishing
  • the object image 206 is preliminarily printed by the DTP 210 on the paper space 208, and the features of the object image 206 are created by the feature creation unit 204. Then, the created features are preliminarily registered into the feature management DB 202 of the server 198. When a large number of object images 206 to be registered exist, the above-described creation and registration of features are repeatedly performed.
  • the application software 188 When a user desiring retrieval acquires the object image 206 from the paper face 208 by using the camera 186 of the mobile phone 184, the application software 188 performs feature extraction of an image from the input image. The application software 188 sends the extracted features to the matching process unit 200 of the server 198. Then, the matching process unit 200 performs matching with the features registered in the feature management DB 202. If a matching result is obtained, then the matching process unit 200 sends information of the matching result to the application software 188 of the mobile phone 184 with the camera 186. The application software 188 displays the result information on the display 190.
  • a plurality of features are extracted from the input image, and a feature set consisting of the features is comparatively matched (subjected to the matching process) with the feature set in units of the preliminarily registered object. Thereby, identification of the identical object is carried out.
  • the feature point in the image in this case refers to that having a difference greater than a predetermined level from an other pixel, for example, contrast in brightness, color, distribution of peripheral pixels, differentiation component value, and inter-feature point arrangement.
  • the features are extracted and are then registered in units of the object. Then, in the event of actual identification, features are extracted by searching the interior of an input image and are compared to the preliminarily registered data. Referring to FIG. 56, the following describes the flow of operation control of an identification process in the matching process unit 200 according to the eleventh application. To begin with, preliminarily registered features of recognition elements of an object Z (object image 206, for example) is read from the feature management DB 202 containing the feature point set (step S238).
  • step S240 the features are input to the matching process unit 200 that performs comparison of the features.
  • step S242 comparative matching between the features and the input features of the object is carried out.
  • step S244 it is determined whether the object Z is identical to the input object.
  • step S246 it is determined whether the number of matching features is greater than or equal to a predetermined value (X (pieces), in the present example) (step S246) . If step S246 is branched to "NO”, then the process returns to step S242. Alternately, if step S246 is branched to "YES", then it is determined that the recognition element of the object Z currently in comparison is identical to the input object (step S248) .
  • step S250 it is determined whether the comparison with all the recognition elements is finished. If step S250 is branched to "NO", the features in the feature set of the next recognition element is input to the matching process unit 200 as comparison data (step S252) , and the process returns to step S242
  • step S250 is branched to "YES"
  • step S254 determines whether the input object is identical to the object Z, and is displayed on the display 190 to be notified to the user (step S256) . Alternately, if step S254 is branched "NO”, then a determination is made that the input object and the object Z are not identical to one another (step S258) .
  • the feature is determined to be a similar feature. Further, an object having a plurality of matched features is determined to be identical to the object of the input image. More specifically, features in an input image and a preliminarily registered feature set are compared with one another as described herebelow.
  • the interior of an object is split into a plurality of elements, and the elements are registered.
  • a determination logic is applied for recognition to determine such that the object is not recognized unless a plurality of elements (three elements, for example) are recognized.
  • the probability of causing erroneous recognition due to the identity of only part of the object can be reduced.
  • a determination reference to be applied particularly when erroneous recognition is desired to be prevented can be specified to be strict.
  • recognition is carried out in accordance with the determination logic described herebelow.
  • features A, B 7 and C are weighted by allocating weights as evaluation scores.
  • the features are weighted as 1.0, 0.5, and 0.3, respectively.
  • the evaluation scores of the recognition elements are manageable together with the features of the recognition elements.
  • the priority of the respective element can be altered, whereby not only “A and B and C,” but also a combination, such as “A and (B or C)” or “A or (B and C)", is possible.
  • the feature A is always essential to achieve successful recognition.
  • evaluation scores and logical expressions can be used by being combined. More specifically, the priorities of the respective logical expressions and weights of the respective elements can be used by being combined.
  • logos of, for example, companies in the competitive relation are identified in the following manner. For example, only when only one of the object OBJl used as the logo of the S company and the object OBJ2 used as the logo of the M company is displayed on the acquired image, the logo is recognized. More specifically, either only one of (A, B, and C) or only one of (E, F, and G) is detected within one image, either the object OBJl or the object OBJ2 is recognized. In other words, any one of (A, B, and C) and any one of (E, F, and G) are detected within one image, neither the object OBJl nor the object OBJ2 is recognized.
  • the result presentation method is altered (expression is made to include uncertainty) .
  • the recognition result is presented to the user in a high-tone expression, such as "The object OBJl has been recognized”.
  • the recognition result is presented to the user in a low-tone expression reducing the conviction, such as "The object is considered to be the object OBJl.”
  • the recognition result is presented to the user in an expression including uncertainty, such as "The object OBJl may have been recognized.”
  • the feature creation unit 204 can be operated in the server 198.
  • the paper space 208 refers to a display surface, but not necessarily be paper.
  • it can be any one of metal, plastic, and like materials, or can even be an image display apparatus, such as a liquid crystal monitor or plasma television.
  • information displayed on those such as described above corresponds to information that is displayed in visible light regions for human beings.
  • the information can be invisible for human beings as long as the information is inputtable into the camera 186.
  • the objects since all those acquirable as images can be objects, the objects may be images such as X-ray images and thermographic images . In FIG.
  • the image including the object image input from the camera 186 is transmitted from the mobile phone 184 with the camera 186 to the matching process unit 200 of the server 198.
  • the image acquired by the camera 186 can of course be transmitted as it is in the form of image data, or can be demagnified and transmitted.
  • features for use in matching can be extracted from the image and can be transmitted.
  • both the image and the features can of course be transmitted.
  • any type of data can be transmitted as long as it is the data derivable from the image.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Library & Information Science (AREA)
  • Multimedia (AREA)
  • Databases & Information Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

L'invention concerne un procédé de mise en correspondance de caractéristiques permettant la reconnaissance d'un objet dans des données d'image bidimensionnelles ou tridimensionnelles. Dans ce procédé : les caractéristiques dans lesquelles un attribut prédéterminé dans les données d'image bidimensionnelles ou tridimensionnelles présente un maximum et/ou un minimum local sont détectées (10); et les caractéristiques existant le long de contours de bords et de lignes à partir des caractéristiques détectées sont exclues (12). Puis, les caractéristiques restantes sont attribuées à un plan, certaines caractéristiques sont sélectionnées parmi les caractéristiques attribuées au moyen d'informations locales et une mise en correspondance des caractéristiques sélectionnées définis comme objets est effectuée (14).
PCT/US2007/003653 2007-02-13 2007-02-13 Procede de mise en correspondance de caracteristiques WO2008100248A2 (fr)

Priority Applications (6)

Application Number Priority Date Filing Date Title
JP2009549056A JP2010518507A (ja) 2007-02-13 2007-02-13 特徴マッチング方法
CN2007800512883A CN101627399B (zh) 2007-02-13 2007-02-13 特征匹配方法
PCT/US2007/003653 WO2008100248A2 (fr) 2007-02-13 2007-02-13 Procede de mise en correspondance de caracteristiques
EP07750486A EP2106599A4 (fr) 2007-02-13 2007-02-13 Procede de mise en correspondance de caracteristiques
US12/539,786 US20100092093A1 (en) 2007-02-13 2009-08-12 Feature matching method
HK10101874.7A HK1134158A1 (en) 2007-02-13 2010-02-23 Feature matching method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/US2007/003653 WO2008100248A2 (fr) 2007-02-13 2007-02-13 Procede de mise en correspondance de caracteristiques

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US12/539,786 Continuation US20100092093A1 (en) 2007-02-13 2009-08-12 Feature matching method

Publications (2)

Publication Number Publication Date
WO2008100248A2 true WO2008100248A2 (fr) 2008-08-21
WO2008100248A3 WO2008100248A3 (fr) 2009-03-19

Family

ID=39690637

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2007/003653 WO2008100248A2 (fr) 2007-02-13 2007-02-13 Procede de mise en correspondance de caracteristiques

Country Status (5)

Country Link
EP (1) EP2106599A4 (fr)
JP (1) JP2010518507A (fr)
CN (1) CN101627399B (fr)
HK (1) HK1134158A1 (fr)
WO (1) WO2008100248A2 (fr)

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102063719A (zh) * 2010-12-23 2011-05-18 西北工业大学 一种三维模型局部匹配方法
WO2011060952A1 (fr) * 2009-11-19 2011-05-26 Schoeller Holding Gmbh Appareil de prise de vues et de représentation d'objets, en particulier jumelles numériques, appareil photographique numérique ou caméra vidéo numérique
WO2012032747A1 (fr) * 2010-09-06 2012-03-15 日本電気株式会社 Système de sélection de point caractéristique, procédé de sélection de point caractéristique, programme de sélection de point caractéristique
US20120076417A1 (en) * 2009-06-18 2012-03-29 Canon Kabushiki Kaisha Image recognition method and image recognition apparatus
JP5108018B2 (ja) * 2007-09-07 2012-12-26 グローリー株式会社 紙葉類識別装置及び紙葉類識別方法
JP2013058174A (ja) * 2011-09-09 2013-03-28 Fujitsu Ltd 画像処理プログラム、画像処理方法および画像処理装置
US8856035B2 (en) 2010-08-23 2014-10-07 Toshiba Tec Kabushiki Kaisha Store system and sales registration method
US9036925B2 (en) 2011-04-14 2015-05-19 Qualcomm Incorporated Robust feature matching for visual search
US9971939B2 (en) 2015-06-23 2018-05-15 Toshiba Tec Kabushiki Kaisha Image processing apparatus, display state determination apparatus, and image processing method
US10115127B2 (en) 2011-12-16 2018-10-30 Nec Corporation Information processing system, information processing method, communications terminals and control method and control program thereof
EP4092559A1 (fr) * 2021-05-21 2022-11-23 Dassault Systèmes Procédé de détection d'extrusion de matériau
CN115775269A (zh) * 2023-02-10 2023-03-10 西南交通大学 一种基于线特征的列车图像精确配准方法

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120109993A1 (en) * 2010-10-28 2012-05-03 Qualcomm Incorporated Performing Visual Search in a Network
JP5485954B2 (ja) * 2011-09-06 2014-05-07 東芝テック株式会社 店舗システム及びプログラム
JP6098065B2 (ja) * 2011-09-13 2017-03-22 株式会社リコー 画像検査装置、画像検査方法、及びプログラム
JP5963609B2 (ja) * 2012-08-23 2016-08-03 キヤノン株式会社 画像処理装置、画像処理方法
JP6002529B2 (ja) * 2012-10-04 2016-10-05 パイオニア株式会社 電子機器、データの生成方法、プログラム、記録媒体、クライアントサーバシステム、クライアント端末、およびサーバ装置
CN104296852A (zh) * 2014-07-17 2015-01-21 赵肖肖 果蔬商品的自助称量方法及装置
JP5933675B1 (ja) * 2014-12-09 2016-06-15 エヌ・ティ・ティ・コムウェア株式会社 情報処理装置、情報処理方法、及びプログラム
US9741133B2 (en) * 2015-09-29 2017-08-22 Adobe Systems Incorporated Identifying shapes in an image by comparing Bézier curves
CN105374241A (zh) * 2015-11-23 2016-03-02 东莞市凡豆信息科技有限公司 一种幼儿自助阅读系统
CN105374240A (zh) * 2015-11-23 2016-03-02 东莞市凡豆信息科技有限公司 一种幼儿自助阅读系统
CN111968243B (zh) * 2020-06-28 2023-04-11 成都威爱新经济技术研究院有限公司 一种ar图像生成方法、系统、设备以及存储介质
CN113112529B (zh) * 2021-03-08 2022-11-18 武汉市土地利用和城市空间规划研究中心 一种基于区域邻近点搜索的密集匹配错配点处理方法
US20230252542A1 (en) 2022-02-04 2023-08-10 Ncr Corporation Multi-item product recognition for checkouts

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6069696A (en) * 1995-06-08 2000-05-30 Psc Scanning, Inc. Object recognition system and method
JPH09161075A (ja) * 1995-12-07 1997-06-20 Sony Corp 特徴点検出装置および特徴点検出方法、領域分割装置、画像符号化装置、並びに記録媒体
US7016532B2 (en) * 2000-11-06 2006-03-21 Evryx Technologies Image capture and identification system and process
US7574045B2 (en) * 2001-07-27 2009-08-11 Matrox Electronic Systems Ltd. Model-based recognition of objects using a calibrated image system
US7751805B2 (en) * 2004-02-20 2010-07-06 Google Inc. Mobile image-based information retrieval system

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See references of EP2106599A4 *

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5108018B2 (ja) * 2007-09-07 2012-12-26 グローリー株式会社 紙葉類識別装置及び紙葉類識別方法
US9852159B2 (en) 2009-06-18 2017-12-26 Canon Kabushiki Kaisha Image recognition method and image recognition apparatus
US20120076417A1 (en) * 2009-06-18 2012-03-29 Canon Kabushiki Kaisha Image recognition method and image recognition apparatus
CN102460508A (zh) * 2009-06-18 2012-05-16 佳能株式会社 图像识别方法和图像识别设备
US10891329B2 (en) 2009-06-18 2021-01-12 Canon Kabushiki Kaisha Image recognition method and image recognition apparatus
WO2011060952A1 (fr) * 2009-11-19 2011-05-26 Schoeller Holding Gmbh Appareil de prise de vues et de représentation d'objets, en particulier jumelles numériques, appareil photographique numérique ou caméra vidéo numérique
US8856035B2 (en) 2010-08-23 2014-10-07 Toshiba Tec Kabushiki Kaisha Store system and sales registration method
WO2012032747A1 (fr) * 2010-09-06 2012-03-15 日本電気株式会社 Système de sélection de point caractéristique, procédé de sélection de point caractéristique, programme de sélection de point caractéristique
CN102063719A (zh) * 2010-12-23 2011-05-18 西北工业大学 一种三维模型局部匹配方法
US9036925B2 (en) 2011-04-14 2015-05-19 Qualcomm Incorporated Robust feature matching for visual search
JP2013058174A (ja) * 2011-09-09 2013-03-28 Fujitsu Ltd 画像処理プログラム、画像処理方法および画像処理装置
US10115127B2 (en) 2011-12-16 2018-10-30 Nec Corporation Information processing system, information processing method, communications terminals and control method and control program thereof
US9971939B2 (en) 2015-06-23 2018-05-15 Toshiba Tec Kabushiki Kaisha Image processing apparatus, display state determination apparatus, and image processing method
EP4092559A1 (fr) * 2021-05-21 2022-11-23 Dassault Systèmes Procédé de détection d'extrusion de matériau
CN115775269A (zh) * 2023-02-10 2023-03-10 西南交通大学 一种基于线特征的列车图像精确配准方法

Also Published As

Publication number Publication date
CN101627399A (zh) 2010-01-13
EP2106599A4 (fr) 2010-10-27
JP2010518507A (ja) 2010-05-27
CN101627399B (zh) 2012-10-10
HK1134158A1 (en) 2010-04-16
WO2008100248A3 (fr) 2009-03-19
EP2106599A2 (fr) 2009-10-07

Similar Documents

Publication Publication Date Title
US20100092093A1 (en) Feature matching method
EP2106599A2 (fr) Procede de mise en correspondance de caracteristiques
Chaudhuri et al. Multilabel remote sensing image retrieval using a semisupervised graph-theoretic method
Eitz et al. An evaluation of descriptors for large-scale image retrieval from sketched feature lines
Cheng et al. Automatic landslide detection from remote-sensing imagery using a scene classification method based on BoVW and pLSA
EP2015166B1 (fr) Reconnaissance et suivi à l'aide de jonctions invisibles
JP5095535B2 (ja) 画像処理方法、画像処理システム、画像処理装置及びプログラム
US8533204B2 (en) Text-based searching of image data
US8144921B2 (en) Information retrieval using invisible junctions and geometric constraints
US9158995B2 (en) Data driven localization using task-dependent representations
EP2015224B1 (fr) Caractéristiques de jonction invisible pour reconnaissance de patch
CN101133429B (zh) 文档和/或图像检索方法、文档和/或图像存储设备和检索设备
JP2013109773A (ja) 特徴マッチング方法及び商品認識システム
Dimitrovski et al. Improving bag-of-visual-words image retrieval with predictive clustering trees
KR20010053788A (ko) 내용기반 이미지 검색 시스템 및 그 방법
CN104991954A (zh) 基于图像搜索的购物搜索系统及其实现方法
Li et al. Logo detection with extendibility and discrimination
EP2442238A1 (fr) Traitement d'un graphique réutilisable dans un document
Martinet et al. A relational vector space model using an advanced weighting scheme for image retrieval
Hu et al. Coherent phrase model for efficient image near-duplicate retrieval
Indu et al. Survey on sketch based image retrieval methods
Diem et al. Semi-automated document image clustering and retrieval
Wong et al. Automatic target recognition based on cross-plot
Mumar Image retrieval using SURF features
Solli Color Emotions in Large Scale Content Based Image Indexing

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 200780051288.3

Country of ref document: CN

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 07750486

Country of ref document: EP

Kind code of ref document: A2

WWE Wipo information: entry into national phase

Ref document number: 2007750486

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2009549056

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE