EP3373197A1 - Image processing apparatus, image processing method, template generation apparatus, object recognition processor, and object recognition processing program - Google Patents

Image processing apparatus, image processing method, template generation apparatus, object recognition processor, and object recognition processing program Download PDF

Info

Publication number
EP3373197A1
EP3373197A1 EP18000141.4A EP18000141A EP3373197A1 EP 3373197 A1 EP3373197 A1 EP 3373197A1 EP 18000141 A EP18000141 A EP 18000141A EP 3373197 A1 EP3373197 A1 EP 3373197A1
Authority
EP
European Patent Office
Prior art keywords
templates
image
template
matching
resolution
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP18000141.4A
Other languages
German (de)
English (en)
French (fr)
Inventor
Yoshinori Konishi
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Omron Corp
Original Assignee
Omron Corp
Omron Tateisi Electronics Co
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Omron Corp, Omron Tateisi Electronics Co filed Critical Omron Corp
Publication of EP3373197A1 publication Critical patent/EP3373197A1/en
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional objects
    • G06V20/647—Three-dimensional objects by matching two-dimensional images to three-dimensional objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/0014—Image feed-back for automatic industrial control, e.g. robot with camera
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/36—Applying a local operator, i.e. means to operate on image points situated in the vicinity of a given point; Non-linear local filtering operations, e.g. median filtering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/758—Involving statistics of pixels or of feature values, e.g. histogram matching
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/24—Character recognition characterised by the processing or recognition method
    • G06V30/248—Character recognition characterised by the processing or recognition method involving plural approaches, e.g. verification by template match; Resolving confusion among similar patterns, e.g. "O" versus "Q"
    • G06V30/2504—Coarse or fine approaches, e.g. resolution of ambiguities or multiscale approaches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G06T2207/30164—Workpiece; Machine component
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/06—Recognition of objects for industrial automation

Definitions

  • the present invention relates to a technique for detecting a target object in an image using template matching.
  • Template matching is a method for detecting a target object in an object recognition image.
  • a model (template) for an object to be detected is prepared, and a template image is manipulated within an input image to calculate the similarities between the template image and the input image at different positions in the input image and evaluate the calculated similarities to detect the position and posture of the object in the image.
  • Object detection by template matching is used in, for example, inspection and picking in factory automation (FA), robot vision, and monitoring cameras.
  • FA factory automation
  • Recent template matching for detecting the position and posture of a target object has shifted from two-dimensional (2D) measurement to more advanced, three-dimensional (3D) measurement.
  • Matching using a search has been developed as template matching using three-dimensional measurement.
  • search individual templates for postures viewed from different points are prepared, and matching with each of these templates is performed.
  • this method uses far more templates than matching using two-dimensional measurement, and increases the processing time (refer to, for example, Patent Literature 1).
  • the coarse-to-fine search is a technique for speeding up the search performed in template matching.
  • the coarse-to-fine search uses an image set (or image pyramid) including images having resolutions that differ from one image to another in a stepwise manner. This technique coarsely searches a lower-resolution image to narrow the search scope based on the search results, and further searches a higher-resolution image within the narrowed search scope repeatedly until the position and posture of a target object are detected in the original-resolution image.
  • Fig. 9 is a diagram showing the basic concept of a coarse-to-fine search using an image pyramid.
  • the coarse-to-fine search uses an image set (image pyramid) including first to k-th image layers (k is an integer greater than or equal to 2) having resolutions that differ from one image to another in a stepwise manner.
  • the first layer has the lowest resolution, and the second to the k-th layers have their resolutions increasing sequentially in the order of the layers.
  • k 3
  • the third layer represents the original image, and the second layer and the first layer have their resolutions decreasing in this order.
  • a first layer image with the lowest resolution is first searched through template matching to detect the position of an object (match candidate) in the first layer (refer to a detected position in the first layer image in Fig. 9 ).
  • a second layer is next searched.
  • a second layer image area corresponding to the detected position in the first layer is defined as a search scope, and this area is searched (refer to a detected position in the second layer image in Fig. 9 ).
  • a search scope is defined in a third layer using the detection result obtained for the second layer. The defined search scope is searched to detect the position of the object in the third layer (original image) (refer to a detected position in the third layer image in Fig. 9 ).
  • the search scope is narrowed stepwise from an image with a lower resolution (or lower-resolution image) toward an image with a higher resolution (or higher-resolution image) in this manner to reduce the number of times template matching is performed, and shorten the processing time.
  • Patent Literature 1 Japanese Unexamined Patent Application Publication No. 2016-207147
  • the coarse-to-fine search uses more templates for matching (or match candidate templates) as the search proceeds from a lower-resolution image to a higher-resolution image.
  • narrowing the search scope can shorten the processing time, the search involves matching performed a greater number of times for each higher-resolution image, increasing the processing time for matching.
  • the coarse-to-fine search may not sufficiently shorten the total processing time.
  • one or more aspects of the present invention are directed to a technique for enabling faster matching in object recognition using template matching.
  • the apparatus includes an image obtaining unit that obtains an input image of the object, a standard template generation unit that generates a plurality of standard templates corresponding to the object viewed from different points, an identification template generation unit that generates a plurality of identification templates representing differences between the plurality of standard templates, and a template matching unit that performs matching of the input image of the object with the plurality of identification templates.
  • the apparatus with the above structure uses the plurality of standard templates corresponding to the object viewed from different points to generate identification templates representing differences between the plurality of standard templates, and performs matching using the generated plurality of identification templates.
  • the matching using the identification templates that represent only the differences between the plurality of standard templates enables faster processing than matching using the standard templates.
  • the identification template generation unit may generate, in addition to the plurality of identification templates, a common template representing a common part of the standard templates, and the template matching unit may perform matching of the input image of the object with the generated common template and the plurality of identification templates.
  • the apparatus may further include an image pyramid generation unit that generates, from the input image of the object, a plurality of images having resolutions that differ from one image to another in a stepwise manner.
  • the plurality of images include a lower-resolution image having a resolution and a higher-resolution image having a resolution higher than the resolution of the lower-resolution image.
  • the standard template generation unit may generate a plurality of standard templates for a lower resolution corresponding to the lower-resolution image, and a plurality of standard templates for a higher resolution corresponding to the higher-resolution image.
  • the identification template generation unit may generate a plurality of identification templates for the higher resolution based on the plurality of standard templates for the higher resolution.
  • the template matching unit may perform matching of the lower-resolution image with the plurality of standard templates for the lower resolution to narrow match candidate standard templates, defines a search scope in the higher-resolution image based on a result of the narrowing, and perform matching of the higher-resolution image within the defined search scope with the plurality of identification templates for the higher resolution.
  • the apparatus may further include a feature extraction unit that extracts a feature quantity from each of the lower-resolution image and the higher-resolution image.
  • the template matching unit may perform matching using the feature quantity of the lower-resolution image and the plurality of standard templates for the lower resolution to narrow match candidate standard templates, define a search scope in the higher-resolution image based on a result of the narrowing, and perform matching using the feature quantity of the higher-resolution image within the defined search scope and the plurality of identification templates for the higher resolution.
  • the apparatus may recognize a position of the object by template matching with a coarse-to-fine search.
  • Another aspect of the present invention provides an image processing method for recognizing an object by template matching.
  • the method includes obtaining an input image of the object, generating a plurality of standard templates corresponding to the object viewed from different points, generating a plurality of identification templates representing differences between the plurality of standard templates, and performing matching of the input image of the object with the plurality of identification templates.
  • Still another aspect of the present invention provides a program for enabling a computer to implement image processing for recognizing an object by template matching.
  • the program enables the computer to implement obtaining an input image of the object, generating a plurality of standard templates corresponding to the object viewed from different points, generating a plurality of identification templates representing differences between the plurality of standard templates, and performing matching of the input image of the object with the plurality of identification templates.
  • the technique according to the aspects of the present invention enables faster matching in object recognition using template matching.
  • FIG. 1 the overall configuration of an object recognition apparatus according to one embodiment of the present invention and its applications will be described.
  • An object recognition apparatus 1 is a system installed on, for example, a production line including a picking system.
  • the object recognition apparatus 1 performs template matching using images captured with a camera 11 to recognize objects 2 in a tray 3.
  • the objects 2 to be recognized are randomly placed in the tray 3.
  • the object recognition apparatus 1 captures an image with the camera 11 at predetermined time intervals, causes an image processing apparatus 10 to recognize the position and posture of each object 2 included in the image, and outputs the recognition result to, for example, a programmable logic controller (PLC) 4 or to a display 12.
  • PLC programmable logic controller
  • the recognition result output from the object recognition apparatus 1 is used in, for example, controlling a picking robot, a machining device, and a printer, or inspecting and measuring the objects 2.
  • the object recognition apparatus 1 mainly includes the camera 11 and the image processing apparatus 10.
  • the camera 11 is an imaging device for capturing a digital image of the objects 2, which is read into the image processing apparatus 10.
  • the camera 11 may be a complementary metal-oxide semiconductor (CMOS) camera or a charge-coupled device (CCD) camera.
  • CMOS complementary metal-oxide semiconductor
  • CCD charge-coupled device
  • the input image may have any resolution, either color or black-and-white mode, either static or dynamic mode, any gray-scale, and any data format, which may be selected in accordance with the type of the objects 2 and the purpose of sensing.
  • a camera dedicated to the image may be used.
  • the image processing apparatus 10 includes a central processing unit (CPU) 110, a main memory 112 used as a working memory, a hard disk 114, which is a fixed storage, a camera interface 116, an input interface 118, a display controller 120, a PLC interface 122, a communication interface 124, and a data reader/writer 126. These components are interconnected by a bus 128 to allow data communication between them.
  • the camera interface 116 allows data transfer between the CPU 110 and the camera 11, and includes an image buffer 116a for temporarily storing image data obtained from the camera 11.
  • the input interface 118 allows data transfer between the CPU 110 and an input unit.
  • the input unit includes a mouse 13, a keyboard, a touch panel, and a jog controller.
  • the display controller 120 is connected to the display 12, such as a liquid crystal monitor, and controls the information that appears on the display.
  • the PLC interface 122 allows data transfer between the CPU 110 and the PLC 4.
  • the communication interface 124 allows data transfer between the CPU 110 and a console or between the CPU 110 and a personal computer or a server.
  • the data reader/writer 126 allows data transfer between the CPU 110 and a memory card 14, which is a recording medium.
  • the image processing apparatus 10 may be a computer with a general-purpose architecture, in which the CPU 110 reads and executes programs stored in the hard disk 114 or a memory card 14 to provide various functions. Such programs are distributed in the form of computer-readable recording mediums storing them, such as a memory card 14 or an optical disc, or provided through the Internet.
  • the program used in the present embodiment may be provided as a single application program or as a module incorporated in another program.
  • the functions of the program may be partly or entirely replaced by a dedicated circuit such as an application specific integrated circuit (ASIC).
  • ASIC application specific integrated circuit
  • Fig. 3 is a functional block diagram of the image processing apparatus 10.
  • the image processing apparatus 10 includes a functional unit serving as a template generation apparatus 20 and a functional unit serving as an object recognition processor 30.
  • the template generation apparatus 20 is a functional unit that generates templates to be used in an object recognition process.
  • the template generation apparatus 20 includes a 3D computer-aided design (CAD) data obtaining unit 201, a standard template generation unit 202, an identification template generation unit 203, and a storage 204.
  • CAD computer-aided design
  • the object recognition processor 30 is a functional unit that recognizes an object in an image captured with the camera 11 by performing template matching of the image with the templates generated by and stored in the template generation apparatus 20.
  • the object recognition processor 30 includes a template information obtaining unit 301, an image obtaining unit 302, an image pyramid generation unit 303, a feature extraction unit 304, a template matching unit 305, and a recognition output unit 306. The functions of the template generation apparatus 20 and the object recognition processor 30 will be described in detail later.
  • Fig. 4 is a diagram showing an example image captured with the camera 11.
  • the image includes a plurality of objects 2, each of which has random positions (XYZ coordinates) and postures (angles to three XYZ axes).
  • XYZ coordinates random positions
  • postures angles to three XYZ axes.
  • different object postures (orientations) relative to the camera 11 cause the object to appear differently (e.g., with different shapes and colors) in a captured image.
  • templates representing different object postures are first prepared for each layer of an image pyramid.
  • a coarse-to-fine search known in the art uses more match candidate templates in searches using higher-resolution images, or a second layer image and a third layer image, although the position and posture of an object are narrowed using a lower-resolution image (e.g., a first layer image). This lowers the processing speed in template matching (refer to the Technical Problem).
  • match candidate templates often have similar features in higher-resolution images, which are obtained by narrowing the positions and the postures of the objects using lower-resolution images.
  • the inventor has then focused on generating identification templates that represent differences between the templates to be used in matching in a higher-resolution image (and also a common template that represents a common part of the templates as appropriate), and performing matching using the identification templates.
  • identification templates increases the processing speed in template matching using higher-resolution images, and thus increases the total processing speed (described in detail later).
  • a template generation process performed by the template generation apparatus 20 will now be described with reference to the flowchart shown in Fig. 5 .
  • the 3D CAD data obtaining unit 201 obtains three-dimensional data representing the three-dimensional shape of an object 2 to be recognized (target object) (step S401).
  • the 3D CAD data obtaining unit 201 can obtain any three-dimensional data that allows recognition of the target object 2 as a three-dimensional figure.
  • the obtained three-dimensional data is three-dimensional CAD data.
  • the 3D CAD data obtaining unit 201 may obtain the three-dimensional CAD data from an external three-dimensional CAD server or a memory (not shown).
  • the standard template generation unit 202 generates two-dimensional images of the object 2 viewed from different points using the three-dimensional data obtained by the 3D CAD data obtaining unit 201, and generates a plurality of templates using the generated two-dimensional images (step S402).
  • the templates generated by the standard template generation unit 202 which correspond to the images of the object 2 viewed from different points, are referred to as standard templates.
  • the templates are data representing the image features of the object 2 to be recognized.
  • Each template can be in any format such as a feature quantity array of a plurality of feature points in the image.
  • the feature quantities include a pixel value (luminance), a luminance gradient orientation, a quantized gradient orientation, histograms of oriented gradients (Hog), HAAR-like, and scale-invariant feature transform (SIFT).
  • luminance gradient orientation is a continuous-value representation of gradient orientations (angles) of luminance in a local area around a feature point.
  • the quantized gradient orientation is a discrete-value representation of gradient orientations (angles) of the luminance in a local area around a feature point (for example, eight directions are represented by 1-byte information corresponding to 0 to 7).
  • the image processing apparatus performs a coarse-to-fine search that uses image layers (or an image pyramid) having resolutions that differ from one image to another in a stepwise manner, for each of which base templates are to be generated.
  • the standard template generation unit 202 generates a plurality of standard templates with different resolutions for different layers.
  • the standard templates with different resolutions may be generated from a plurality of object images with different resolutions, or may be generated by using one standard template generated from the object image with the highest resolution and lowering the resolution of that standard template to generate other standard templates with intended resolutions. In this case, the resolution may be lowered by integrating neighboring feature points based on the positional relationship between the feature points.
  • the standard templates with different resolutions may be generated with any methods.
  • the standard template generation unit 202 generates the plurality of standard templates for each of the image layers having resolutions that differ from one image to another in a stepwise manner, and provides the generated standard templates for each layer to the identification template generation unit 203.
  • the identification template generation unit 203 generates a common template and identification templates for each layer based on the standard templates for each layer provided by the standard template generation unit 202 (step S403). More specifically, the identification template generation unit 203 generates, for each layer (except the highest first layer), the common template that represents a common part of the plurality of standard templates, and the identification templates that represent differences between the plurality of templates.
  • the identification template generation unit 203 may simply generate the identification templates for each layer (except the highest first layer) without generating the common template.
  • Fig. 6 is a diagram describing template matching using only standard templates (example known in the art)
  • Fig. 7 is a diagram describing template matching using standard templates, a common template, and identification templates (present embodiment).
  • the lower-resolution image is the first layer image with the lowest resolution
  • the higher-resolution image is the second layer image with the second lowest resolution.
  • the second layer image which has a resolution twice that of the first layer image, undergoes matching with all the standard templates for the second layer, or 20 standard templates.
  • a match candidate template is narrowed in the first layer image to one standard template with 2 ⁇ 2 pixels as in the above example known in the art.
  • the lower level layers have detection candidates determined by narrowing the candidates in the upper layer images (lower-resolution images).
  • the lower level layers include no unrelated detection candidate.
  • the template matching in the lower level layers may be handled as a classification problem (identification problem), and thus use various identification techniques with learning samples (e.g., Fisher's linear discriminant, logistic regression, support vector machine (SVM), random forests, support vector machine, and neural networks).
  • detection candidates are narrowed in the upper layer images (lower-resolution images).
  • the candidate templates in the lower level layers thus tend to have more common parts and fewer differences (feature quantities used for identification). With this tendency, high-speed matching is performed with the common template and the identification templates in the lower level layers other than the highest first layer.
  • the identification template generation unit 203 generates, for every image layer except the first layer (higher-resolution images), the common template that represents a common part of the plurality of standard templates, and the identification templates that represent differences between the plurality of templates, and stores these templates into the storage 204 (step S404).
  • the identification template generation unit 203 stores the standard templates generated by the standard template generation unit 202 into the storage 204 without generating a common template and identification templates (step S404).
  • the standard templates, the common templates, and the identification templates stored in the storage 204 may be collectively referred to as template information when they may not be differentiated from one another.
  • the template information obtaining unit 301 first obtains the template information stored in the storage 204 in the template generation apparatus 20, and provides the template information to the template matching unit 305 (step S501).
  • the image obtaining unit 302 obtains an input image from the camera 11 (step S502). This image includes objects 2 captured with any postures.
  • the image pyramid generation unit 303 generates lower-resolution images based on the obtained input image (original image) to generate an image pyramid (step S503). For example, the image pyramid generation unit 303 generates an image pyramid including a first layer image with 160 ⁇ 120 pixels and a second layer image with 320 ⁇ 240 pixels.
  • the feature extraction unit 304 extracts feature quantities from the image of each layer included in the image pyramid (step S504).
  • the feature quantities to be extracted are the same as the feature quantities of the template, and include quantized gradient orientations.
  • the first layer image which is the highest layer, undergoes the feature extraction to generate an image having the same resolution as the first layer image and having feature quantity data (feature values) extracted as pixel values at different pixel positions in the first layer image (hereafter, a first layer feature image).
  • the second layer image undergoes the feature extraction to generate a second layer feature image.
  • the template matching unit 305 performs template matching using template information provided by the template information obtaining unit 301 and the feature quantities corresponding to the template information calculated by the feature extraction unit 304.
  • a highest-layer template matching unit 305a first performs matching using the first layer feature image and the templates for the first layer (step S505).
  • the highest-layer template matching unit 305a uses standard templates as the templates for the lowest-resolution first layer (refer to the first layer in Fig. 7 ).
  • the highest template matching unit 305a performs matching using the standard templates to detect one match candidate standard template (2 ⁇ 2 pixels), and transmits the detection result to a lower-layer template matching unit 305b.
  • the lower-layer template matching unit 305b uses the detection result received from the highest-layer template matching unit 305a to define a search scope in the second layer feature image, and performs matching using the second layer feature image and the templates for the second layer (step S506).
  • the lower-layer template matching unit 305b uses, as lower layer templates, the common template among the templates and the identification templates representing the differences between the templates (refer to the second layer in Fig. 7 ). In the example shown in Fig. 7 , the lower-layer template matching unit 305b performs matching using one common template (4 ⁇ 4 pixels) that is common to 20 standard templates and 20 identification templates (1 pixel) for identifying the standard templates.
  • the matching for a lower-resolution image using the standard templates produces a reliable matching result
  • lower level image layers may undergo matching using only identification templates without matching using a common template.
  • the matching using the common template (16 times) may be eliminated, and the matching using only the identification templates (20 times) may be performed. This enables even faster matching.
  • the lower-layer template matching unit 305b performs the matching of the second layer image in the above example, the lower-layer template matching unit 305b performs the same matching of any other layer images, such as a third layer image and a fourth layer image.
  • the lower-layer template matching unit 305b uses a matching result obtained for the second layer image to define a search scope in a third layer feature image and performs the matching using the third layer feature image and the templates for the third layer.
  • the template matching unit 305 recognizes the position and posture of an object, and then outputs the resultant recognition information to the recognition output unit 306 (step S507).
  • the recognition output unit 306 outputs the recognition information received from the template matching unit 305 to an external device or a liquid crystal panel.
  • the recognition information is used to, for example, inspect and measure the objects 2 and to control a picking robot.
  • template matching including a coarse-to-fine search is performed using the common template representing a common part of the templates and the identification templates representing the differences between the templates in lower level layers. Matching using the identification templates representing only the differences between the plurality of standard templates enables faster processing than matching using the standard templates.
  • match candidate templates are narrowed to a single template for the highest level layer (lower-resolution image) (refer to the first layer in Fig. 7 ) in the above embodiment, the embodiment is not limited to this structure.
  • all the templates with a probability of matching with the lower-resolution image (the first layer image) greater than or equal to a threshold may be selected as match candidates, and may be used in matching with lower level layers (higher-resolution images) to obtain the intended match (e.g., the position and posture of an object).
  • this matching result may be combined with the matching results obtained using a common template and identification templates in higher-resolution images to improve the accuracy of object recognition.
  • the templates generated from three-dimensional CAD data are used to perform three-dimensional recognition of the position and posture of an object (or three-dimensional object recognition)
  • this technique is also applicable to two-dimensional object recognition.
  • the technique is applicable to a bin picking system for recognizing the positions and postures of randomly stacked, overlapping objects (three-dimensional object recognition) and picking them one by one, or to a picking system for recognizing the positions and postures (rotations) of non-overlapping objects that are placed without being stacked on one another (two-dimensional object recognition) and picking them one by one.
  • the technique is used in template matching using a coarse-to-fine search, the embodiment is not limited to the coarse-to-fine search for narrowing the position of an object.
  • the technique is also applicable to hierarchical matching for narrowing the type and the posture of an object in a stepwise manner (without narrowing the position of an object).
  • unit does not merely refer to a physical component, but also to the function of the unit implemented by software.
  • the function of a single unit or apparatus may be implemented by two or more physical components or apparatuses, or the functions of two or more units or apparatuses may be implemented by a single physical component or apparatus.
  • the program for implementing each processing described herein may be stored in a recording medium.
  • the recording medium may be used to install the program in the image processing apparatus 10.
  • the recording medium storing the program may be a non-transitory recording medium.
  • the non-transitory recording medium may be any recording medium, such as a CD-ROM.
  • An image processing apparatus for recognizing an object by template matching, the apparatus comprising at least one hardware processor, the hardware processor being configured to execute a predetermined program to function as:
  • An image processing method for recognizing an object by template matching the method being implemented by at least one hardware processor, the method comprising:

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Nonlinear Science (AREA)
  • Robotics (AREA)
  • Image Analysis (AREA)
EP18000141.4A 2017-03-10 2018-02-14 Image processing apparatus, image processing method, template generation apparatus, object recognition processor, and object recognition processing program Withdrawn EP3373197A1 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
JP2017046114A JP6732214B2 (ja) 2017-03-10 2017-03-10 画像処理装置、画像処理方法、テンプレート作成装置、物体認識処理装置及びプログラム

Publications (1)

Publication Number Publication Date
EP3373197A1 true EP3373197A1 (en) 2018-09-12

Family

ID=61226359

Family Applications (1)

Application Number Title Priority Date Filing Date
EP18000141.4A Withdrawn EP3373197A1 (en) 2017-03-10 2018-02-14 Image processing apparatus, image processing method, template generation apparatus, object recognition processor, and object recognition processing program

Country Status (4)

Country Link
US (1) US20180260669A1 (zh)
EP (1) EP3373197A1 (zh)
JP (1) JP6732214B2 (zh)
CN (1) CN108573273B (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210240995A1 (en) * 2020-02-05 2021-08-05 Sony Interactive Entertainment Inc. Information processing apparatus, information processing method, and program

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6554900B2 (ja) * 2015-04-28 2019-08-07 オムロン株式会社 テンプレート作成装置及びテンプレート作成方法
JP2017099616A (ja) * 2015-12-01 2017-06-08 ソニー株式会社 手術用制御装置、手術用制御方法、およびプログラム、並びに手術システム
JP6976270B2 (ja) * 2016-04-08 2021-12-08 オービタル インサイト インコーポレイテッド 地理的領域におけるコンテナ内に格納された量の遠隔決定
EP3836027A4 (en) 2018-08-10 2022-07-06 Yamaha Corporation METHOD AND APPARATUS FOR GENERATION OF FREQUENCY COMPONENT VECTOR OF TIME SERIES DATA
JP7193979B2 (ja) * 2018-10-29 2022-12-21 富士フイルムヘルスケア株式会社 医用撮像装置、画像処理装置、および、画像処理方法
JP7279341B2 (ja) * 2018-11-06 2023-05-23 富士フイルムビジネスイノベーション株式会社 識別システム、識別装置、データ取得システム及びデータ取得装置
JP7277116B2 (ja) * 2018-11-29 2023-05-18 キヤノン株式会社 情報処理装置、情報処理方法及びプログラム
CN109829502B (zh) * 2019-02-01 2023-02-07 辽宁工程技术大学 一种面向重复纹理及非刚性形变的像对高效稠密匹配方法
CN110998663B (zh) * 2019-11-22 2023-12-01 驭势（上海）汽车科技有限公司 一种仿真场景的图像生成方法、电子设备和存储介质
JP7459769B2 (ja) 2020-11-25 2024-04-02 オムロン株式会社 埋設物情報管理装置およびこれを備えた埋設物情報管理システム、埋設物情報管理方法、埋設物情報管理プログラム
CN112784857B (zh) * 2021-01-29 2022-11-04 北京三快在线科技有限公司 一种模型训练以及图像处理方法及装置
CN113361651B (zh) * 2021-03-05 2022-01-04 牧今科技 生成用于物体检测的安全空间列表的方法和计算系统
JPWO2022234742A1 (zh) * 2021-05-06 2022-11-10
CN114398978B (zh) * 2022-01-13 2022-10-11 深圳市腾盛精密装备股份有限公司 一种模板匹配方法、装置、存储介质以及电子设备
CN117115487B (zh) * 2023-10-23 2024-03-08 睿励科学仪器（上海）有限公司 模板匹配方法、模板匹配系统及存储介质

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016207147A (ja) 2015-04-28 2016-12-08 オムロン株式会社 テンプレート作成装置及びテンプレート作成方法

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH11144057A (ja) * 1997-11-12 1999-05-28 Ricoh Co Ltd 画像認識装置および画像認識方法
JP3680826B2 (ja) * 2002-08-21 2005-08-10 三菱電機株式会社 画像処理装置
DE602007003849D1 (de) * 2007-10-11 2010-01-28 Mvtec Software Gmbh System und Verfahren zur 3D-Objekterkennung
JP5593217B2 (ja) * 2010-12-28 2014-09-17 クラリオン株式会社 車両用外界認識装置およびそれを用いた車両システム
CN102654902B (zh) * 2012-01-16 2013-11-20 江南大学 基于轮廓向量特征的嵌入式实时图像匹配方法
EP2808828B1 (en) * 2013-05-31 2020-08-05 Omron Corporation Image matching method, image matching device, model template generation method, model template generation device, and program
JP6560852B2 (ja) * 2014-10-10 2019-08-14 日本電信電話株式会社 画像処理装置、方法、及びプログラム
JP2016130979A (ja) * 2015-01-14 2016-07-21 株式会社リコー 検知装置、検知処理プログラム及び検知システム
CN106251353A (zh) * 2016-08-01 2016-12-21 上海交通大学 弱纹理工件及其三维位姿的识别检测方法及系统

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016207147A (ja) 2015-04-28 2016-12-08 オムロン株式会社 テンプレート作成装置及びテンプレート作成方法

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
BORGEFORS G: "Hierarchical Chamfer Matching: A Parametric Edge Matching Algorithm", IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, IEEE COMPUTER SOCIETY, USA, vol. 10, no. 6, 1 November 1989 (1989-11-01), pages 849 - 865, XP002114858, ISSN: 0162-8828, DOI: 10.1109/34.9107 *
GAVRILA D M: "A Bayesian, Exemplar-Based Approach to Hierarchical Shape Matching", IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, IEEE COMPUTER SOCIETY, USA, vol. 29, no. 8, 1 August 2007 (2007-08-01), pages 1408 - 1421, XP011185895, ISSN: 0162-8828, DOI: 10.1109/TPAMI.2007.1062 *
KONISHI YOSHINORI ET AL: "Fast 6D Pose Estimation from a Monocular Image Using Hierarchical Pose Trees", 17 September 2016, MEDICAL IMAGE COMPUTING AND COMPUTER-ASSISTED INTERVENTION - MICCAI 2015 : 18TH INTERNATIONAL CONFERENCE, MUNICH, GERMANY, OCTOBER 5-9, 2015; PROCEEDINGS; [LECTURE NOTES IN COMPUTER SCIENCE; LECT.NOTES COMPUTER], SPRINGER INTERNATIONAL PUBLISHING, CH, ISBN: 978-3-642-38287-1, ISSN: 0302-9743, XP047355263 *
LIU HUI ET AL: "Template-Based Multiple Codebooks Generation for Fine-Grained Shopping Classification and Retrieval", 2014 5TH INTERNATIONAL CONFERENCE ON DIGITAL HOME, IEEE, 28 November 2014 (2014-11-28), pages 293 - 298, XP032715326, DOI: 10.1109/ICDH.2014.63 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210240995A1 (en) * 2020-02-05 2021-08-05 Sony Interactive Entertainment Inc. Information processing apparatus, information processing method, and program
US11580744B2 (en) * 2020-02-05 2023-02-14 Sony Interactive Entertainment Inc. Information processing apparatus, information processing method, and program

Also Published As

Publication number Publication date
JP6732214B2 (ja) 2020-07-29
JP2018151748A (ja) 2018-09-27
US20180260669A1 (en) 2018-09-13
CN108573273A (zh) 2018-09-25
CN108573273B (zh) 2022-02-18

Similar Documents

Publication Publication Date Title
EP3373197A1 (en) Image processing apparatus, image processing method, template generation apparatus, object recognition processor, and object recognition processing program
EP3258441B1 (en) Template creation device and template creation method
US10713530B2 (en) Image processing apparatus, image processing method, and image processing program
US10366307B2 (en) Coarse-to-fine search method, image processing device and recording medium
JP7151805B2 (ja) 学習データ生成装置、学習データ生成方法、およびプログラム
EP3502958B1 (en) Object recognition processing apparatus, object recognition processing method, and program
US8526679B2 (en) Image processing apparatus and image processing method
JP5936561B2 (ja) 画像における外観及びコンテキストに基づく物体分類
US10311564B2 (en) Image processing device, image sensor, and image processing method
EP3410396B1 (en) Moving object tracking apparatus, moving object tracking method, and computer-readable medium
JP2012123631A (ja) 注目領域検出方法、注目領域検出装置、及びプログラム
US10706319B2 (en) Template creation apparatus, object recognition processing apparatus, template creation method, and program
JP2023535005A (ja) 電子画像内の物体を認識するための方法及びシステム又は装置
JP5814714B2 (ja) 画像類似度判定装置、画像類似度判定方法及び画像類似度判定プログラム
WO2015136716A1 (ja) 画像処理装置、画像センサ、画像処理方法
Al-khaleeli et al. Automatic Machining Setup via Deep Learning and Image Processing
Li et al. Image mosaic using SIFT for bridge cable surface flaw detection
Yao et al. Catching Data from Displayers by Machine Vision

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20190312

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20190823

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20200303