JP7251631B2 - Template creation device, object recognition processing device, template creation method, object recognition processing method, and program - Google Patents

Description

The present invention relates to a template creation device, an object recognition processing device, a template creation method, an object recognition processing method, and a program.

Template matching is one of methods for recognizing (detecting) an object from an image. Template matching is a method of detecting an object contained in an input image by preparing a model (template) of an object to be recognized in advance and evaluating the matching degree of image features between the input image and the model. be. Object recognition by template matching is used in a wide variety of fields, such as inspection and picking in FA (Factory Automation), robot vision, and surveillance cameras.

Such a template can be created using 3D CAD data of an object to be recognized (hereinafter also referred to as "target object"). However, there are cases where the 3D CAD data cannot be used or is not suitable, such as when there is no 3D CAD data of the target object or when there is a large difference between the 3D CAD data and the actual target object. In such a case, a template is created using three-dimensional measurement data obtained by three-dimensionally measuring an actual target object.

Conventionally, when integrating three-dimensional point cloud data measured from various viewpoints, it was necessary to integrate neighboring point cloud data in areas where fields of view overlap. Therefore, there is a problem that detailed shape information is lost by averaging data including position and orientation recognition errors, noise, and the like. (See Patent Document 1, for example).

Further, in the technique described in Patent Literature 2, three-dimensional measurement data from each viewpoint is weighted and accumulated in a voxel (cubic box) set in a three-dimensional space. For this reason, measurement errors, noise, and viewpoint position/orientation recognition errors are accumulated, and corners and fine shapes are rounded or lost. In addition, the texture of the surface of the object is similarly averaged and thus blurred or disappears.

If a template for object recognition is created based on the three-dimensional shape created by integration in this way, there is a problem that a template with a different shape and texture from the real thing is created, resulting in a decrease in recognition accuracy. .

Japanese Patent No. 6298035 U.S. Pat. No. 9,053,571

SUMMARY OF THE INVENTION The present invention has been made in view of the problems described above. To provide a technology capable of creating a template that is close to the real thing without

The present invention for solving the above problems is
A template creation device for creating a template corresponding to a target object viewed from a viewpoint,
a template creation viewpoint position/orientation data acquisition unit that acquires data relating to the position and orientation of a template creation viewpoint as the viewpoint for creating the template;
a three-dimensional measurement data acquisition unit that acquires three-dimensional measurement data of the target object;
a measurement viewpoint position/orientation data acquisition unit that acquires data relating to the position and orientation of a measurement viewpoint as a viewpoint for three-dimensionally measuring the target object;
The three-dimensional measurement data corresponding to the different measurement viewpoints are projected onto a predetermined plane with respect to the template creation viewpoint, and basic three-dimensional data for extracting a feature amount is generated based on the projected three-dimensional measurement data. a basic three-dimensional data generation unit to
with
The basic three-dimensional data generation unit selects the measurement viewpoint having a closer distance to the template creation viewpoint, and converts the three-dimensional measurement data corresponding to the selected measurement viewpoint to generate the basic three-dimensional data. This template creation device is characterized in that it is used preferentially to

According to the present invention, when generating basic three-dimensional data for extracting a feature amount from three-dimensional measurement data measured at different measurement viewpoints, three-dimensional data corresponding to a measurement viewpoint having a closer distance to the template creation viewpoint is generated. Since the measurement data is preferentially used, fine shapes and textures are not lost by smoothing three-dimensional measurement data measured from different measurement viewpoints. Therefore, a template close to the real thing can be created.

In the present invention, the distance of the measurement viewpoint to the template creation viewpoint can be evaluated in various ways. For example, it can be evaluated by the size of the angle formed by the vector from the template creation viewpoint to the center of the target object and the vector from the measurement viewpoint to the center of the target object, or by the Euclidean distance or geodesic distance between the two viewpoints. You may evaluate, but are not limited to these.

Further, in the present invention, three-dimensional measurement data is data obtained by three-dimensional measurement and composed of a plurality of points each having three-dimensional information. Like point cloud data, each point may have a three-dimensional coordinate value in the camera coordinate system, or each point (each pixel) of a two-dimensional image is associated with a depth value (depth distance information). format data.

Further, in the present invention, the predetermined plane with respect to the template creation viewpoint is an image such as a CCD plane with respect to the focal point of the CCD camera when the image acquisition unit for acquiring the image to be object recognition by the template is composed of a CCD camera. It is flat. Three-dimensional measurement data corresponding to different measurement viewpoints are projected onto a predetermined plane with respect to the template creation viewpoint, and basic three-dimensional data to be used for extracting feature amounts is generated based on the projected three-dimensional measurement data. In this case, the amount of memory used can be reduced and the processing time can be shortened as compared with the case where the three-dimensional measurement data are integrated and generated in the three-dimensional space.

In the present invention,
The basic three-dimensional data generation unit converts a portion of the basic three-dimensional data projected onto the predetermined plane, which is not included in the three-dimensional measurement data with higher priority, into the three-dimensional measurement data as the predetermined may be supplemented by data projected onto the plane of .

In this way, when generating the basic three-dimensional data, the three-dimensional measurement data corresponding to the measurement viewpoint with higher priority is selectively used, so that the three-dimensional measurement data corresponding to different measurement viewpoints are leveled. without losing detailed shapes and textures of the target object. Therefore, a template close to the real thing can be created.

In addition, the present invention
an image acquisition unit that acquires an image including the target object;
a feature extraction unit that extracts a feature amount from the acquired image;
a template matching unit that recognizes the target object included in the acquired image by template matching using a template created by the template creation device;
and an object recognition processing device.

According to the present invention, object recognition is performed using a template that is close to the real thing and that reflects the detailed shape and texture of the target object, so object recognition can be performed with higher accuracy.

In addition, the present invention
A template creation method for creating a template corresponding to a target object viewed from a viewpoint, comprising:
obtaining information about the position and orientation of a template creation viewpoint as the viewpoint for creating the template;
obtaining three-dimensional measurement data of the target object;
acquiring data on the position and orientation of a measurement viewpoint as a viewpoint for three-dimensionally measuring the target object;
selecting the measurement viewpoint having a closer distance to the template creation viewpoint;
a step of projecting the three-dimensional measurement data corresponding to the selected measurement viewpoint onto a predetermined plane with respect to the template creation viewpoint;
When generating basic three-dimensional data for extracting a feature amount based on the projected three-dimensional measurement data, giving priority to the three-dimensional measurement data corresponding to the measurement viewpoint closer to the distance. generating basic three-dimensional data using
It is a template creation method including

According to the present invention, when generating basic three-dimensional data for extracting a feature amount from three-dimensional measurement data measured at different measurement viewpoints, three-dimensional data corresponding to a measurement viewpoint having a closer distance to the template creation viewpoint is generated. Since the measurement data is preferentially used, fine shapes and textures are not lost by smoothing three-dimensional measurement data measured from different measurement viewpoints. Therefore, a template close to the real thing can be created.
Also, the order of each step included in the present invention is not limited to the described order.

In the present invention, the distance of the measurement viewpoint to the template creation viewpoint can be evaluated in various ways. For example, it can be evaluated by the size of the angle formed by the vector from the template creation viewpoint to the center of the target object and the vector from the measurement viewpoint to the center of the target object, or by the Euclidean distance or geodesic distance between the two viewpoints. You may evaluate, but are not limited to these.

Further, in the present invention, three-dimensional measurement data is data obtained by three-dimensional measurement and composed of a plurality of points each having three-dimensional information. Like point cloud data, each point may have a three-dimensional coordinate value in the camera coordinate system, or each point (each pixel) of a two-dimensional image is associated with a depth value (depth distance information). format data.

In addition, the present invention
An object recognition processing method for recognizing a target object using a template,
obtaining an image containing the target object;
extracting features from the acquired image;
a step of recognizing the target object by template matching using a template created by the template creating method;
An object recognition processing method including

According to the present invention, object recognition is performed using a template that is close to the real thing and that reflects the detailed shape and texture of the target object, so object recognition can be performed with higher accuracy.

In addition, the present invention
A program for causing a computer to execute each step of the template creation method.

In addition, the present invention
A program for causing a computer to execute each step of the object recognition method.

According to the present invention, when creating a template for object recognition using three-dimensional measurements measured from different viewpoints, a technique is provided that can create a template that is close to the real thing without losing detailed shapes and textures. can be provided.

It is a figure showing an example of whole composition of an object recognition device in an embodiment of the present invention. 1 is a functional block diagram of an object recognition device according to an embodiment of the present invention; FIG. 6 is a flowchart showing an example of template creation processing according to the embodiment of the present invention; FIG. 4 is a diagram illustrating a configuration for capturing an image of a target object and acquiring an image capturing viewpoint position and orientation according to the embodiment of the present invention; 4A and 4B illustrate the relationship between the template creation viewpoint and the shooting viewpoint in the embodiment of the present invention. FIG. 4 is a diagram schematically showing integration processing on an image plane for an embodiment of the present invention; 4 is a flowchart showing an example of object recognition processing according to the embodiment of the present invention; It is a figure explaining the example of application of the present invention.

[Example of application]
Hereinafter, application examples of the present invention will be described with reference to the drawings.
FIG. 8 is a diagram for explaining the principle of the template creation method according to the present invention.
FIG. 8A is an example of creating a template of the target object 227 from the viewpoint VP. Here, an arrow from the viewpoint VP to the center 227c of the target object 227 is displayed as a solid line.

Conventionally, when creating a template of the target object 227 from the viewpoint VP, the target object 227 is measured from a plurality of different viewpoints of IMPa, IMPb, and IMPc by the three-dimensional sensor 220, and the three-dimensional measurement data is captured in the three-dimensional space. was integrated with Three-dimensional measurement data IMa, IMb, and IMc respectively obtained from viewpoints IMPa, IMPb, and IMPc are schematically shown on the left side of FIG. 8(B). At this time, the three-dimensional measurement data IMa, IMb, and IMc as shown on the left side of FIG. not necessarily match. In the data IM0 obtained by integrating such three-dimensional measurement data in a three-dimensional space, as shown on the right side of FIG. It will be done.

On the other hand, in the present invention, when creating a template of the target object 227 from the viewpoint VP, the target object 227 from a plurality of different viewpoints is selected from the three-dimensional measurement data obtained by the three-dimensional sensor 220, and the viewpoint position is the data closest to the position of the template creation viewpoint VP, projected onto the image plane of the template creation viewpoint VP, and used preferentially. Pixels not projected onto the image plane of the template creation viewpoint VP are compensated for by projecting the three-dimensional measurement data of the target object from a viewpoint position far from the position of the template creation viewpoint VP.
Therefore, although there is an error in recognition of the viewpoint position and orientation of the three-dimensional sensor 220, the shape information of the target object 227 is not lost.
In addition, the three-dimensional measurement data acquired by the three-dimensional sensor is not integrated in the three-dimensional space, but is integrated as two-dimensional data projected onto the image plane of the template creation viewpoint VP. Processing time can be shortened.

[Embodiment]
(Overall configuration of object recognition device)
An object recognition device according to an embodiment of the present invention will be described with reference to FIG.

The object recognition device 2 is installed in a production line that assembles and processes articles, and uses data captured from the sensor unit 20 to recognize the position and orientation of an object 27 loaded on a tray 26 by template matching ( 3D object recognition). Objects 27 to be recognized are randomly stacked on the tray 26 .

The object recognition device 2 is roughly composed of a sensor unit 20 and an image processing device 21 . A wired or wireless connection is established between the sensor unit 20 and the image processing device 21 , and the output of the sensor unit 20 is taken into the image processing device 21 . The image processing device 21 is a device that performs various processes using data captured from the sensor unit 20 . The processing of the image processing device 21 may include, for example, distance measurement (distance measurement), three-dimensional shape recognition, object recognition, scene recognition, and the like. A recognition result of the object recognition device 2 is output to a PLC (programmable logic controller) 25, a display 22, or the like, for example. The recognition result is used, for example, for control of the picking robot 28, control of processing devices and printing devices, inspection and measurement of the target object 27, and the like.

(sensor unit)
The sensor unit 20 has at least a camera for taking an optical image of the target object 27 . Furthermore, the sensor unit 20 may include components (sensors, illumination devices, light projection devices, etc.) necessary for three-dimensional measurement of the target object 27 . For example, when the depth distance is measured by stereo matching (also called stereo vision, stereo camera method, etc.), the sensor unit 20 is provided with a plurality of cameras. In the case of active stereo, the sensor unit 20 is further provided with a light projection device for projecting pattern light onto the target object 27 . When three-dimensional measurement is performed by the space-encoded pattern projection method, the sensor unit 20 is provided with a light projection device for projecting pattern light and a camera. In addition, any method such as a photometric stereo method, a TOF (time of flight) method, a phase shift method, or the like may be used as long as the method can acquire the three-dimensional information of the target object 27 .

(Image processing device)
The image processing device 21 is configured by a computer including, for example, a CPU (processor), RAM (memory), nonvolatile storage device (hard disk, SSD, etc.), input device, output device, and the like. In this case, the CPU develops a program stored in the nonvolatile storage device in the RAM and executes the program, thereby realizing various configurations described later. However, the configuration of the image processing device 21 is not limited to this, and all or part of the configuration described later may be realized by a dedicated circuit such as FPGA or ASIC, or by cloud computing or distributed computing. You may

FIG. 2 is a block diagram showing the configuration of the image processing device 21. As shown in FIG. The image processing device 21 has the configuration of the template creation device 30 and the configuration of the object recognition processing device 31 .
The template generation device 30 is configured to generate a template used in object recognition processing, and includes a multi-viewpoint three-dimensional measurement data acquisition unit 300, an imaging viewpoint three-dimensional position and orientation data acquisition unit 301, and a template creation viewpoint position information acquisition unit. 302 , a three-dimensional data generation unit 303 for template creation viewpoints, a feature extraction unit 304 , a template creation unit 305 , and a template information output unit 306 . In this embodiment, the multi-viewpoint three-dimensional measurement data acquisition section 300 corresponds to the "three-dimensional measurement data acquisition section" of the present invention. Further, in this embodiment, the shooting viewpoint three-dimensional position/orientation data acquisition unit 301 corresponds to the “measurement viewpoint position/orientation data acquisition unit” of the present invention. Further, in this embodiment, the template creation viewpoint position information acquisition unit 302 corresponds to the "template creation viewpoint position/orientation data acquisition unit" of the present invention. In this embodiment, the three-dimensional data generation unit 303 of the template creation viewpoint corresponds to the "basic three-dimensional data generation unit" of the present invention.
The object recognition processing device 31 is configured to execute object recognition processing by template matching, and includes an image acquisition unit 310, an image pyramid creation unit 311, a feature extraction unit 312, a template information acquisition unit 313, a template matching unit 314, a recognition It has a result output unit 315 . In this embodiment, the image acquisition section 310, the feature extraction section 312, and the template matching section 314 respectively correspond to the "image acquisition section", the "feature extraction section", and the "template matching section" of the present invention.

(Template creation process)
An example of template creation processing by the template creation device 30 will be described with reference to the flowchart of FIG.

In step S101, the template creation viewpoint 3D data generation unit 303 acquires 3D measurement data of a target object photographed from different viewpoints, information relating to the 3D position and orientation of the shooting viewpoint, and information relating to the template creation viewpoint position. Here, the multi-viewpoint three-dimensional measurement data acquisition unit 300 acquires the three-dimensional measurement data of the target object from a plurality of different viewpoints. Therefore, the three-dimensional data generation unit 303 of the template creation viewpoint acquires the three-dimensional measurement data of the target object photographed from different viewpoints from the multi-viewpoint three-dimensional measurement data acquisition unit 300 . Further, when the multi-viewpoint three-dimensional measurement data acquisition unit photographs a target object from a plurality of different viewpoints and acquires the three-dimensional measurement data of the target object, the three-dimensional position and orientation data of the shooting viewpoints is the three-dimensional position of the shooting viewpoint. Acquired by the posture data acquisition unit 301 . Therefore, the three-dimensional data generation unit 303 of the template creation viewpoint acquires information on the three-dimensional position and orientation of the shooting viewpoint from the shooting viewpoint three-dimensional position and orientation data acquisition unit 301 . Since the template creation viewpoint position information acquisition unit 302 acquires data on the position and orientation of the template creation viewpoint, the template creation viewpoint 3D data generation unit 303 obtains the data on the template creation viewpoint position and orientation from the template creation viewpoint. Acquired from the position information acquisition unit 302 . In this embodiment, the shooting viewpoint corresponds to the "measurement viewpoint" of the present invention.

FIG. 4 is a diagram illustrating an outline of a configuration for acquiring three-dimensional measurement data of a target object from a plurality of different viewpoints and information on the three-dimensional position and orientation of the shooting viewpoints.
A three-dimensional sensor 220 is arranged at a predetermined position and orientation with respect to the target object 27 to acquire (image) three-dimensional measurement data of the target object 27 and to acquire position and orientation information of the imaging viewpoint. When the three-dimensional sensor 220 captures images of the target object from a plurality of different viewpoints and acquires three-dimensional measurement data, the three-dimensional sensor 220 may be fixed and the posture of the target object 27 may be changed. The target object 27 may be fixed and the posture of the three-dimensional sensor 220 may be changed, or the relative positional relationship between the three-dimensional sensor 220 and the target object 27 may be changed. FIG. 4 shows an example in which the three-dimensional sensor 220 captures an image of the target object 27 placed on the placement surface 32 to obtain three-dimensional measurement data. Here, the three-dimensional sensor 220 may be fixed and the posture of the placement surface 32 may be changed, or the posture of the placement surface 32 may be fixed and the posture of the three-dimensional sensor 220 may be changed.
In addition, the three-dimensional position and orientation of the shooting viewpoint are the plane marker 33 displayed on the mounting surface 32 of the target object, and the reference object (whose shape is known by 3D CAD data or the like) fixedly arranged with respect to the mounting surface. It may be obtained by recognizing the target object), or may be obtained by providing the mounting surface 32 of the target object on the goniometer stage.

FIG. 5 shows an example of template creation viewpoints and shooting viewpoints. In this example, 42 vertices of the regular octasahedron containing the target object 27 are used as template creation viewpoints VP, and here, one vertex VP1 (indicated by a black circle) is set as the template creation viewpoint. The number and arrangement of the template creation viewpoints VP may be appropriately set according to the required resolution, the shape of the target object 27, possible poses, and the like. The number and positions of the template creation viewpoints VP may be specified by the user and acquired by the template creation viewpoint position information acquisition unit 302, or may be automatically set by the template creation viewpoint position information acquisition unit 302. It may be held. In the example shown in FIG. 5, the target object is placed at the center of the regular octahedron, so by specifying the position of the vertex of this regular octahedron, the position of the viewpoint for creating the template and its posture are uniquely determined. However, the method of setting the position and orientation of the template creation viewpoint is not limited to this. In FIG. 5, among the vertices of the regular octasahedron, the vertices existing on the front side of the paper surface are indicated by white circles.

In FIG. 5, an arrow IMP1, an arrow IMP2, and an arrow IMP3 indicate the first shooting viewpoint, the second shooting viewpoint, and the third shooting viewpoint, respectively. Three shooting viewpoints are shown here as an example, but the present invention is not limited to this.

Returning to the flowchart of FIG. 3, in step S102, a viewpoint position for which no template has been created is selected. Here, it is assumed that the viewpoint position indicated by the black circle in FIG. 5 is selected as the viewpoint position for which no template is created.

Next, in step S103, angles between the template creation viewpoint vector and all the shooting viewpoint vectors are calculated. Here, assume that all shooting viewpoints are arrow IMP1, arrow IMP2, and arrow IMP3.
The template creation viewpoint vector is a vector directed from the template creation viewpoint VP to the center of the target object. For the template creation viewpoint indicated by the black circle in FIG. 5, the template creation viewpoint vector is the center of the target object arranged so as to coincide with the center of the octahedron from the vertex indicated by the black circle of the regular octahedron. is a vector pointing to The imaging viewpoint vector is a vector that points from the imaging viewpoint to the center of the target object when the target object 27 is captured by the multi-viewpoint three-dimensional measurement data acquisition unit 300 . In FIG. 5, the shooting viewpoint vectors corresponding to the first shooting viewpoint, the second shooting viewpoint, and the third shooting viewpoint are direction vectors of arrows IMP1, IMP2, and IMP3, respectively.
Here, the angles between the vectors pointing from the vertices of the regular octahedron indicated by black circles to the center and the vectors in the directions of arrows IMP1, IMP2, and IMP3 are calculated.

In step S104, the one with the smallest angle is selected from the unprojected imaging viewpoints and measurement data. Here, it is assumed that all shooting viewpoints are unprojected. In the example of FIG. 5, since the angle between the shooting viewpoint vector and the template creation viewpoint vector with respect to the shooting viewpoint of the arrow IMP1 is the smallest, the first shooting viewpoint indicated by the arrow IMP1 and the shooting from this first shooting viewpoint are obtained. Select 3D measurement data.

Next, in step S105, the three-dimensional measurement data captured by the multi-viewpoint three-dimensional measurement data acquisition unit 300 from the first shooting viewpoint indicated by the arrow IMP1 is converted into an image for the template creation viewpoint VP1 selected in step S102. Project onto a plane. FIG. 6A schematically shows an image obtained by projecting three-dimensional measurement data taken from a first shooting viewpoint indicated by an arrow IMP1 onto an image plane. In FIGS. 6A to 6C, a solid line indicates a portion corresponding to three-dimensional measurement data from each shooting viewpoint, and a dashed line indicates integrated data for comparison. In this embodiment, the image plane corresponds to the "predetermined plane" of the invention.

In this way, the three-dimensional measurement data shot from different shooting viewpoints are projected onto the image plane for the template creation viewpoint, and integrated processing is performed on this plane, so the amount of memory used is small and the processing time is shortened. can.

Then, in step S106, it is determined whether or not the three-dimensional measurement data taken from all the shooting viewpoints have been projected onto the image plane with respect to the template creation viewpoint VP1.

If it is determined No in step S106, that is, among all the shooting viewpoints, among the three-dimensional measurement data taken from the shooting viewpoint, there is a shooting viewpoint that is not projected onto the image plane for the template creation viewpoint VP1. If so, the process returns to step S104. In the example of FIG. 5, the shooting viewpoint vector from the third shooting viewpoint indicated by the arrow IMP3, which has the smallest angle with the template creation viewpoint vector among the unprojected shooting viewpoints, and The photographed three-dimensional measurement data is selected, and the process proceeds to step S105. Similarly, when returning from step S106 to step S104, the imaging viewpoint vector from the second imaging viewpoint indicated by the arrow IMP2 and the three-dimensional measurement data captured from the second imaging viewpoint are selected. be done. FIG. 6B schematically shows an image obtained by projecting three-dimensional measurement data taken from a third shooting viewpoint indicated by an arrow IMP3 onto an image plane. FIG. 6C schematically shows an image obtained by projecting the three-dimensional measurement data taken from the second shooting viewpoint indicated by the arrow IMP2 onto the image plane. FIG. 6D schematically shows an image in which the three-dimensional measurement data from the first shooting viewpoint to the third shooting viewpoint are projected onto the image plane and integrated in this way.

In this way, the shooting viewpoints closest to the template creation viewpoint are projected onto the image plane with respect to the template creation viewpoint, and the pixels for which no shooting point cloud data is present, that is, the portions not included in the higher priority shooting point cloud data are Supplement the point cloud data from the shooting viewpoint. As a result, it is possible to create a template that is close to the real thing without losing the detailed shape and texture of the target object.

If it is determined Yes in step S106, that is, if it is determined that the three-dimensional measurement data captured from all the shooting viewpoints has been projected onto the image plane for the template creation viewpoint VP1, The process proceeds to step S107.

In step S107, the feature extraction unit 304 extracts an image of the target object 27 based on the three-dimensional measurement data of the target object 27 from all shooting viewpoints projected onto the image plane with respect to the template creation viewpoint VP1 selected in step S102. Extract features. Image features include, for example, brightness, color, brightness gradient direction, quantization gradient direction, HoG (Histogram of Oriented Gradients), surface normal direction, HAAR-like, SIFT (Scale-Invariant Feature Transform), etc. can be done. The intensity gradient direction represents the direction (angle) of the intensity gradient in a local area centered on a feature point as a continuous value. Gradient directions are represented by discrete values (for example, 8 directions are stored as 1-byte information from 0 to 7). The feature extraction unit 304 may obtain image features for all points (pixels) of the image 60 projected onto the image plane with respect to the template creation viewpoint, or may obtain image features for some points sampled according to a predetermined rule. may A point from which an image feature is obtained is called a feature point.

In step S108, the template creation unit 305 creates a template corresponding to the template creation viewpoint VP1 based on the image features extracted in step S107. A template is, for example, a data set including coordinate values of each feature point and extracted image features.

The processing of steps S102 to S108 is performed for all template creation viewpoints VP acquired in step S101 (step S109). When template creation is completed for all template creation viewpoints VP, the template creation unit 305 outputs template data to the object recognition processing device 31 via the template information output unit 306 (step S110).

(Object recognition processing)
An example of the object recognition processing method by the object recognition processing device 31 will be described with reference to the flowchart of FIG.

First, in step S<b>201 , the template information acquisition unit 313 acquires template data output from the template information output unit 306 of the template creation device 30 and supplies the data to the template matching unit 314 .

Next, in step S202, the image acquisition unit 310 acquires an image that includes the target object and is the target of object recognition processing. This image may be read from the internal storage device of the image processing apparatus 21, or may be obtained from an external storage or the like via a network. Also, an image captured by the sensor unit 20 may be acquired. When the recognition result is used to control the picking robot 28, the sensor unit 20 may be attached to the arm of the picking robot 28. FIG.

Next, in step S203, the image pyramid creation unit 311 creates a low-resolution image from the image acquired by the image acquisition unit 310 to create an image pyramid. For example, the first layer image is 160 pixels x 120 pixels, and the second layer image is 320 pixels x 240 pixels. can be configured.

Next, in step S204, the feature extraction unit 312 extracts image features for each layer image forming the image pyramid. The image features to be extracted are feature amounts of the same type as the image features used when creating the template. As a result of image feature extraction for the first layer image (the image with the lowest resolution), which is the top layer of the image pyramid, a An image having feature data as pixel values (hereinafter also referred to as a "first layer feature image") is obtained. Similarly, extraction of image features for the second layer image results in a second layer feature image.

Next, in step S205, the template matching unit 314 uses the template data for each viewpoint supplied from the template information acquisition unit 313 and each feature amount calculated by the feature extraction unit 312 corresponding to each template data. Do template matching.
Here, the template matching unit 314 first performs matching processing using the first layer feature image and the template for each viewpoint for the first layer. When the template matching unit 314 detects a template that is a correct candidate as a result of the matching process using the template for each viewpoint for the first layer, the template matching unit 314 sets the search range for the second layer feature image based on the detection result. Matching processing is performed using the 2nd layer feature image and the template for each viewpoint for the 2nd layer. If a third layer image and a fourth layer image exist, similar processing is performed on these layer images. As a result of such processing, the existence position and orientation of the object in the lowest layer (the layer with the highest image resolution) can be recognized.

Next, in step S<b>206 , the template matching unit 314 outputs recognition information indicating the recognition result to the recognition result output unit 315 after recognizing the existence position, posture, etc. of the target object. A recognition result output unit 315 outputs the recognition information supplied from the template matching unit 314 to an external device, a liquid crystal panel, or the like. The recognition information is used, for example, for inspection/measurement of the target object, control of the picking robot 28, and the like.

(Advantages of this embodiment)
In the configuration and processing described above, the three-dimensional measurement data of the target object photographed from different viewpoints is not integrated in a three-dimensional space, but the three-dimensional measurement data of the target object photographed from different viewpoints is used as an image for the template creation viewpoint. The template data is generated by extracting the image features of the target object based on the data obtained by projecting onto the plane. In this way, since the integration process is performed on the image plane, the amount of memory used is small and the processing time can be shortened. Therefore, template creation processing can be speeded up.
In addition, among the three-dimensional measurement data of the target object photographed from different viewpoints, the photographed point group data of viewpoints close to the template creation viewpoint are preferentially used and integrated. In this way, the captured point cloud data of the viewpoints closest to the template creation viewpoint are sequentially projected onto the image plane, and only the pixels where no point cloud exists are supplemented with the captured point cloud data from other capturing viewpoints. Therefore, a template that is close to the real object can be created without losing the detailed shape and texture of the target object.

<Others>
The above-described embodiment is merely an example of the configuration of the present invention. The present invention is not limited to the specific forms described above, and various modifications are possible within the technical scope of the present invention.

In addition, the template creation viewpoint may be created by adopting the vertices of the regular polyhedron as the viewpoint as described above, or by an arbitrary method of sampling the celestial sphere centered at the center of the target object at equal intervals. .
The index of the distance between the template creation viewpoint and the shooting viewpoint is not limited to the angle between the template creation viewpoint vector and the shooting viewpoint vector, and the Euclidean distance or the geodesic distance between the template creation viewpoint and the shooting viewpoint may be used.
If the template creation viewpoint is far from the shooting viewpoint, for example, if the angle formed by the template creation viewpoint vector and the shooting viewpoint vector is greater than or equal to a predetermined threshold value, step S105 is performed for such a shooting viewpoint. may not be projected. Since it is possible to determine whether or not a point is visible from the template creation viewpoint based on the normal vector of the measurement point of the target object, the projection processing in step S105 may not be performed for points that are not visible from the template creation viewpoint. good.

In order to allow comparison between the constituent elements of the present invention and the configurations of the embodiments, the constituent elements of the present invention will be described below with reference numerals in the drawings.
<Invention 1>
A template creation device (30) for creating a template corresponding to a target object (27) viewed from a viewpoint,
a template creation viewpoint position and orientation data acquisition unit (302) for acquiring data relating to the position and orientation of the template creation viewpoint as the viewpoint for creating the template;
a three-dimensional measurement data acquisition unit (300) for acquiring three-dimensional measurement data of the target object (27);
a measurement viewpoint position/orientation data acquisition unit (301) for acquiring data relating to the position and orientation of a measurement viewpoint as a viewpoint for three-dimensionally measuring the target object (37);
The three-dimensional measurement data corresponding to the different measurement viewpoints are projected onto a predetermined plane with respect to the template creation viewpoint, and basic three-dimensional data for extracting a feature amount is generated based on the projected three-dimensional measurement data. A basic three-dimensional data generation unit (303) to
with
The basic three-dimensional data generation unit (303) selects the measurement viewpoint having a closer distance to the template creation viewpoint, and converts the three-dimensional measurement data corresponding to the selected measurement viewpoint into the basic three-dimensional data. A template creation device (30) characterized by being preferentially used for creating.

2: Object recognition device 21: Image processing device 27: Target object 30: Template creation device 31: Object recognition processing device 300: Multi-viewpoint 3D measurement data acquisition unit 301: Shooting viewpoint 3D position and orientation data acquisition unit 302: Template creation Viewpoint position information acquisition unit 303: Template creation Viewpoint three-dimensional data generation unit 310: Image acquisition unit 312: Feature extraction unit 314: Template matching unit

Claims (7)

A template creation device for creating a template corresponding to a target object viewed from a viewpoint,
a template creation viewpoint position/orientation data acquisition unit that acquires data relating to the position and orientation of a template creation viewpoint as the viewpoint for creating the template;
a three-dimensional measurement data acquisition unit that acquires three-dimensional measurement data of the target object;
a measurement viewpoint position/orientation data acquisition unit that acquires data relating to the position and orientation of a measurement viewpoint as a viewpoint for three-dimensionally measuring the target object;
The three-dimensional measurement data corresponding to the different measurement viewpoints are projected onto a predetermined plane with respect to the template creation viewpoint, and basic three-dimensional data for extracting a feature amount is generated based on the projected three-dimensional measurement data. a basic three-dimensional data generation unit to
a feature extraction unit that extracts the feature amount based on the basic three-dimensional data;
a template creation unit that creates the template for each of the template creation viewpoints based on the feature amount extracted for the template creation viewpoint;
with
The basic three-dimensional data generation unit selects the measurement viewpoint having a closer distance to the template creation viewpoint, and converts the three-dimensional measurement data corresponding to the selected measurement viewpoint to generate the basic three-dimensional data. A template creation device characterized in that it is used preferentially to The basic three-dimensional data generation unit converts a portion of the basic three-dimensional data projected onto the predetermined plane, which is not included in the three-dimensional measurement data with higher priority, into the three-dimensional measurement data as the predetermined 2. The template creation apparatus according to claim 1, wherein the data projected onto the plane of the template is supplemented. an image acquisition unit that acquires an image including the target object;
a feature extraction unit that extracts the feature amount from the acquired image;
A template matching unit that recognizes the target object included in the acquired image by template matching using a template created by the template creation device according to claim 1 or 2;
object recognition processing device. A template creation method for creating a template corresponding to a target object viewed from a viewpoint, comprising:
obtaining information about the position and orientation of a template creation viewpoint as the viewpoint for creating the template;
obtaining three-dimensional measurement data of the target object;
acquiring data on the position and orientation of a measurement viewpoint as a viewpoint for three-dimensionally measuring the target object;
selecting the measurement viewpoint having a closer distance to the template creation viewpoint;
a step of projecting the three-dimensional measurement data corresponding to the selected measurement viewpoint onto a predetermined plane with respect to the template creation viewpoint;
When generating basic three-dimensional data for extracting a feature amount based on the projected three-dimensional measurement data, giving priority to the three-dimensional measurement data corresponding to the measurement viewpoint closer to the distance. generating basic three-dimensional data using
a step of extracting the feature amount based on the basic three-dimensional data;
creating the template for each of the template creation viewpoints based on the feature amount extracted for the template creation viewpoint;
Template creation method including. An object recognition processing method for recognizing a target object using a template,
obtaining an image containing the target object;
extracting the feature quantity from the acquired image;
a step of recognizing the target object by template matching using a template created by the template creation method according to claim 4;
An object recognition processing method comprising: A program for causing a computer to execute each step of the template creation method according to claim 4. A program for causing a computer to execute each step of the object recognition processing method according to claim 5.

Images