JP6041710B2 - Image recognition method - Google Patents

Description

  The present invention relates to an image recognition method.

  When picking up workpieces stacked in bulk with a robot or the like, it is necessary to accurately detect the position and posture of the workpiece located at the top. Conventionally, in such a case, a workpiece to be picked is detected by performing pattern matching using a two-dimensional grayscale image from a CCD camera.

  For example, Patent Literature 1 describes a method for detecting the position and posture of workpieces stacked using pattern matching. In this method, first, an image obtained by picking up the stacked workpieces with a two-dimensional CCD camera is matched with the contour shape of the workpiece taken in advance from a plurality of directions, and the relative posture between the workpiece and the CCD camera is determined. Ask. Then, based on the obtained relative posture, the position and posture of the workpiece are measured by a three-dimensional visual sensor moved to a position and posture that can clearly capture the feature of the target workpiece. Thereby, the position and orientation of the workpiece can be accurately measured.

  Patent Document 2 describes that a photographed image of a workpiece is divided into icons (patches) and the relative positional relationship of each icon is tagged.

US Pat. No. 7,177,459 U.S. Pat. No. 8,150,165

  However, in a captured image in which a plurality of workpieces are overlapped in a complicated manner, a change in luminance in the image is random and features are ambiguous. Therefore, the contour shape cannot be extracted by the pattern matching method, and there are cases where the exact position and orientation of the workpiece cannot be specified.

  In view of the above, an object of the present invention is to provide an image recognition method capable of obtaining the position and posture of a workpiece in a complicatedly overlapped state with high reliability.

  The present invention is an image recognition method for recognizing the position and orientation of a workpiece from a captured image, the step of generating a polygon mesh from a three-dimensional shape model of the workpiece, and actually capturing the workpiece with an imaging means And a sample reference assumed image assumed to be obtained when the imaging means captures a three-dimensional shape model of the workpiece from the same direction as the direction captured by the imaging means. A step of associating, for each polygon mesh in the sample reference assumed image, a step of associating the brightness of the surface of the sample actual image associated with the polygon mesh, from the direction different from the direction actually captured by the imaging means, Sample concept assumed to be obtained when a three-dimensional shape model of a piece is imaged by the imaging means For each polygon mesh constituting each surface of the sample assumed image with respect to the image, the normal direction of the polygon mesh, and the normal direction and correspondence of the polygon mesh in the sample reference assumed image corresponding to the polygon mesh A step of obtaining a virtual luminance based on the luminance, a step of dividing the sample reference assumed image and the sample assumed image into patches to obtain a plurality of sample patch images, The step of classifying on the basis of the virtual luminance, and the workpiece when the image is taken or assumed to be taken with the position information indicating the reference position of the workpiece for each of the plurality of sample patch images A preparatory process consisting of a step of storing the posture information indicating the posture of the device in association with each other, and a plurality of the work items The plurality of patch images are captured in the same manner as the imaging step of capturing a piece with the imaging unit and dividing the captured image into patches to obtain a plurality of patch images, and the method of classifying the plurality of sample patch images. A classification process for classifying based on luminance, and an accumulation process for collecting, for each of the plurality of patch images, the position information and posture information of the workpiece stored corresponding to the sample patch image of the same classification as the patch image. And a workpiece estimation step of estimating a reference position and orientation of at least one workpiece in the patch image based on the accumulation result of the position information and the orientation information.

  According to the present invention, since the workpiece existing below is covered with another workpiece existing above the workpiece, the position of the position information and the posture information accumulated in the accumulation step is accumulated at the uppermost workpiece. Will grow. Therefore, it is estimated that a workpiece having a reference position at a position represented by large positional information and having a posture in which the accumulation is represented by large posture information is higher.

  As described above, according to the present invention, since the position information and the posture information corresponding to the sample patch images of the same classification as the patch images obtained by dividing the image obtained by imaging a plurality of workpieces are accumulated, the classification or association can be performed satisfactorily. Even if there is nothing, the position and orientation of the reference position of the workpiece having a small covering portion can be finally estimated with high reliability.

  Further, the virtual actual brightness of the sample reference assumed image is obtained by associating the sample actual image obtained by actually imaging the workpiece with the imaging means and the sample reference assumed image obtained from the three-dimensional shape model of the workpiece. After obtaining, a sample assumed image is obtained from the three-dimensional shape model of the workpiece. Therefore, it is not necessary to obtain a sample image by actually imaging the workpiece from a large number of directions by the imaging means, so that the preparation process is simplified.

  In the present invention, a covering portion estimation step of estimating the covering portion of the workpiece, which has been obtained at the reference position and orientation in the workpiece estimation step, based on the patch image accumulated in the position information and posture information of the workpiece. It is preferable that it is further included.

  In this case, when the accumulation is estimated to be low and a part of the workpiece is covered, the patch image accumulated in the position information and the posture information indicating the workpiece is reversely developed in the covering portion estimation step. Estimate the covering part. Thereby, it can be estimated whether or not the part picked up by the robot or the like is covered.

It is a figure which shows the structure of a sample data acquisition apparatus. (A) is a figure which shows the state which divided | segmented the sample image into the patch, (b) shows an example of a part of sample patch image database. It is a figure which shows the structure of the picking up system provided with the workpiece position and orientation recognition apparatus. It is explanatory drawing which shows the method of estimating the position of a workpiece based on the accumulation result of position information. It is a flowchart of an image recognition method.

  The sample data acquisition apparatus 10 used in the preparation process in the embodiment of the present invention will be described.

  As shown in FIG. 1, the sample data acquisition device 10 captures an image of a workpiece 11 installed on the installation table 11 from above, an installation table 11 on which the workpiece W can be installed in a known posture. An imaging device 12 that controls the position of the installation base 11 or the imaging device 12.

  The installation base 11 is, for example, a rotary stage, and is configured to be controlled by the control unit 13. Here, the imaging device 12 is a single CCD camera, and generates data of a two-dimensional captured image. The imaging device 12 is supported above a support column or ceiling (not shown) and disposed above the installation table 11.

  The control unit 13 controls the posture of the installation table 11 or the imaging device 12, and turns the workpiece W installed on the installation table 11 as necessary to cause the imaging device 12 to image the workpiece W.

  The sample data acquisition device 10 further includes a sample actual image generation unit 21, a polygon mesh generation unit 22, a sample reference assumption image generation unit 23, a sample image association unit 24, a sample assumption image generation unit 25, an image division unit 26, and , Provided with a tag assigning unit 27 and connected to the storage device 14.

  The sample actual image generating unit 21 obtains sample actual image data by causing the imaging device 12 to capture an image of the workpiece W from a specific direction set in advance. The specific directions are, for example, six directions of up, down, right, left, front, and back. The obtained sample actual image data is stored in the storage device 14 in association with the posture information indicating the posture of the workpiece W at the time of imaging.

  As shown in FIG. 2A, one reference position O and one reference posture (reference direction) are determined in advance on the workpiece W. The reference position O may be set, for example, at the center of gravity of the workpiece W, the center of the hole formed in the workpiece W, or the like. The posture information associated with each sample actual image data is information indicating the direction deviation of the posture of the workpiece W when the sample actual image is captured with reference to the reference posture. In addition, in order to obtain posture information, you may image with the imaging device 12 with the checker pattern which positioned the workpiece W in the vicinity.

  The polygon mesh generation unit 22 generates a polygon mesh from 3D CAD data indicating a 3D shape model of the workpiece W. The three-dimensional CAD data is, for example, design data of the workpiece W, and is stored in advance in the storage device 14. The method for generating the polygon mesh from the three-dimensional CAD data is not particularly limited, and may be performed by a known method, and commercially available software may be used.

  The sample reference assumed image generation unit 23 is assumed to be obtained when the three-dimensional shape model of the workpiece W is imaged by the imaging device 12 and the workpiece W is imaged from the same direction as the actual imaging direction by the imaging device 12. Sample reference assumed image data is generated.

  Specifically, the sample reference assumption image generation unit 23 generates a 3D image of the workpiece W based on the normal direction of the face (face) indicated by the polygon mesh for each polygon mesh generated by the polygon mesh generation unit 22. The shading, outline, etc. obtained when the shape model is imaged by the imaging device 12 are obtained, and sample reference assumed image data is generated. The sample reference assumption image data generated by the sample reference assumption image generation unit 23 is stored in the storage device 14 in association with the posture information indicating the posture of the workpiece W when imaging is assumed.

  The sample image association unit 24 associates sample actual image data associated with the same posture information and sample reference assumed image data. Specifically, the sample image association unit 24 extracts a contour line or the like from the sample actual image and the sample reference assumed image, and performs association by correlating the extracted contour line or the like.

  The sample image association unit 24 associates, for each polygon mesh in the sample reference assumed image, the luminance on the surface of the sample actual image corresponding to the surface indicated by the polygon mesh.

  The sample assumed image generation unit 25 is assumed to be obtained when the three-dimensional shape model of the workpiece W is captured by the imaging device 12 and the workpiece W is captured from a direction different from the direction in which the workpiece W is actually captured. Sample assumed image data is generated.

  Specifically, the sample assumed image generation unit 25 generates a workpiece from a three-dimensional shape model for each polygon mesh generated by the polygon mesh generation unit 22 based on the normal direction of the face (face) indicated by the polygon mesh. A shade, an outline, and the like obtained when W is imaged by the imaging device 12 are obtained, and sample assumed image data is generated. The sample assumed image data generated by the sample assumed image generation unit 25 is stored in the storage device 14 in association with the posture information indicating the posture of the workpiece W when imaging is assumed.

  The image dividing unit 26 performs processing for dividing each sample reference assumed image and sample assumed image into patches (small regions) to obtain a plurality of sample patch images. The patch is, for example, a rectangular or square area corresponding to a predetermined number of pixels × the number of pixels.

  The tag assigning unit 27 assigns the positional relationship information of the workpiece W, posture information, and reliability information regarding the positional relationship information and posture information to each sample patch image divided by the image dividing unit 26 as a tag. Specifically, the positional relationship information is relative coordinate information (for example, XYZ axis coordinates) with respect to the reference position O of the workpiece W of the part related to the sample patch image. The posture information is relative angle information with respect to the reference angle of the workpiece W of the part related to the sample patch image, as in the positional relationship information. For example, the rotation angle around three axes of Rx, Ry, and Rz, or 3 × 3 It is a rotation matrix.

  Depending on the sample patch image, the relationship to the reference position O of the workpiece W may or may not be uniquely determined, and therefore the tag assigning unit 27 assigns a weighting coefficient as reliability information.

  For example, referring to FIG. 2A, in the sample patch image obtained by imaging a part of the side portion of the workpiece W, it is not clear which region of the side portion is specifically captured. In such a case, since the reliability is low, the weighting coefficient is lowered as much. More specifically, when it is known that a certain sample patch image is obtained by imaging one of the two parts of the workpiece W from a specific direction, the reliability is 50%. The weighting coefficient given to is assumed to be 0.5. On the other hand, in the sample patch image obtained by imaging a characteristic part such as a corner of the workpiece W from a specific direction, the reference position O and the posture are uniquely determined, so the reliability is high, and the weighting coefficient is 1. 0.

  Further, the tag assigning unit 27 sorts the sample patch image according to whether or not it satisfies a conditional expression for sequentially comparing the luminance of each pixel (pixel) constituting the sample patch image with a predetermined threshold value. Classify by law. Then, the tag assigning unit 27 labels each sample patch image with a number according to the classification.

  Specifically, the tag assigning unit 27, for example, “0” when the luminance of each pixel is equal to or higher than a threshold value in a predetermined order starting from the upper left pixel, and “1” when the luminance is lower than the threshold value. ". As a result, several sequences such as “100101...” Are obtained for each sample patch image. Then, the sample patch images are rearranged in this numerical sequence, and a number is assigned to each sample patch image.

  Further, the brightness of a pixel at a specific address is compared with the brightness of a pixel at another specific address, and when the brightness of a specific address pixel is equal to or higher than the brightness of a pixel at another specific address, “0”. If the number is less than “1”, the pixels to be compared may be appropriately selected sequentially to obtain a number sequence such as “001101...”, And this number sequence may be added to the number sequence described in the previous paragraph. In this way, the question for obtaining the numerical sequence may be any answer as long as it can be answered correctly or not, and the answer can be classified into binary values of 0 and 1.

  By sorting using the binary tree method in this way, a huge number of patch sample image classifications can be reduced to simple binary classification repetitions. Therefore, if an electronic computing unit is used, it becomes possible to classify in a short time. However, the classification is not limited to using the binary tree method, and classification may be performed by any known image classification method such as extracting feature parameters from a sample patch image.

  Then, as shown in FIG. 2B, the tag assigning unit 27 associates the positional relationship information, the posture information, the reliability information, and the numerical sequence in the order of the assigned numbers, and assigns each sample patch image to the sample patch image database. Is stored in the storage device 14.

  Next, the picking up system 40 provided with the workpiece position and orientation recognition device 30 used in the image recognition method according to the embodiment of the present invention will be described.

  As shown in FIG. 3, the workpiece position / posture recognition device 30 is a device that obtains the reference position, posture, and distance of the workpieces W stacked in a hopper (storage tank) 41. Here, the workpieces W have the same shape, and a large number of workpieces W are stacked in an arbitrary position and posture three-dimensionally in the hopper 41.

  The workpiece position / posture recognition apparatus 30 determines a workpiece W that is located near the top of the stacked workpieces W and is suitable for being picked up. Note that the workpiece position / posture recognition device 30 is not limited to the case where the workpieces W are randomly stacked, but can be similarly applied to the case where the workpieces W are aligned and arranged.

  The picking-up system 40 includes a robot 42 that picks up the workpieces W one by one in accordance with the reference position O, posture, and distance of the workpiece W supplied from the workpiece position / posture recognition device 30. The robot 42 includes an arm 43, a hand 44 that is an end effector of the arm 43, and a robot control unit 45 that controls the operation of the arm 43 and the hand 44.

  The robot control unit 45 converts the reference position information, posture information, and distance information of the workpiece W to be picked up supplied from the workpiece position / posture recognition device 30 into a robot coordinate system, and uses the workpiece W as a hand. The arm 43 and the hand 44 are controlled so as to be picked up at 44.

  The picking-up system 40 further includes one or a plurality of imaging devices 46 that image the workpieces W stacked in the hopper 41 from above. The imaging device 46 is, for example, two CCD cameras, and generates two-dimensional captured images from different angles. The two image pickup devices 46 are supported by a post or a ceiling (not shown), and are spaced apart and arranged above the hopper 41. The two imaging devices 46 image the piled workpieces W in an overlapping manner.

  The workpiece position and orientation recognition device 30 includes an image processing unit 31, an image dividing unit 32, a corresponding sample patch image classification unit 33, a stacking unit 34, a workpiece estimation unit 35, a covering unit estimation unit 36, an actual workpiece determination unit 37, And a pickup determining unit 38, which is connected to the storage device 14.

  The image processing unit 31 takes in images taken by the two imaging devices 46 and stores the image data subjected to predetermined processing in the storage device 14.

  The image dividing unit 32 divides the image processed by the image processing unit 31 into a plurality of patches (small regions), and performs processing for obtaining a plurality of patch images. This patch is an area having the same number of pixels × number of pixels as the patch into which the sample image is divided by the image dividing unit 26 of the sample data acquisition apparatus 10. Here, the scale of the patch divided by the image dividing unit 32 is adjusted to be substantially the same as the scale of the patch divided by the image dividing unit 26 of the sample data acquisition apparatus 10.

  The corresponding sample patch image classifying unit 33 stores which sample image stored in the storage device 14 indicates which of the sample patch images divided by the image dividing unit 26 is most similar to each of the patch images divided by the image dividing unit 32. Search from the patch image database.

  Here, the corresponding sample patch image classification unit 33 obtains a number sequence corresponding to each patch image divided by the image dividing unit 32 in the same manner as the number sequence obtained by the tag assignment unit 27 described above. The corresponding sample patch image classification unit 33 searches for a sample patch image whose numerical sequence matches the numerical sequence corresponding to each patch image.

  The corresponding sample patch image classifying unit 33 obtains a number sequence by the binary tree method in the same manner as the tag assigning unit 27, and compares the obtained number sequence with the number sequence given by the tag assigning unit 27. It becomes possible to search.

  Then, the corresponding sample patch image classification unit 33 selects the sample patch image in which the number sequence and the number sequence corresponding to each patch image all match, or the sample patch image in which the number sequence equal to or more than the threshold number matches, An image. There are cases where two workpieces W overlap and there is a branch in the patch image. In such a case, there is no sample patch image whose number sequence is identical or similar.

  The corresponding sample patch image classification unit 33 classifies the sample patch image most similar to the patch image as the sample patch image corresponding to the patch image.

  The accumulating unit 34 accumulates where the point indicated as the most reference position by each patch image is. Specifically, the weighting coefficient is accumulated for each reference position information labeled in the sample patch area searched by the corresponding sample patch image classification unit 33.

  Specifically, the accumulating unit 34 corresponds to the sample patch images searched by the corresponding sample patch image classifying unit 33 respectively corresponding to all patch images obtained by dividing the images captured by the two imaging devices 46. For each of the stored reference position information, a total value of weighting coefficients (hereinafter, this total value is referred to as “score”) is calculated.

  As shown in FIG. 4, the workpiece estimation unit 35 estimates the reference position O of the workpiece W based on the scores accumulated by the accumulation unit 34. The workpiece W that is relatively below the hopper 41 is partially or entirely hidden by the upper workpiece W. Therefore, the reference position O of the workpiece W existing below cannot obtain a high score, and the work piece W located at the uppermost part can obtain a score with a high reference position O.

  Therefore, it can be estimated that the workpiece W in the posture indicated by the posture information at the reference position O indicated by the reference position information obtained with a high score is near the top and exposed. Therefore, when the score calculated by the accumulating unit 34 exceeds a predetermined threshold value, the workpiece estimation unit 35 determines the posture indicated by the posture information at the reference position O indicated by the reference position information that acquired the score (according to these). The workpiece W estimated to be in the “estimated position / posture” is selected. The threshold value is set so that, for example, the selected workpieces W include those that are all exposed and those that are not partially exposed.

  FIG. 4 conceptually shows a state in which scores are accumulated only from the reference position information that is the total value of the weighting coefficients in order to facilitate visual understanding. Thus, by ranking the scores in the screen by height, it is possible to narrow down the search to only the number of parts to be detected, thereby enabling faster detection. For example, when it is desired to detect three, it is only necessary to calculate the estimated position and orientation from the top three with the highest score accumulation.

  By the way, when the score accumulated in the accumulation unit 34 is higher than the threshold value but low to some extent, it is considered that at least a part of the workpiece W selected by obtaining the score is covered with another workpiece W or the like. . When the covering portion (non-exposed portion) of the selected workpiece W is the pickup portion (gripping portion) of the robot 42, the workpiece W cannot be picked up by the robot 42. Therefore, it is desirable to specify the covering portion of the workpiece W.

  Therefore, the covering portion estimation unit 36 uses the sample patch image database stored in the storage device 14 again to form the shape of the workpiece W when the workpiece W is in the estimated position and posture (consisting of a patch image at any location). The area obtained by subtracting a set of patch images in which weighting coefficients are actually accumulated is estimated as a covering portion.

  For example, if the posture estimated by the workpiece estimation unit 35 is a position / posture composed of sample patches A, B, C, D, and E, A and B are also detected in the workpiece W photographed by the imaging device 46. , C, D, and E are expected to accumulate reference position information from patch images. However, for example, from the area of the patch image corresponding to the B sample patch, if the weighting coefficient is not accumulated using the reference position O of the workpiece W as the reference position information, it is estimated that the B area is covered.

  Since the postures of the workpieces W actually piled up are arbitrary, there are postures that do not coincide with the postures in which the sample images are captured. That is, since the posture indicated by the posture information of the selected workpiece W is the most similar posture selected from the discrete database, it may be slightly different from the actual posture of the workpiece W. Therefore, the estimated position / posture of the workpiece W selected by the workpiece estimation unit 35 is different from the actual position / posture, which may affect the pickup by the robot 42.

  Therefore, the actual workpiece determination unit 37 uses the actual reference position O and posture (collectively referred to as “actual position and posture”) of the workpiece W selected by the workpiece estimation unit 35 as the actual workpiece W. The image captured by the two imaging devices 46 is obtained by performing stereo image processing.

  Specifically, the actual workpiece determination unit 37 selects a workpiece W that approximates the position and orientation of the reference position O to the estimated position and orientation from the images captured by the two imaging devices 46, and performs differential processing on the image. The edge is extracted by a known method such as applying a contour, and the contour line of the entire workpiece W is obtained.

  At this time, since the position and orientation of the reference position O of the workpiece W to be subjected to image processing are estimated in advance, the corresponding points between the images captured by the two imaging devices 46 can be associated without erroneous recognition. The amount of image processing required for searching for corresponding points is also small.

  Further, the actual workpiece determining unit 37 obtains the distance to the workpiece W (vertical direction position in FIG. 3). The distance Z from the imaging device 46 to the corresponding point (reference position O of the workpiece W) is the distance L between the two imaging devices 46, the focal length f of the imaging device 46, and the corresponding points between the two imaging devices 46. Can be obtained by the following equation (1).

Z = L × f / d (1)
As a result, the workpiece W selected by the robot 42 can be picked up.

  The actual workpiece determination unit 37 may specify the covering portion by obtaining the contour line of the workpiece W from the image obtained by capturing the actual workpiece W with the two imaging devices 46. This image processing is wasted if it is found to be a part. Therefore, the actual workpiece determination unit 37 only needs to obtain the actual position and orientation with respect to the workpiece W that is estimated by the covering unit estimation unit 36 that the covering unit is not a pickup part.

  The pickup determination unit 38 determines a workpiece W suitable for removal by the robot 42 based on the actual position / posture and distance of the actual workpiece W obtained by the actual workpiece determination unit 37. Then, information indicating the determined actual position / posture and distance of the workpiece W is transmitted to the robot controller 45. The method for determining the workpiece W to be taken out is not particularly limited. For example, a workpiece W that can be easily controlled by the robot 42 when picking up or a workpiece W that does not need to be turned over after picking up may be selected.

  Hereinafter, an image recognition method according to the embodiment of the present invention will be described with reference to FIG.

  First, a preparation process is performed using the sample data acquisition apparatus 10 described above (S10).

  In this preparation process, one workpiece W is first installed on the installation table 11. And while rotating the installation stand 11 by the control part 13, the workpiece W is imaged with the imaging device 12 from the preset specific direction, and the sample real image generation part 21 obtains sample real image data ( S11). The sample actual image data is stored in the storage device 14 in association with the posture information of the workpiece W.

  Next, the polygon mesh generation unit 22 generates a polygon mesh from the three-dimensional CAD data indicating the three-dimensional shape model of the workpiece W (S12).

  Then, it is assumed that the sample reference assumed image generation unit 23 is obtained when the three-dimensional shape model of the workpiece W is captured by the imaging device 12 and the workpiece W is captured from the same direction as the direction in which the workpiece W is actually captured. Sample reference assumed image data to be generated is generated (S13).

  Next, the sample image association unit 24 associates the sample actual image associated with the same posture information with the sample reference assumed image (S14).

  Then, for each polygon mesh in the sample reference assumed image, the sample image association unit 24 associates the luminance on the surface of the sample actual image associated in S14 with the surface indicated by the polygon mesh (S15).

  Next, the sample assumed image generation unit 25 is obtained when the three-dimensional shape model of the workpiece W is captured by the imaging device 12 and the workpiece W is captured from a direction different from the direction actually captured by the imaging device 12. Assumed sample assumed image data is generated (S16).

  Next, the image dividing unit 26 divides each sample reference assumed image and sample assumed image into patches to obtain a plurality of sample patch images (S17).

  Next, the tag assigning unit 27 assigns the positional relationship information, posture information, and reliability information regarding the positional relationship information and posture information to each sample patch image obtained in S16 as a tag, A number sequence is obtained from the conditional expressions described above and numbered to perform labeling. Further, the tag assigning unit 27 associates the positional relationship information, the posture information, the reliability information, and the numerical sequence in the order of the assigned numbers, and stores each sample patch image data in the storage device 14 as a sample patch image database (S18). ).

  Then, after the preparation step (S10) is completed, an image recognition step (S20) of the workpiece W is performed using the above-described pick-up system 40.

  In this image recognition process, first, the workpieces W stacked at random from above the hopper 41 are imaged by the two imaging devices 46 (S21). This S21 corresponds to the imaging step of the present invention.

  The image processing unit 31 stores the image data captured by the two imaging devices 46 in the storage device 14 (S22).

  Next, the image dividing unit 32 divides the image processed in S22 into a plurality of patches to obtain a plurality of patch images (S23).

  Next, the corresponding sample patch image classifying unit 33 determines from the sample patch image database stored in the storage device 14 which of the sample patch images divided in S13 is most similar to each of the patch images divided in S23. Explore. Each patch image is classified based on the search result (S24). This S24 corresponds to the classification step of the present invention.

  Next, the accumulating unit 34 obtains a score obtained by accumulating the sum of the weighting coefficients for each reference position information and posture information labeled in the sample patch area searched by the corresponding sample patch image classifying unit 33 (S25). This S25 corresponds to the integration process of the present invention.

  Next, the workpiece estimation unit 35 selects a workpiece W for which the score accumulated in S25 exceeds the threshold, and estimates the reference position O and posture of the selected workpiece W (S26). This S26 corresponds to the workpiece estimation step of the present invention.

  Next, the covering portion estimation unit 36 estimates the covering portion of the workpiece W having a low score among the workpieces W selected in S26 (S27). This S27 corresponds to the covering portion estimation step of the present invention.

  Next, the actual workpiece determination unit 37 obtains the actual reference position O and posture of the workpiece W selected in S26, except for the workpiece W in which the covering portion is estimated to be a pickup part in S27 (S28). . Further, the actual workpiece determining unit 37 obtains the actual distance of the workpiece W (S29).

  Then, the pickup determining unit 38 determines a workpiece W suitable for removal by the robot 42 from the actual reference position O, posture, and distance of the workpiece W obtained in S28 and S29 (S30).

  And finally, the pick-up process (S31) which picks up the workpiece W determined by S30 using the robot 42 is performed.

  Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and can be appropriately modified and applied without departing from the gist of the present invention.

  For example, in the present embodiment, the workpiece position / posture recognition apparatus 30 includes the covering portion estimation unit 36, and the image recognition method has been described as including the step of estimating the covering portion of the workpiece W (S27). However, you may not provide the process which estimates the coating | coated part estimation part 36 and a coating | coated part.

  DESCRIPTION OF SYMBOLS 10 ... Sample data acquisition apparatus, 11 ... Installation stand, 12 ... Imaging device (imaging means), 13 ... Control part, 14 ... Memory | storage device, 21 ... Sample real image generation part, 22 ... Polygon mesh generation part, 23 ... Sample reference | standard Assumed image generating unit, 24 ... Sample image associating unit, 25 ... Sample assumed image generating unit, 26 ... Image dividing unit, 27 ... Tag giving unit, 30 ... Workpiece position / posture recognition device, 31 ... Image processing unit, 32 ... Image segmentation unit 33 ... Corresponding sample patch image classification unit 34 ... Accumulation unit 35 ... Workpiece estimation unit 36 ... Cover part estimation unit 37 ... Actual workpiece determination unit 38 ... Pickup determination unit 40 ... Picking up -System, 41 ... Hopper, 42 ... Robot, 43 ... Arm, 44 ... Hand, 46 ... Imaging device (imaging means), W ... Workpiece.

Claims (2)

An image recognition method for recognizing the position and orientation of a workpiece from a captured image,
Generating a polygon mesh from a three-dimensional shape model of the workpiece;
Obtained when a sample actual image obtained by actually imaging the workpiece by the imaging means and a three-dimensional shape model of the workpiece taken by the imaging means from the same direction as the image taken by the imaging means A process of associating the assumed sample reference assumed image with
Correlating the brightness of the surface of the sample actual image associated with the polygon mesh for each polygon mesh in the sample reference assumed image;
With respect to the sample assumed image assumed to be obtained when the three-dimensional shape model of the workpiece is imaged by the imaging unit from a direction different from the direction actually captured by the imaging unit, For each polygon mesh constituting the surface, based on the normal direction of the polygon mesh, the normal direction of the polygon mesh in the sample reference assumed image corresponding to the polygon mesh, and the corresponding luminance, the virtual luminance The process of seeking
Dividing the sample reference assumed image and the sample assumed image into patches to obtain a plurality of sample patch images;
Classifying the plurality of sample patch images based on the luminance or the virtual luminance; and
For each of the plurality of sample patch images, the position information indicating the reference position of the workpiece and the posture information indicating the posture of the workpiece when captured or assumed to be captured are stored in association with each other. A preparation process comprising:
An imaging step of capturing a plurality of workpieces with the imaging means, and dividing the captured images into patches to obtain a plurality of patch images;
A classification step of classifying the plurality of patch images based on luminance in the same method as the method of classifying the plurality of sample patch images;
An accumulation step of accumulating position information and posture information of the workpiece stored for each of the plurality of patch images corresponding to the sample patch images of the same classification as the patch images;
An image recognition method comprising: a workpiece estimation step of estimating a reference position and orientation of at least one workpiece in the patch image based on the accumulation result of the position information and the orientation information.   And further including a covering portion estimation step of estimating the covering portion of the workpiece for which the reference position and orientation are obtained in the workpiece estimation step based on the patch image accumulated in the position information and posture information of the workpiece. The image recognition method according to claim 1.