JP7502641B2 - Recognition System - Google Patents

Description

The disclosure in this specification relates to a recognition system.

Patent Document 1 discloses a three-dimensional measuring device that calculates the position and orientation of a measurement object based on the distance from an imaging means to the measurement object and a three-dimensional shape model. The contents of the prior art documents are incorporated by reference as explanations of the technical elements in this specification.

JP 2011-27724 A

The reflection of light is not stable at the edge of the object to be measured. Therefore, when the entire object to be measured is irradiated with light to measure the three-dimensional shape, the brightness around the edge of the object to be measured varies, and the measured data is likely to have large errors. In the configuration of the prior art document, the illumination pattern is irradiated only to the area for distance measurement determined by mask processing, rather than the entire screen, and three-dimensional information is calculated. It is also disclosed that the area for distance measurement is an area in the captured image where the change in shading is small and the edge is difficult to detect. However, since the illumination pattern needs to be irradiated locally, the hardware configuration of the three-dimensional measuring device is likely to become complicated. In the above-mentioned viewpoints, or in other viewpoints not mentioned, further improvements are required in the recognition system.

One disclosed objective is to provide a recognition system capable of highly accurate position and orientation recognition.

The first recognition system disclosed herein is a recognition system that performs position and orientation recognition for measuring a shape of a measurement object (41, 241) and recognizing the position and orientation of the measurement object,
A measurement unit (20) that captures a surface image of a measurement object and calculates a three-dimensional point cloud corresponding to the surface image;
a model plane data acquisition unit (72) for acquiring model plane data obtained by extracting data of a plane portion from model data indicating a three-dimensional shape of a measurement object;
a measurement data acquisition unit (73) that acquires measurement data including a surface image captured by the measurement unit and a three-dimensional point cloud calculated by the measurement unit;
a measurement plane data extraction unit (74) that extracts three-dimensional coordinates in a plane portion from the three-dimensional point cloud as measurement plane data based on the surface image acquired by the measurement data acquisition unit;
a matching unit (79) for matching the model plane data with the measurement plane data to recognize the position and orientation of the measurement object ;
a feature point extraction unit (75) for extracting edge portions as feature points from the surface image;
an offset position setting unit (77) that sets an offset position that is a position offset in a direction away from the edge portion ,
The measurement plane data extraction unit extracts, from the three-dimensional point cloud, three-dimensional coordinates of a position corresponding to the offset position as measurement plane data .
The second recognition system disclosed herein is a recognition system that performs position and orientation recognition for measuring a shape of a measurement object (41, 241) and recognizing the position and orientation of the measurement object,
A measurement unit (20) that captures a surface image of a measurement object and calculates a three-dimensional point cloud corresponding to the surface image;
a model plane data acquisition unit (72) for acquiring model plane data obtained by extracting data of a plane portion from model data indicating a three-dimensional shape of a measurement object;
a measurement data acquisition unit (73) that acquires measurement data including a surface image captured by the measurement unit and a three-dimensional point cloud calculated by the measurement unit;
a measurement plane data extraction unit (74) that extracts three-dimensional coordinates in a plane portion from the three-dimensional point cloud as measurement plane data based on the surface image acquired by the measurement data acquisition unit;
a matching unit (79) for matching the model plane data with the measurement plane data to recognize the position and orientation of the measurement object;
a feature point extraction unit (75) for extracting edge portions as feature points from the surface image;
a mask range setting unit (278) for extracting a surrounding area including an edge portion as a mask range;
The measurement plane data extraction unit extracts, as measurement plane data, three-dimensional coordinates of positions excluding the mask range from the three-dimensional point cloud.

The disclosed recognition system includes a measurement plane data extraction unit that extracts three-dimensional coordinates of the planar portion of the measurement object from the three-dimensional point cloud as measurement plane data based on the surface image acquired by the measurement data acquisition unit, and a matching unit that matches the model plane data with the measurement plane data to recognize the position and orientation of the measurement object. Therefore, the three-dimensional coordinates of the planar portion, which has a higher measurement accuracy than the edge portion, can be used for matching. Therefore, a recognition system capable of highly accurate position and orientation recognition can be provided.

The various aspects disclosed in this specification employ different technical means to achieve their respective objectives. The claims and the reference characters in parentheses in this section are illustrative of the corresponding relationships with the embodiments described below, and are not intended to limit the technical scope. The objectives, features, and advantages disclosed in this specification will become clearer with reference to the detailed description that follows and the accompanying drawings.

FIG. 1 is a configuration diagram showing a schematic configuration of a recognition system and a robot. FIG. 2 is a block diagram relating to control of the recognition system. 11 is a flowchart relating to control of the recognition system. FIG. 11 is an explanatory diagram for explaining a method for setting an offset position. FIG. 13 is a configuration diagram showing a schematic configuration of a recognition system and a robot according to a second embodiment. FIG. 11 is a block diagram relating to control of a recognition system in a second embodiment. 10 is a flowchart relating to control of a recognition system in a second embodiment. FIG. 11 is an explanatory diagram for explaining a method for setting a mask range. FIG. 11 is an explanatory diagram for explaining a method of extracting measurement plane data. FIG. 13 is a block diagram relating to control of the recognition system in the third embodiment. 13 is a flowchart relating to control of a recognition system in a third embodiment.

Several embodiments will be described with reference to the drawings. In several embodiments, functionally and/or structurally corresponding and/or associated parts may be given the same reference symbol or reference symbols differing in the hundredth or higher digit. For corresponding and/or associated parts, the descriptions of other embodiments may be referred to. In the following, the three mutually orthogonal directions are referred to as the X direction, Y direction, and Z direction. In addition, the Z direction may be described as corresponding to the up and down direction.

1, the recognition system 1 can be used for various tasks using a robot 10. Examples of tasks using the robot 10 include an assembly task for assembling a part in a specific location, a picking task for picking a part, an appearance inspection for inspecting whether the part is scratched or chipped, and a gripping task for gripping a part. However, the recognition system 1 may also be used for tasks that do not use the robot 10.

The robot 10 comprises a base 11, a robot arm 15, and a robot hand 16. The robot 10 is a vertical articulated robot. However, a horizontal articulated robot or the like may also be used as the robot 10. The base 11 is fixed to the installation surface on which the robot 10 is installed using bolts or the like.

The robot arm 15 is composed of multiple shaft parts connected by joint parts. The joint parts include motors, and the angles of the joint parts can be freely controlled by controlling the motors. The multiple shaft parts are connected to each other via the joint parts so that they can rotate relative to each other. The robot arm 15 is connected to the base 11.

The robot hand 16 is attached to the tip of the robot arm 15. The robot hand 16 includes a motor. The robot hand 16 is configured to be able to rotate relatively around the axis of the shaft part provided on the robot arm 15 by controlling the motor.

The robot hand 16 has a claw portion. This claw portion rotates around the rotation axis of the robot hand 16. The robot hand 16 is configured to be able to grasp a measurement object 41 placed on the work table 31 by performing an opening and closing operation to expand and reduce the distance between the claw portions. The robot 10 can also move the robot arm 15 and the robot hand 16 while the robot hand 16 is grasping the measurement object 41. This allows the position and orientation of the measurement object 41 to be changed. However, in order to grasp the measurement object 41, it is necessary to accurately recognize the position and orientation of the measurement object 41. The position and orientation of the measurement object 41 refer to the position and orientation of the measurement object 41. The position and orientation of the measurement object 41 are recognized using the shape measurement results using the measurement unit 20 described later.

In the following, an example will be described in which the recognition system 1 is applied to a task of recognizing the position and orientation of a measurement object 41 placed on a work table 31. The work table 31 is a rectangular plate member. However, the type and shape of the work table 31 are not limited to the above example. A box-shaped parts box may be used as the work table 31. A movable device such as a belt conveyor may also be used as the work table 31. The number of measurement objects 41 placed on the work table 31 is not limited to one. For example, multiple measurement objects 41 may be placed on the work table 31 in a state where they overlap each other. The measurement object 41 is located at a standard position set approximately in the center of the work table 31. The standard position is a standard position where the measurement object 41 is placed. The standard position is set in advance in the recognition system 1.

The measurement object 41 can be an object of any shape, but as an example, an object having a shape of two disks of different diameters arranged side by side on the same axis is used. The measurement object 41 is placed on the work table 31 so that the disk part with the smaller diameter is located on the upper side. By recognizing the position and orientation of the measurement object 41, the robot 10 can lift and move the measurement object 41 from the work table 31 as necessary.

The measurement unit 20 is a system for measuring the shape of the measurement object 41. The measurement unit 20 has a function of capturing a surface image of the measurement object 41 and a function of calculating a three-dimensional point cloud of the measurement object 41. The three-dimensional point cloud is data obtained by calculating the three-dimensional coordinates of a plurality of measurement points of the measurement object 41. In other words, the measurement points are each point that expresses the coordinates of the measurement object 41. The three-dimensional coordinates have X coordinates, Y coordinates, and Z coordinates as coordinate components. An example of the three-dimensional point cloud is the brightness data of each measurement point. For example, a distance image sensor can be used as the measurement unit 20. The measurement unit 20 is aligned with the measurement object 41 in the Z direction. In other words, the alignment direction of the measurement unit 20 and the measurement object 41 is along the Z direction, and can also be said to be along the up-down direction.

The measurement unit 20 captures a surface image of the upper surface of the measurement object 41. The amount of light reflected from the surface of the measurement object 41 varies depending on the shape of the measurement object 41. Therefore, the brightness in the surface image varies depending on the shape of the measurement object 41. More specifically, in the edge portion, the brightness in the surface image varies more significantly than the surrounding brightness. In addition, in the flat portion, the brightness in the surface image varies very little compared to the surrounding brightness. An example of an edge portion is the outer edge portion of the measurement object 41. An example of an edge portion is a step portion such as a rib or an opening. The surface image measured by the measurement unit 20 is data that can be used to estimate the edge portion and the flat portion. In estimating the edge portion and the flat portion, the magnitude of the change in brightness in the surface image is used. However, it is also possible to estimate only the edge portion without estimating the flat portion from the surface image.

The measurement unit 20 calculates a three-dimensional point cloud on the measurement object 41. The three-dimensional point cloud on the measurement object 41 can be obtained by a phase shift method, a method using a stereo camera, or the like. When the phase shift method is adopted, the measurement unit 20 is provided with a light projecting device that projects a pattern light onto the measurement object 41, and a light receiving device that receives the pattern light reflected by the measurement object 41. When the method using a stereo camera is adopted, the measurement unit 20 is provided with multiple cameras that capture images of the measurement object 41 from different positions. The measurement unit 20 calculates the three-dimensional coordinates of the entire top surface of the measurement object 41 to obtain a three-dimensional point cloud. Therefore, the three-dimensional coordinates are calculated for multiple measurement points including the edge portions and flat portions of the measurement object 41.

When the phase shift method is adopted as the measurement method of the measurement unit 20, the pattern light reflected from the surface of the measurement object 41 is received. Therefore, if the reflected pattern light cannot be received accurately, the three-dimensional coordinates cannot be calculated accurately, resulting in errors. One example of a case in which the reflected pattern light cannot be received accurately is when the projected pattern light is scattered by the edge of the measurement object 41. In other words, light is less likely to be scattered on the flat surface of the measurement object 41, and the three-dimensional coordinates can be calculated with high accuracy.

When a method using a stereo camera is adopted as the measurement method of the measurement unit 20, the surface of the measurement object 41 is simultaneously imaged by multiple cameras. Therefore, if the surface shape of the measurement object 41 is not easily reflected in the captured surface image, the three-dimensional coordinates cannot be calculated accurately, resulting in errors. One example of a case in which the surface shape of the measurement object 41 is not easily reflected in the surface image is when the edges of the surface are crushed and no unevenness can be confirmed. In other words, in flat parts of the measurement object 41 that are free of unevenness, the three-dimensional coordinates can be calculated with high precision.

In summary, whether the phase shift method or the stereo camera method is used, it is easier to calculate three-dimensional coordinates with high accuracy for flat areas compared to edge areas. In other words, the accuracy of three-dimensional shape measurement is likely to decrease when edge areas are included in the calculation of three-dimensional coordinates.

The surface image captured by the measurement unit 20 and the calculated three-dimensional coordinates are associated in a pixel-by-pixel coordinate system. Therefore, for a position estimated to be an edge portion from the surface image, the three-dimensional coordinates of the corresponding position can be selected and extracted.

In FIG. 2, a control unit 70 that controls the recognition system 1 is connected to a measurement unit 20. The control unit 70 controls the measurement unit 20 to measure the shape of the measurement object 41. The control unit 70 acquires the measurement data measured by the measurement unit 20.

The control unit 70 includes a model plane data acquisition unit 72, a measurement data acquisition unit 73, a measurement plane data extraction unit 74, and a matching unit 79.

The model plane data acquisition unit 72 acquires model plane data, which is data obtained by extracting only the planar portion from the model data of the measurement object 41. Here, the model data is data forming a three-dimensional design drawing of the measurement object 41. The model plane data acquisition unit 72 is not limited to a configuration in which it acquires model plane data extracted in advance from the model data outside the recognition system 1. For example, it may be configured to include a model data acquisition unit that acquires model data, and a model plane data extraction unit that extracts model plane data from the model data.

The measurement data acquisition unit 73 acquires the measurement data measured by the measurement unit 20. The acquired measurement data includes surface image data and three-dimensional point cloud data.

The measurement plane data extraction unit 74 extracts, from the three-dimensional point cloud, the three-dimensional coordinates of a portion that can be estimated as a planar portion away from the edge portion based on the edge portion determined from the surface image, as measurement plane data. The planar portion away from the edge portion can be estimated from the magnitude of the brightness change in the surface image. The method of extracting the measurement plane data will be described in detail later. The measurement plane data extraction unit 74 includes a feature point extraction unit 75, a center point extraction unit 76, and an offset position setting unit 77.

The feature point extraction unit 75 extracts feature points from the measurement data. Here, a feature point is a point that indicates a feature of the shape of the measurement object 41. A feature point is, for example, an end portion, i.e., an edge portion, of the measurement object 41.

The center point extraction unit 76 extracts the center point of the measurement object 41. The offset position setting unit 77 sets the offset position. Here, the offset position refers to a portion of the measurement object 41 that can be estimated as a flat portion. The offset position is set, for example, at a position away from the edge portion of the measurement object 41, i.e., an offset position. It is not necessary to set an offset position corresponding to all edge portions. For example, the offset position may be set for only a portion of the edge portion.

The matching unit 79 compares the model plane data with the measurement plane data to obtain the position and orientation when the model plane data approximately matches the measurement plane data. By performing matching, the matching unit 79 can recognize the position and orientation of the measurement object 41.

The flow of position and orientation recognition, which is the task of recognizing the position and orientation of the measurement object 41 using the recognition system 1, is described below. However, before position and orientation recognition begins, calibration of the measurement unit 20 of the recognition system 1 is completed. This makes it possible to calculate accurate measurement data of the measurement object 41. Calibration can be performed initially after the robot 10 is installed, but calibration can also be performed repeatedly at regular intervals as necessary.

In FIG. 3, when control related to position and orientation recognition is started, the model plane data acquisition unit 72 acquires model plane data in step S101. The model plane data is data obtained by extracting only the planar portions from the model data forming the three-dimensional design drawing of the measurement object 41. The model data includes the planar portions and edge portions of the measurement object 41. On the other hand, the model plane data includes only the planar portions of the measurement object 41, and does not include the edge portions. Therefore, the model plane data has a smaller data size than the model data. The model plane data does not need to include all the planar portions of the measurement object 41, but only needs to include some of the planar portions. The model plane data includes data on the planar portions that correspond to the measurement plane data described later.

The model plane data is extracted in advance from the model data outside the recognition system 1. Therefore, the recognition system 1 itself does not need to perform processing to extract the model plane data from the model data. This simplifies the processing related to position and orientation recognition in the recognition system 1. After obtaining the model plane data, proceed to step S111.

In step S111, the control unit 70 uses the measurement unit 20 to measure the shape of the measurement object 41. In shape measurement by the measurement unit 20, a surface image of the top surface of the measurement object 41 is captured. Furthermore, in shape measurement by the measurement unit 20, three-dimensional coordinates are measured for each measurement point on the top surface of the measurement object 41, and a three-dimensional point cloud is calculated. After measuring the measurement object 41, the process proceeds to step S112.

In step S112, the measurement data acquisition unit 73 acquires measurement data. The measurement data includes surface image data obtained by capturing an image of the top surface of the measurement object 41. The measurement data includes a three-dimensional point cloud on the top surface of the measurement object 41. Therefore, the measurement data acquisition unit 73 acquires the surface image and the three-dimensional point cloud as the measurement data. After acquiring the measurement data, the process proceeds to step S121.

In step S121, the feature point extraction unit 75 extracts edge portions as feature points from the surface image. Edge portions can be extracted from changes in luminance in the surface image. More specifically, portions where the luminance change is greater than the surrounding area are regarded as edge portions and are extracted. After the edge portions are extracted, the process proceeds to step S122.

In step S122, the center point extraction unit 76 extracts a center point from the surface image. The outline of the measurement object 41 is estimated from the outer periphery, which is part of the edge of the measurement object 41, and the center point can be extracted from that outline. Since the surface image is an image of the measurement object 41 captured from above, the center point in the XY plane of the measurement object 41 is extracted. After the center point is extracted, proceed to step S131.

In step S131, the offset position setting unit 77 acquires offset information. The offset information is information for setting an offset position that is a position offset from the edge portion in a direction approaching the center point. An example of the offset information is an offset amount indicating the distance from the edge portion to a portion that can be estimated as a flat portion. An example of the offset information is an offset magnification indicating the magnification from the edge portion to a portion that can be estimated as a flat portion. The offset amount and offset magnification can be determined from the model data or the model flat data. A method for setting the offset position based on the offset information will be described in detail later. After acquiring the offset information, proceed to step S132.

In step S132, the offset position setting unit 77 sets the offset position. The offset position is a position away from the edge portion, and is therefore a position that is likely to be a flat portion. The process of setting the offset position can also be considered as a process of selecting a flat portion in the measurement object 41.

When an offset amount is acquired as offset information, the edge portion is offset by a predetermined offset amount in a direction approaching the center point. As a result, in the surface image, which is an image of the XY plane, the coordinates of the offset position are closer to the center point than the coordinates of the edge portion. If the distance from the edge portion to the center point is shorter than the offset amount, the center point is set as the offset position rather than offsetting beyond the center point. The offset amount is always constant regardless of the coordinates of the edge portion. Here, if the offset is made in a direction away from the outer edge portion toward the center point, the offset position is set outside the measurement object 41. In other words, the plane portion cannot be set as the offset position. For this reason, it is preferable to offset in a direction approaching the center point from the edge portion.

In Figure 4, the center point is indicated as CP and the offset amount is indicated as LO. The edge portion of the lower disc portion of the measurement object 41 is the outer edge portion. The edge portion of the upper disc portion of the measurement object 41 is the inner edge portion. In the figure, the outer edge portion is indicated as OE and the inner edge portion is indicated as IE. Both the outer edge portion and the inner edge portion are circular. The inner edge portion is circular with a smaller diameter than the outer edge portion.

The position approaching the center point from the outer edge portion by the offset amount is the outer edge offset position. The position approaching the center point from the inner edge portion by the offset amount is the inner edge offset position. In the figure, the outer edge offset position is indicated by an alternate long and short dash line as OP and the inner edge offset position is indicated by an alternate long and short dash line as IP. Both the outer edge offset position and the inner edge offset position are circular. The inner edge offset position is circular with a smaller diameter than the outer edge offset position. The diameters decrease in the order of the outer edge portion, the outer edge offset position, the inner edge portion, and the inner edge offset position. The offset positions of the measurement object 41 are made up of the outer edge offset position and the inner edge offset position.

The planar portion including the outer edge offset position is a different planar portion from the planar portion including the inner edge offset position. More specifically, the planar portion including the outer edge offset position and the planar portion including the inner edge offset position are partitioned as planar portions separated from each other by the inner edge portion. In addition, the planar portion including the outer edge offset position and the planar portion including the inner edge offset position are planar portions having different Z coordinates.

If an offset magnification has been acquired as offset information, the edge portion is offset by a specified offset magnification in a direction approaching the center point so as to shrink. As a result, in the surface image, which is an image of the XY plane, the coordinates of the offset position are closer to the coordinates of the center point than the coordinates of the edge portion. The farther the edge portion is from the center point, the greater the amount of movement from the edge portion to the offset position. The offset magnification is always constant, regardless of the coordinates of the edge portion.

The method for setting the offset position is not limited to using an offset amount or an offset magnification, and various methods can be used. After setting the offset position, proceed to step S141.

In step S141, the measurement plane data extraction unit 74 extracts the three-dimensional coordinates of the offset position. More specifically, the three-dimensional coordinates of the position corresponding to the offset position determined based on the surface image are extracted and used as the measurement plane data. Here, the surface image and the three-dimensional point cloud correspond to each other in a pixel-by-pixel coordinate system. Therefore, it is possible to extract the three-dimensional coordinates corresponding to the offset position in the surface image from the three-dimensional point cloud.

The offset positions set based on the surface image include outer edge offset positions and inner edge offset positions. Therefore, the measurement plane data also includes an outer edge offset point group corresponding to the outer edge offset positions, and an inner edge offset point group corresponding to the inner edge offset positions. The coordinates of each point that makes up the outer edge offset point group are approximately equal to each other in Z coordinates. Furthermore, the coordinates of each point that makes up the inner edge offset point group are approximately equal to each other in Z coordinates. However, the outer edge offset positions and the inner edge offset positions are different in position in the Z direction. Therefore, the Z coordinate of the inner edge offset point group is a different value from the Z coordinate of the outer edge offset point group. After extracting the three-dimensional coordinates corresponding to the offset positions as measurement plane data, proceed to step S151.

In step S151, the matching unit 79 performs matching between the model plane data and the measurement plane data. Both the model plane data and the measurement plane data are three-dimensional coordinate data. In matching, the three-dimensional coordinates of multiple measurement points included in the measurement plane data are matched with the three-dimensional coordinate data of multiple points included in the model plane data. For example, after matching the coordinates of one specific point, the process of matching the coordinates of points adjacent to that point is repeated to match the coordinates of all points. However, since the measurement plane data contains some errors, it is not necessary to match the coordinates of each point perfectly. For example, the matching result may be the position and orientation that minimizes the sum of the distances between corresponding points in the model plane data and the measurement plane data.

Depending on the shape of the measurement object 41, the measurement plane data may contain an outer edge offset point cloud and an inner edge offset point cloud whose Z coordinates are different from each other. In this case, when matching the outer edge offset point cloud and the inner edge offset point cloud with the model plane data, in addition to matching the X and Y coordinates, the Z coordinates must also match. Therefore, since the Z coordinate of each point is constant, a larger amount of information is used for matching compared to matching using only two coordinates, the X and Y coordinates. By performing matching, position and orientation recognition is completed, and control related to position and orientation recognition is terminated. After position and orientation recognition is completed, the process transitions to the desired control, such as assembly control using the robot 10.

The effects of the above-described embodiment will be described below. According to the above-described embodiment, the recognition system 1 includes a measurement plane data extraction unit 74 that extracts three-dimensional coordinates in the planar portion of the measurement object 41 as measurement plane data based on the surface image. Furthermore, the recognition system 1 includes a matching unit 79 that matches the model plane data with the measurement plane data to recognize the position and orientation of the measurement object 41. For this reason, matching can be performed using the measurement plane data, which is data that does not include three-dimensional coordinates of edge portions, which have lower measurement accuracy compared to the planar portion. Therefore, it is possible to prevent the matching from including many three-dimensional coordinates with low measurement accuracy, which would result in a decrease in the accuracy of position and orientation recognition. Therefore, it is possible to provide a recognition system 1 that is capable of highly accurate position and orientation recognition.

The more steps there are on the surface of the measurement object 41, the greater the proportion of edge portions in the measurement data. For this reason, the accuracy of position and orientation recognition using measurement data that includes edge portions is likely to decrease. Therefore, a configuration that performs position and orientation recognition using measurement plane data in which the three-dimensional coordinates of the plane portion are extracted is particularly useful when there are many steps on the surface of the measurement object 41.

The matching unit 79 recognizes the position and orientation of the measurement object 41 by matching the model plane data with the measurement plane data. Therefore, the number of three-dimensional coordinates to be matched is smaller than when matching model data with measurement data. In other words, the data size of the data required for matching is small. Therefore, the processing required for matching can be reduced, and matching can be completed quickly.

The measurement plane data extraction unit 74 extracts the measurement plane data from the measurement data. Therefore, there is no need to limit the area to be measured in order to prevent the edge portion from being included in the measurement area from the beginning. Therefore, there is no need to provide the measurement unit 20 with a separate configuration for controlling the light projection area to match the shape of the measurement object 41, making it easier to simplify the configuration of the measurement unit 20. Furthermore, it is easier to start measurement quickly compared to a configuration that limits the measurement area to match the shape of the measurement object 41. Furthermore, when the number of data points in the measurement plane data is small and matching cannot be performed appropriately, it is possible to extract other measurement plane data from the measurement data. The other measurement plane data is, for example, three-dimensional coordinate data at a position with a different offset amount from the edge portion. Therefore, when matching cannot be performed appropriately, there is no need to measure the measurement object 41 again.

The offset position setting unit 77 sets an offset position that is an offset position in a direction away from the edge portion. Furthermore, the measurement plane data extraction unit 74 extracts the three-dimensional coordinates of a position corresponding to the offset position from the three-dimensional point cloud as measurement plane data. Therefore, the three-dimensional coordinates of a position away from the edge portion can be used for matching.

The offset position setting unit 77 sets an offset position that is offset from the edge portion toward the center point. Therefore, at least the offset position is not set outside the outer edge portion. Therefore, it is easy to set the flat portion as the offset position.

The offset position setting unit 77 sets a position that is a predetermined offset amount away from the edge portion in a direction toward the center point as the offset position. Therefore, the amount of movement from the edge portion to the offset position is small, making it easier to prevent three-dimensional coordinates with low measurement accuracy at positions close to the edge portion from being included in the measurement plane data.

The offset position setting unit 77 sets a position that is reduced from the edge portion by a predetermined offset magnification in a direction toward the center point as the offset position. This makes it possible to prevent the offset position for the edge portion close to the center point from exceeding the center point.

The feature point extraction unit 75 extracts the positions of edge parts, which are feature points, based on changes in luminance of the surface image. This allows the positions of edge parts to be extracted with greater accuracy than when the positions of edge parts, which are feature points, are extracted from a three-dimensional point cloud.

The matching unit 79 performs matching for each coordinate component of the three-dimensional coordinates including the coordinate component in the Z direction, which is the alignment direction of the measurement unit 20 and the measurement target object 41. Here, it is assumed that the measurement plane data includes data of plane parts with different Z coordinates. An example of a plane part with different Z coordinates is a case where the Z coordinates of the outer edge offset point group and the inner edge offset point group are different. Alternatively, a case where the plane part at the outer edge offset position is inclined in the Z direction, and the Z coordinates are different in the outer edge offset point group. Alternatively, a case where the plane part at the inner edge offset position is inclined in the Z direction, and the Z coordinates are different in the inner edge offset point group. In this case, the matching unit 79 performs matching so that the model plane data and the measurement plane data match for each coordinate component of the X coordinate, Y coordinate, and Z coordinate. Therefore, it is possible to perform matching using many coordinate components compared to a case where matching is performed using only two coordinate components, the X coordinate and the Y coordinate. Therefore, it is easy to suppress erroneous recognition and improve the accuracy of position and orientation recognition.

The measurement plane data includes three-dimensional coordinates of a plane portion that includes the outer edge offset position and three-dimensional coordinates of a plane portion that includes the inner edge offset position. In other words, the measurement plane data extraction unit 74 extracts the measurement plane data so as to include plane portions with different Z coordinates in the model plane data. For this reason, data of plane portions with different Z coordinates can be used for matching, making it easier to suppress erroneous recognition.

The measurement plane data also includes three-dimensional coordinates of a plane portion that includes the outer edge offset position and three-dimensional coordinates of a plane portion that includes the inner edge offset position. In other words, the measurement plane data extraction unit 74 extracts the measurement plane data so as to include multiple plane portions that are separated from each other. Therefore, it is easy to suppress erroneous recognition by using data of multiple plane portions for matching.

Although the above description takes as an example a case where the offset position is set in a direction approaching the center point from the edge portion, the offset direction is not limited to the above example. For example, the offset direction may be set in a direction away from the center point. Alternatively, the offset direction may be set in a direction that is not based on the center point. An example of a direction that is not based on the center point is parallel movement in the X direction or Y direction. However, if the offset direction is set in a direction away from the center point or in the X direction, at least a portion of the position offset from the outer edge portion will include a portion that is not a flat surface. For this reason, of the positions offset from the edge portion, only positions that are located on flat surfaces will be set as offset positions, and positions that are not located on flat surfaces will not be set as offset positions.

Second embodiment This embodiment is a modified example based on the preceding embodiment. In this embodiment, a mask range is set based on a surface image, and measurement plane data is extracted from a three-dimensional point cloud. In addition, an operation panel 239 is provided that allows the mask range to be set.

In FIG. 5, the measurement object 241 can be an object of any shape, but as an example, an elliptical disk portion is used with two small diameter disks on top. The two small disks on the measurement object 241 are spaced apart from each other and lined up in the X direction. The measurement object 241 is placed on the workbench 31 so that the two small disk portions are located on the upper side.

In FIG. 6, the control unit 70 that controls the recognition system 1 is connected to an operation panel 239. The operation panel 239 is a device for accepting input control by a user. The user can input settings for position and orientation recognition of the recognition system 1 using the operation panel 239. An example of a setting in the recognition system 1 is mask radius information, which will be described later. An example of a setting in the recognition system 1 is offset information. The control unit 70 acquires the settings operated on the operation panel 239.

The measurement plane data extraction unit 74 includes a feature point extraction unit 75 and a mask range setting unit 278. The mask range setting unit 278 sets a mask range, which will be described later, based on the information on the feature points. The measurement plane data extraction unit 74 extracts measurement plane data by excluding the three-dimensional point cloud at the position corresponding to the mask range set by the mask range setting unit 278.

In FIG. 7, when control related to position and orientation recognition is started, the model plane data acquisition unit 72 acquires model plane data in step S101. Then, the process proceeds to step S111, where the measurement target object 241 is measured. Then, the process proceeds to step S112, where measurement data is acquired. Then, the process proceeds to step S121, where edge portions are extracted as feature points from the surface image. Then, the process proceeds to step S231.

In step S231, the mask range setting unit 278 acquires the mask radius. The mask radius is the radius of the mask range that is set with the edge part as the center. The mask range will be described later. The larger the mask radius, the larger the mask range will be, and the mask range will extend to positions farther from the edge part. The mask radius can be changed by the user operating the operation panel 239. After acquiring the mask radius, the process proceeds to step S232.

In step S232, the mask range setting unit 278 sets a mask range. The mask range indicates a range that should not be used for matching as measurement plane data. The mask range is set as a range that includes the edge portion. The mask range is a range inside a specified mask radius centered on the edge portion.

In FIG. 8, the edge portion of the lower disk portion of the measurement object 241 is the outer edge portion. Also, the edge portions of the two upper disk portions of the measurement object 241 are the inner edge portions. In the figure, the outer edge portion is indicated as OE, and the inner edge portion is indicated as IE.

In the figure, the mask radius is shown as MR and the mask range as MA. In the figure, the mask range is the range surrounded by a two-dot chain line and indicated by dots. The mask range includes an annular range surrounded by a specified mask radius around the outer edge portion. It also includes an annular range surrounded by a specified mask radius around the inner edge portion. In summary, the mask range includes three ranges: an annular range corresponding to the outer edge portion and two annular ranges corresponding to the two inner edge portions. The mask range corresponding to the outer edge portion and the mask range corresponding to the inner edge portion may overlap. In this case, the mask range is not three separate ranges, but one continuous range.

The larger the mask radius, the larger the mask range. The more edge parts there are, the larger the mask range. The further apart the edge parts are, the smaller the area where the mask ranges overlap. For this reason, the further apart the edge parts are, the larger the mask range is likely to be. In this way, the mask range is determined by the mask radius and the shape of the edge parts. For this reason, if the measurement object 241 has many edge parts, the mask range is likely to be wide, so it is preferable to set the mask radius small. Also, if the measurement object 241 has few edge parts, the mask range is likely to be narrow, so it is preferable to set the mask radius large. After setting the mask range, proceed to step S241.

In step S241, the measurement plane data extraction unit 74 extracts three-dimensional coordinates corresponding to the position excluding the mask range as measurement plane data. The position excluding the mask range is the extraction range that indicates the extraction position of the measurement plane data. The measurement plane data extraction unit 74 extracts the three-dimensional coordinates of the position corresponding to the extraction range.

In Figure 9, the extraction range is shown as EA. The extraction range is the area surrounded by a two-dot chain line and indicated by dots. The extraction range is set in a range away from the edge portion. In other words, the extraction range is set in the flat portion.

The larger the mask range, the smaller the extraction range and the smaller the data size of the measurement plane data. Therefore, if you want to reduce the data size of the measurement plane data, you will set the mask radius to be large. On the other hand, if you want to increase the data size of the measurement plane data, you will set the mask radius to be small. The mask radius setting can be changed by the user operating the operation panel 239. However, the mask radius may be changed automatically so that the data size of the measurement plane data becomes appropriate. After extracting the three-dimensional coordinates corresponding to the position excluding the mask range as measurement plane data, proceed to step S151 and perform matching.

The smaller the data size of the measurement plane data, the faster the matching process can be completed. However, the larger the data size of the measurement plane data, the easier it is to accurately recognize the position and orientation in the matching process. Therefore, it is preferable to adjust the data size of the measurement plane data while considering the balance between processing speed and accuracy in position and orientation recognition. In order to prevent a significant decrease in the accuracy of position and orientation recognition, it is preferable that the data size of the measurement plane data is at least half the data size of the three-dimensional point cloud of the measurement data.

The effects of the above-described embodiment will be described below. According to the above-described embodiment, the measurement plane data extraction unit 74 extracts the three-dimensional coordinates of a position excluding the mask range from the three-dimensional point cloud as measurement plane data. Therefore, the three-dimensional coordinates of a position that does not include edge portions and their peripheral portions can be used as measurement plane data. Therefore, matching can be performed using measurement plane data that is data that does not include the three-dimensional coordinates of edge portions, which have lower measurement accuracy compared to plane portions. This prevents the matching from including many three-dimensional coordinates with low measurement accuracy, which would result in a decrease in the accuracy of position and orientation recognition.

The user can set the mask radius by operating the operation panel 239. In other words, the user can set the size of the mask range by operating the operation panel 239. For example, if the measurement object 241 has many edge portions, the mask range can be made small by setting the mask radius small. Alternatively, if the measurement object 241 has few edge portions, the mask range can be made large by setting the mask radius large. Therefore, by appropriately setting the mask radius to match the shape of the measurement object 241, the data size of the measurement plane data can be made appropriate.

Third Embodiment This embodiment is a modification of the preceding embodiment as a basic form. In this embodiment, a planar portion is extracted from a surface image, and the three-dimensional coordinates of a position corresponding to the planar portion are used as measurement plane data.

In FIG. 10, the measurement plane data extraction unit 74 includes a feature point extraction unit 375. The feature point extraction unit 375 extracts feature points from the measurement data. The feature points are, for example, planar portions of the measurement object 41.

In FIG. 11, when control related to position and orientation recognition is started, the model plane data acquisition unit 72 acquires model plane data in step S101. Then, the process proceeds to step S111, where the measurement target object 41 is measured. Then, the process proceeds to step S112, where measurement data is acquired. Then, the process proceeds to step S321.

In step S321, the feature point extraction unit 375 extracts planar parts from the surface image as feature points. Planar parts can be extracted from changes in luminance in the surface image. More specifically, parts with smaller changes in luminance compared to the surrounding area are regarded as planar parts and extracted. After extracting the planar parts, the process proceeds to step S341.

In step S341, the measurement plane data extraction unit 74 extracts the three-dimensional coordinates of the planar portion. More specifically, the three-dimensional coordinates of the positions corresponding to the planar portion determined based on the surface image are extracted and used as the measurement plane data. The three-dimensional coordinates of the positions corresponding to the entire positions extracted as the planar portion are extracted as the measurement plane data. However, it is also possible to extract a portion of the positions extracted as the planar portion and extract the three-dimensional coordinates of the corresponding positions as the measurement plane data. After extracting the three-dimensional coordinates, the process proceeds to step S151, where matching is performed.

The effects of the above-described embodiment will be described below. According to the above-described embodiment, the measurement plane data extraction unit 74 extracts, as measurement plane data, the three-dimensional coordinates of the position corresponding to the plane portion extracted by the feature point extraction unit 375 from the three-dimensional point cloud. This makes it easier to reduce the processing required to extract the measurement plane data compared to the case where the edge portion is extracted and then the three-dimensional coordinates are extracted so as not to include the edge portion. This reduces the processing load in controlling position and orientation recognition, and allows matching to be completed quickly.

Other embodiments The disclosure in this specification and drawings, etc. is not limited to the exemplified embodiments. The disclosure includes the exemplified embodiments and modifications by those skilled in the art based thereon. For example, the disclosure is not limited to the combination of parts and/or elements shown in the embodiments. The disclosure can be implemented by various combinations. The disclosure can have additional parts that can be added to the embodiments. The disclosure includes the omission of parts and/or elements of the embodiments. The disclosure includes the substitution or combination of parts and/or elements between one embodiment and another embodiment. The disclosed technical scope is not limited to the description of the embodiments. Some disclosed technical scopes are indicated by the description of the claims, and should be interpreted as including all modifications within the meaning and scope equivalent to the description of the claims.

The disclosure in the specification and drawings, etc. is not limited by the claims. The disclosure in the specification and drawings, etc. encompasses the technical ideas described in the claims, and extends to more diverse and extensive technical ideas than the technical ideas described in the claims. Therefore, various technical ideas can be extracted from the disclosure in the specification and drawings, etc., without being bound by the claims.

The control unit and the method described in the present disclosure may be realized by a dedicated computer having a processor programmed to execute one or more functions embodied in a computer program. Alternatively, the device and the method described in the present disclosure may be realized by a dedicated hardware logic circuit. Alternatively, the device and the method described in the present disclosure may be realized by one or more dedicated computers configured by a combination of a processor that executes a computer program and one or more hardware logic circuits. In addition, the computer program may be stored in a computer-readable non-transient tangible recording medium as instructions executed by the computer.

1 Recognition system, 10 Robot, 11 Base, 15 Robot arm, 16 Robot hand, 20 Measurement unit, 31 Work table, 41 Measurement object, 70 Control unit, 72 Model plane data acquisition unit, 73 Measurement data acquisition unit, 74 Measurement plane data extraction unit, 75 Feature point extraction unit, 76 Center point extraction unit, 77 Offset position setting unit, 79 Matching unit, 239 Operation panel, 241 Measurement object, 278 Mask range setting unit, 375 Feature point extraction unit

Claims (9)

A recognition system for performing position and orientation recognition for measuring a shape of a measurement object (41, 241) and recognizing a position and orientation of the measurement object, comprising:
A measurement unit (20) that captures a surface image of the measurement object and calculates a three-dimensional point cloud corresponding to the surface image;
a model plane data acquisition unit (72) for acquiring model plane data obtained by extracting data of a plane portion from model data indicating a three-dimensional shape of the measurement object;
a measurement data acquisition unit (73) that acquires measurement data including the surface image captured by the measurement unit and the three-dimensional point cloud calculated by the measurement unit;
a measurement plane data extraction unit (74) that extracts, as measurement plane data, three-dimensional coordinates in the plane portion from the three-dimensional point cloud based on the surface image acquired by the measurement data acquisition unit;
a matching unit (79) that matches the model plane data with the measurement plane data to recognize the position and orientation of the measurement object ;
a feature point extraction unit (75) for extracting edge portions as feature points from the surface image;
an offset position setting unit (77) that sets an offset position that is a position offset in a direction away from the edge portion ,
The measurement plane data extraction unit is a recognition system that extracts, from the three-dimensional point cloud, the three-dimensional coordinates of a position corresponding to the offset position as the measurement plane data . a center point extraction unit (76) for extracting a center point from the surface image;
The recognition system according to claim 1 , wherein the offset position setting unit sets a position offset from the edge portion in a direction approaching the center point as the offset position. The recognition system according to claim 2 , wherein the offset position setting unit sets, as the offset position, a position that is closer to the center point from the edge portion by an offset amount in a direction from the edge portion to the center point. The recognition system according to claim 2 , wherein the offset position setting unit sets, as the offset position, a position reduced from the edge portion by an offset magnification in a direction from the edge portion toward the center point. A recognition system for performing position and orientation recognition for measuring a shape of a measurement object (41, 241) and recognizing a position and orientation of the measurement object, comprising:
A measurement unit (20) that captures a surface image of the measurement object and calculates a three-dimensional point cloud corresponding to the surface image;
a model plane data acquisition unit (72) for acquiring model plane data obtained by extracting data of a plane portion from model data indicating a three-dimensional shape of the measurement object;
a measurement data acquisition unit (73) that acquires measurement data including the surface image captured by the measurement unit and the three-dimensional point cloud calculated by the measurement unit;
a measurement plane data extraction unit (74) that extracts, as measurement plane data, three-dimensional coordinates in the plane portion from the three-dimensional point cloud based on the surface image acquired by the measurement data acquisition unit;
a matching unit (79) that matches the model plane data with the measurement plane data to recognize the position and orientation of the measurement object ;
a feature point extraction unit (75) for extracting edge portions as feature points from the surface image;
a mask range setting unit (278) for extracting a surrounding area including the edge portion as a mask range ,
The measurement plane data extraction unit is a recognition system that extracts, as the measurement plane data, the three-dimensional coordinates of a position excluding the mask range from the three-dimensional point cloud . an operation panel (239) through which a user can input settings for the position and orientation recognition;
The recognition system according to claim 5 , wherein the user can set the size of the mask range by operating the operation panel. 7. The recognition system according to claim 1 , wherein the feature point extraction unit extracts the positions of the feature points based on a change in luminance of the surface image. 8. The recognition system according to claim 1, wherein, when a planar portion having different coordinate components in an arrangement direction of the measurement unit and the measurement object exists in the model planar data, the measurement planar data extraction unit extracts the measurement planar data so as to include three-dimensional coordinates of a position corresponding to the planar portion having different coordinate components in the arrangement direction. 9. A recognition system according to claim 1, wherein the measurement plane data extraction unit extracts the measurement plane data so as to include three-dimensional coordinates of positions corresponding to the plurality of planar portions that are separated from each other when the plurality of planar portions are present in the model plane data and are separated from each other.

Images