JP2018001333A - Calibration board, robot and detection method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a calibration board that can accurately detect a position corresponding to a geometrical physical relationship of four or more markers.SOLUTION: A calibration board includes four or more markers. The four or more markers respectively position at an apex of convex hull that does not have rotational symmetry property.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a calibration board, a robot, and a detection method.

  Research and development have been conducted on techniques for imaging an object by an imaging unit and causing a robot to perform a predetermined operation based on the captured image. In such a technique, there is a case where calibration is performed to associate the coordinates in the robot coordinate system, which is the coordinate system of the robot, with the coordinates in the imaging unit coordinate system, which is the coordinate system of the imaging unit. In such calibration, for example, a calibration pattern is captured by the imaging unit, and coordinates in the imaging unit coordinate system and coordinates in the robot coordinate system are associated with each other based on the pattern included in the captured image.

  In this regard, a calibration pattern having three or more first markers arranged in a straight line and one second marker arranged at a position orthogonal to the straight line is known (see Patent Document 1).

JP 2003-244521 A

  However, such a calibration pattern sometimes erroneously detects the second marker. As a result, an apparatus that detects the position indicated by the geometric positional relationship between the first marker and the second marker may not be able to detect the position with high accuracy.

In order to solve at least one of the above problems, one embodiment of the present invention includes four or more markers, and each of the four or more markers is located at the apex of a convex hull that does not have rotational symmetry. A calibration board.
With this configuration, in the calibration board, each of the four or more markers is positioned at the apex of the convex hull that does not have rotational symmetry. Thereby, the calibration board can detect the position according to four or more markers accurately.

According to another aspect of the present invention, in the calibration board, each of the four or more markers may be a centroid marker.
With this configuration, in the calibration board, each of the four or more markers is a centroid marker. Thereby, the calibration board can detect the position according to four or more centroid markers accurately.

According to another aspect of the present invention, in the calibration board, each of the centroid markers may have a shape in which three or more centroids overlap each other.
With this configuration, in the calibration board, each centroid marker has a figure in which three or more centroids overlap. Thereby, the calibration board can detect the centroid marker which has a figure which is four or more markers and three or more centroids overlap from an image without an error.

According to another aspect of the present invention, the calibration board may have a configuration in which the length of the side including the first marker in the convex hull is longer than the length of the other side in the convex hull.
With this configuration, in the calibration board, the length of the side including the first marker in the convex hull is longer than the length of the other side in the convex hull. As a result, the calibration board can efficiently detect positions corresponding to four or more markers by using the first marker as a reference.

In another aspect of the present invention, the calibration board may have a configuration in which an origin is provided inside the convex hull.
With this configuration, the calibration board has an origin inside the convex hull. Thereby, the calibration board can express the position according to four or more markers more precisely by the origin.

According to another aspect of the present invention, the calibration board includes a first reference portion and a second reference portion outside the convex hull, a straight line passing through the origin and the first reference portion, and the origin. A configuration orthogonal to a straight line passing through the second reference portion may be used.
With this configuration, in the calibration board, the straight line passing through the origin and the first reference portion is orthogonal to the straight line passing through the origin and the second reference portion. Thus, the calibration board can more accurately represent the posture of the calibration board by the first reference portion and the second reference portion.

In another aspect of the present invention, a configuration in which the number of the markers is five in the calibration board may be used.
With this configuration, the calibration board has five markers. Thereby, the calibration board can detect the position according to the five markers with high accuracy.

According to another aspect of the present invention, a movable portion is provided, and the movable portion operates based on an image obtained by capturing an image of a pattern including four or more markers, each of which is located at the apex of the convex hull. Is a robot.
With this configuration, the robot operates based on an image in which a pattern including four or more markers each positioned at the apex of the convex hull is captured by the imaging unit. Thereby, the robot can perform a predetermined operation with high accuracy based on a position corresponding to four or more markers and detected from an image captured by the imaging unit.

In another aspect of the present invention, the robot uses a configuration in which calibration of the coordinate system of the robot and the coordinate system of the imaging unit is performed based on an image obtained by imaging the pattern by the imaging unit. May be.
With this configuration, the robot can calibrate the coordinate system of the robot and the coordinate system of the imaging unit based on an image obtained by capturing an image of a pattern composed of four or more markers, each of which is located at the apex of the convex hull. Done. Thereby, the robot can accurately calculate the position in the coordinate system of the robot, which is a position corresponding to the position of the imaging unit in the coordinate system.

In another aspect of the present invention, in the robot, a configuration in which each of the four or more markers is a centroid marker may be used.
With this configuration, the robot operates based on an image in which a pattern including four or more centroid markers each positioned at the apex of the convex hull is captured by the imaging unit. As a result, the robot can accurately perform a predetermined operation based on the position corresponding to the four or more centroid markers and the position detected from the image captured by the imaging unit.

According to another aspect of the present invention, in the robot, a configuration in which each of the centroid markers has a figure in which three or more centroids overlap each other may be used.
With this configuration, the robot operates based on an image in which a pattern of centroid markers each having a figure in which three or more centroids overlap is captured by the imaging unit. Thereby, the robot is predetermined based on the position detected from the image picked up by the image pickup unit at a position corresponding to the centroid mark marker having four or more markers and three or more centroids overlapping each other. Can be performed with high accuracy.

In another aspect of the present invention, the robot may have a configuration in which a length of the side including the first marker in the convex hull is longer than a length of the other side in the convex hull.
According to this configuration, the robot is based on an image in which a pattern including four or more markers whose side length including the first marker in the convex hull is longer than the length of the other side in the convex hull is captured by the imaging unit. Operate. Thus, the robot efficiently detects positions corresponding to the four or more markers from the image captured by the imaging unit using the first marker as a reference, and performs a predetermined operation based on the detected positions. It can be performed with high accuracy.

Further, in another aspect of the present invention, in the robot, a configuration in which an origin is provided inside the convex hull may be used.
With this configuration, the robot operates based on an image captured by the imaging unit, which is a pattern composed of four or more markers each positioned at the apex of the convex hull and having an origin inside the convex hull. As a result, the robot can accurately perform a predetermined operation based on a position represented by the origin provided inside the convex hull and detected from the image captured by the imaging unit.

In another aspect of the present invention, the robot includes a first reference portion and a second reference portion outside the convex hull, a straight line passing through the origin and the first reference portion, the origin, and the A configuration orthogonal to the straight line passing through the second reference portion may be used.
With this configuration, the robot operates based on an image obtained by capturing an image of a pattern composed of four or more markers, each of which is located at the apex of a convex hull having a first reference portion and a second reference portion outside. To do. As a result, the robot can accurately detect the posture of the calibration board and perform a predetermined operation based on the detected posture.

In another aspect of the present invention, the robot may have a configuration in which the number of the markers is five.
With this configuration, the robot operates based on an image in which a pattern including five markers each positioned at the apex of the convex hull is captured by the imaging unit. Thereby, the robot can accurately perform a predetermined operation based on the position corresponding to the five markers and detected from the image captured by the imaging unit.

Further, according to another aspect of the present invention, a predetermined number of markers each including four or more markers located at the vertices of a convex hull that does not have rotational symmetry are selected from marker candidates included in an image captured by an imaging unit. And detecting the predetermined number of candidates that match a reference pattern consisting of:
With this configuration, the detection method includes a predetermined number of markers, each of which is four or more located at the vertices of the convex hull that does not have rotational symmetry, from candidate markers included in the image captured by the imaging unit. A predetermined number of the candidates that match the reference pattern are detected. Thereby, the detection method can detect the position according to four or more markers accurately.

According to another aspect of the present invention, in the detection method, the result of homography conversion of the first pattern including the predetermined number of the candidates selected from the marker candidates included in the image captured by the imaging unit, A configuration that detects the predetermined number of the candidates based on consistency with a reference pattern may be used.
With this configuration, the detection method is based on the consistency between the result of homography conversion of a predetermined number of first patterns selected from the marker candidates included in the image captured by the imaging unit and the reference pattern. Then, a predetermined number of the candidates are detected. As a result, the detection method can correctly detect a pattern composed of four or more markers even when an incorrect marker is included in the marker candidates included in the image.

According to another aspect of the present invention, in the detection method, a configuration may be used in which the consistency is represented by a difference between a result of the homography conversion and the reference pattern.
With this configuration, in the detection method, consistency with the reference pattern is expressed by a difference between the result of homography conversion of the first pattern and the reference pattern. As a result, even if the marker candidate included in the image includes an incorrect marker, the detection method is based on the consistency represented by the difference between the result of homography conversion of the first pattern and the reference pattern. A pattern composed of more than one marker can be detected correctly.

As described above, in the calibration board, each of the four or more markers is positioned at the apex of the convex hull that does not have rotational symmetry. Thereby, the calibration board can detect the position according to four or more markers accurately.
Further, the robot operates based on an image obtained by capturing an image of a pattern including four or more markers, each of which is located at the apex of the convex hull. Thereby, the robot can perform a predetermined operation with high accuracy based on a position corresponding to four or more markers and detected from an image captured by the imaging unit.
In addition, the detection method is based on the consistency between the reference pattern and the result of homography conversion of the first pattern including a predetermined number of candidates selected from the marker candidates included in the image captured by the imaging unit. A predetermined number of the candidates are detected. As a result, the detection method can correctly detect a pattern composed of four or more markers even when an incorrect marker is included in the marker candidates included in the image.

It is a figure showing an example of composition of robot system 1 concerning an embodiment. It is a figure which shows an example of pattern PN. It is a figure for demonstrating a centroid. It is a figure which shows marker MK1 shown in FIG. It is a figure which shows an example of the geometric positional relationship of the marker MK1-marker MK5 in pattern PN. 2 is a diagram illustrating an example of a hardware configuration of a robot control device 30. FIG. 2 is a diagram illustrating an example of a functional configuration of a robot control device 30. FIG. It is a flowchart which shows an example of the flow of the process which the robot control apparatus 30 performs calibration. It is a figure which shows an example of a reference | standard pattern. 10 is a table showing X and Y coordinates of markers MK1 to MK5 in the two-dimensional coordinate system BC shown in FIG. It is a figure which shows an example of a part of the captured image which the image acquisition part 43 acquired. It is a figure which shows an example of the 0th combination which may be determined as the combination which satisfy | fills a predetermined condition. It is a figure which shows an example of the 0th combination determined with the combination which satisfy | fills a predetermined condition. It is a table which shows an example of the X coordinate and Y coordinate which were specified by the detection part 45, and were in the imaging part coordinate system CC of each of marker candidate DP1-marker candidate DP6. It is a figure which shows an example of the coordinate in the two-dimensional coordinate system BC of each marker candidate contained in the 1st combination after homography conversion was performed in step S160. It is a figure which shows the other example of the coordinate in the two-dimensional coordinate system BC of each marker candidate contained in the 1st combination after homography conversion was performed in step S160. It is a figure showing an example of composition of robot system 1 when making robot 20 perform predetermined work. 4 is a flowchart illustrating an example of a flow of processing in which the robot control device 30 causes the robot 20 to perform a predetermined operation. It is a flowchart which shows the other example of the flow of the process which the robot control apparatus 30 performs calibration. It is a figure which shows an example of the distance which is the distance between the marker candidates contained in each marker candidate pair, and was calculated by the detection part 45. FIG. It is a figure which shows an example of the calibration board CB which concerns on the modification 2 of embodiment.

<Embodiment>
Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<Robot system configuration>
First, the configuration of the robot system 1 will be described.
FIG. 1 is a diagram illustrating an example of a configuration of a robot system 1 according to the embodiment. The robot system 1 includes an imaging unit 10, a robot 20, and a robot control device 30.

  The imaging unit 10 is a camera including, for example, a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like that is an imaging device that converts collected light into an electrical signal. The imaging unit 10 is installed at a position where the range including the calibration board CB placed on the upper surface of the work table TB can be imaged.

  The work table TB is a table such as a table, for example. Note that the work table TB may be another table as long as it is a table on which the calibration board CB can be placed instead of the table. The calibration board CB is used for calibration for associating the imaging unit coordinate system CC with the robot coordinate system RC. More specifically, in the calibration, the coordinates in the imaging unit coordinate system CC and the coordinates in the robot coordinate system RC are associated. The imaging unit coordinate system CC is a coordinate system that indicates a position on a captured image captured by the imaging unit 10. The robot coordinate system RC is a coordinate system that serves as a reference when the robot control device 30 moves the TCP (Tool Center Point) of the robot 20 to the robot 20. The shape of the calibration board CB shown in FIG. 1 is a plate shape, but may be other shapes such as a cubic shape, a rectangular parallelepiped shape, a triangular prism shape, a triangular pyramid shape, a cylindrical shape, and a conical shape. The calibration board CB will be described later.

  The imaging unit 10 is connected to the robot control device 30 through a cable so as to be communicable. Note that wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB (Universal Serial Bus), for example. The imaging unit 10 may be configured to be connected to the robot control device 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  The robot 20 is a single-arm robot including an arm A and a support base B that supports the arm A. The single arm robot is a robot having one arm (arm) like the arm A in this example. The robot 20 may be a multi-arm robot instead of the single-arm robot. The multi-arm robot is a robot having two or more arms (for example, two or more arms A). Of the multi-arm robot, a robot having two arms is also referred to as a double-arm robot. That is, the robot 20 may be a double-arm robot having two arms or a multi-arm robot having three or more arms (for example, three or more arms A). The robot 20 may be another robot such as a SCARA robot, a rectangular coordinate robot, or a cylindrical robot. The orthogonal coordinate robot is, for example, a gantry robot.

The arm A includes an end effector E and a manipulator M. The arm A is an example of a movable part.
In this example, the end effector E is an end effector including a finger portion that can grip an object. The end effector E may be an end effector capable of lifting an object by air suction, a magnetic force, a jig or the like, or another end effector, instead of the end effector including the finger portion.

  The end effector E is communicably connected to the robot control device 30 via a cable. Thereby, the end effector E performs an operation based on the control signal acquired from the robot control device 30. Note that wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. The end effector E may be configured to be connected to the robot control device 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  The manipulator M includes six joints. Each of the six joints includes an actuator (not shown). That is, the arm A including the manipulator M is a 6-axis vertical articulated arm. The arm A performs an operation with six degrees of freedom by a coordinated operation by the support base B, the end effector E, the manipulator M, and the actuators of each of the six joints included in the manipulator M. The arm A may be configured to operate with a degree of freedom of 5 axes or less, or may be configured to operate with a degree of freedom of 7 axes or more.

  Each of the six actuators (provided at the joints) included in the manipulator M is communicably connected to the robot controller 30 via a cable. Thereby, the actuator operates the manipulator M based on the control signal acquired from the robot control device 30. Note that wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. In addition, a part or all of the six actuators included in the manipulator M may be connected to the robot control device 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  In this example, the robot control device 30 is a controller that controls (operates) the robot. The robot control device 30 causes the imaging unit 10 to image the range including the calibration board CB. Then, the robot control device 30 acquires a captured image captured by the imaging unit 10 from the imaging unit 10. The robot control device 30 performs calibration for associating the imaging unit coordinate system CC with the robot coordinate system RC based on the acquired captured image. Specifically, the robot control device 30 performs calibration that associates the robot coordinate system RC with a calibration board coordinate system that is a coordinate system representing a position on the calibration board CB, based on the captured image. Here, when the calibration for associating the calibration board coordinate system with the robot coordinate system RC has been performed on the robot control device 30 in advance, the robot control device 30 determines the imaging unit coordinates based on the results of these two calibrations. Calibration that correlates the system CC and the robot coordinate system RC can be performed. The method of performing calibration for associating the calibration board coordinate system with the robot coordinate system RC with respect to the robot control device 30 may be an existing method such as a method including a touch-up operation such as direct teaching. It may be a new method. Hereinafter, as an example, a case will be described in which the robot control device 30 is previously calibrated to associate the calibration board coordinate system with the robot coordinate system RC. After calibration that associates the imaging unit coordinate system CC with the robot coordinate system RC is performed, the robot control device 30 causes the robot 20 to perform a predetermined operation based on the captured image captured by the imaging unit 10.

<Configuration of calibration board>
Hereinafter, the calibration board CB will be described. The calibration board CB is provided with a pattern PN composed of four or more markers. That is, the pattern PN includes these four or more markers. In the pattern PN, each of the four or more markers is located at the apex of the convex hull. A convex hull is the smallest convex polygon that encompasses all given points. Thereby, any combination of three markers that can be selected from four or more markers constituting the pattern PN is not aligned on a straight line. In this example, in the pattern PN, the first marker that is one of the four or more markers has characteristics different from those of other markers. Specifically, the length of the side including the first marker among the four or more markers constituting the pattern PN is the length of the other side in the convex hull where each of the four or more markers is located at the apex. Longer than that. Note that the four or more markers included in the pattern PN may not include the first marker having the characteristics. However, when the first marker is included in four or more markers included in the pattern PN, the calibration board CB is compared with the case where the first marker is not included in four or more markers included in the pattern PN. Thus, the calibration board CB can be detected in a short time.

  Hereinafter, as an example, the case where the pattern PN includes the markers MK1 to MK5 that are five markers will be described. The five are examples of a predetermined number. Hereinafter, a case where the marker MK1 is the first marker will be described. Hereinafter, a case will be described in which all of the markers MK1 to MK5 are markers having the same shape, and all of the markers are markers having the same size. The first marker may be any one of the markers MK2 to MK5 instead of the marker MK1. Further, some or all of the markers MK1 to MK5 may be markers having different shapes or may be markers having different sizes.

  FIG. 2 is a diagram illustrating an example of the pattern PN. As shown in FIG. 2, the pattern PN includes markers MK1 to MK5. In this example, each of the markers MK1 to MK5 is a centroid marker. Note that some or all of the markers MK1 to MK5 may be other markers instead of the centroid markers.

  Here, the centroid of the centroid marker will be described. FIG. 3 is a diagram for explaining the centroid. FIG. 3 shows a figure F0 whose contour is formed by a curve, a two-dimensional coordinate system for representing the position of each point constituting the figure F0, and a vector r_i indicating the position of each point. . Here, in FIG. 3, the vector r is represented by attaching an arrow on r. Further, in FIG. 3, “i” added after “_” of the vector r_i is represented by a subscript i of r. The subscript i is a label for distinguishing each point (for example, pixel) constituting the figure F0. In FIG. 3, an arrow extending from the origin of the two-dimensional coordinate system to the figure F0 is an arrow indicating the size and direction of the vector r_i.

  The centroid is the center of the figure. Only one centroid is uniquely determined for one figure. The position of the centroid of the figure F0 shown in FIG. 3 is represented by the vector c defined by the following equation (1) using the vector r_i.

  In equation (1), the vector c is expressed by attaching an arrow on c. N is the number of points constituting the graphic F0 and is an integer of 1 or more. That is, the position of the centroid of the figure F0 shown in FIG. 3 is obtained by a vector (average of N vectors) obtained by dividing the sum of the vectors representing the positions of the points constituting the figure F0 by the number of each point. expressed. In FIG. 3, the centroid has been described using a two-dimensional figure F0, a two-dimensional coordinate system, and a two-dimensional vector. However, the position of the centroid is not limited to three-dimensional figures or more. It is represented by the vector c defined by the above equation (1).

  The centroid marker is a marker constituted by a combination of three or more figures. That is, the centroid of each of the three or more figures constituting the centroid marker falls within (includes) a predetermined range. For example, in a centroid marker configured by a combination of the figure A0, the figure B0, and the figure C0, the centroid A1 of the figure A0, the centroid B1 of the figure B0, and the centroid C1 of the figure C0 are predetermined. Within (included) the range. The predetermined range is, for example, a circular range having a radius of about several pixels, but may be a circular range having a radius of less than one pixel, or may be a circular range having a radius of several pixels or more, It may be a range of another shape different from a circle such as a rectangle.

  Next, the centroid marker will be described using the marker MK1 as an example among the markers MK1 to MK5 shown in FIG. FIG. 4 is a diagram showing the marker MK1 shown in FIG. As shown in FIG. 4, the marker MK1 is composed of three or more figures. Specifically, each of the three or more figures constituting the marker MK1 is detected as a partial region of only white or black in the binarized image. That is, the marker MK1 includes a figure M11 represented by a black partial area having a circular outline, a figure M12 represented by a white partial area having a ring shape, and a black partial area having a ring shape. It is comprised by three figures with the figure M13 represented. The marker MK1 is formed so that the centroids of the figure M11, the figure M12, and the figure M13 are within a predetermined range. For this reason, marker MK1 comprised by the combination of figure M11, figure M12, and figure M13 is a centroid marker. Note that the contours of the graphics M11 to M13 in the marker MK1 constitute concentric circles, and the centroid is located at the center of the concentric circles. The centroid marker may be a combination in which a combination of three or more figures constituting the centroid marker does not constitute a concentric circle. However, even in this case, the centroid of each of the three or more figures constituting the centroid marker falls within a predetermined range. When each of the markers MK1 to MK5 is a centroid marker, it is unlikely that the three centroids will accidentally fall within a predetermined range. Therefore, the calibration board CB detects the false detection of each of the markers MK1 to MK5. , And each of the markers MK1 to MK5 can be detected with high accuracy.

  Markers MK1 to MK5 that are such centroid markers are located at the apexes of the convex hull that does not have rotational symmetry as shown in FIG. 5 in the pattern PN. FIG. 5 is a diagram illustrating an example of a geometric positional relationship between the markers MK1 to MK5 in the pattern PN. That is, each of the markers MK1 to MK5 is arranged at each vertex of a convex hull that is a convex pentagon having no rotational symmetry in the pattern PN. The convex hull CH1 shown in FIG. 5 is an example of a convex hull that does not have rotational symmetry in which each of the markers MK1 to MK5 is located at the apex. Due to the geometric positional relationship of the markers MK1 to MK5, any combination of three markers that can be selected from the markers MK1 to MK5 is not aligned. Further, as described above, the length of the side including the marker MK1 which is the first marker in the convex hull CH1 is longer than the length of the other side in the convex hull CH1.

  Returning to FIG. The pattern PN in this example further includes n black circles BC1 to BCn, a marker OP, a marker XP1, a marker XP2, a marker YP1, and a marker YP2. Here, n is an arbitrary integer. The pattern PN does not include some or all of the black circles BC1 to BCn, which are n black circles, the marker OP, the marker XP1, the marker XP2, the marker YP1, and the marker YP2. There may be. Further, at least one of the marker XP1 and the marker XP2 is an example of a first reference portion. Further, at least one of the marker YP1 and the marker YP2 is an example of a second reference portion.

  Each of the black circles BC1 to BCn is arranged at each lattice point when the region in the pattern PN is divided into a square lattice. Further, at a position where a black circle among the black circles BC1 to BCn overlaps any one of the markers MK1 to MK5, the black circle is deleted and the marker is arranged at the position. Such black circles BC1 to BCn are used, for example, when distortion correction of the imaging unit 10 is performed by the calibration board CB. Such black circles BC1 to BCn can also be used to improve the accuracy with which the robot controller 30 detects the posture of the calibration board CB. Note that some or all of the black circles BC1 to BCn may have other shapes, other sizes, or other colors. In this case, the part or the whole must be identifiable even when compared with any of the markers MK1 to MK5, the marker OP, the marker XP1, the marker XP2, the marker YP1, and the marker YP2.

  The marker OP is a marker indicating the origin of the coordinate system representing the position on the pattern PN. The marker OP shown in FIG. 2 is represented by a black circle having a larger diameter than each of the plurality of black circles BC1 to BCn. The marker OP may be a black circle having a smaller diameter than each of the plurality of black circles BC1 to BCn instead of the black circle, may be a marker of another shape, or may be a marker of another color. There may be. The marker OP is located inside (inside) the convex hull CH1 in which each of the markers MK1 to MK5 is located at the apex (see FIG. 5). In this example, the position of the marker OP is the position of the midpoint between two markers that are the farthest away from the markers MK1 to MK5. That is, the marker OP represents a position corresponding to the geometric positional relationship between the markers MK1 to MK5. Here, at a position where the marker OP overlaps a certain black circle among the black circles BC1 to BCn, the black circle is deleted, and the marker OP is arranged at the position. The marker OP may be configured not to be included in the pattern PN. However, when the marker OP is included in the pattern PN, the calibration board CB can suppress erroneous detection of positions according to the geometric positional relationship between the markers MK1 to MK5 and can detect the positions with high accuracy. it can. Further, the marker OP may be configured to represent a position unrelated to the geometric positional relationship between the markers MK1 to MK5. That is, the position of the marker OP may be any position as long as it is inside the convex hull CH1.

  In this example, the shape and size of the marker XP1 and the marker XP2 are the same shape and size as the shape and size of the marker OP. In this example, the marker XP1 and the marker XP2 are arranged such that the marker OP is positioned on a straight line connecting the marker XP1 and the marker XP2. In addition, the marker XP1 and the marker XP2 are arranged so that one of the sides of the square lattice described above is parallel to the straight line. That is, the straight line connecting the marker XP1 and the marker XP2 represents the X axis passing through the marker OP which is the origin on the pattern PN. Here, at a position where the marker XP1 overlaps a certain black circle among the black circles BC1 to BCn, the black circle is deleted, and the marker XP1 is arranged at the position. Further, at the position where the position of the marker XP2 overlaps with a certain black circle among the black circles BC1 to BCn, the black circle is deleted, and the marker XP2 is arranged at the position. The marker XP1 and the marker XP2 may be configured not to be included in the pattern PN. However, since the marker XP1 and the marker XP2 are included in the pattern PN, the calibration board CB is an X axis corresponding to the geometric positional relationship between the markers MK1 to MK5 and represents a posture of the calibration board CB. This makes it possible to suppress erroneous detection of the X axis and to detect the X axis with high accuracy. Further, one or both of the marker XP1 and the marker XP2 may have other shapes, other sizes, or other colors.

  In this example, the shape and size of the marker YP1 and the marker YP2 are the same shape and size as the shape and size of the marker OP. In this example, the marker YP1 and the marker YP2 are arranged so that the marker OP is positioned on a straight line connecting the marker YP1 and the marker YP2. The marker YP1 and the marker YP2 are arranged so that a straight line connecting the marker XP1 and the marker XP2 and a straight line connecting the marker YP1 and the marker YP2 are orthogonal to each other. That is, the straight line connecting the marker YP1 and the marker YP2 represents the Y axis passing through the marker OP that is the origin on the pattern PN. Here, at a position where the marker YP1 overlaps a certain black circle among the black circles BC1 to BCn, the black circle is deleted, and the marker YP1 is arranged at the position. Further, at a position where the marker YP2 overlaps a certain black circle among the black circles BC1 to BCn, the black circle is deleted, and the marker YP2 is arranged at the position. The marker YP1 and the marker YP2 may be configured not to be included in the pattern PN. However, since the marker YP1 and the marker YP2 are included in the pattern PN, the calibration board CB is a Y-axis corresponding to the geometric positional relationship between the markers MK1 to MK5 and represents a posture of the calibration board CB. In this way, it is possible to suppress erroneous detection of the Y axis and to detect the Y axis with high accuracy. Further, either one or both of the marker YP1 and the marker YP2 may have another shape, may have another size, or may have another color.

<Hardware configuration of robot controller>
Hereinafter, the hardware configuration of the robot control device 30 will be described with reference to FIG. FIG. 6 is a diagram illustrating an example of a hardware configuration of the robot control device 30.

  The robot control device 30 includes, for example, a CPU (Central Processing Unit) 31, a storage unit 32, an input receiving unit 33, a communication unit 34, and a display unit 35. These components are connected to each other via a bus Bus so that they can communicate with each other. Further, the robot control device 30 communicates with each of the imaging unit 10 and the robot 20 via the communication unit 34.

The CPU 31 executes various programs stored in the storage unit 32.
The storage unit 32 includes, for example, a hard disk drive (HDD), a solid state drive (SSD), an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), and a random access memory (RAM). The storage unit 32 may be an external storage device connected via a digital input / output port such as a USB instead of the one built in the robot control device 30. The storage unit 32 stores various information processed by the robot control device 30, various programs including an operation program for operating the robot 20, various images, and the like.

The input receiving unit 33 is, for example, a touch panel configured integrally with the display unit 35. The input receiving unit 33 may be a keyboard, a mouse, a touch pad, or other input device.
The communication unit 34 includes, for example, a digital input / output port such as USB, an Ethernet (registered trademark) port, and the like.
The display unit 35 is, for example, a liquid crystal display panel or an organic EL (ElectroLuminescence) display panel.

<Functional configuration of robot controller>
Hereinafter, the functional configuration of the robot control device 30 will be described with reference to FIG. FIG. 7 is a diagram illustrating an example of a functional configuration of the robot control device 30.

  The robot control device 30 includes a storage unit 32 and a control unit 36.

  The control unit 36 controls the entire robot control device 30. The control unit 36 includes an imaging control unit 41, an image acquisition unit 43, a detection unit 45, a calibration unit 47, a position / orientation calculation unit 49, and a robot control unit 51. These functional units included in the control unit 36 are realized, for example, when the CPU 31 executes various programs stored in the storage unit 32. Further, some or all of the functional units may be hardware functional units such as LSI (Large Scale Integration) and ASIC (Application Specific Integrated Circuit).

The imaging control unit 41 causes the imaging unit 10 to capture a range that can be captured by the imaging unit 10.
The image acquisition unit 43 acquires the captured image captured by the imaging unit 10 from the imaging unit 10.
Based on the captured image acquired by the image acquisition unit 43, the detection unit 45 detects marker candidates that are candidates for the markers MK1 to MK5 included in the captured image. The detection unit 45 detects a pattern PN composed of the markers MK1 to MK5 based on the detected marker candidates. Further, the detection unit 45 detects a marker OP included in the captured image based on the captured image acquired by the image acquisition unit 43. Further, the detection unit 45 detects the marker XP1 and the marker XP2 included in the captured image based on the captured image acquired by the image acquisition unit 43. The detection unit 45 detects the marker YP1 and the marker YP2 included in the captured image based on the captured image acquired by the image acquisition unit 43.

The calibration unit 47 performs calibration for associating the imaging unit coordinate system CC with the robot coordinate system RC based on the pattern PN detected by the detection unit 45.
The position / orientation calculation unit 49 calculates the position and orientation corresponding to the geometric positional relationship between the markers MK1 to MK5 based on the pattern PN detected by the detection unit 45. The position / orientation calculation unit 49 may be configured to calculate a position or orientation corresponding to the geometric positional relationship between the markers MK1 to MK5 based on the pattern PN detected by the detection unit 45.
The robot control unit 51 causes the robot 20 to perform a predetermined operation based on the position and orientation calculated by the position / orientation calculation unit 49.

<Process where robot controller performs calibration>
Hereinafter, with reference to FIG. 8, a process in which the robot control apparatus 30 performs calibration will be described. The calibration is calibration that associates the imaging unit coordinate system CC with the robot coordinate system RC. FIG. 8 is a flowchart illustrating an example of a flow of processing in which the robot control device 30 performs calibration. As described above, a case will be described below in which the robot controller 30 has been previously calibrated to associate the calibration board coordinate system with the robot coordinate system RC.

  The detection unit 45 reads the reference pattern stored in advance in the storage unit 32 from the storage unit 32 (step S105). Here, the process of step S105 will be described. The reference pattern is an image to be compared that is used for the detection unit 45 to detect the markers MK1 to MK5 included in the captured image. FIG. 9 is a diagram illustrating an example of a reference pattern. The reference pattern BP shown in FIG. 9 is an example of such a reference pattern. In the reference pattern BP, the marker MK1, the marker MK2, the marker MK3, the marker MK4, and the marker MK5 are arranged so as to have a predetermined relative positional relationship. The positional relationship is the same positional relationship as the geometric positional relationship of each of the markers MK1 to MK5 included in the pattern PN. That is, each of the markers MK1 to MK5 is located at each vertex of the convex hull BCH, which is a convex hull having the same shape as the convex hull CH1 described above.

  In addition, the reference pattern BP is associated with a two-dimensional coordinate system BC that represents the position and orientation according to the geometric positional relationship between the markers MK1 to MK5 included in the reference pattern BP. The position is represented by the position of the origin in the robot coordinate system RC in the two-dimensional coordinate system BC. The posture is represented by the direction in the robot coordinate system RC of each coordinate axis in the two-dimensional coordinate system BC. In addition, although the position of the origin of the two-dimensional coordinate system BC shown in FIG. 9 is located inside the convex hull BCH, it may be configured to be located outside the convex hull BCH.

  More specifically, each of the markers MK1 to MK5 constituting the reference pattern BP is arranged on the reference pattern BP based on the coordinates shown in FIG. FIG. 10 is a table showing the X and Y coordinates of the markers MK1 to MK5 in the two-dimensional coordinate system BC shown in FIG. In the example shown in FIGS. 9 and 10, the positive direction of the X axis in the two-dimensional coordinate system BC coincides with the direction from the marker MK4 to the marker MK5. Further, the positive direction of the Y axis in the two-dimensional coordinate system BC coincides with the direction from the marker MK3 toward the marker MK2. Note that the X-axis positive direction and the Y-axis positive direction in the two-dimensional coordinate system BC may coincide with other directions. The values of the X coordinate and the Y coordinate shown in FIG. 10 are calculated by setting the relative distance between the marker closest to the origin of the two-dimensional coordinate system BC among the markers MK1 to MK5 and the origin of the two-dimensional coordinate system BC as 10. Value. For this reason, the X coordinate and Y coordinate values shown in FIG. 10 do not indicate units.

  After the process of step S105 is performed, the imaging control unit 41 causes the imaging unit 10 to capture an image that can be captured by the imaging unit 10 (step S110). Next, the image acquisition unit 43 acquires the captured image captured by the imaging unit 10 in step S110 from the imaging unit 10 (step S120). Next, the detection unit 45 detects marker candidates that are candidates for each of the markers MK1 to MK5 included in the captured image based on the captured image acquired by the image acquisition unit 43 in step S120 (step S130). Here, the process of step S130 will be described.

  In step S130, the detection unit 45 detects, as a marker candidate, a graphic that can be determined to be the centroid marker shown in FIG. 3 from a plurality of graphics included in the captured image acquired by the image acquisition unit 43 in step S120. . That is, the detection unit 45 detects a combination of figures that can be determined that three or more centroids are within a predetermined range from the plurality of figures as marker candidates. The combination includes noise. In this example, the noise is another marker, figure, pattern, or the like that is erroneously detected by the detection unit 45 from the captured image as the centroid marker shown in FIG. Below, the case where the detection part 45 detects each of marker candidate DP1-marker candidate DP6 which is six marker candidates as an example is demonstrated. In this case, since the markers to be detected are the five markers MK1 to MK5, any one of the marker candidates DP1 to DP6 is noise erroneously detected as a marker candidate. The detection unit 45 may be configured to detect, as a marker candidate, a figure that can be determined as the centroid marker by pattern matching from among a plurality of figures included in the captured image. In this case, the detection unit 45 reads a model image of the centroid marker stored in advance in the storage unit 32, and detects a marker candidate based on the read model image. The detection unit 45 also detects the position (X coordinate and Y coordinate) of the marker candidate in the imaging unit coordinate system CC when detecting the marker candidate.

  Each of the marker candidates DP1 to DP6 shown in FIG. 11 is a marker candidate detected by the detection unit 45 in this way. FIG. 11 is a diagram illustrating an example of a part of the captured image acquired by the image acquisition unit 43. The part is a part where each of the markers MK1 to MK5 is captured in the captured image. An image P1 shown in FIG. 11 is an example of the part. In the example shown in FIG. 11, the marker candidate DP5 is noise detected as the marker candidate described above. Therefore, in FIG. 11, the marker candidate DP5 is represented by a dotted line in order to distinguish it from other marker candidates.

  After the process of step S130 is performed, the detection unit 45 extracts each of combinations of five marker candidates that can be selected from the six marker candidates detected in step S130 as the 0th combination. And the detection part 45 extracts the combination which satisfy | fills a predetermined condition from the extracted 1 or more 0th combination as a 1st combination (step S140). In this example, the predetermined condition is that each of the five marker candidates included in the first combination is located at the vertex of the convex hull. Here, the process of step S140 will be described.

  The detecting unit 45 detects a convex hull based on the five marker candidates included in the 0th combination for each of the extracted 0th combinations. The method by which the detection unit 45 detects the convex hull for each of the extracted 0th combinations may be an existing method or a method to be developed in the future. When the method for detecting the convex hull is an existing method, for example, the detection unit 45 selects one marker candidate satisfying a predetermined selection condition from the 0th combination for each 0th combination. The predetermined selection condition is to satisfy both the conditions 1) and 2) shown below.

1) The marker candidate located at the top of the captured image acquired by the image acquisition unit 43 in step S120 (the position where the Y-axis coordinate is the largest in the imaging unit coordinate system CC) 2) The condition of 1) When there are two or more marker candidates that satisfy the condition, the marker candidate is located on the rightmost (position where the X-axis coordinate is the largest in the imaging unit coordinate system CC) among the two or more marker candidates.

  The predetermined selection condition may be to satisfy other conditions instead of the above conditions 1) and 2). Further, the predetermined selection condition may be to satisfy other conditions in addition to the above conditions 1) and 2). Further, the predetermined selection condition may satisfy only the condition 1) described above.

  The detection unit 45 calculates a vector from the selected marker candidate that is the selected marker candidate to each of the other marker candidates. The detection unit 45 regularizes each calculated vector into a vector of length 1. Here, if the outer product “vector CV1 × vector CV2” of a vector CV1 included in these regularized vectors and a vector CV2 included in the regularized vectors is negative, the marker candidate indicated by the vector CV1 is , The X-axis coordinate in the imaging unit coordinate system CC is smaller than the marker candidate indicated by the vector CV2. Therefore, the detection unit 45 calculates a cross product between the regularized vectors, and determines whether or not the result of the calculated cross product is negative, thereby specifying a marker candidate having the smallest X-axis coordinate. . The detection unit 45 selects the identified marker candidate as the next selection marker candidate. The detection unit 45 repeats such selection of the selection marker candidate and repeats until the selection marker candidate selected first is selected again as the selection marker candidate. Then, the detection unit 45 detects a shape formed when the marker candidates selected as the selection marker candidates are connected by a straight line in the order of selection from the selection marker candidates selected first, that is, a convex hull. The detection unit 45 performs such detection of the convex hull for each of the 0th combinations. The detection unit 45 calculates the outer product of the regularized vectors, and determines whether the calculated result of the outer product is positive, thereby identifying the marker candidate having the largest X-axis coordinate. It may be a configuration.

  Based on the detected convex hull, the detection unit 45 extracts each of the 0th combinations that satisfy the predetermined condition as the first combination. The predetermined condition is that each of the marker candidates included in the 0th combination is located at the vertex of the convex hull detected based on the respective candidates. That is, the detection unit 45 excludes convex hulls having four or less convex hull vertices from each detected convex hull. And the detection part 45 specifies the convex hull which remained without being excluded, ie, the convex hull whose number of vertices is five. The detection unit 45 extracts, for each identified convex hull, a combination of marker candidates constituting the convex hull as a 0th combination that satisfies a predetermined condition, that is, a first combination.

  Here, the 0th combination that satisfies the predetermined condition will be described with reference to FIGS. FIG. 12 is a diagram illustrating an example of a 0th combination that may be determined to be a combination that satisfies a predetermined condition. In the 0th combination shown in FIG. 12, whether or not a predetermined condition is satisfied changes depending on the position in the imaging unit coordinate system CC where the marker candidate DP3 is detected. For example, when the outer product of the marker candidate DP3 and the marker candidate DP2 is 0 or positive, the detection unit 45 performs the marker candidate DP2, the marker candidate DP4, the marker candidate DP5, based on the 0th combination illustrated in FIG. A convex hull having each of the four marker candidates of the marker candidate DP6 as a vertex is detected. The convex hull CH2 shown in FIG. 12 is an example of such a convex hull. In this case, since the number of vertices of the convex hull is 4, the detection unit 45 does not extract the 0th combination illustrated in FIG. 12 as the first combination. On the other hand, when the outer product of the marker candidate DP3 and the marker candidate DP2 is negative, the detection unit 45 detects each of the five marker candidates: the marker candidate DP2, the marker candidate DP3, the marker candidate DP4, the marker candidate DP5, and the marker candidate DP6. Detects the convex hull with the vertex at. In this case, since the number of vertices of the convex hull is 5, the detection unit 45 extracts the 0th combination illustrated in FIG. 12 as the first combination.

  On the other hand, FIG. 13 is a diagram illustrating an example of a 0th combination that is determined to be a combination that satisfies a predetermined condition. In the example illustrated in FIG. 13, the detection unit 45 selects the combination of the marker candidates DP1 to DP4 and the marker candidate DP5 as the 0th combination. In this case, the detection unit 45 detects the convex hull CH3 based on the zeroth combination. As shown in FIG. 13, the number of vertices of the convex hull CH3 is five. For this reason, the detection unit 45 determines that the 0th combination illustrated in FIG. 13 is a combination that satisfies the predetermined condition, and extracts the 0th combination as the first combination.

  In step S140, the detection unit 45 associates an order with each marker candidate included in the first combination for each of the extracted first combinations. Here, taking the first combination shown in FIG. 13 as an example, the process of associating the order with each marker candidate will be described. The detecting unit 45 associates each marker candidate included in the first combination with the order in which each marker candidate is selected as the aforementioned selection marker candidate. In the example illustrated in FIG. 13, the marker candidate that is first selected as the selection marker candidate is the marker candidate DP1. For this reason, the detection unit 45 associates 1 as the number representing the first with respect to the marker candidate DP1. In the example, the marker candidate selected as the second selection marker candidate is the marker candidate DP4. For this reason, the detection unit 45 associates 2 as the number representing the second with respect to the marker candidate DP4. By repeating this, the detection unit 45 associates 3 as the number indicating the third order with the marker candidate DP3, associates 4 with the number indicating the fourth order with the marker candidate DP2, and sets the fifth as the marker candidate DP6. 5 is associated with the number representing the order.

  After the process of step S140 is performed, the detection unit 45 repeatedly performs the processes of step S155 to step S170 for each of the one or more first combinations extracted in step S140 (step S150).

  The detection unit 45 selects first marker candidates that are candidates for the first marker one by one from the marker candidates included in the first combination selected in step S150. And the detection part 45 repeats the process of step S160-step S170 for every selected 1st marker candidate (step S155). More specifically, the detection unit 45 selects in step S150 such that the first marker candidate selected in step S155 matches the first marker included in the reference pattern BP (in this example, the marker MK1). Homography conversion is performed in which each of the five marker candidates included in the first combination is mapped to each of the markers MK1 to MK5 included in the reference pattern BP.

  Next, the detection unit 45 compares the result of the homography conversion performed in step S160 with the reference pattern BP, and uses the result and the reference pattern BP as a value indicating the consistency between the result and the reference pattern BP. Is calculated (step S170). The method for calculating the residual may be a known method or a method developed in the future. Here, with reference to FIG. 14 to FIG. 16, the processing of step S160 to step S170 will be described. Here, the case where the 0th combination shown in FIG. 13 is the first combination selected in step S150 will be described as an example.

  FIG. 14 is a table showing an example of the X coordinate and Y coordinate detected by the detection unit 45 in step S130 and the X coordinate and Y coordinate in the imaging unit coordinate system CC of each of the marker candidates DP1 to DP6.

  FIG. 15 is a diagram illustrating an example of coordinates in the two-dimensional coordinate system BC of each marker candidate included in the first combination after the homography conversion is performed in step S160. In the example illustrated in FIG. 15, the detection unit 45 selects the marker candidate DP1 as the first marker candidate. In this case, in step S170, the detection unit 45 calculates the residual between the homography conversion result shown in FIG. 15 and the reference pattern BP as 3.26.

  On the other hand, FIG. 16 is a diagram illustrating another example of coordinates in the two-dimensional coordinate system BC of each marker candidate included in the first combination after the homography conversion is performed in step S160. In the example illustrated in FIG. 16, the detection unit 45 selects the marker candidate DP6 as the first marker candidate. In this case, in step S170, the detection unit 45 calculates the residual between the homography conversion result shown in FIG. 16 and the reference pattern BP as 0.0158.

  After the residual corresponding to each combination of the first combination and the first marker candidate is calculated by the repeated processing of step S150 to step S170, the detection unit 45 identifies the combination corresponding to the smallest residual To do. In addition, the detection part 45 can exclude the 1st combination which comprises the pattern which does not match with reference | standard pattern BP as shown in FIG. 12 by specifying the said combination based on such a homography transformation. For this reason, even if it is a case where the said 1st combination cannot be excluded in step S140, the detection part 45 can exclude the said 1st combination by the repetition process of step S150-step S170. Then, the detection unit 45 detects five marker candidates included in the first combination in the combination corresponding to the specified smallest residual as marker candidates constituting the pattern PN that is a pattern that matches the reference pattern BP. (Step S180).

  Next, the detection unit 45 determines whether or not the residual corresponding to the first combination including the combination of marker candidates detected in step S180 is less than a predetermined threshold (step S190). When it is determined that the residual is not less than the predetermined threshold value (step S190-NO), the detection unit 45 cannot detect the pattern PN based on the captured image acquired by the image acquisition unit 43 in step S120. judge. Then, the imaging control unit 41 transitions to step S110, and causes the imaging unit 10 to capture again a range that can be imaged by the imaging unit 10. On the other hand, when it is determined that the residual is less than the predetermined threshold (step S190-YES), the detection unit 45 detects (identifies) the five marker candidates detected in step S180 as the markers MK1 to MK5. That is, in this case, the detection unit 45 detects the pattern PN composed of the marker candidate (step S195).

  Next, the detection unit 45 detects a position in the imaging unit coordinate system CC that is a position corresponding to the geometric positional relationship of the markers MK1 to MK5 included in the pattern PN detected in step S195 (step S200). ). Specifically, the detecting unit 45 converts the position of the origin of the two-dimensional coordinate system BC associated with the reference pattern BP in the imaging unit coordinate system CC to homography conversion (inverse homology to the homography conversion performed in step S160). (Graphic conversion). In this example, the pattern PN includes a marker OP. For this reason, the detection unit 45 detects the position of the marker OP in the imaging unit coordinate system CC as a position corresponding to the geometric positional relationship of the markers MK1 to MK5 included in the pattern PN detected in step S195. You can also. In this case, the detection unit 45 detects the marker OP present inside the convex hull CH1 in which each of the markers MK1 to MK5 constituting the pattern PN is located at the apex. For example, the detection unit 45 detects the marker OP by pattern matching. Then, the detecting unit 45 detects the position of the detected marker OP in the imaging unit coordinate system CC as a position corresponding to the geometric positional relationship between the markers MK1 to MK5 included in the pattern PN detected in step S195. . Thereby, the detection part 45 can detect the said position with high precision irrespective of the precision of homography conversion. The detection unit 45 may be configured to detect the marker OP by another method instead of pattern matching.

  Next, the detection unit 45 detects each of the X axis and the Y axis on the pattern PN detected in step S195 and the X axis and the Y axis in the imaging unit coordinate system CC (step S210). Specifically, the detection unit 45 converts the direction of each coordinate axis in the imaging unit coordinate system CC in the two-dimensional coordinate system BC associated with the reference pattern BP to homography conversion (inverse of the homography conversion performed in step S160). Detect by homography transformation). In this example, the pattern PN includes each of the marker XP1, the marker XP2, the marker YP1, and the marker YP2. Therefore, the detection unit 45 is based on each of the marker XP1, the marker XP2, the marker YP1, and the marker YP2, which is the X axis and the Y axis on the pattern PN detected in step S195, and the X in the imaging unit coordinate system CC. Each of the axis and the Y axis can also be detected. In this case, the detection unit 45 detects each of the marker XP1, the marker XP2, the marker YP1, and the marker YP2 existing outside the convex hull CH1 in which each of the markers MK1 to MK5 constituting the pattern PN is located at the apex. The detection unit 45 detects each of the marker XP1, the marker XP2, the marker YP1, and the marker YP2 by pattern matching, for example. And the detection part 45 detects the straight line which passes both the detected marker XP1 and the marker XP2 as the said X-axis. Further, the detection unit 45 detects a straight line passing through both the detected marker YP1 and the marker YP2 as the Y axis. Thereby, the detection part 45 can detect the said X-axis and the said Y-axis with high precision irrespective of the precision of homography conversion.

  Next, the calibration unit 47 uses the imaging unit coordinate system CC based on the pattern PN detected in step S195, the position of the origin detected in step S200, and the X axis and Y axis detected in step S210. Is associated with the robot coordinate system RC (step S220), and the process ends. Specifically, the calibration unit 47 is a coordinate system that represents a position on the robot coordinate system RC and the calibration board CB based on the pattern PN, the position of the origin, and the X axis and the Y axis. Calibration is performed in association with the calibration board coordinate system (in this example, the two-dimensional coordinate system BC). The calibration unit 47 associates the imaging unit coordinate system CC with the robot coordinate system RC based on the result of the calibration and the result of association between the robot coordinate system RC and the calibration board coordinate system.

  As described above, the robot control device 30 changes the reference pattern BP including a predetermined number of markers, each of which is four or more, located at the vertex of the convex hull, from the marker candidates included in the captured image captured by the imaging unit 10. A predetermined number of matching marker candidates are detected. Thereby, the robot control apparatus 30 can detect the position according to four or more markers accurately.

<Processing performed by the robot controller when the robot performs a predetermined work>
Hereinafter, with reference to FIGS. 17 and 18, processing performed by the robot control device 30 when the robot 20 performs a predetermined operation will be described.

  FIG. 17 is a diagram illustrating an example of the configuration of the robot system 1 when the robot 20 performs a predetermined work. The configuration shown in FIG. 17 is different from the configuration shown in FIG. 1 in the object placed on the work table TB. In the configuration shown in FIG. 17, the object O is placed on the upper surface of the work table TB instead of the calibration board CB. The object O is, for example, an industrial part or member such as a plate, screw, or bolt that is assembled to a product. The object O may be other objects such as daily necessities and living bodies instead of industrial parts and members. In the example illustrated in FIG. 1, the object O is represented as a cubic object. The shape of the object O may be another shape instead of the cubic shape.

  A pattern PN2 is provided on the surface of the object O that faces the surface that is in contact with the upper surface of the work table TB. The pattern PN2 is a pattern in which n black circles BC1 to BCn are omitted from the pattern PN. That is, the pattern PN2 includes each of the markers MK1 to MK5, the marker OP, the marker XP1, the marker XP2, the marker YP1, and the marker YP2. The pattern PN2 may be the same pattern as the pattern PN.

  In the example illustrated in FIG. 17, the predetermined operation that the robot control device 30 causes the robot 20 to perform is the operation of gripping the object O and supplying the gripped object O to a supply area (not illustrated). Here, with reference to FIG. 18, a process in which the robot control device 30 causes the robot 20 to perform a predetermined operation will be described. FIG. 18 is a flowchart illustrating an example of a process flow in which the robot control device 30 causes the robot 20 to perform a predetermined operation. Note that the processing in steps S105 to S195 in the flowchart shown in FIG. 17 is the same as the processing in steps S105 to S195 in the flowchart shown in FIG.

  The position / orientation calculation unit 49 detects the position of the origin of the coordinate system representing the position on the pattern PN detected in step S195 and the position in the imaging unit coordinate system CC. Then, the position / orientation calculation unit 49 calculates the position of the object O in the robot coordinate system RC based on the detected position (step S310).

  Next, the position / orientation calculation unit 49 detects each of the X axis and the Y axis on the pattern PN detected in step S195 and the X axis and the Y axis in the imaging unit coordinate system CC. Then, the position / orientation calculation unit 49 calculates, as the orientation of the object O, the detected direction in the robot coordinate system RC in each of the X-axis direction and the Y-axis direction (step S320).

  Next, the robot control unit 51 determines the robot 20 based on the position of the object O in the robot coordinate system RC calculated in step S310 and the posture of the object O in the robot coordinate system RC calculated in step S320. To perform a predetermined operation (step S330), and the process is terminated.

  As described above, the robot control apparatus 30 is based on the captured image obtained by capturing the pattern PN including four or more markers each positioned at the apex of the convex hull by the processing of the flowchart illustrated in FIG. Then, the robot 20 is operated. Thereby, the robot control device 30 can cause the robot 20 to perform a predetermined operation with high accuracy based on the position corresponding to the four or more markers and the position detected from the image captured by the imaging unit. .

<Modification 1 of Embodiment>
Hereinafter, Modification 1 of the embodiment will be described with reference to FIGS. 19 and 20. In the first modification of the embodiment, the robot control apparatus 30 performs the process of step S510 illustrated in FIG. 19 instead of the process of step S155 illustrated in FIG. FIG. 19 is a flowchart illustrating another example of the flow of processing in which the robot control device 30 performs calibration. Note that the processes in steps S105 to S105 and the processes in steps S160 to S220 shown in FIG. 19 are respectively the processes in steps S105 to S105 and the processes in steps S160 to S220 shown in FIG. Since it is the same process as each of the above, description thereof is omitted.

  In step S510, the detection unit 45 specifies a first marker from marker candidates included in the first combination selected in step S150 (step S510). Here, the process of step S510 will be described. The detection unit 45 extracts a combination of two marker candidates that can be selected from marker candidates included in the first combination as a marker candidate pair. The detection unit 45 calculates a distance between two marker candidates included in the marker candidate pair for each of the extracted marker candidate pairs.

  FIG. 20 is a diagram illustrating an example of a distance calculated between the marker candidates included in each marker candidate pair and calculated by the detection unit 45. The detection unit 45 identifies a marker candidate pair corresponding to the longest distance among the distances calculated for each marker candidate pair and a marker candidate pair corresponding to the second longest distance among the distances calculated for each marker candidate pair. To do. And the detection part 45 specifies the marker candidate contained in common with both of two specified marker candidate pairs as a 1st marker. In the example shown in FIG. 20, the distance corresponding to the marker candidate pair including the marker candidate DP6 and the marker candidate DP1 is the longest, and the distance corresponding to the marker candidate pair including the marker candidate DP2 and the marker candidate DP6 is the second. long. And the marker candidate contained in common in both of these marker candidate pairs is marker candidate DP6. Accordingly, the detection unit 45 specifies the marker candidate DP6 as the first marker in the example illustrated in FIG.

  Thus, since the robot control apparatus 30 specifies a 1st marker by the process of step S510, it becomes unnecessary to perform the repetition process of step S155, and performs matching with the imaging part coordinate system CC and robot coordinate system RC. The time required for calibration can be shortened. And the robot control apparatus 30 can acquire the effect similar to embodiment.

<Modification 2 of Embodiment>
Hereinafter, a second modification of the embodiment will be described with reference to FIG. In the second modification of the embodiment, on the calibration board CB, two or more markers XP1 are arranged side by side as the first reference portion XP11 on a straight line connecting the marker XP1 and the marker OP. In the calibration board CB, two or more markers XP2 are arranged side by side as a first reference portion XP12 on a straight line connecting the markers XP2 and the marker OP. In the calibration board CB, two or more markers YP1 are arranged side by side as a second reference portion YP21 on a straight line connecting the markers YP1 and OP. In the calibration board CB, two or more markers YP2 are arranged side by side as a second reference portion YP22 on a straight line connecting the markers YP2 and OP.

  FIG. 21 is a diagram illustrating an example of the calibration board CB according to the second modification of the embodiment. In the example shown in FIG. 21, in the first reference portion XP11, 13 markers XP1 are arranged on a straight line. In the first reference portion XP12, 13 markers XP2 are arranged in a straight line. In the second reference portion YP11, 13 markers YP1 are arranged on a straight line. In the second reference portion YP22, 13 markers YP2 are arranged on a straight line.

  When the calibration board CB according to the second modification of the embodiment is used, the detection unit 45 has the direction in which the markers XP1 are arranged in the first reference portion XP11 and the marker XP2 in the first reference portion XP12. At least one of the directions is detected as the X-axis direction on the pattern PN. In this case, the detecting unit 45 determines at least one of the direction in which the marker YP1 is arranged in the second reference portion YP21 and the direction in which the marker YP2 is arranged in the second reference portion YP22 as Y on the pattern PN. Detect as axial direction.

  When the calibration board CB includes the first reference portion XP11, the first reference portion XP12, the second reference portion YP11, and the second reference portion YP12, the robot control device 30 captures an image captured by the imaging unit 10. Even if only a part of the calibration board CB is included in the image, based on the part, the imaging unit so that the entire pattern PN of the calibration board CB is included in a range that the imaging unit 10 can capture. The direction of the ten optical axes can be changed. However, in this case, a mechanism unit that changes the direction of the optical axis of the imaging unit 10 is provided in the imaging unit 10, or a moving unit that can move the imaging unit 10 (for example, the arm A provided in the robot 20). ) Needs to be equipped with the imaging unit 10. Below, the case where the mechanism part which changes the direction of the optical axis of the imaging part 10 is provided in the imaging part 10 as an example is demonstrated.

  For example, the range including all of the markers MK1 to MK5 in the calibration board CB is not included in the captured image captured by the imaging unit 10, and only the range including the first reference portion XP11 in the calibration board CB. Is included in the captured image, the robot control device 30 detects the first reference portion XP11 from the captured image. Then, the robot control device 30 moves the direction of the optical axis of the imaging unit 10 along the direction in which the markers XP1 included in the detected first reference portion XP11 are arranged. At this time, every time the movement amount of the optical axis reaches a predetermined movement amount, the robot control device 30 causes the imaging unit 10 to capture a range that can be captured by the imaging unit 10. The robot controller 30 moves the optical axis until the captured image captured by the imaging unit 10 includes a range including all of the markers MK1 to MK5 in the calibration board CB. repeat. Thereby, even if only a part of the calibration board CB is included in the captured image captured by the imaging unit 10, the robot control device 30 is within the range that the imaging unit 10 can capture based on the part. It is possible to change the direction of the optical axis of the imaging unit 10 so that the entire pattern PN of the calibration board CB is included.

  As described above, the calibration board CB includes the first reference portion XP11, the first reference portion XP12, the second reference portion YP21, and the second reference portion YP22. Y axis misdetection can be suppressed. Further, even when the calibration board CB according to the second modification of the embodiment is used, the robot control device 30 can obtain the same effects as those of the embodiment.

  In the embodiment described above, the first marker may be larger in size than other markers. Further, the first marker may have a shape different from that of other markers. Moreover, the structure which is a color different from another marker may be sufficient as the color of a 1st marker. Thus, the robot control device 30 can easily detect the first marker.

  As described above, in the calibration board CB, each of the four or more markers (in this example, the markers MK1 to MK5) is the vertex of the convex hull (in this example, the convex hull CH1) that does not have rotational symmetry. Located in. Thereby, the calibration board CB can accurately detect positions corresponding to four or more markers.

  In the calibration board CB, each of the four or more markers is a centroid marker. As a result, the calibration board CB can accurately detect positions corresponding to four or more centroid markers.

  In the calibration board CB, each centroid marker has a figure in which three or more centroids overlap. Thereby, the calibration board CB can detect the centroid marker which has a figure which is four or more markers and three or more centroids overlap from an image without an error.

  In the calibration board CB, the length of the side including the first marker (in this example, the marker MK1) in the convex hull is longer than the length of the other side in the convex hull. Thereby, the calibration board CB can efficiently detect positions corresponding to four or more markers by using the first marker as a reference.

  The calibration board CB includes an origin (in this example, a marker OP) inside the convex hull. As a result, the calibration board CB can more accurately represent the positions corresponding to the four or more markers by the origin.

  Further, in the calibration board CB, a straight line passing through the origin and the first reference portion (in this example, the marker XP1, the marker XP2, the first reference portion XP11, the first reference portion XP12), the origin and the second reference portion (this In one example, a straight line passing through the marker YP1, the marker YP2, the second reference portion YP21, and the second reference portion YP22) is orthogonal. Thus, the calibration board CB can more accurately represent the attitude of the calibration board by the first reference portion and the second reference portion.

  In the calibration board CB, the number of markers is five. As a result, the calibration board CB can accurately detect the positions corresponding to the five markers.

  In addition, the robot 20 has an image (a pattern PN in this example) that is made up of four or more markers each positioned at the apex of the convex hull captured by the imaging unit (the imaging unit 10 in this example). In one example, the operation is based on a captured image. Thereby, the robot 20 can perform a predetermined operation with high accuracy based on a position corresponding to four or more markers and detected from an image captured by the imaging unit.

  In addition, the robot 20 is based on an image in which a pattern including four or more markers each positioned at the vertex of the convex hull is captured by the imaging unit, and a robot coordinate system (in this example, the robot coordinate system RC) Calibration with the coordinate system of the imaging unit (in this example, the imaging unit coordinate system CC) is performed. Thereby, the robot 20 can accurately calculate the position in the coordinate system of the robot, which is a position corresponding to the position of the imaging unit in the coordinate system.

  Further, the robot 20 operates based on an image obtained by capturing an image of a pattern composed of four or more centroid markers each positioned at the apex of the convex hull. As a result, the robot 20 can accurately perform a predetermined operation based on the position corresponding to the four or more centroid markers and detected from the image captured by the imaging unit.

  In addition, the robot 20 operates based on an image in which a pattern made of centroid markers having a figure in which three or more centroids overlap is captured by the imaging unit. Thereby, the robot 20 is a position corresponding to the centroid marker having a figure that is four or more markers and three or more centroids overlapped, and is based on the position detected from the image captured by the imaging unit. Predetermined work can be performed with high accuracy.

  In addition, the robot 20 operates based on an image in which a pattern including four or more markers whose side including the first marker in the convex hull is longer than the length of the other side in the convex hull is captured by the imaging unit. To do. As a result, the robot 20 efficiently detects positions corresponding to four or more markers by using the first marker as a reference from the image captured by the imaging unit, and performs predetermined work based on the detected positions. Can be performed with high accuracy.

  The robot 20 operates based on an image obtained by capturing an image of a pattern including four or more markers each positioned at the apex of the convex hull and having an origin in the convex hull. Thereby, the robot 20 can perform a predetermined operation with high accuracy based on a position represented by an origin provided inside the convex hull and detected from an image captured by the imaging unit.

  In addition, the robot 20 operates based on an image in which a pattern including four or more markers each positioned at the apex of a convex hull provided with a first reference portion and a second reference portion is captured by the imaging unit. . As a result, the robot can accurately detect the posture of the calibration board and perform a predetermined operation based on the detected posture.

  Further, the robot 20 operates based on an image obtained by capturing an image of a pattern including five markers, each of which is located at the apex of the convex hull. Thereby, the robot 20 can perform a predetermined operation with high accuracy based on the positions corresponding to the five markers and detected from the image captured by the imaging unit.

  In addition, the detection method includes a reference pattern composed of a predetermined number of markers, each of which is four or more located at the vertices of a convex hull that does not have rotational symmetry, from candidate markers included in the image captured by the imaging unit. A predetermined number of the candidates that match the above are detected. Thereby, the detection method can detect the position according to four or more markers accurately.

  In addition, the detection method is based on the consistency between the reference pattern and the result of homography conversion of the first pattern including a predetermined number of candidates selected from the marker candidates included in the image captured by the imaging unit. A predetermined number of the candidates are detected. As a result, the detection method can correctly detect a pattern composed of four or more markers even when an incorrect marker is included in the marker candidates included in the image.

  In the detection method, the consistency with the reference pattern is expressed by the difference between the result of homography conversion of the first pattern and the reference pattern. As a result, even if the marker candidate included in the image includes an incorrect marker, the detection method is based on the consistency represented by the difference between the result of homography conversion of the first pattern and the reference pattern. A pattern composed of more than one marker can be detected correctly.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and changes, substitutions, deletions, and the like are possible without departing from the gist of the present invention. May be.

  Further, a program for realizing the function of an arbitrary component in the above-described apparatus (for example, robot control apparatus 30) is recorded on a computer-readable recording medium, and the program is read into a computer system and executed. You may make it do. Here, the “computer system” includes hardware such as an OS (Operating System) and peripheral devices. The “computer-readable recording medium” refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD (Compact Disk) -ROM, or a storage device such as a hard disk built in the computer system. . Furthermore, “computer-readable recording medium” means a volatile memory (RAM) inside a computer system that becomes a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

In addition, the above program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
Further, the above program may be for realizing a part of the functions described above. Further, the program may be a so-called difference file (difference program) that can realize the above-described functions in combination with a program already recorded in the computer system.

DESCRIPTION OF SYMBOLS 1 ... Robot system, 10 ... Imaging part, 20 ... Robot, 30 ... Robot control apparatus, 31 ... CPU, 32 ... Memory | storage part, 33 ... Input reception part, 34 ... Communication part, 35 ... Display part, 36 ... Control part, DESCRIPTION OF SYMBOLS 41 ... Imaging control part, 43 ... Image acquisition part, 45 ... Detection part, 47 ... Calibration part, 49 ... Position and orientation calculation part, 51 ... Robot control part

Claims (18)

With four or more markers,
Each of the four or more markers is located at the apex of a convex hull that does not have rotational symmetry,
Calibration board. Each of the four or more markers is a centroid marker,
The calibration board according to claim 1. Each of the centroid markers has a figure in which three or more centroids overlap.
The calibration board according to claim 2. The length of the side including the first marker in the convex hull is longer than the length of the other side in the convex hull.
The calibration board according to any one of claims 1 to 3. An origin is provided inside the convex hull,
The calibration board according to any one of claims 1 to 4. A first reference portion and a second reference portion are provided outside the convex hull,
A straight line passing through the origin and the first reference portion is orthogonal to a straight line passing through the origin and the second reference portion.
The calibration board according to claim 5. The number of the markers is five.
The calibration board according to any one of claims 1 to 6. With moving parts,
The movable part operates based on an image in which a pattern made of four or more markers each positioned at the apex of a convex hull that does not have rotational symmetry is imaged by the imaging unit.
robot. Based on the image obtained by imaging the pattern by the imaging unit, calibration of the coordinate system of the robot and the coordinate system of the imaging unit is performed.
The robot according to claim 8. Each of the four or more markers is a centroid marker,
The robot according to claim 9. Each of the centroid markers has a figure in which three or more centroids overlap.
The robot according to claim 10. The length of the side including the first marker in the convex hull is longer than the length of the other side in the convex hull.
The robot according to any one of claims 8 to 11. An origin is provided inside the convex hull,
The robot according to any one of claims 8 to 12. A first reference portion and a second reference portion are provided outside the convex hull,
A straight line passing through the origin and the first reference portion;
A straight line passing through the origin and the second reference portion is orthogonal to
The robot according to claim 13. The number of the markers is five.
The robot according to any one of claims 8 to 14. The predetermined number that matches a reference pattern composed of a predetermined number of markers that are four or more positioned at the vertices of a convex hull that does not have rotational symmetry, from candidate markers included in the image captured by the imaging unit Detecting the candidate of
Detection method. Based on the consistency between the reference pattern and the result of homography conversion of the first pattern of the predetermined number of candidates selected from the marker candidates included in the image captured by the imaging unit, the predetermined number of Detecting the candidate,
The detection method according to claim 16. The consistency is represented by a difference between the result of the homography transformation and the reference pattern.
The detection method according to claim 17.

Images