JP6561757B2 - Image processing apparatus, robot, and marker - Google Patents

Description

The present invention relates to an image processing apparatus, robot, relates 及 Beauty marker.

  Research and development of robots that detect markers and objects from images and perform operations according to the detected markers and objects are being conducted.

  In this regard, there is known a two-dimensional code including a marker composed of a first color region and a second color region adjacent to each other and a data matrix composed of binary data cells arranged in a matrix. (See Patent Document 1).

JP2013-84031A

  However, since this two-dimensional code must be detected based on the first color region and the second color region from the captured image including the two-dimensional code, dirt on these regions and changes with time of the regions In some cases, the detection may fail due to external factors such as changes in imaging conditions such as ambient light when the captured image is captured.

In order to solve at least one of the above problems, one embodiment of the present invention is an image processing apparatus for detecting a marker, wherein the marker includes a first graphic having three or more centroids and the first graphic. And a data area provided inside or outside the figure, and after detecting the first figure, the image processing apparatus detects the data area based on the centroid of the first figure.
With this configuration, after detecting the first graphic, the image processing apparatus detects the data area based on the centroid of the first graphic. Thereby, the image processing apparatus can suppress the failure of marker detection due to an external factor for the marker.

According to another aspect of the present invention, in the image processing apparatus, a configuration in which the data area is detected based on a direction passing through the centroid of the first graphic may be used.
With this configuration, the image processing apparatus detects the data area based on the direction passing through the centroid of the first graphic. As a result, the image processing apparatus can detect the data area more reliably based on the direction passing through the centroid of the first graphic.

In another aspect of the present invention, in the image processing apparatus, a configuration in which the direction is changed around the centroid of the first graphic and the data area is detected may be used.
With this configuration, the image processing apparatus detects the data area by changing the direction passing through the centroid of the first graphic around the centroid of the first graphic. Thereby, the image processing apparatus can detect the data area even when the first graphic does not indicate the direction passing through the centroid of the first graphic.

According to another aspect of the present invention, in the image processing apparatus, the data area may include a graphic having a smaller number of overlapping centroids than the first graphic.
With this configuration, after detecting the first graphic, the image processing apparatus detects a data area having a graphic with a small number of centroids that overlap with the first graphic. Accordingly, the image processing apparatus can suppress erroneous detection of the graphic included in the data area as the first graphic.

According to another aspect of the present invention, in the image processing apparatus, the marker includes a second graphic having three or more centroids overlapped, and the data area includes the first graphic and the second graphic. A configuration located between the two may be used.
With this configuration, after detecting the first graphic and the second graphic, the image processing apparatus detects the data area based on the centroid of the first graphic and the centroid of the second graphic. Thereby, the image processing apparatus can detect the data area more reliably.

In another aspect of the present invention, in the image processing apparatus, the data area is detected based on a direction connecting the centroid of the first graphic and the centroid of the second graphic. May be.
With this configuration, the image processing apparatus detects the data area based on the direction connecting the centroid of the first graphic and the centroid of the second graphic. Thereby, the image processing apparatus can detect the data area earlier based on the direction connecting the centroid of the first graphic and the centroid of the second graphic.

Another aspect of the present invention is a robot including the image processing apparatus according to any one of the above.
With this configuration, after detecting the first graphic, the robot detects the data area based on the centroid of the first graphic. Thereby, the robot can suppress the failure of marker detection due to an external factor for the marker.

Moreover, the other aspect of this invention is a robot system provided with the imaging part which images the captured image containing the said marker, and the robot of Claim 7.
With this configuration, after detecting the first graphic, the robot system detects the data area based on the centroid of the first graphic. Thereby, the robot system can suppress the failure of marker detection due to an external factor for the marker.

Another aspect of the present invention is a marker having a first graphic having three or more overlapping centroids and a data area provided outside the first graphic.
With this configuration, the marker detects the first graphic and then detects the data area based on the centroid of the first graphic. Thereby, the marker can suppress the failure of marker detection due to an external factor for the marker.

According to another aspect of the present invention, in the marker, the marker has a second graphic with three or more centroids overlapping, and the data area is located between the first graphic and the second graphic. A configuration may be used.
With this configuration, the marker detects the first graphic and the second graphic, and then detects the data area based on the centroid of the first graphic and the centroid of the second graphic. Thereby, the marker can detect a data area more reliably.

As described above, after detecting the first graphic, the image processing device, the robot, and the robot system detect the data area based on the centroid of the first graphic. Thereby, the image processing apparatus, the robot, and the robot system can suppress a failure in marker detection due to an external factor with respect to the marker.
The marker detects the first graphic and then detects the data area based on the centroid of the first graphic. Thereby, the marker can suppress the failure of marker detection due to an external factor for the marker.

It is a block diagram which shows an example of the robot 20 which concerns on this embodiment. 2 is a diagram illustrating an example of a hardware configuration of a robot control device 30. FIG. 2 is a diagram illustrating an example of a functional configuration of a robot control device 30. FIG. 4 is a flowchart illustrating an example of a flow of processing in which a control unit causes a robot to perform a predetermined operation. It is a figure which shows the specific example of the marker Mk shown in FIG. It is a figure for demonstrating the centroid of a figure. It is a figure which shows the 1st figure F1 as an example of a centroid marker. It is a figure which shows the 2nd figure F2 as another example of a centroid marker. It is a figure which shows an example of the centroid marker with the multiplicity of 3 comprised by three figures which do not comprise the concentric circle. It is a figure which shows an example of the centroid marker with the multiplicity of 4 comprised by the four figures which do not comprise the concentric circle. It is a figure which shows the other example of the centroid marker with the multiplicity of 4 comprised by the four figures which do not comprise the concentric circle. It is a flowchart which shows the flow of the process in which the robot control apparatus 30 specifies which of the 16 types of code figures each of the four code figures detected by the robot control apparatus 30 is. It is a flowchart which shows an example of the flow of the marker detection process in step S120 shown in FIG. It is a figure which illustrates the centroid of the 1st figure F1 which the marker Mk has, the centroid of the 2nd figure F2, and the centroid of each of the four code figures contained in a data area. It is a figure which shows an example of the marker in the modification 1 of embodiment. It is a flowchart which shows an example of the flow of the marker detection process which the image process part 44 performs in the modification 1 of embodiment. FIG. 17 is a diagram for visually explaining an overview of processing in steps S410 to S430 in the flowchart illustrated in FIG. 16. It is a figure which shows an example of the marker in the modification 2 of embodiment. It is a flowchart which shows an example of the flow of the marker detection process which the image process part 44 performs in the modification 2 of embodiment. It is a figure which shows an example of the marker in the modification 3 of embodiment. It is a flowchart which shows an example of the flow of the marker detection process which the image process part 44 performs in the modification 3 of embodiment. It is a figure which shows an example of the mark which shows the data area | region which the 2nd detection process part 51 searches from the circumference | surroundings of the 1st figure F1 in step S420c. It is a figure which shows an example of the marker in the modification 4 of embodiment. It is a flowchart which shows an example of the flow of the marker detection process which the image process part 44 performs in the modification 4 of embodiment. It is a figure which shows an example of the marker Mk2 distorted to such an extent that it can be corrected by affine transformation in the captured image. It is a figure which shows an example of the normal line direction N1 of the marker Mk2 which is not distorted. It is a figure which shows an example of the marker Mk in the picked-up image when the marker Mk is distorted and imaged to such an extent that it can be corrected by affine transformation. It is a table | surface which shows an example of the information regarding the marker Mk3 and the marker Mk4.

<Embodiment>
Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a configuration diagram illustrating an example of a robot 20 according to the present embodiment.

<Configuration of robot 20>
First, the configuration of the robot 20 will be described.
The robot 20 is a double-arm robot including a first arm, a second arm, a support base that supports the first arm and the second arm, and a robot control device 30. The double-arm robot is a robot having two arms (arms) such as the first arm and the second arm in this example. The robot 20 may be a single arm robot instead of the double arm robot. A single arm robot is a robot provided with one arm. For example, the single-arm robot includes one of a first arm and a second arm. Further, the robot 20 may be a multi-arm robot having three or more arms instead of the double-arm robot.

  The first arm includes a first end effector E1 and a first manipulator M1.

  In this example, the first end effector E1 is an end effector including a claw that can grip an object. The first end effector E1 may be another end effector such as an end effector provided with an electric screwdriver, instead of the end effector provided with the claw portion.

  The first end effector E1 is communicably connected to the robot controller 30 via a cable. Thus, the first end effector E1 performs an operation based on the control signal acquired from the robot control device 30. Note that wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB (Universal Serial Bus), for example. The first end effector E1 may be configured to be connected to the robot control device 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  The first manipulator M1 includes seven joints and a first imaging unit 21. Each of the seven joints includes an actuator (not shown). That is, the first arm including the first manipulator M1 is a 7-axis vertical articulated arm. The first arm performs a motion with seven degrees of freedom by a coordinated operation of the support, the first end effector E1, the first manipulator M1, and the actuators of the seven joints included in the first manipulator M1. The first arm may be configured to operate with a degree of freedom of 6 axes or less, or may be configured to operate with a degree of freedom of 8 axes or more.

  When the first arm operates with a degree of freedom of seven axes, the posture that the first arm can take is increased as compared with a case where the first arm operates with a degree of freedom of six axes or less. Thereby, for example, the first arm can be smoothly operated, and interference with an object existing around the first arm can be easily avoided. In addition, when the first arm operates with a degree of freedom of 7 axes, the control of the first arm is easy and requires a smaller amount of calculation than when the first arm operates with a degree of freedom of 8 axes or more.

  Each of the seven actuators (provided at the joints) included in the first manipulator M1 is connected to the robot controller 30 via a cable so as to be communicable. Accordingly, the actuator operates the first manipulator M1 based on the control signal acquired from the robot control device 30. Note that wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. Further, some or all of the seven actuators included in the first manipulator M1 may be connected to the robot control device 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark). Good.

  The first imaging unit 21 is a camera including, for example, a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like, which is an imaging element that converts collected light into an electrical signal. In this example, the first imaging unit 21 is provided in a part of the first manipulator M1. Therefore, the first imaging unit 21 moves according to the movement of the first arm. In addition, the range in which the first imaging unit 21 can capture an image changes according to the movement of the first arm. The first imaging unit 21 may be configured to capture a still image in the range, or may be configured to capture a moving image in the range.

  The first imaging unit 21 is communicably connected to the robot control device 30 via a cable. Wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. The first imaging unit 21 may be configured to be connected to the robot control device 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  The second arm includes a second end effector E2 and a second manipulator M2.

  In this example, the second end effector E2 is an end effector including a claw portion that can grip an object. The second end effector E2 may be another end effector such as an end effector including an electric screwdriver, instead of the end effector including the claw portion.

  The second end effector E2 is communicably connected to the robot controller 30 via a cable. Accordingly, the second end effector E2 performs an operation based on the control signal acquired from the robot control device 30. Note that wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. The second end effector E2 may be configured to be connected to the robot control device 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  The second manipulator M <b> 2 includes seven joints and the second imaging unit 22. Each of the seven joints includes an actuator (not shown). That is, the second arm provided with the second manipulator M2 is a 7-axis vertical articulated arm. The second arm performs an operation with seven degrees of freedom by a coordinated operation of the support, the second end effector E2, the second manipulator M2, and the actuators of the seven joints included in the second manipulator M2. The second arm may be configured to operate with a degree of freedom of 6 axes or less, or may be configured to operate with a degree of freedom of 8 axes or more.

  When the second arm operates with a degree of freedom of 7 axes, the posture that the second arm can take is increased as compared with a case where the second arm operates with a degree of freedom of 6 axes or less. Thereby, for example, the operation of the second arm becomes smooth, and interference with an object existing around the second arm can be easily avoided. Further, when the second arm operates with a degree of freedom of 7 axes, the control of the second arm is easy and requires a smaller amount of calculation than when the second arm operates with a degree of freedom of 8 axes or more.

  Each of the seven actuators (provided at the joints) included in the second manipulator M2 is communicably connected to the robot controller 30 via a cable. Thus, the actuator operates the second manipulator M2 based on the control signal acquired from the robot control device 30. Note that wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. Further, some or all of the seven actuators included in the second manipulator M2 may be configured to be connected to the robot controller 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark). Good.

  The second imaging unit 22 is, for example, a camera provided with a CCD, a CMOS, or the like that is an imaging element that converts the collected light into an electrical signal. In this example, the second imaging unit 22 is provided in a part of the second manipulator M2. Therefore, the second imaging unit 22 moves according to the movement of the second arm. Further, the range that can be imaged by the second imaging unit 22 changes according to the movement of the second arm. The second imaging unit 22 may be configured to capture a still image within the range, or may be configured to capture a moving image within the range.

  The second imaging unit 22 is connected to the robot control device 30 via a cable so as to be communicable. Wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. The second imaging unit 22 may be configured to be connected to the robot control device 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

The robot 20 includes a third imaging unit 23 and a fourth imaging unit 24.
The third imaging unit 23 is, for example, a camera that includes a CCD, a CMOS, or the like that is an imaging element that converts collected light into an electrical signal. The third image pickup unit 23 is provided in a region where the fourth image pickup unit 24 can take an image together with the fourth image pickup unit 24 in a region where stereo image pickup is possible. The 3rd imaging part 23 is connected with the robot control apparatus 30 so that communication is possible with the cable. Wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. The third imaging unit 23 may be configured to be connected to the robot control device 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  The fourth imaging unit 24 is, for example, a camera that includes a CCD, a CMOS, or the like that is an imaging element that converts collected light into an electrical signal. The fourth imaging unit 24 is provided in a region where the third imaging unit 23 can capture images together with the third imaging unit 23 in a region where stereo imaging can be performed. The fourth imaging unit 24 is connected to the robot control device 30 through a cable so as to be communicable. Wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. The fourth imaging unit 24 may be configured to be connected to the robot control device 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  Each of these functional units included in the robot 20 described above acquires a control signal from the robot control device 30 built in the robot 20 in this example. Then, each functional unit performs an operation based on the acquired control signal. Note that the robot 20 may be configured to be controlled by the robot control device 30 installed outside, instead of the configuration incorporating the robot control device 30. In this case, the robot 20 and the robot control device 30 constitute a robot system. Further, the robot 20 may be configured not to include a part of the first imaging unit 21, the second imaging unit 22, the third imaging unit 23, and the fourth imaging unit 24.

  The robot control device 30 operates the robot 20 by transmitting a control signal to the robot 20. As a result, the robot control device 30 causes the robot 20 to perform a predetermined operation.

<Outline of predetermined work performed by the robot 20>
Hereinafter, an outline of predetermined work performed by the robot 20 will be described.
In this example, the robot 20 grips an object placed in a work area that can be worked by both the first arm and the second arm, and supplies the gripped object to a feed area (not shown). The (arrangement) work is performed as a predetermined work. Instead of this, the robot 20 may be configured to perform other work as a predetermined work. Further, the work area may be an area in which work can be performed by either one of the first arm and the second arm.

  In the example shown in FIG. 1, the work area is an area including the upper surface of the work table TB. The work table TB is, for example, a table. The work table TB may be another object such as a floor surface or a shelf instead of the table. An object O to be gripped by the robot 20 is disposed on the upper surface of the work table TB. In addition, the structure by which the 2 or more target object O is arrange | positioned at the upper surface of the worktable TB may be sufficient.

  The object O is, for example, an industrial part or member such as a plate, screw, or bolt that is assembled to a product. In FIG. 1, the object O is represented as a rectangular parallelepiped object for simplification of the drawing. The object O may be other objects such as daily necessities and living bodies instead of industrial parts and members. Further, the shape of the object O may be another shape instead of the rectangular parallelepiped shape.

<Outline of processing performed by robot controller 30>
Hereinafter, an outline of processing performed by the robot control device 30 to cause the robot 20 to perform a predetermined operation will be described.

  The robot control device 30 causes the third imaging unit 23 and the fourth imaging unit 24 to perform stereo imaging using the range including the work area as the imaging range. Then, the robot control device 30 acquires a captured image captured by the third imaging unit 23 and the fourth imaging unit 24 in stereo. The robot control device 30 detects a marker affixed in the work area based on the acquired captured image. Note that the robot control device 30 has a first imaging unit 21, a second imaging unit 22, and a third imaging unit instead of the configuration in which the third imaging unit 23 and the fourth imaging unit 24 perform stereo imaging as the imaging range. 23 and the fourth imaging unit 24 may be configured to perform stereo imaging of the imaging range using any two combinations excluding the combination of the third imaging unit 23 and the fourth imaging unit 24.

  The marker in the present embodiment includes a graphic indicating the position and orientation of the marker and a data area including a graphic indicating other information different from the position and orientation of the marker. The graphic indicating the position and orientation of the marker includes one or more graphic. In this example, a case will be described in which the graphics indicating the position and orientation of the marker are two graphics, a first graphic and a second graphic. The graphic indicating the other information is a code graphic that is a graphic representing a code (symbol) indicating the other information. The data area may include one or more code figures. When a plurality of code figures are included in the data area, each code figure represents a code indicating a part or all of the other information. In this example, four code figures are included in the data area, and each of the four code figures represents a code indicating a part of the other information. A case where a code indicating all of the information is expressed will be described.

  In this example, the code indicates robot operation information indicating the operation of the robot 20. The robot operation information is information indicating, for example, an operation of gripping the object O based on the position and posture of the marker and feeding a material area (not shown) among the operations of the robot 20. Each of the four code figures representing the code indicating the robot operation information is a figure representing a code of several bits. In this example, a case where one code figure represents a 4-bit code will be described. That is, the four code figures represent a 16-bit code in total.

  The codes represented by the four code figures may be information indicating other operations such as an operation of causing the robot 20 to process the object O among the operations of the robot 20 as the robot operation information. The codes represented by the four code figures are information indicating a character string (for example, a comment indicating information on the object O to which the marker is attached) to be displayed on the display of the robot control device 30 instead of the robot operation information. The structure which shows other information may be sufficient.

  The position and posture of the marker Mk indicated by the first graphic and the second graphic included in the marker indicate the position and posture of the object O in the robot coordinate system. Note that the position and orientation of the marker Mk are not the configuration indicating the position and orientation of the object O in the robot coordinate system, but the position of the via point that the robot 20 passes through the TCP of the first arm and the TCP of the second arm. The structure which shows other positions and attitude | positions, such as an attitude | position, may be sufficient.

  Here, the centroid of the first graphic is detected based on the shape of the first graphic, and the centroid of the second graphic is detected based on the shape of the second graphic. For this reason, the centroid of the first graphic and the centroid of the second graphic are more sensitive to changes in ambient light, changes over time of the marker Mk, etc. than in the case where a marker including a colored graphic is detected based on the color. It is hard to be mistakenly detected due to technical factors. Therefore, when the robot control device 30 detects the marker in the process of causing the robot 20 to perform a predetermined operation, the robot control device 30 detects at least the first graphic of the first graphic and the second graphic included in the marker, and then detects the first graphic. Based on the centroid, the data area, that is, four code figures included in the data area are detected. Thereby, the robot control apparatus 30 can suppress the failure of marker detection due to an external factor for the marker. Below, the process in which the robot control apparatus 30 detects the said marker from the captured image containing a marker is demonstrated in detail.

  In this example, the marker affixed in the work area is the marker Mk affixed to the object O arranged on the upper surface of the work table TB in the work area. That is, the robot control device 30 detects the marker Mk from the captured image captured by the third imaging unit 23 and the fourth imaging unit 24. In this detection, the robot controller 30 detects at least the first graphic F1 of the first graphic F1 and the second graphic F2 which are the first graphic and the second graphic of the marker Mk, and then the first graphic F1. Based on the centroid, the data area, that is, four code figures included in the data area are detected. In FIG. 1, the marker Mk is represented by a black square attached to the object O in order to simplify the drawing. Alternatively, the marker Mk may be configured to be affixed to another object different from the object O in the work area, the upper surface of the work table TB, or the like.

  After detecting the data area, the robot control device 30 calculates the position and orientation of the detected marker Mk as the position and orientation of the object O in the robot coordinate system. Then, the robot control device 30 causes the first arm of the robot 20 to grip the calculated object O of the position and orientation based on the code represented by the code figure included in the data area of the marker Mk, and supplies the material (not shown) Feed the area. Instead of this, the robot control device 30 may have a configuration in which the object O is gripped by the second arm, or may be a configuration in which the object O is gripped by both the first arm and the second arm. It may be configured as follows.

<Hardware Configuration of Robot Control Device 30>
Hereinafter, the hardware configuration of the robot control device 30 will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a hardware configuration of the robot control device 30. The robot control device 30 includes, for example, a CPU (Central Processing Unit) 31, a storage unit 32, an input receiving unit 33, a communication unit 34, and a display unit 35. Further, the robot control device 30 communicates with the robot 20 via the communication unit 34. These components are connected to each other via a bus Bus so that they can communicate with each other.

The CPU 31 executes various programs stored in the storage unit 32.
The storage unit 32 includes, for example, a hard disk drive (HDD), a solid state drive (SSD), an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), and a random access memory (RAM). The storage unit 32 may be an external storage device connected via a digital input / output port such as a USB instead of the one built in the robot control device 30. The storage unit 32 stores various information, images, programs, information indicating the position of a material supply area (not shown), etc., which are processed by the robot control device 30.

The input receiving unit 33 is, for example, a teaching pendant provided with a keyboard, a mouse, a touch pad, or the like, or other input device. The input receiving unit 33 may be configured integrally with the display unit 35 as a touch panel.
The communication unit 34 includes, for example, a digital input / output port such as USB, an Ethernet (registered trademark) port, and the like.
The display unit 35 is, for example, a liquid crystal display panel or an organic EL (ElectroLuminescence) display panel.

<Functional Configuration of Robot Control Device 30>
Hereinafter, the functional configuration of the robot control device 30 will be described with reference to FIG. 3. FIG. 3 is a diagram illustrating an example of a functional configuration of the robot control device 30. The robot control device 30 includes a storage unit 32 and a control unit 36.

  The control unit 36 controls the entire robot control device 30. The control unit 36 includes an imaging control unit 40, an image acquisition unit 42, an image processing unit 44, and a robot control unit 46. These functional units included in the control unit 36 are realized, for example, when the CPU 31 executes various programs stored in the storage unit 32. Some or all of these functional units may be hardware functional units such as LSI (Large Scale Integration) and ASIC (Application Specific Integrated Circuit).

The imaging control unit 40 causes the third imaging unit 23 and the fourth imaging unit 24 to perform stereo imaging of the imaging range including the work area.
The image acquisition unit 42 acquires captured images captured by the third imaging unit 23 and the fourth imaging unit 24 in stereo from the third imaging unit 23 and the fourth imaging unit 24.

  The image processing unit 44 performs various types of image processing on the captured image acquired by the image acquisition unit 42. The image processing unit 44 includes a first detection processing unit 50, a second detection processing unit 51, a position / orientation calculation unit 52, and a code detection unit 53.

The first detection processing unit 50, based on the captured image acquired by the image acquisition unit 42, one or more figures (for example, the first in this example) indicating the position and orientation of the marker included in the marker included in the captured image. A graphic and a second graphic) are detected from the captured image.
The second detection processing unit 51 is based on the captured image acquired by the image acquisition unit 42 and one or more graphics indicating the position and orientation of the marker detected by the first detection processing unit 50. One or more code figures included in is detected from the captured image.

The position / orientation calculation unit 52 determines the position and orientation of the marker based on the graphic indicating the position and orientation of the marker detected by the first detection processing unit 50, and the position of the target to which the marker is attached in the robot coordinate system. And calculated as an attitude.
The code detection unit 53 detects a code from one or more code figures included in the data area detected by the second detection processing unit 51.

  The robot control unit 46 causes the robot 20 to perform a predetermined operation based on the position and posture of the marker calculated by the position / orientation calculation unit 52 and the code detected by the code detection unit 53.

<About the flow of processing in which the control unit 36 causes the robot 20 to perform a predetermined work>
Hereinafter, with reference to FIG. 4, a process in which the control unit 36 causes the robot 20 to perform a predetermined operation will be described. FIG. 4 is a flowchart illustrating an example of a process flow in which the control unit 36 causes the robot 20 to perform a predetermined operation. Below, the case where the control part 36 has read the information which shows the position of a material supply area | region from the memory | storage part 32 previously is demonstrated. Note that the control unit 36 may be configured to read information indicating the position of the material supply region from the storage unit 32 at an arbitrary timing in the processing from step S100 to step S130.

  The imaging control unit 40 causes the third imaging unit 23 and the fourth imaging unit 24 to perform stereo imaging of the imaging range including the work area illustrated in FIG. 1 (Step S100). Next, the image acquisition unit 42 acquires, from the third imaging unit 23 and the fourth imaging unit 24, a captured image that is captured in stereo by the third imaging unit 23 and the fourth imaging unit 24 in step S100 (step S110). Next, each functional unit included in the image processing unit 44 executes a marker detection process for detecting the marker Mk included in the captured image based on the captured image acquired by the image acquisition unit 42 in Step S110 (Step S120). .

  Next, the robot control unit 46 acquires the result of the marker detection process in step S120 from the image processing unit 44. Then, based on the result of the marker detection process, the robot control unit 46 causes the first arm to grip the object O and feed it to a feed area (not shown) (step S130).

  Here, the process of step S130 will be described. When the detection of the marker Mk is successful in the marker detection process in step S120, the result of the marker detection process includes the position and orientation of the object O in the robot coordinate system calculated based on the position and orientation of the marker Mk. , And a code represented by four code figures included in the data area of the marker Mk. In this case, the robot control unit 46 causes the first arm to grip the object O based on the position and orientation and the code, and feeds the object O to a feeding area (not shown). Thereafter, the robot control unit 46 ends the process.

  On the other hand, when the detection of the marker Mk has failed in the marker detection process in step S120, the result of the marker detection process includes information indicating that the detection of the marker Mk has failed. In this case, the robot control unit 46 does not control the robot 20 and ends the process. At this time, the robot control unit 46 may be configured to display information indicating that the detection of the marker Mk has failed on the display unit 35.

<Details of marker>
Hereinafter, details of the marker detected by the robot control device 30 from the captured image will be described. The marker in this example (for example, the marker Mk) has a first graphic and a second graphic that are graphic indicating the position and orientation of the marker, and a data area including four code graphic representing codes. In the marker in this example, a data area is arranged between the first graphic and the second graphic. Here, the marker in this example will be described with reference to FIGS.

  FIG. 5 is a diagram showing a specific example of the marker Mk shown in FIG. Similar to the marker Mk shown in FIG. 5 as an example of the marker, the marker in this example is divided into three regions. Therefore, here, the marker Mk will be described as an example. The marker Mk is divided into three regions: a region W1 including a first graphic F1 that is the first graphic of the marker Mk, a region W2 including a second graphic F2 that is the second graphic of the marker Mk, and a data region W3. ing. The data area W3 includes four code figures representing codes. The robot control device 30 detects the first figure F1, the second figure F2, and four code figures representing codes based on the centroid of each figure.

  Here, the centroid of the figure will be described in detail. FIG. 6 is a diagram for explaining the centroid of the figure. FIG. 6 shows a figure F0 whose contour is formed by a curve, a two-dimensional coordinate system for representing the position of each point constituting the figure F0, and a vector r_i indicating the position of each point. . Here, in FIG. 6, the vector r is represented by attaching an arrow on r. Further, “i” added after “_” of the vector r_i indicates the subscript i of the vector r shown in FIG. The subscript i is a label for distinguishing each point (for example, pixel) constituting the figure F0. In FIG. 6, an arrow extending from the origin of the two-dimensional coordinate system to the figure F0 is an arrow indicating the magnitude and direction of the vector r_i.

  The centroid is the center of the figure. Only one centroid is uniquely determined for one figure. The position of the centroid of the figure F0 shown in FIG. 6 is represented by the vector c defined by the following equation (1) using the vector r_i.

  In equation (1), the vector c is expressed by attaching an arrow on c. N is the number of points constituting the graphic F0 and is an integer of 1 or more. That is, the position of the centroid of the figure F0 shown in FIG. 6 is obtained by a vector (average of N vectors) obtained by dividing the sum of vectors representing the positions of the points constituting the figure F0 by the number of the points. expressed. In FIG. 6, the centroid has been described using the two-dimensional figure F0, the two-dimensional coordinate system, and the two-dimensional vector. However, the position of the centroid is not limited to three-dimensional figures or more. It is represented by the vector c defined by the above equation (1).

  Here, each of the first graphic and the second graphic in this example is a centroid marker that is a graphic constituted by a combination of three or more graphics. The centroid marker is a marker in which the centroids of each of the three or more figures constituting the centroid marker fall within a predetermined range. For example, the centroid marker constituted by the combination of the figure A0, the figure B0, and the figure C0 has a predetermined range between the figure A1 of the figure A0, the centroid B1 of the figure B0, and the centroid C1 of the figure C0. (Contains). The predetermined range is, for example, a circular range having a radius of about several pixels, but may be a circular range having a radius of less than one pixel, or may be a circular range having a radius of several pixels or more, It may be a range of another shape different from a circle such as a rectangle. Hereinafter, for convenience of explanation, the number of centroids included in a predetermined range will be referred to as multiplicity. That is, the centroid marker configured by the combination of the figure A0, the figure B0, and the figure C0 is a centroid marker with a multiplicity of 3. Note that either one or both of the first graphic and the second graphic may be replaced by a graphic constituted by one or more graphics.

  FIG. 7 is a diagram illustrating the first graphic F1 as an example of a centroid marker. As shown in FIG. 7, three or more figures constituting the centroid marker are detected as partial areas of only white or black in the binarized image. The first figure F1 includes a figure F11 represented by a white partial area having a circular outline, a figure F12 represented by a black partial area having a ring shape, and a white partial area having a ring shape. The figure is composed of four figures: a figure F13 represented and a figure F14 whose contour is represented by a black partial region having a ring shape. The first graphic F1 is formed such that the centroids of the graphic F11, the graphic F12, the graphic F13, and the graphic F14 fall within a predetermined range. For this reason, the first graphic F1 configured by a combination of the graphic F11, the graphic F12, the graphic F13, and the graphic F14 is a centroid marker with a multiplicity of 4. In addition, the figures F11 to F14 in the first figure F1 are concentric circles, and the centroid is located at the center of the concentric circles.

  FIG. 8 is a diagram showing the second figure F2 as another example of the centroid marker. The second figure F2 includes a figure F21 represented by a black partial area having a circular outline, a figure F22 represented by a white partial area having a ring shape, and a black partial area having a ring shape. It is comprised by three figures with the figure F23 represented. The second graphic F2 is formed such that the centroids of the graphic F21, the graphic F22, and the graphic F23 are within a predetermined range. For this reason, the second graphic F2 configured by a combination of the graphic F21, the graphic F22, and the graphic F23 is a centroid marker with a multiplicity of 3. In addition, the figures F21 to F23 in the second figure F2 are concentric circles, and the centroid is located at the center of the concentric circles.

  In the first graphic F1 and the second graphic F2 shown in FIGS. 7 and 8, a combination of three or more graphics constituting the centroid marker constitutes a concentric circle. However, the centroid marker is not limited to one in which a combination of three or more figures constituting the centroid marker constitutes a concentric circle. The centroid marker shown in FIGS. 9 to 11 is an example in the case where a combination of three or more figures constituting the centroid marker does not constitute a concentric circle. FIG. 9 is a diagram illustrating an example of a centroid marker having a multiplicity of 3 constituted by three figures that do not constitute concentric circles. FIG. 10 is a diagram illustrating an example of a centroid marker having a multiplicity of 4 configured by four figures that do not form concentric circles. FIG. 11 is a diagram illustrating another example of a centroid marker having a multiplicity of 4 configured by four figures that do not constitute concentric circles.

  In this example, the position and orientation of an object (for example, the object O shown in FIG. 1) to which a marker having such a centroid marker as the first graphic or the second graphic is attached is the first graphic of the marker. Expressed by the position and orientation of the centroid. The position of the centroid of the first graphic is represented by a vector (average of three or more vectors) obtained by dividing the sum of vectors indicating the positions of three or more centroids included in a predetermined range by the number of centroids. The Alternatively, the position of the object to which the marker is attached may be represented by another position (offset position) associated with the position of the centroid of the first graphic instead. Good. Further, the position and posture of the object to which the marker having the centroid marker as the first graphic or the second graphic is attached may be represented by the position and posture of the centroid of the second graphic of the marker. .

  The orientation of the centroid of the first graphic is determined based on the centroid of the first graphic and the centroid of the second graphic, with the normal direction to the first graphic at the position of the centroid as the Z axis. Each of the three coordinate axes with the X axis as the direction determined based on the centroid of the first graphic and the graphic contained in the data area and the Y axis as the direction perpendicular to the Z axis and the X axis. It is represented by the direction in the robot coordinate system. Alternatively, the posture of the object to which the marker is attached may be represented by another posture (offset posture) associated with the posture of the centroid of the first graphic instead. Good.

  Next, four code figures representing codes will be described. In this example, any one of the four types of code figures FD1 to FD16 shown in FIG. 12 is used for each of these four code figures. FIG. 12 is a flowchart showing a flow of processing in which the robot control device 30 specifies which of the 16 types of code graphics each of the four code graphics detected by the robot control device 30 is. If the robot control device 30 can distinguish the code graphic, a configuration in which another code graphic is included instead of a part or all of the 16 types of code graphic shown in FIG. It may be. Further, the number of code figures may be less than 15 instead of 16, and may be 17 or more. Further, the robot control device 30 replaces each detected code figure with any of the 16 types of code figures shown in FIG. 12 by performing other processes such as pattern matching instead of the process of the flowchart shown in FIG. It may be configured to specify whether or not there is.

  In addition, as shown in FIG. 12, each of the 16 types of code figures in this example is a figure represented by a black partial area in an area in which 32 square cells are arranged in 8 rows and 4 columns, And a figure represented by a white partial area surrounded by the partial areas. Further, in each of the code figures, among the 32 cells arranged in the 8 rows and 4 columns, all the 8 cells included in the fourth column from the left indicated by the arrow A11 in FIG. 12 are all white. In this example, the robot control device 30 does not detect a white partial region including white cells included in the fourth column in the code graphic as a graphic constituting the code graphic.

  First, the robot control apparatus 30 detects the centroid of the figure which comprises the object code figure, after detecting the object code figure which is the code figure used as the object which specifies which figure FD1-figure FD16. Then, the robot controller 30 determines whether the multiplicity of the centroid detected in the area of 8 rows and 4 columns is 1 or 2 (step S200). By this determination, the robot control device 30 determines whether the target code figure is any one of the figures FD1 to FD3 and whether the target code figure is any one of the figures FD4 to FD16. To do.

  When it is determined in step S200 that the multiplicity of centroids is 2 (step S200-2), the robot control device 30 specifies that the target code figure is any one of the figures FD1 to FD3. Then, the robot control device 30 determines whether the area of the black partial region of the target code figure is large, medium, or small (step S210).

  Here, the process of step S210 will be described. In step S210, when the number of black cells representing the area of the black partial region in the target code figure is less than 18, the robot control apparatus 30 determines that the area is small. In addition, when the number of black cells representing the area of the black partial region in the target code figure is 18 or more and less than 22, the robot control device 30 determines that the area is medium. The robot control device 30 determines that the area is large when the number of black cells representing the area of the black partial region in the target code graphic is 22 or more.

  In this example, the figure FD1 is determined to have a small area in step S210 because the number of black cells is ten. Note that the graphic FD1 is composed of two graphics as shown in FIG. 12, and the multiplicity is 2 because the centroid of each graphic is within a predetermined range. Since the figure FD2 has 18 black cells, it is determined that the area is medium in step S210. Note that the graphic FD2 is composed of two graphics as shown in FIG. 12, and the multiplicity is 2 because the centroid of each graphic is within a predetermined range. Since the figure FD3 has 22 black cells, it is determined in step S210 that the area is large. Note that the graphic FD3 is composed of two graphics as shown in FIG. 12, and the multiplicity is 2 because the centroid of each graphic is within a predetermined range.

  That is, when it is determined that the area of the black partial region of the target code graphic is small (step S210—small), the robot control device 30 specifies that the target code graphic is the graphic FD1. On the other hand, when it is determined that the area of the black partial region of the target code graphic is medium (in step S210-), the robot control device 30 specifies that the target code graphic is the graphic FD2. on the other hand. When it is determined that the area of the black partial region of the target code graphic is large (step S210—large), the robot control device 30 specifies that the target code graphic is the graphic FD3.

  On the other hand, when it is determined in step S200 that the multiplicity of the centroid is 1 (step S200-1), the robot control device 30 specifies that the target code figure is any one of the figures FD4 to FD16. Then, the robot controller 30 determines whether the number of centroids of the target code figure detected in step S200 is 1 or 2 (step S220). By this determination, the robot control device 30 determines whether the target code figure is any one of the figures FD4 to FD10 and whether the target code figure is any one of the figures FD11 to FD16. To do.

  If it is determined in step S220 that the number of centroids is 1 (step S220-1), the robot control device 30 specifies that the target code figure is any one of the figures FD4 to FD10. Then, the robot control device 30 determines whether the area of the black partial region of the target code figure is large, medium, or small (step S230).

  Here, the process of step S230 will be described. In step S230, when the number of black cells representing the area of the black partial region in the target code figure is less than 8, the robot control apparatus 30 determines that the area is small. In addition, when the number of black cells representing the area of the black partial region in the target code figure is 8 or more and less than 16, the robot control device 30 determines that the area is medium. In addition, when the number of black cells representing the area of the black partial region in the target code figure is 16 or more, the robot control device 30 determines that the area is large.

  In this example, the figure FD4 and the figure FD5 have a small area in step S230 because the number of black cells is less than eight. The number of black cells in the figure FD4 is 2, and the number of black cells in the figure FD5 is 6. Further, the graphic FD4 and the graphic FD5 are composed of one graphic as shown in FIG. Since the figures FD6 and FD7 have 8 or more and less than 16 black cells, the area is determined to be medium in step S230. Note that the number of black cells in the figure FD6 is 12, and the number of black cells in the figure FD7 is 8. Further, as shown in FIG. 12, the graphic FD6 and the graphic FD7 are composed of one graphic, and the multiplicity is 1. Since the figures FD8 to FD10 have 16 or more black cells, the area is determined to be large in step S230. Note that the number of black cells in the figure FD8 is 16, the number of black cells in the figure FD9 is 20, and the number of black cells in the figure FD10 is 24. Further, the figures FD8 to FD10 are composed of one figure as shown in FIG.

  That is, in step S230, when it is determined that the area of the black partial region of the target code figure is small (step S230-small), the robot control device 30 determines whether the target code figure is one of the figures FD4 and FD5. Identifies it. Then, the robot controller 30 determines whether the aspect ratio of the black partial area of the target code figure is large or small (step S240).

  Here, the process of step S240 will be described. In step S240, the robot controller 30 calculates an aspect ratio obtained by dividing the number of black cells arranged in the vertical direction of the black partial area in the target code figure by the number of black cells arranged in the horizontal direction. . In this example, the vertical direction is a direction in which eight cells are arranged in each column of 32 cells arranged in 8 rows and 4 columns, and the horizontal direction is 32 cells arranged in 8 rows and 4 columns. This is a direction in which four cells are arranged in each row of cells. When the calculated aspect ratio is 1 or more, the robot controller 30 determines that the aspect ratio is large. Further, when the calculated aspect ratio is less than 1, the robot control apparatus 30 determines that the aspect ratio is small.

  In this example, in the figure FD4, the number of black cells arranged in the vertical direction is 2, and the number of black cells arranged in the horizontal direction is 1, so that it is determined in step S240 that the aspect ratio is large. . In the figure FD5, since the number of black cells arranged in the vertical direction is 2, and the number of black cells arranged in the horizontal direction is 3, it is determined in step S240 that the aspect ratio is small.

  That is, in step S240, when it is determined that the aspect ratio of the black partial region of the target code graphic is large (step S240-large), the robot control device 30 specifies that the target code graphic is the graphic FD4. On the other hand, when it is determined that the aspect ratio of the black partial area of the target code graphic is small (step S240—small), the robot control device 30 specifies that the target code graphic is the graphic FD5.

  On the other hand, if it is determined in step S230 that the area of the black partial region of the target code figure is medium (in step S230-in), the robot control device 30 determines whether the target code figure is one of the figures FD6 and FD7. Identifies it. Then, the robot control apparatus 30 determines the size of the aspect ratio of the black partial area of the target code figure (step S250).

  Here, the process of step S250 will be described. In step S250, the robot controller 30 calculates an aspect ratio obtained by dividing the number of black cells arranged in the vertical direction of the black partial area in the target code figure by the number of black cells arranged in the horizontal direction. . When the calculated aspect ratio is 8, the robot controller 30 determines that the aspect ratio is large. Further, when the calculated aspect ratio is less than 8, the robot control apparatus 30 determines that the aspect ratio is small.

  In this example, in the figure FD7, the number of black cells arranged in the vertical direction is 4, and the number of black cells arranged in the horizontal direction is 3, so it is determined in step S250 that the aspect ratio is small. . In the figure FD8, the number of black cells arranged in the vertical direction is 8, and the number of black cells arranged in the horizontal direction is 1. Therefore, it is determined in step S250 that the aspect ratio is large.

  That is, when it is determined in step S250 that the aspect ratio of the black partial area of the target code graphic is small (step S250-small), the robot control device 30 specifies that the target code graphic is the graphic FD7. On the other hand, when it is determined that the aspect ratio of the black partial area of the target code graphic is large (step S250—large), the robot control device 30 specifies that the target code graphic is the graphic FD8.

  On the other hand, when it is determined in step S230 that the area of the black partial region of the target code figure is large (step S230-large), the robot control apparatus 30 determines that the target code figure is any one of the figures FD8 to FD10. Identifies it. Then, the robot control device 30 determines whether the area of the black partial region of the target code figure is large, medium, or small (step S260).

  Here, the process of step S260 will be described. In step S260, when the number of black cells representing the area of the black partial region in the target code figure is less than 20, the robot control apparatus 30 determines that the area is small. In addition, when the number of black cells representing the area of the black partial region in the target code figure is 20 or more and less than 24, the robot control device 30 determines that the area is medium. The robot control device 30 determines that the area is large when the number of black cells representing the area of the black partial region in the target code graphic is 24 or more.

  In this example, since the figure FD8 has 16 black cells, the area is determined to be small in step S260. Since the figure FD9 has 20 black cells, it is determined in step S260 that the area is medium. Since the figure FD10 has 24 black cells, it is determined in step S230 that the area is large.

  That is, in step S260, when it is determined that the area of the black partial region of the target code figure is small (step S260-small), the robot control device 30 specifies that the target code figure is the figure FD8. On the other hand, when it is determined that the area of the black partial region of the target code graphic is medium (in step S260-), the robot control device 30 specifies that the target code graphic is the graphic FD9. On the other hand, when it is determined that the area of the black partial region of the target code graphic is large (step S260—large), the robot control device 30 specifies that the target code graphic is the graphic FD10.

  On the other hand, when it is determined in step S220 that the number of centroids is 2 (step S220-2), the robot control device 30 specifies that the target code figure is any one of the figures FD11 to FD16. Then, the robot controller 30 determines that the area of the first column from the left in the 32 cells arranged in 8 rows and 4 columns of the area of the black partial region of the target code figure is the area of the third column from the left side. Or the areas of the first and third columns from the left side are the same size, or the area of the third column from the left side is larger than the area of the first column from the left side. It is determined whether or not there is (step S270).

  Here, the process of step S270 will be described. In step S270, the robot controller 30 counts the number of black cells constituting the area of the first column from the left in the 32 cells arranged in 8 rows and 4 columns of the area of the black partial region of the target code figure. Is larger than the number of black cells constituting the area of the third column from the left side, from the left side of the 32 cells arranged in 8 rows and 4 columns of the area of the black partial region of the target code figure. It is determined that the area of the first column is larger than the area of the third column from the left side. Further, the robot control device 30 includes the number of black cells constituting the area of the first column from the left in the 32 cells arranged in 8 rows and 4 columns of the area of the black partial region of the target code figure, If the number of black cells constituting the area of the third column from the left side is the same, one column from the left side of the 32 cells arranged in 8 rows and 4 columns of the area of the black partial region of the target code figure It is determined that the areas of the eyes and the third column have the same size. Further, the robot control apparatus 30 has the number of black cells constituting the area of the third column from the left in the 32 cells arranged in 8 rows and 4 columns of the area of the black partial region of the target code figure. If the number of black cells constituting the area of the first column from the left side is larger, three columns from the left side of the 32 cells arranged in 8 rows and 4 columns out of the area of the black partial region of the target code figure It is determined that the eye area is larger than the area of the first column from the left side.

  In this example, the figure FD11 and the figure FD12 have the number of black cells constituting the area of the first column from the left in the 32 cells arranged in 8 rows and 4 columns in the area of the black partial region. Since the number of black cells constituting the area of the third column from the left side is larger, the area of the first column from the left side in the 32 cells arranged in 8 rows and 4 columns of the area of the black partial region is It is determined that the area is larger than the area in the third column from the left side. In the figure FD11, the number of black cells constituting the area of the first column from the left in the 32 cells arranged in 8 rows and 4 columns of the area of the black partial region is 2, and the left side The number of black cells constituting the area of the third column from the first is eight. In the figure FD12, the number of black cells constituting the area of the first column from the left side in the 32 cells arranged in 8 rows and 4 columns in the area of the black partial region is 6, and the left side The number of black cells constituting the area of the third column from the first is eight. Further, the figure FD11 and the figure FD12 are composed of two figures, and the centroid of each figure is not within a predetermined range, so the multiplicity is 1 and the number of centroids is 2.

  The figure FD13 and the figure FD14 are the number of black cells constituting the area of the first column from the left side in the 32 cells arranged in 8 rows and 4 columns of the area of the black partial region, and three columns from the left side. Since the number of black cells constituting the area of the eye is the same, each area of the first and third columns from the left in 32 cells arranged in 8 rows and 4 columns of the area of the black partial region is It is determined that they are the same size. In the figure FD13, the number of black cells constituting the area of the first column from the left in the 32 cells arranged in 8 rows and 4 columns of the area of the black partial region is 2, and the left side The number of black cells constituting the area of the third column from the second is 2. In the figure FD14, the number of black cells constituting the area of the first column from the left in the 32 cells arranged in 8 rows and 4 columns in the area of the black partial region is 8, and the left side The number of black cells constituting the area of the third column from the first is eight. Further, the figure FD13 and the figure FD14 are composed of two figures, and since the centroid of each figure is not within a predetermined range, the multiplicity is 1 and the number of centroids is 2.

  In the figures FD15 and FD16, the number of black cells constituting the area of the third column from the left side in 32 cells arranged in 8 rows and 4 columns in the area of the black partial region is one column from the left side. Since there are more black cells constituting the area of the eye, the area of the third column from the left in 32 cells arranged in 8 rows and 4 columns of the area of the black partial region is one column from the left side. It is determined that the area is larger than the eye area. In the figure FD15, the number of black cells constituting the area of the first column from the left in the 32 cells arranged in 8 rows and 4 columns in the area of the black partial region is 8, and the left side The number of black cells constituting the area of the third column from the second is 2. In the figure FD16, the number of black cells constituting the area of the first column from the left side in the 32 cells arranged in 8 rows and 4 columns in the area of the black partial region is 8, and the left side The number of black cells constituting the area of the third column from the first row is 6. Further, the figure FD15 and the figure FD16 are composed of two figures, and the centroid of each figure is not within a predetermined range, so the multiplicity is 1 and the number of centroids is 2.

  That is, in step S270, it was determined that the area of the first column from the left in the 32 cells arranged in 8 rows and 4 columns of the area of the black partial region is larger than the area of the third column from the left side. In the case (step S270-3 column size), the robot control device 30 specifies that the target code figure is either the figure FD11 or the figure FD12. Then, the robot control device 30 determines the size of the area of the black partial region of the target code figure (step S280).

  Here, the process of step S280 will be described. In step S280, when the number of black cells representing the area of the black partial region in the target code figure is less than 18, the robot control apparatus 30 determines that the area is small. The robot control device 30 determines that the area is large when the number of black cells representing the area of the black partial region in the target code graphic is 18 or more.

  In this example, the figure FD11 is determined to have a small area in step S280 because the number of black cells is ten. Since the figure FD12 has 18 black cells, the area is determined to be large in step S280.

  That is, when it is determined in step S280 that the area of the black partial region of the target code graphic is small (step S280-small), the robot control device 30 specifies that the target code graphic is the graphic FD11. On the other hand, when it is determined that the area of the black partial region of the target code graphic is large (step S280—large), the robot control device 30 specifies that the target code graphic is the graphic FD12.

  On the other hand, when it is determined that the areas of the first column and the third column from the left in 32 cells arranged in 8 rows and 4 columns of the area of the black partial region have the same size (step S270- The robot control device 30 specifies that the target code figure is either the figure FD13 or the figure 14. Then, the robot control device 30 determines the size of the area of the black partial region of the target code figure (step S290).

  Here, the process of step S290 will be described. In step S290, when the number of black cells representing the area of the black partial region in the target code figure is less than 16, the robot control apparatus 30 determines that the area is small. In addition, when the number of black cells representing the area of the black partial region in the target code figure is 16 or more, the robot control device 30 determines that the area is large.

  In this example, the figure FD13 is determined to have a small area in step S290 because the number of black cells is four. Since the figure FD14 has 16 black cells, the area is determined to be large in step S290.

  That is, if it is determined in step S290 that the area of the black partial region of the target code graphic is small (step S290-small), the robot control device 30 specifies that the target code graphic is the graphic FD13. On the other hand, when it is determined that the area of the black partial region of the target code graphic is large (step S290—large), the robot control device 30 specifies that the target code graphic is the graphic FD14.

  On the other hand, when it is determined that the area of the third column from the left in the 32 cells arranged in 8 rows and 4 columns of the area of the black partial region is larger than the area of the first column from the left (step S270). (First column size), the robot control device 30 specifies that the target code figure is either the figure FD15 or the figure FD16. Then, the robot control device 30 determines the size of the area of the black partial region of the target code figure (step S295).

  Here, the process of step S295 will be described. In step S295, when the number of black cells representing the area of the black partial region in the target code figure is less than 18, the robot control apparatus 30 determines that the area is small. The robot control device 30 determines that the area is large when the number of black cells representing the area of the black partial region in the target code graphic is 18 or more.

  In this example, the figure FD15 is determined to have a small area in step S295 because the number of black cells is ten. Since the figure FD16 has 18 black cells, the area is determined to be large in step S295.

  That is, if it is determined in step S295 that the area of the black partial region of the target code graphic is small (step S295-small), the robot control device 30 specifies that the target code graphic is the graphic FD15. On the other hand, when it is determined that the area of the black partial region of the target code graphic is large (step S295-large), the robot control device 30 specifies that the target code graphic is the graphic FD16.

  As described above, the robot control device 30 can specify which of the 16 types of code figures is the target code figure. For this reason, by considering the figures FD1 to FD16 as hexadecimal numbers 1 to F in order, one code figure can represent 4-bit information. That is, the four code figures included in the data area can represent a 16-bit code.

  Further, in the marker (for example, marker Mk) in this example, each of the four code figures included in the data area is arranged on a straight line passing through the centroid of the first figure and the centroid of the second figure. Arranged so that each centroid is located. By doing in this way, when there are a plurality of figures between the first figure candidate and the second figure candidate, the robot controller 30 determines whether or not these figures are markers in this example. It is possible to suppress the erroneous detection that a combination of figures that are not the marker is a marker.

  As described above, the marker in this example includes the first graphic and the second graphic, and the data area including the four code graphics. Accordingly, after the first graphic is detected, the marker can detect the data area based on the centroid of the first graphic. As a result, the marker detection failure due to an external factor for the marker can be detected. Can be suppressed.

<Outline of centroid marker detection method>
Hereinafter, a method for detecting a centroid marker by the robot controller 30 according to the present embodiment will be described.
In this example, in the centroid marker detection method by the robot control device 30, when detecting the centroid marker from the captured image, the robot control device 30 converts the captured image into a grayscale image (grayscale processing). . For example, the robot controller 30 converts the captured image into a grayscale image of 256 gradations in the grayscale processing of the captured image. Note that the robot control device 30 may be configured to convert the captured image into a grayscale image with other gradations instead of being converted into a grayscale image with 256 gradations.

  Then, the robot control device 30 converts the grayscale image obtained by converting the captured image by the grayscale processing into a binarized image (monochrome image) (binarization processing). In the binarization process, the color of a pixel having a luminance value equal to or higher than a predetermined threshold (binarization threshold) on a grayscale image is converted to white, and the pixel having a luminance value less than the threshold is converted to black. This is processing for converting a grayscale image into a binarized image. In this example, the binarization threshold is determined in advance. The binarization threshold may be another configuration determined by a method based on a captured image such as the Otsu method. The robot controller 30 detects the centroid marker from the binarized image obtained by converting the grayscale image by the binarization process.

<Flow of marker detection process>
Hereinafter, the marker detection process in step S120 shown in FIG. 4 will be described with reference to FIG. FIG. 13 is a flowchart showing an example of the flow of marker detection processing in step S120 shown in FIG.

  The image processing unit 44 reads out marker information stored in advance in the storage unit 32 from the storage unit 32 (step S300). In this example, the marker information includes at least information indicating the marker and information indicating each of the first graphic and the second graphic included in the marker. For example, the marker information includes information in which information indicating the marker Mk is associated with information indicating each of the first graphic F1 and the second graphic F2 included in the marker Mk.

  In this example, the information indicating the marker is a marker ID. Alternatively, the information indicating the marker may be other information as long as it is information that can identify the marker such as a marker name. In this example, the information indicating each of the first graphic and the second graphic is a model image used for detecting each of the first graphic and the second graphic by pattern matching. Note that the information indicating each of the first graphic and the second graphic may be other information that can be used for processing for detecting each of the first graphic and the second graphic from the captured image instead. Good. Note that the processing in step S300 may be performed by any of the functional units illustrated in FIG. 3 included in the image processing unit 44, and functions that are not illustrated in FIG. The structure which a part performs may be sufficient.

  Next, each functional unit included in the image processing unit 44 selects one marker ID included in the marker information read from the storage unit 32 in step S300 one by one, and uses the marker indicated by the selected marker ID as a target marker. The processing from step S320 to step S370 is repeated for each marker (step S310). Hereinafter, for convenience of explanation, a case will be described in which the marker ID included in the marker information is only one of the marker IDs indicating the marker Mk. In this case, the image processing unit 44 performs the processing from step S320 to step S370 only once for the marker Mk.

  Next, the first detection processing unit 50 obtains information indicating each of the first graphic F1 and the second graphic F2 associated with the marker ID indicating the marker Mk selected as the target marker in step S310 from the marker information. Extract. And the 1st detection process part 50 is based on the extracted information which shows each of the said 1st figure F1 and the 2nd figure F2, and the detection method of the centroid marker mentioned above in step S110 shown in FIG. The first graphic F1 and the second graphic F2 are detected from the captured image acquired by the image acquisition unit 42 (step S320).

  Here, the process of step S320 will be described. The first detection processing unit 50 detects a centroid marker having a multiplicity of 3 or more from the captured image as a candidate for the first graphic F1 and the second graphic F2. The multiplicity (number of overlapping centroids) of the four code figures included in the data area is 2 at the maximum, and the multiplicity is less than that of each of the first figure F1 and the second figure F2. For this reason, the first detection processing unit 50 is prevented from erroneously detecting these code figures as candidates for the first figure F1 and the second figure F2. The first detection processing unit 50 has a multiplicity of 3 or more detected as candidates for the first graphic F1 and the second graphic F2 based on the model images of the first graphic F1 and the second graphic F2 extracted from the marker information. The first graphic F1 and the second graphic F2 are detected from the centroid marker by pattern matching. Since the conventionally known method is used for the pattern matching process, the description thereof is omitted.

  Next, the second detection processing unit 51 detects the data area of the marker Mk based on the first graphic F1 and the second graphic F2 detected by the first detection processing unit 50 from the captured image in step S320 (step S320). S330). Here, the process of step S330 will be described with reference to FIGS. FIG. 14 is a diagram illustrating the centroid of the first graphic F1, the centroid of the second graphic F2, and the centroid of each of the four code graphics included in the data area of the marker Mk. In FIG. 14, the four code figures are arranged in the order of the figure FD5, the figure FD10, the figure FD16, and the figure FD1 from the first figure F1 to the second figure F2.

  Further, in the marker Mk shown in FIG. 14, as described above, the centroid C5 of the figure FD5, which is the four code figures, the centroid C10 of the figure FD10, and the figure FD16 (having two centroids). The centroid C161 and centroid C162 of FIG. 2 and the centroid C1 of the figure FD1 are positioned on a straight line CL connecting the centroid F1C of the first figure F1 and the centroid F2C of the second figure F2. FD5, figure FD10, figure FD16, and figure FD1 are arranged.

  The second detection processing unit 51 detects a plurality of figures included between the first figure F1 and the second figure F2 detected in step S330 as data areas of the marker Mk as candidates for four code figures. Execute the process. In this process, the second detection processing unit 51 calculates centroids of a plurality of figures included between the first figure F1 and the second figure F2. Then, the second detection processing unit 51 connects a straight line passing through the centroid of the first graphic F1 and the centroid of the second graphic F2 (that is, the centroid of the first graphic F1 and the centroid of the second graphic F2). When four figures having a centroid located on CL are detected, these four figures are detected as four code figures included in the data area of the marker Mk.

  After the process for detecting the data area is executed in step S330, the second detection processing unit 51 determines whether or not the data area has been successfully detected in step S330 (step S340). When it is determined that the detection of the data area is not successful (failed) (No at Step S340), the second detection processing unit 51 proceeds to Step S310 and selects the next marker. In this example, since the only marker that can be selected in step S310 is the marker Mk, if the second detection processing unit 51 determines that the detection of the data area has not been successful, it indicates that the marker detection has failed. Information to be shown is generated as a result of the marker detection process, and the process ends. On the other hand, when the second detection processing unit 51 determines that the detection of the data area is successful (step S340-Yes), the position / orientation calculation unit 52 is based on the first graphic F1 and the second graphic F2 detected in step S320. Then, the position and orientation of the marker Mk are calculated as the position and orientation of the object O in the robot coordinate system (step S350). The position on the captured image and the position in the robot coordinate system are associated in advance by calibration.

  Here, the process of calculating the posture of the marker Mk in the process of step S350 will be described. In this example, the position / orientation calculation unit 52 sets the normal direction to the first figure F1 in the centroid F1C of the first figure F1 as the Z axis, and the centroid F1C of the first figure F1 from the centroid F2C of the second figure F2. The direction toward the X axis is the X axis, and the direction perpendicular to the Z axis and the X axis is the Y axis, so that the posture of the first figure F1 is the posture of the marker Mk (in this example, the posture of the object O). Calculate as Alternatively, the position / orientation calculation unit 52 may be configured to calculate the orientation of the marker Mk by another method based on the centroid F1C of the first graphic F1.

  After calculating the position and orientation of the object O in the robot coordinate system in step S350, the code detection unit 53 detects the data area detected by the second detection processing unit 51 in step S330, that is, four data included in the data area. Based on the code figure and the processing of the flowchart shown in FIG. 12, it is specified which of these four code figures is the figure FD1 to the figure FD16. Then, the code detection unit 53 detects the code represented by the four specified code figures (step S360).

  Next, the code detection unit 53 determines whether or not the code detected in step S360 is appropriate information (step S370). In this example, the appropriate information is robot operation information that does not include an error and can be operated by the robot control apparatus 30. When the code detection unit 53 determines that the code is not appropriate information, the code detection unit 53 generates information indicating that the detection of the marker Mk has failed as a result of the marker detection process. Then, the image processing unit 44 proceeds to step S310 and selects the next marker. In this example, the only marker that can be selected in step S310 is the marker Mk. For this reason, the image processing unit 44 ends the process after generating the result of the marker detection process. On the other hand, if it is determined that the code is appropriate information, the code detection unit 53 includes information including the position and orientation of the marker Mk calculated by the position / orientation calculation unit 52 in step S350 and the code detected in step S360. Are generated as a result of the marker detection process. Then, the image processing unit 44 proceeds to step S310 and selects the next marker. In this example, the image processing unit 44 ends the process after generating a result of the marker detection process.

  As described above, after detecting the first graphic F1, the robot control device 30 detects the data area based on the centroid F1C of the first graphic F1. Thereby, the robot control apparatus 30 can suppress the failure in detection of the marker Mk due to an external factor with respect to the marker Mk.

  In the process of the flowchart shown in FIG. 13, the resolution of the captured image including the marker Mk is such that 32 cells constituting each of the four code figures included in the data area can be distinguished by the second detection processing unit 51. It was explained on the assumption. In order to guarantee this premise, the first detection processing unit 50 has the size of one or both of the detected first graphic F1 and second graphic F2 in the process of step S320 of the flowchart shown in FIG. It may be configured to determine whether or not the size is greater than or equal to a predetermined size. When the first detection processing unit 50 determines that one or both of the detected first graphic F1 and second graphic F2 are not larger than a predetermined size, the image processing unit 44 determines whether the marker Mk Information indicating that the detection has failed is generated as a result of the marker detection process, and the process ends. On the other hand, when the first detection processing unit 50 determines that one or both of the detected first graphic F1 and second graphic F2 are equal to or larger than a predetermined size, the second detection processing unit 51 Then, the process of step S330 of the flowchart shown in FIG. 13 is executed. The predetermined size is represented by, for example, the number of pixels constituting one or both of the detected first graphic F1 and second graphic F2, and is a value determined according to the resolution of the imaging unit that captures the captured image. It is.

<Modification 1 of Embodiment>
Hereinafter, a first modification of the embodiment of the present invention will be described with reference to the drawings. The marker attached to the object O described in the above embodiment may be the marker Mk1 described in this example instead of the marker Mk. That is, in the first modification of the embodiment, the robot control device 30 detects the marker Mk1 instead of the marker Mk. The marker Mk1 is a marker having a configuration in which only the second graphic F2 is omitted (deleted) from the marker Mk.

<Details of Marker in Modification 1 of Embodiment>
FIG. 15 is a diagram illustrating an example of a marker in the first modification of the embodiment. As shown in FIG. 15, unlike the marker Mk, the marker Mk1 does not have the second graphic F2, but has only the first graphic F1. That is, the marker Mk1 has the same characteristics as the marker Mk except that it does not have the second graphic F2. In the example shown in FIG. 15, the marker Mk1 includes the figure FD5, the figure FD10, the figure FD16, and the figure FD1 as the code figure in the data area.

<Regarding Flow of Marker Detection Processing in Modification 1 of Embodiment>
Hereinafter, the flow of the marker detection process in the first modification of the embodiment will be described. In the first modification of the embodiment, the image processing unit 44 performs the process of step S120a described below instead of the process of step S120 in the flowchart illustrated in FIG. FIG. 16 is a flowchart illustrating an example of the flow of marker detection processing executed by the image processing unit 44 in the first modification of the embodiment. In FIG. 16, the processing in step S310 and the processing in steps S350 to S370 are the same as the processing in step S310 and the processing in steps S350 to S370 shown in FIG. Omitted.

  The image processing unit 44 reads marker information including information indicating the first graphic of each of the one or more markers stored in advance in the storage unit 32 from the storage unit 32 (step S400). In this example, the marker information includes at least information indicating the marker and information indicating the first graphic included in the marker. For example, the marker information includes information in which information indicating the marker Mk1 is associated with information indicating the first graphic F1 included in the marker Mk1. Hereinafter, for convenience of explanation, a case where only the information is included in the marker information will be described.

  Next, the first detection processing unit 50 extracts information indicating the first graphic F1 associated with the marker ID indicating the marker Mk1 selected as the target marker in step S310 from the marker information. And the 1st detection process part 50 is based on the information which showed the extracted 1st figure F1, and the detection method of the marker mentioned above from the captured image which the image acquisition part 42 acquired in step S110 shown in FIG. The first graphic F1 is detected (step S405). Since the process of detecting the first graphic F1 from the captured image in step S405 is similar to the process of detecting the first graphic F1 of the first graphic F1 and the second graphic F2 in step S320, the description thereof is omitted.

  Next, the second detection processing unit 51 sets the search direction starting from the centroid F1C of the first graphic F1 detected in step S405 to 0 ° to 360 ° counterclockwise with the predetermined direction being 0 °. And the processes in steps S420 to S430 are repeated for each selected search direction (step S410). The predetermined direction is, for example, a positive direction on the X axis of the image sensor used for capturing the captured image. The predetermined direction may be another direction such as a randomly determined direction instead of this. The search direction is an example of a direction passing through the centroid of the first graphic.

  More specifically, the second detection processing unit 51 selects 0 °, which is a predetermined direction, as the search direction, performs the processing of Steps S420 to S430, and then adds the predetermined angle AG to 0 °. The direction (0 ° + angle AG) is selected as the search direction in the processing of the next step S420 to step S430. Then, the second detection processing unit 51 performs the processing of steps S420 to S430 with (0 ° + angle AG) as the search direction, and then adds (0 ° + angle AG) to the predetermined angle AG. (0 ° + 2AG) is selected as the search direction in the processing of the next step S420 to step S430. By repeating this, the second detection processing unit 51 repeatedly performs the processing of step S420 to step S430 for each search direction selected within the range of 0 ° to 360 °. The predetermined angle is, for example, 0.1 °. The predetermined angle may be another angle instead of this.

  After the search direction is selected in step S410, the second detection processing unit 51 includes the data area from the centroid F1C of the first graphic F1 detected in step S405 toward the search direction selected in step S410. The four code figures are detected (step S420). More specifically, the second detection processing unit 51 performs four code figures included in the data area from the centroid F1C of the first figure F1 detected in step S405 toward the search direction selected in step S410. Detect candidates for. When four candidate graphics are detected in the search direction, the second detection processing unit 51 calculates the centroid of each of the four graphics. Then, when all the calculated centroids are located on a straight line from the centroid F1C of the first graphic F1 toward the search direction, the second detection processing unit 51 sets the four detected graphics as data areas. It specifies that it is four code figures contained.

  Next, the second detection processing unit 51 determines whether or not the data area included in the marker Mk1 in step S420, that is, the four code figures included in the data area has been successfully detected (step S430). When the second detection processing unit 51 determines that the four code figures have been successfully detected (step S430-Yes), the position / orientation calculation unit 52 transitions to step S350, and determines the position and orientation of the marker Mk1 as the object O. The position and orientation in the robot coordinate system are calculated. On the other hand, when it is determined that the four code figures have not been successfully detected (failed) (No at Step S430), the second detection processing unit 51 proceeds to Step S410, and the next search direction from the unselected search direction. Select the search direction.

  FIG. 17 is a diagram for visually explaining an outline of the processing in steps S410 to S430 in the flowchart shown in FIG. From the centroid F1C of the marker Mk1 shown in FIG. 17, two arrows are drawn in directions different from each other. An arrow S1, which is one of the two arrows, extends from the centroid F1C in the positive direction of the X axis of the coordinate system shown in FIG. On the other hand, an arrow S2, which is one of the two arrows, extends in a direction rotated from the arrow S1 by an angle R1 counterclockwise around the centroid F1C.

  An arrow S1 represents a predetermined direction that is first selected as a search direction in step S410 shown in FIG. 16 in FIG. As shown in FIG. 17, the four code figures included in the data area of the marker Mk1 do not exist in the direction indicated by the arrow S1. For this reason, the second detection processing unit 51 changes the search direction counterclockwise from the direction of the arrow S1, and detects the code figure for each changed search direction. And the 2nd detection process part 51 can detect the said code figure, when the direction of arrow S2 is selected in step S410.

  As described above, the robot control device 30 includes the position and orientation of the marker and the data area included in the marker 1 even when the marker attached to the object O does not have the second graphic. The code indicated by the above code figure can be detected, and the same effect as in the embodiment can be obtained.

<Modification 2 of Embodiment>
Hereinafter, Modification 2 of the embodiment of the present invention will be described with reference to the drawings. The marker attached to the object O described in the above embodiment may be the marker Mk2 described in this example instead of the marker Mk and the marker Mk1. That is, in the second modification of the embodiment, the robot control device 30 detects the marker Mk2 instead of the marker Mk and the marker Mk1. The marker Mk2 is a marker having a configuration in which the data area of the marker Mk includes only one code graphic FC described below instead of four code graphics.

<Details of Marker in Modification 2 of Embodiment>
FIG. 18 is a diagram illustrating an example of a marker in the second modification of the embodiment. As shown in FIG. 18, unlike the marker Mk, the marker Mk2 has only one code figure FC in the data area W3. That is, the marker Mk2 has the same characteristics as the marker Mk except for the code figure included in the data area W3. Unlike the four code figures included in the data area of the marker Mk, one code figure FC represents a code.

  In the example shown in FIG. 18, the code figure FC is a two-dimensional code in which 24 square cells are arranged in 4 rows and 6 columns. Each of the cells is either a white cell or a black cell. The code graphic FC includes a fixed code used as a mark for detecting the code graphic FC. The fixed code is configured by a cell in which the color is fixed (the color is not changed) among the cells constituting the code graphic FC. That is, in this example, the fixed code is not a part of the code represented by the code graphic FC.

  In the example shown in FIG. 18, among the cells representing the fixed code, white cells are the first row, first column, first row, third column, first row, fifth column, third row, first column in the 4 rows and 6 columns, Five in the fourth row and the fifth column. Among the cells representing the fixed code, the black cells are the first row, second column, first row, third column, first row, fifth column, second row, first column, fourth row, first column in the fourth row and sixth column. These are five. Note that the white cell or the black cell representing the fixed code may be replaced by another position in the 4 rows and 6 columns.

  The code figure FC is obtained by considering white and black as binary numbers of 0 and 1 among the colors of the remaining 12 cells excluding cells representing the fixed code in the 4 rows and 6 columns. Represents a bit code. That is, the code represented by the code figure FC is entirely represented by these 12 cells (as described above, a part of the code is not represented by the fixed code). In FIG. 18, each of these 12 cells is included in a hatched area CD1. In the hatched area CD1 shown in FIG. 18, the colors of these 12 cells are omitted.

<Regarding Flow of Marker Detection Processing in Modification 2 of Embodiment>
Hereinafter, the flow of marker detection processing in Modification 2 of the embodiment will be described. In the second modification of the embodiment, the image processing unit 44 performs the process of step S120b described below instead of the process of step S120 in the flowchart illustrated in FIG.

  FIG. 19 is a flowchart illustrating an example of a marker detection process performed by the image processing unit 44 in the second modification of the embodiment. In FIG. 19, the processing from step S300 to step S320, the processing from step S340 to step S350, and the processing from step S370 are the processing from step S300 to step S320 and the processing from step S340 to step S350 shown in FIG. Since this process is the same as the process in step S370, description thereof will be omitted.

  After the first graphic F1 and the second graphic F2 are detected in step S320, the second detection processing unit 51 detects the data area of the marker Mk2 based on the first graphic F1 and the second graphic F2 ( Step S330b). Here, the process of step S330b will be described. The second detection processing unit 51 performs a process of detecting a graphic included between the first graphic F1 and the second graphic F2 as a code graphic FC candidate as a data area of the marker Mk2. In this process, the second detection processing unit 51 searches (detects) a fixed code from a graphic included between the first graphic F1 and the second graphic F2. Then, when a fixed code is detected from the graphic, the second detection processing unit 51 detects this graphic as a code graphic FC included in the data area of the marker Mk2. In addition, when the second detection processing unit 51 detects a fixed code, the model code of the fixed code is read from the storage unit 32 in advance, and the fixed code is detected by pattern matching using the read model graphic. The fixed code is detected by a known method.

  After the position and orientation of the marker Mk2 are calculated as the position and orientation of the object O in the robot coordinate system in step S350, the code detection unit 53 uses the 12 fixed codes from the code figure FC detected in step S330b. The code represented by the code figure FC is detected by detecting the position and color of each of the remaining 12 cells excluding the cell (step S360b). Note that the code detection method in step S360b is the same as the conventionally known two-dimensional code detection method, and a description thereof will be omitted.

  As described above, the robot control device 30 determines the position and orientation of the marker and the data held by the marker even when the marker attached to the object O is a marker having the same characteristics as the marker Mk2. The code indicated by the code graphic included in the region can be detected, and the same effect as in the embodiment can be obtained.

  In the process of the flowchart shown in FIG. 19, it is assumed that the resolution of the captured image including the marker Mk2 is such that each cell of the code figure FC included in the data area can be distinguished by the second detection processing unit 51. explained. In order to guarantee this premise, the first detection processing unit 50 has a size of one or both of the detected first graphic F1 and second graphic F2 in the process of step S320 of the flowchart shown in FIG. It may be configured to determine whether or not the size is greater than or equal to a predetermined size. When the first detection processing unit 50 determines that one or both of the detected first graphic F1 and second graphic F2 are not larger than a predetermined size, the image processing unit 44 determines whether the marker Mk2 Information indicating that the detection has failed is generated as a result of the marker detection process, and the process ends. On the other hand, when the first detection processing unit 50 determines that one or both of the detected first graphic F1 and second graphic F2 are equal to or larger than a predetermined size, the second detection processing unit 51 Then, the process of step S330b of the flowchart shown in FIG. 19 is executed.

<Modification 3 of embodiment>
Hereinafter, Modification 3 of the embodiment of the present invention will be described with reference to the drawings. The marker affixed to the object O described in the above embodiment may be the marker Mk3 described in this example instead of the marker Mk, the marker Mk1, and the marker Mk2. That is, in the third modification of the embodiment, the robot control device 30 detects the marker Mk3 instead of the marker Mk, the marker Mk1, and the marker Mk2. The marker Mk3 is a marker having a configuration in which the data area of the marker Mk1 includes only one code graphic FC described in the second modification of the embodiment instead of the four code graphics.

<Details of Markers in Modification 3 of Embodiment>
FIG. 20 is a diagram illustrating an example of the marker in the third modification of the embodiment. As shown in FIG. 20, unlike the marker Mk1, the marker Mk3 has only one code figure FC in the data area W3. That is, the marker Mk3 has the same characteristics as the marker Mk1 except for the code figure included in the data area W3.

<Regarding Flow of Marker Detection Processing in Modification 3 of Embodiment>
Hereinafter, the flow of the marker detection process in the third modification of the embodiment will be described. In the third modification of the embodiment, the image processing unit 44 performs the process of step S120c described below instead of the process of step S120 in the flowchart illustrated in FIG.

  FIG. 21 is a flowchart illustrating an example of a marker detection process performed by the image processing unit 44 in the third modification of the embodiment. In FIG. 21, the processing in step S310, the processing in steps S350 to S370, the processing in step S400, the processing in step S405, and the processing in step S430 are the processing in step S310 shown in FIG. Since the processing is the same as the processing in steps S350 to S370, the processing in step S400, the processing in step S405, and the processing in step S430, the description thereof is omitted.

  After the first graphic F1 is detected in step S405, the second detection processing unit 51 detects the data area of the marker Mk3 based on the first graphic F1 (step S420c). Here, the process of step S420c will be described. First, the second detection processing unit 51 searches for (detects) a part of the code graphic FC as a mark indicating the data area from around the first graphic F1 in order to detect the data area of the marker Mk3. Run. In this example, the mark is a part of the fixed code included in the code graphic FC. The mark may be the entire fixed code instead of a part of the fixed code, or may be a part of the code figure FC different from the fixed code.

  In this example, the mark indicating the data area is the cell included in the column closest to the first graphic F1 among the cells arranged in 4 rows and 6 columns constituting the code graphic FC, that is, the cell included in the fixed code. Of these, four cells (part of the fixed code) in the first row, first column, second row, first column, third row, first column, and fourth row, first column. FIG. 22 is a diagram illustrating an example of a mark indicating a data area that the second detection processing unit 51 searches from around the first graphic F1 in step S420c. As shown in FIG. 22, the mark is the column closest to the first graphic F1 among the cells arranged in 4 rows and 6 columns constituting the code graphic FC, that is, the column in the region W5 shown in FIG. Is a part of the fixed code included in

  Further, the periphery of the first graphic F1 searched by the second detection processing unit 51 is included in the expanded area when, for example, the area of the first graphic F1 detected in step S405 is expanded by a predetermined value. It is a range area. The predetermined value is, for example, about 4 times. The predetermined value is a value corresponding to the first graphic F1 detected from the captured image and may be any value that can detect a mark (a part of the fixed code described above) indicating the data area in this example. May be the value. When the second detection processing unit 51 detects the mark, the second detection processing unit 51 previously reads a part of the model figure of the fixed code serving as the mark from the storage unit 32 and performs pattern matching using the read model figure. The mark is detected by a conventionally known method such as detecting.

  After detecting the mark indicating the data area, the second detection processing unit 51 searches the entire fixed code of the code graphic FC from the predetermined search range in the direction from the centroid F1C of the first graphic F1 to the mark ( Detection). Then, when a fixed code is detected from the search range, the second detection processing unit 51 detects a graphic including the detected fixed code as a code graphic FC.

  Thus, even when the marker attached to the object O is a marker having the same characteristics as the marker Mk3, the robot controller 30 can detect the position and orientation of the marker and the data the marker has. The code indicated by the code graphic included in the region can be detected, and the same effect as in the embodiment can be obtained.

<Modification 4 of embodiment>
Hereinafter, Modification 4 of the embodiment of the present invention will be described with reference to the drawings. The marker attached to the object O described in the above embodiment may be the marker Mk4 described in this example instead of the marker Mk, the marker Mk1, the marker Mk2, and the marker Mk3. That is, in the fourth modification of the embodiment, the robot control device 30 detects the marker Mk4 instead of the marker Mk, the marker Mk1, the marker Mk2, and the marker Mk3.

<Details of Marker in Modification 4 of Embodiment>
FIG. 23 is a diagram illustrating an example of a marker in Modification 4 of the embodiment. As shown in FIG. 23, the marker Mk4 is different from each of the marker Mk, the marker Mk1, the marker Mk2, and the marker Mk3, and has a first graphic F1 and a data area CR arranged in the first graphic F1. . In the example shown in FIG. 23, the first graphic F1 is a centroid marker having a centroid multiplicity of 4.

  In the example shown in FIG. 23, the data area CR includes the black square frame shape graphic shown in FIG. 23 among the four figures constituting the first graphic F1, and FIG. It is an area | region on a white figure between the black ring-shaped figures arrange | positioned inside the said figure shown. Two code figures are arranged in the data area CR. In the example shown in FIG. 23, there are two code figures, code figure FC1 and code figure FC2.

  Each of the code graphic FC1 and the code graphic FC2 is a two-dimensional code in which nine square cells are arranged in three rows and three columns. Therefore, each of the code graphic FC1 and the code graphic FC2 can represent a 9-bit code by the color (white or black) of each cell arranged in the 3 rows and 3 columns. Each of the code graphic FC1 and the code graphic FC2 has a different fixed code. These fixed codes are marks for detecting each of the code graphic FC1 and the code graphic FC2. The robot control device 30 can detect and distinguish between the code graphic FC1 and the code graphic FC2 based on the fixed code. For example, the fixed code of the code graphic FC1 is a black cell in the first row and the first column, a white cell in the second row and the first column, and a black cell in the third row and the first column among the cells arranged in the third row and the third column. The fixed code which the code figure FC2 has is three cells, the black cell in the second row and the first column, the white cell in the second row and the second column, the second row and the third column among the cells arranged in the third row and the third column. The black cells are three cells. In addition, the position of the cell of the fixed code which each of the code figure FC1 and the code figure FC2 has may be another position instead of this. Further, the number of cells of the fixed code included in each of the code graphic FC1 and the code graphic FC2 may be other numbers instead. In FIG. 23, for simplification of the drawing, cells other than the fixed code (cells representing codes) in the code graphic FC1 and the code graphic FC2 are shown by hatching, and the colors of the cells are omitted.

<Regarding Flow of Marker Detection Processing in Modification 4 of Embodiment>
Hereinafter, the flow of the marker detection process in Modification 4 of the embodiment will be described. In the fourth modification of the embodiment, the image processing unit 44 performs the process of step S120d described below instead of the process of step S120 in the flowchart illustrated in FIG.

  FIG. 24 is a flowchart illustrating an exemplary flow of a marker detection process executed by the image processing unit 44 in the fourth modification of the embodiment. In FIG. 24, the processing in step S310, the processing in step S340, the processing in step S360b, the processing in step S370, and the processing in step S400 are the processing in step S310 and the processing in step S340 shown in FIG. The process is the same as the process, the process of step S360b, the process of step S370, and the process of step S400 shown in FIG.

  After the marker Mk4 is selected as the target marker in step S310, the first detection processing unit 50 extracts information indicating the first graphic F1 associated with the marker ID indicating the marker Mk4 from the marker information. And the 1st detection process part 50 is the imaging which the image acquisition part 42 acquired in step S110 shown in FIG. 4 based on the information which shows the extracted 1st figure F1, and the detection method of the centroid marker mentioned above. The first figure F1 is detected from the image (step S320d).

  Here, the process of step S320d will be described. The first detection processing unit 50 detects a centroid marker having a multiplicity of 3 or more from the captured image as the first graphic F1. The second detection processing unit 51 allows the positions of the centroids of the code graphic FC1 and the code graphic FC2 included in the data area to fall within a predetermined range together with the positions of the centroids of the four graphics constituting the first graphic F1. Therefore, the code graphic FC1 and the code graphic FC2 can be ignored when detecting the first graphic F1. Further, since the multiplicity of the centroids of the code graphic FC1 and the code graphic FC2 is 1, the first detection processing unit 50 does not erroneously detect the code graphic FC1 and the code graphic FC2 as the first graphic F1. .

  Next, based on the first graphic F1 detected in step S320d, the second detection processing unit 51 detects the data area included in the marker Mk4, that is, each of the code graphic FC1 and the code graphic FC2 included in the data area. (Step S330d). Here, the process of step S330d will be described. The second detection processing unit 51 searches the inside of the first graphic F1 detected in step S320d, and detects a fixed code included in the code graphic FC1. The second detection processing unit 51 searches for a code that is included in the code graphic FC1 from the centroid F1C of the first graphic F1 in the direction of the fixed code and is different from the fixed code. Detect the whole. Further, the second detection processing unit 51 searches the inside of the first graphic F1 detected in step S320d, and detects the fixed code included in the code graphic FC2. The second detection processing unit 51 searches for a code that is included in the code graphic FC2 from the centroid F1C of the first graphic F1 in the direction of the fixed code and is different from the fixed code. Detect the whole.

  After it is determined in step S340 that the data area has been successfully detected, the position / orientation calculation unit 52 calculates the position and orientation of the marker Mk4 based on the centroid F1C of the first graphic F1 (step S350d). Here, the process of calculating the posture of the marker Mk4 in the process of step S350d will be described. In this example, the position / orientation calculation unit 52 sets the normal direction of the first figure F1 at the centroid F1C of the first figure F1 to the first figure F1 as the Z axis, and the code figure FC2 detected from the inside of the first figure F1 in step S330d. The direction toward the code graphic FC1 is the X axis, and the direction orthogonal to the Z axis and the X axis is the Y axis, so that the posture of the first graphic F1 is the posture of the marker Mk4 (in this example, the object O posture).

  Thus, even if the marker affixed to the object O is a marker having the same characteristics as the marker Mk4, the robot control device 30 can detect the position and orientation of the marker and the data the marker has. The code indicated by the code graphic included in the region can be detected, and the same effect as in the embodiment can be obtained.

  In this example, the code detection unit 53 may have other configurations such as a configuration in which the areas of the corresponding cells in the code graphic FC1 and the code graphic FC2 are integrated and the integrated value is detected as a code.

<Regarding Correction of Distortion in Embodiment and Modified Examples 1 to 3 of Embodiment>
In the embodiment described above and the first to third modifications of the embodiment, the first graphic F1 detected by the first detection processing unit 50 from the captured image, or each of the first graphic F1 and the second graphic F2 is distorted. In other words, the marker to be detected from the captured image may be distorted in the captured image.

  For example, when the surface of the marker Mk2 is imaged by the imaging unit in a state where the surface of the marker Mk2 is inclined with respect to the optical axis of the imaging unit, the marker Mk2 is distorted in the captured image. That is, the surface of the marker Mk2 is imaged obliquely with respect to the optical axis. The surface of the marker Mk2 is the surface on the side on which the first graphic F1 is drawn. A captured image obtained by distorting the marker Mk2 may be corrected (converted) into an image in which the marker Mk2 is not distorted by affine transformation (affine warp).

  For example, when the marker Mk2 in the captured image described above is sufficiently smaller than the overall size of the captured image, the captured image captured with the marker Mk2 distorted is approximately distorted by the affine transformation. It is possible to correct to an image that is not. Note that the captured image obtained by distorting the marker Mk2 may be the entire captured image or a part including the marker Mk2 in the captured image. In such a case, when the second detection processing unit 51 detects the data area of the marker Mk2, the second detection processing unit 51 corrects the captured image obtained by distorting the marker Mk2 by affine transformation into an image in which the marker Mk2 is not distorted. .

  Here, the images of the first graphic F1 and the second graphic F2 included in the marker Mk2 that are distorted to a degree that can be corrected by affine transformation in the captured image are not circular but elliptical. FIG. 25 is a diagram illustrating an example of a marker Mk2 that is distorted to a degree that can be corrected by affine transformation in a captured image. In FIG. 25, the second graphic F2 included in the marker Mk2 is omitted. In order to correct the captured image captured by distorting the marker Mk2 illustrated in FIG. 25, the second detection processing unit 51 uses the major axis MA1 and the short axis MA1 of the first graphic F1 that is distorted from a circular shape to an elliptical shape in the captured image. Each direction with the diameter MA2 is calculated as the main axis direction, and the ratio of the length between the major axis and the minor axis is calculated as the main axis ratio. Then, the first detection processing unit 50 calculates a normal direction N1 with respect to the first graphic F1 at the centroid F1C of the first graphic F1 based on the main axis direction, the main shaft ratio, and geometry.

  The normal direction N1 is a direction in a virtual three-dimensional space whose position and orientation are represented by the three-dimensional coordinate system shown in FIG. In FIG. The Z-axis direction of the three-dimensional coordinate system coincides with the direction in which the normal direction N1 faces when the marker Mk2 is not distorted in the captured image. The second detection processing unit 51 performs affine transformation based on the normal direction N1 so that the normal direction N1 matches the Z-axis direction of the three-dimensional coordinate system as shown in FIG. The first figure F1 in the picked-up image picked up with distortion is approximately matched with the model image of the first figure F1, which is information indicating the first figure F1 in this example described above. FIG. 26 is a diagram illustrating an example of the normal direction N1 of the undistorted marker Mk2. Thereby, the captured image is corrected to an image in which the marker Mk2 is not substantially distorted (not approximately distorted). As a result, the second detection processing unit 51 can accurately detect the data region from the corrected image.

  However, a captured image that is captured with the marker Mk2 distorted may not be corrected by affine transformation. In such a case, the second detection processing unit 51 corrects the captured image to an image in which the marker Mk2 is not distorted by performing homography conversion (homography warp) on the captured image instead of affine transformation. To do.

  In this case, the second detection processing unit 51 calculates the normal direction N1 at the centroid F1C of the first graphic F1 as in the case of performing the affine transformation described above. Further, as described in step S330b in the flowchart illustrated in FIG. 19, the second detection processing unit 51 detects the mark of the code graphic FC and detects the entire code graphic FC. The second detection processing unit 51 sets an initial value for homography conversion.

  After setting the initial value of the homography conversion, the second detection processing unit 51 normalizes the captured image by performing bilinear interpolation processing on the captured image captured with the marker Mk2 being distorted. Based on the normalized captured image, the non-linear optimization method (ESM), and the model image of the first graphic F1, the second detection processing unit 51 includes a captured image obtained by distorting the marker Mk2. The coefficient of homography conversion that minimizes the matching error between the first figure F1 and the model image of the first figure F1, which is information indicating the first figure F1 in this example described above, is derived. Then, when the matching error is equal to or less than a predetermined error value, the second detection processing unit 51 determines that an undistorted marker Mk2 image has been obtained. If the matching error is not less than or equal to a predetermined error value, it is determined that an image of the undistorted marker Mk2 is not obtained, and the homography conversion coefficient that minimizes the matching error is derived from the bilinear interpolation process described above. Repeat the process up to.

  By using an image obtained by such a homography transformation and an image in which the marker Mk2 is not distorted, the code detection unit 53 can perform high accuracy based on the code figure FC included in the data area of the marker Mk2. Can detect the code.

  When homography conversion is performed on a captured image captured with the marker Mk2 distorted into an image without the marker Mk2 being distorted, iterative processing using ESM is required as compared with the case of performing affine transformation. The time required for detection increases. Since the homography conversion has a smaller error when correcting a distorted image to an undistorted image as compared with the case of performing affine conversion, the detection accuracy of the marker Mk2 is increased. The above affine transformation and homography transformation are not limited to application to the marker Mk2, but can be applied to any of the marker Mk, the marker Mk1, and the marker Mk3.

  When the marker to be detected is the marker Mk or the marker Mk1, even if the marker in the captured image is distorted to a level that can be corrected by affine transformation, the second detection processing unit 51 does not use affine transformation. A data area of the marker can be detected. FIG. 27 is a diagram illustrating an example of the marker Mk in the captured image in a case where the marker Mk is captured with a distortion that can be corrected by affine transformation.

  As shown in FIG. 27, even when the marker Mk is imaged in a distorted manner, the centroids (centroid C5, centroid C10, centroid C161, and centroid of each of the four code figures that the marker Mk has. C162, centroid C1) is not deviated from a straight line CL passing through the centroid F1C of the first graphic F1 and the centroid F2C of the second graphic F2.

  From this, even if the second detection processing unit 51 distorts the marker Mk in the captured image to the extent that it can be corrected by affine transformation, the second detection processing unit 51 uses the method described in FIG. It can be determined whether or not the plurality of images included between the two figures F2 are each of the four code figures included in the data area of the marker Mk.

<Regarding Correction of Distortion in Modification 4 of Embodiment>
In the modification 4 of the embodiment described above, when the first graphic F1 detected from the captured image by the first detection processing unit 50 is distorted, that is, the marker to be detected from the captured image is distorted in the captured image. There may be.

  For example, when the surface of the marker Mk4 is imaged by the imaging unit in a state where the surface of the marker Mk4 is inclined with respect to the optical axis of the imaging unit, the marker Mk4 is distorted in the captured image. That is, the surface of the marker Mk4 is imaged obliquely with respect to the optical axis. The surface of the marker Mk4 is the surface on the side on which the first graphic F1 is drawn. A captured image obtained by distorting the marker Mk4 may be corrected (converted) into an image in which the marker Mk4 is not distorted by affine transformation (affine warp).

  For example, when the marker Mk4 in the captured image described above is sufficiently smaller than the entire captured image, the marker Mk2 is not distorted by affine transformation of the captured image captured with the marker Mk2 distorted. As corrected to an image, a captured image obtained by distorting the marker Mk4 can be corrected to an image in which the marker Mk4 is not approximately distorted by affine transformation. Note that the captured image obtained by distorting the marker Mk4 may be the entire captured image or a part including the marker Mk4 in the captured image. In such a case, when the second detection processing unit 51 detects the data area of the marker Mk4, the second detection processing unit 51 corrects the captured image obtained by distorting the marker Mk4 by affine transformation into an image in which the marker Mk4 is not distorted. .

  However, in order to correct the captured image captured with the marker Mk4 distorted into an image without the marker Mk4 distorted, it may be necessary to further perform radon conversion. Here, a flow of processing in which the second detection processing unit 51 performs radon conversion will be described. Note that the processing for the case where the affine transformation is performed on the captured image captured with the marker Mk4 distorted is the same processing as the processing for the case where the affine transformation is performed on the captured image captured with the marker Mk2 distorted. Therefore, the description is omitted.

  The second detection processing unit 51 obtains an image in which the marker Mk4 is not approximately distorted by affine transformation, and then the degree to which the image matches the model image of the first graphic F1 (similarity). Detection of a peak representing is performed. When the peak is not detected, the second detection processing unit 51 determines that it is difficult to correct the captured image captured with the marker Mk4 being distorted into an image without the marker Mk4 being distorted. End the process. In this case, the second detection processing unit 51 generates information indicating that the detection of the marker Mk4 has failed.

  On the other hand, when the peak is detected, the second detection processing unit 51 performs radon conversion in a predetermined angle range with respect to the direction from the code graphic FC2 of the marker Mk4 toward the code graphic FC1 based on the peak. The angle and the size (scale) are corrected so that the peak which is the similarity between the image in which the marker Mk4 obtained by the affine transformation is not substantially distorted and the model image of the first graphic F1 is maximized. I do. Further, the second detection processing unit 51 performs radon conversion in a predetermined angle range with respect to a direction orthogonal to the direction from the code graphic FC2 of the marker Mk4 toward the code graphic FC1 based on the peak, and performs affine conversion. The angle and size (scale) are corrected so that the peak, which is the similarity between the obtained image in which the marker Mk4 is not approximately distorted and the model image of the first graphic F1, is maximized.

  By using an image obtained by such Radon transformation and an image in which the marker Mk4 is not distorted, the code detection unit 53 is based on the code graphic FC1 and the code graphic FC2 included in the data area of the marker Mk4. The code can be detected with high accuracy.

<Error correction of the code indicated by the code figure>
In addition to the fixed code, the code figures (code figure FC, code figure FC1 and code figure FC2) included in the markers (each of the markers Mk1 to Mk4) according to the modified examples 2 to 4 of the embodiment described above, The configuration may include a check bit or a hamming code. For example, of the 16-bit information represented by the code figure, 11 bits of information indicate a code, and the remaining 5 bits are used for a Hamming code, thereby correcting a 1-bit error and detecting a 2-bit error. it can. Here, the marker Mk3 and the marker Mk4 will be described as examples. Note that the content described below is also applicable to the marker Mk2.

  FIG. 28 is a table showing an example of information related to the marker Mk3 and the marker Mk4. As shown in FIG. 28, the marker Mk3 is created in a size (size) of, for example, 10 mm × 28 mm, 7 mm × 20 mm, and 5 mm × 14 mm. Note that the size of the marker Mk3 may be other sizes. When the number of bits representing the code of the code figure FC that the marker Mk3 has is set to 12 bits without using the check bit, the number of bits that can be error-corrected is 0. On the other hand, when the number of bits representing the code of the code figure FC included in the marker Mk3 is set to 11 bits using one check bit, the number of bits that can be error-corrected is 1.

  As shown in FIG. 28, the marker Mk4 is created with a size (size) of, for example, 10 mm × 21 mm, 7 mm × 15 mm, and 5 mm × 11 mm. Note that the size of the marker Mk4 may be other sizes. When the number of bits representing the codes of the code figure FC1 and the code figure FC2 included in the marker Mk4 is set to 18 bits without using the check bit or the Hamming code, the number of bits that can be corrected for error is zero. On the other hand, when the number of bits representing the codes of the code graphic FC1 and the code graphic FC2 included in the marker Mk4 is 16 bits using two check bits, the number of bits that can be error-corrected is 2. On the other hand, when the number of bits representing the codes of the code graphic FC1 and code graphic FC2 included in the marker Mk4 is 11 bits using a 5-bit Hamming code, the number of bits that can be error-corrected is 7.

  As described above, the robot control device 30 of the robot 20 in the embodiment (or the image processing device separate from the robot control device 30) detects the first graphic (in this example, the first graphic F1), A data area is detected based on the centroid of the first graphic (in this example, the centroid F1C). Thereby, the robot control apparatus 30 can suppress the failure of marker detection due to an external factor with respect to the marker (for example, each of the marker Mk and the markers Mk1 to Mk4 in the present embodiment).

  Further, the robot control device 30 detects the data area based on the direction passing through the centroid of the first graphic (in this example, the search direction). Thereby, the robot control apparatus 30 can perform processing based on information represented by the data area.

  Further, the robot control device 30 detects the data area by changing the direction passing through the centroid of the first graphic around the centroid of the first graphic. Thereby, the robot controller 30 can detect the data area even when the first graphic does not indicate the direction passing through the centroid of the first graphic.

  In addition, after detecting the first graphic, the robot control device 30 has a figure with fewer centroids than the first graphic (in this example, the code graphic FD1 to the code graphic FD16, the code graphic FC, A data area having a code graphic FC1 and a code graphic FC2) is detected. Thereby, the robot control apparatus 30 can suppress erroneously detecting the graphic included in the data area as the first graphic.

  Further, after detecting the first graphic and the second graphic (in this example, the second graphic F2), the robot controller 30 detects the centroid of the first graphic and the centroid of the second graphic (in this example, The data area is detected based on the centroid F2C). Thereby, the robot control apparatus 30 can detect a data area more reliably based on the direction passing through the centroid of the first graphic.

  Also, the robot control device 30 detects the data area based on the direction connecting the centroid of the first graphic and the centroid of the second graphic. Thereby, the robot control apparatus 30 can detect a data area earlier based on the direction connecting the centroid of the first graphic and the centroid of the second graphic.

  In addition, the marker (for example, each of the marker Mk and the markers Mk1 to Mk4) in the embodiment detects the first graphic, and then detects the data area based on the centroid of the first graphic. Thereby, the marker can suppress the failure of marker detection due to an external factor for the marker.

  The marker detects the first graphic and the second graphic, and then detects the data area based on the centroid of the first graphic and the centroid of the second graphic. Thereby, the marker can detect a data area more reliably.

  Note that the image processing unit 44 in the control unit 36 may be separate from the robot control device 30 as an image processing device. In this case, the robot 20, the robot control device 30, and the image processing device constitute a robot system. Further, in this case, the image processing apparatus outputs information based on the detected marker to another apparatus different from the robot control apparatus 30 and, for example, to the other apparatus such as AR (Augmented Reality). It may be configured to display information. The information based on the detected marker is, for example, information associated with the marker.

  In addition, in the data area of the marker (marker Mk, each of the markers Mk1 to Mk4) detected by the robot control device 30, other markers such as color markers, barcodes, QR codes (registered trademark) are provided as code figures. May be included.

  In addition, some or all of the first imaging unit 21, the second imaging unit 22, the third imaging unit 23, and the fourth imaging unit 24 may be separate imaging units from the robot 20. Good. In this case, the robot 20, the imaging unit separate from the robot 20, the robot control device 30, and the image processing device constitute a robot system.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and changes, substitutions, deletions, and the like are possible without departing from the gist of the present invention. May be.

  In addition, a computer-readable program for realizing the functions of arbitrary components in the above-described apparatus (for example, the robot control apparatus 30 of the robot 20 or an image processing apparatus separate from the robot control apparatus 30) is possible. The program may be recorded on a recording medium, and the program may be read into a computer system and executed. Here, the “computer system” includes hardware such as an OS (Operating System) and peripheral devices. The “computer-readable recording medium” refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD (Compact Disk) -ROM, or a storage device such as a hard disk built in the computer system. . Furthermore, “computer-readable recording medium” means a volatile memory (RAM) inside a computer system that becomes a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

In addition, the above program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
Further, the above program may be for realizing a part of the functions described above. Further, the program may be a so-called difference file (difference program) that can realize the above-described functions in combination with a program already recorded in the computer system.

DESCRIPTION OF SYMBOLS 20 ... Robot, 21 ... 1st imaging part, 22 ... 2nd imaging part, 23 ... 3rd imaging part, 24 ... 4th imaging part, 30 ... Robot control apparatus, 31 ... CPU, 32 ... Memory | storage part, 33 ... Input Reception unit 34 ... Communication unit 35 ... Display unit 36 ... Control unit 40 ... Imaging control unit 42 ... Image acquisition unit 44 ... Image processing unit 46 ... Robot control unit 50 ... First detection processing unit 51: second detection processing unit, 52: position and orientation calculation unit, 53: code detection unit

Claims (4)

An image processing apparatus for detecting a marker,
The marker is
A first graphic composed of three or more first regions whose centroids overlap at a first point ;
A second graphic composed of three or more second regions whose centroids overlap at the second point;
A plurality of third figures configured by a third area, and a data area provided between the first figure and the second figure ,
A plurality of the third figures are arranged in the data area such that the centroid of the third area is located on a straight line passing through the first point and the second point;
The number of centroids of the third region overlapping in the third graphic is less than the number of centroids of the first region overlapping at the first point in the first graphic, and Less than the number of centroids of the second region overlapping the second point in the second graphic,
Detecting the data area based on the first point and the second point after detecting the first figure and the second figure ;
Image processing device. Based on the direction toward the first point or found before Symbol second point, it detects the data area,
The image processing apparatus according to claim 1 . The image processing apparatus according to claim 1 or 2 , comprising:
robot. A first figure composed of three or more first region centroid becomes heavy in the first point,
A second graphic composed of three or more second regions whose centroids overlap at the second point;
A plurality of third graphics composed of third regions, a data region provided between the first graphics and the second graphics ,
A plurality of the third figures are arranged in the data area such that the centroid of the third area is located on a straight line passing through the first point and the second point;
The number of centroids of the third region overlapping in the third graphic is less than the number of centroids of the first region overlapping at the first point in the first graphic, and Less than the number of the centroids of the second region overlapping the second point in the second graphic,
marker.

Images