JP2015136764A - Control device, robot system, robot and robot control method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a control device capable of continuously operating a robot even when a feature amount of a work object cannot be detected or is erroneously detected in a captured image, and further to provide a robot system, a robot, a robot control method and the like.SOLUTION: A control device 100 includes: a processing section 150 operating a robot 300 provided with an end effector 310 moving a work object; and a captured image reception section 130 receiving a captured image obtained by the work object and a marker on the work object or the end effector 310 being imaged by an image section 200. Further, the processing section 150 images the work object and the marker a plurality of times while a posture of the robot 300 is being changed from a first posture to a second posture different from the first posture and operates the robot 300 even when a feature amount of the work object cannot be detected in the captured image.

Description

  The present invention relates to a control device, a robot system, a robot, a robot control method, and the like.

  In recent years, industrial robots have been increasingly introduced in order to mechanize and automate the work that humans have done at production sites. However, precise positioning is a prerequisite for positioning the robot, which is a barrier to robot introduction.

  Here, visual servoing is one of the methods for positioning the robot. Visual servo is a technique for controlling a robot based on a difference between a reference image (goal image, target image) and a captured image (current image). Certain types of visual servos are useful in that they do not require precision in calibration, and are attracting attention as a technology that lowers the barrier for robot introduction.

  As a technique related to this visual servo, there is a conventional technique described in Patent Document 1, for example.

JP 2003-305675 A

  In visual servoing, for example, a feature amount detection process of a work target is performed on a captured image, and the robot is controlled based on the feature amount of the work target in the detected captured image and the feature amount of the work target in the reference image.

  However, the feature amount of the work target in the captured image is not always detected normally. For example, the feature quantity of the work target may not be detected or may be erroneously detected due to the flicker of the light source or the position and orientation of the work target. In these cases, the intended work cannot be accurately performed by visual servoing.

  According to one embodiment of the present invention, a processing unit that operates a robot including an end effector that moves a work target, the work target, and the marker on the work target or the end effector are captured by an imaging unit. A captured image receiving unit that receives the captured image, and the processing unit detects the feature amount of the work object in the captured image even if the robot is in a first posture from the first posture. The present invention relates to a control device that operates the robot by imaging the work object and the marker a plurality of times while moving to a second posture different from the posture.

  In one aspect of the present invention, even when the feature amount of the work target in the captured image cannot be detected, processing for continuing visual servoing is performed based on the result of the marker detection processing.

  Thereby, even when the feature amount of the work target is not detected in the captured image, visual servoing can be continuously performed. Then, when the feature amount of the work target is not detected in the captured image, for example, it is determined that an error has occurred in the visual servo, and the robot is prevented from stopping before moving to the target position. It becomes possible.

  In the aspect of the invention, the processing unit performs a feature amount detection process of the work object in the captured image, and the feature amount of the work object cannot be detected in the feature amount detection process, or Even when the feature amount of the work object is erroneously detected, the work object and the marker are imaged a plurality of times while the robot moves from the first posture to the second posture, The robot may be operated based on the marker detection processing result of the marker.

  Thereby, even when the feature amount of the work target is not detected or erroneously detected in the captured image, visual servoing can be continuously performed.

  In the aspect of the invention, the processing unit may detect the marker detected in the marker detection process, although the feature amount of the work object in the captured image is not detected in the feature amount detection process. May be operated based on the result of the marker detection process.

  As a result, it is possible to determine a case where the feature amount of the work target cannot be detected even though the work target is reflected in the captured image.

  In the aspect of the invention, the processing unit detects the feature amount of the work object in the captured image in the feature amount detection process, and determines the number of the markers detected in the marker detection process. The position and orientation relationship between the marker and the work object is specified based on the result of the marker detection process and the result of the feature amount detection process when the value is equal to or greater than a given threshold, and the specified position Based on the posture relationship, it may be determined in the feature amount detection process whether or not the feature amount of the work target is erroneously detected.

  As a result, it is possible to determine a case where the feature amount of the work target is erroneously detected in the captured image.

  Moreover, in one aspect of the present invention, the processing unit specifies a position and orientation relationship between the marker and the work object based on a result of the marker detection process and a result of the feature amount detection process. Registration processing of the specified position and orientation relationship may be performed.

  This makes it possible to compare the position and orientation relationship between the marker and the work target in captured images acquired at different timings.

  In the aspect of the invention, the processing unit performs the marker detection process and the feature amount detection process based on the second captured image received by the captured image reception unit after the registration process. The second position / orientation relationship between the marker and the work object is specified based on the result of the marker detection process in the second captured image and the result of the feature amount detection process, and the second The position / posture relationship is compared with the registered position / posture relationship, and when it is determined that the second position / posture relationship corresponds to the registered position / posture relationship, You may perform the said registration process of 2nd position and orientation relation.

  As a result, it is possible to determine when the feature amount of the work target is not erroneously detected in the captured image, and to update the position and orientation relationship between the marker and the work target.

  In the aspect of the invention, when the processing unit determines in the comparison process that the second position / posture relationship does not correspond to the registered position / posture relationship, the marker Based on the result of the detection process, the feature quantity of the work object in the second captured image may be specified, and the robot may be operated based on the specified feature quantity.

  As a result, it is possible to determine that an erroneous detection of the feature quantity of the work target has occurred in the captured image, and even in this case, visual servoing can be continued.

  In one aspect of the present invention, the processing unit determines that the feature amount of the work object in the captured image has not been detected or has been erroneously detected in the feature amount detection process. When the number of the markers detected in the marker detection process is equal to or greater than a given threshold, based on the result of the marker detection process, the process for specifying the feature amount of the work object is performed, and the specified The robot may be operated based on the feature amount.

  This makes it possible to continue visual servoing or the like even when the feature amount of the work target is not detected or is erroneously detected.

  Moreover, in one aspect of the present invention, the processing unit, when the marker is not detected in the marker detection process, or when the number of the detected markers is equal to or less than a given threshold, It may be determined that the work object is located outside the imaging range.

  Thereby, it is possible to determine whether or not the work target is located outside the imaging range.

  In one aspect of the present invention, when it is determined that at least one of the feature amount detection process and the marker detection process has failed, the processing unit performs a change process of a start position of the robot operation. Also good.

  As a result, even when the work target is out of frame, it is possible to move the end effector to a position where the work target appears in the captured image and restart the visual servo.

  In the aspect of the invention, the processing unit may perform a process of controlling the robot so that the captured image matches or approaches the reference image as the visual servo.

  As a result, the position and orientation of the robot can be set to the target position and orientation.

  In another aspect of the present invention, a processing unit that operates a robot including an end effector that moves a work object, the work object, and a plurality of markers on the work object or the end effector include an imaging unit. A captured image receiving unit that receives a captured image obtained by imaging, wherein the processing unit is identified by the plurality of markers detected by marker detection processing of a marker in the captured image. Based on the detection result of the overlapping area between the area and the second area specified based on the feature quantity of the work object detected by the feature quantity detection process of the work object, the marker and the work object The present invention relates to a control device that specifies a position / posture relationship of an object and operates the robot based on the specified position / posture relationship.

  As a result, for example, when it is determined that an overlapping region exists in a captured image in a state where the end effector is holding the work object, it is possible to determine that the feature amount of the work object is not erroneously detected. become.

  In another aspect of the present invention, the processing unit performs a registration process of the identified position and orientation relationship, and after the registration process, based on a second captured image received by the captured image reception unit, A marker detection process and the feature quantity detection process are performed, and the marker and the work object are determined based on the result of the marker detection process in the second captured image and the result of the feature quantity detection process. The second position / posture relationship is identified, the second position / posture relationship is compared with the registered position / posture relationship, and the second position / posture relationship is compared with the registered position / posture relationship. In the feature amount detection process, it is determined that the feature amount of the work object has not been erroneously detected, and the second position and orientation relationship and the registered Position and posture relationship When it is determined not to be compatible, in the feature amount detection process it may determine that the feature amount of the working object is detected by mistake.

  As a result, it is possible to determine when the feature amount of the work target is not erroneously detected in the captured image, and to update the position and orientation relationship between the marker and the work target.

  In another aspect of the present invention, when the first area can be specified, the processing unit determines that the work target is located within an imaging range, and the first area cannot be specified. Alternatively, it may be determined that the work object is located outside the imaging range.

  Thereby, it is possible to determine whether or not the work target is located outside the imaging range.

  In another aspect of the present invention, a robot including an end effector that moves a work object, a marker on the work object or the end effector, and an imaging unit that images the work object and the marker, The robot is different from the first posture from the first posture even when the imaging unit cannot detect the feature amount of the work subject in the picked-up image obtained by picking up the work subject and the marker. The present invention relates to a robot system that operates the robot by imaging the work object and the marker a plurality of times while moving to a second posture.

  Further, another aspect of the present invention includes an end effector that moves a work object, and in the captured image obtained by imaging the work object and the marker on the work object or the end effector, the feature of the work object Even when the amount cannot be detected, the present invention relates to a robot that operates by imaging the work object and the marker a plurality of times while moving from the first posture to the second posture different from the first posture.

  According to another aspect of the present invention, there is provided a robot control method for operating a robot including an end effector that moves a work object, and images the work object and the marker on the work object or the end effector. And even if the feature quantity of the work object cannot be detected in the captured image obtained by picking up the work object and the marker, the robot is moved from the first position to the second position different from the first position. The robot control method includes: imaging the work object and the marker a plurality of times while moving to and operating the robot.

  Another aspect of the present invention relates to a program that causes a computer to function as each of the above-described units.

  According to some aspects of the present invention, a control device, a robot system, a robot, and a robot capable of continuously operating a robot even when a feature amount of a work target cannot be detected in a captured image or when it is erroneously detected A robot control method and the like can be provided.

The system configuration example of this embodiment. Explanatory drawing of the mode at the time of visual servo implementation of this embodiment. FIG. 3A to FIG. 3E are explanatory diagrams of the result of the feature amount detection process of the work target and the result of the marker detection process. The flowchart explaining the outline | summary of the process of this embodiment. FIG. 5A to FIG. 5F are explanatory diagrams of markers. 6 (A) and 6 (B) are explanatory diagrams of marker placement positions. FIG. 7A to FIG. 7E are explanatory diagrams of the result of the feature amount detection process of the work target in the captured image and the result of the marker detection process. FIG. 8A and FIG. 8B are explanatory diagrams of the process of specifying the feature quantity of the work target based on the result of the marker detection process. FIG. 9A to FIG. 9C are explanatory diagrams of the position and orientation relationship between the work target and the marker. FIG. 10A and FIG. 10B are explanatory diagrams of the change process of the start position of the visual servo. The flowchart explaining the flow of preparation of the visual servo of this embodiment. The flowchart explaining the flow of the process in the visual servo of this embodiment. FIG. 13A and FIG. 13B are configuration examples of a robot system and a robot. 1 is a configuration example of a robot system that controls a robot via a network. Flow chart of position-based visual servo. 16A and 16B are explanatory diagrams of a reference image and a captured image. Flow chart of feature-based visual servo.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. The method of this embodiment Visual servo is a technology that controls the robot so that the feature quantity of the reference image matches the feature quantity obtained from the captured image. As a premise, the feature quantity of the work target shown in the captured image is stable. Thus, it is required that it can be detected (extracted).

  However, when performing actual image processing to detect the feature quantity of the work target, the feature quantity of the work target may not be detected depending on the quality of the captured image and the appearance of the work target. It cannot be said that the target feature amount can be detected. If the feature quantity of the work target cannot be detected correctly in the captured image, the intended work cannot be accurately performed by the visual servo.

  Therefore, in the present embodiment, when visual servoing is performed using such feature quantities of a work target, stable visual servoing is realized by using a combination of marker feature quantities that are easier to detect.

  Specifically, FIG. 1 shows a configuration example of the control device 100 of the present embodiment. The control device 100 according to the present embodiment receives a captured image obtained by imaging the reference image storage unit 110, the work object, and the marker on the work object or the end effector 310 by the imaging unit 200. And a processing unit 150 that operates a robot 300 including an end effector 310 that moves a work object.

  Then, even when the feature amount of the work object cannot be detected in the captured image, the processing unit 150 is configured to operate the work object while the robot 300 moves from the first position to the second position different from the first position. Then, the robot 300 is operated by imaging the marker a plurality of times.

  Specifically, first, the reference image storage unit 110 stores a reference image that is an image representing the target state of the robot 300. For example, a reference image RIM as shown in FIG. The function of the reference image storage unit 110 can be realized by a memory such as a RAM (Random Access Memory) or an HDD (Hard Disk Drive).

  Next, the captured image reception unit 130 obtains a captured image obtained by placing and imaging at least one marker on the work target of the robot 300 or the end effector 310 of the robot 300. For example, the captured image PIM1 shown in FIG. 3B captured by the imaging unit CM in the state shown in FIG. Details of FIG. 3B will be described later.

  Then, the processing unit 150 performs a feature amount detection process of the work target in the captured image, and based on the feature amount of the work target in the detected captured image and the reference image (the feature amount of the work target in the reference image). Do visual servo. For example, in the example of FIG. 3B described later, the contour line OL1 of the workpiece WK is detected as the feature amount of the work target in the captured image PIM1, and the detected contour line OL1 and the reference of FIG. Visual servoing is performed based on the outline OL of the work WK in the image RIM.

  Further, the processing unit 150 causes the robot 300 to move from the first posture to the second position even when the feature amount of the work target cannot be detected or the feature amount of the work target is erroneously detected in the feature amount detection process. The work object and the marker are imaged a plurality of times while moving to the posture, and the robot 300 is operated based on the result of the marker detection process of the marker. That is, in the feature amount detection process, the processing unit 150 continues based on the result of the marker detection process of the marker even when the feature amount of the work target in the captured image is not detected or is erroneously detected. Perform visual servoing. The function of the processing unit 150 can be realized by hardware such as various processors (CPU and the like), ASIC (gate array and the like), a program, and the like.

  For example, in the example of FIG. 3C described later, the contour line OL2 of the workpiece WK is not detected. At this time, the processing unit 150 estimates the contour line OL2 of the workpiece based on the results (MD1 and MD2) of the marker detection process, and the estimated contour line OL2 and a reference image RIM in FIG. Based on the contour OL, visual servoing is continuously performed. That is, even when the feature amount of the work target in the captured image is not detected or is erroneously detected, it is determined that an error has occurred, and the visual servo can be continued without ending the visual servo. Process.

  Note that the control device 100 is not limited to the configuration shown in FIG. 1, and various modifications such as omitting some of these components or adding other components are possible. Further, the control device 100 of this embodiment is included in the robot 300 as shown in FIG. 13B described later, and may be configured integrally with the robot 300. Furthermore, as illustrated in FIG. 14 described later, the function of the control device 100 may be realized by a server 500 and a terminal device 330 included in each robot 300.

  For example, when the control device 100 and the imaging unit 200 are connected by a network including at least one of wired and wireless, the captured image reception unit 130 is a communication unit (interface unit) that communicates with the imaging unit 200. It may be. Furthermore, when the control apparatus 100 includes the imaging unit 200, the captured image reception unit 130 may be the imaging unit 200 itself.

  Next, a specific example of the operation of the present embodiment will be described with reference to the flowcharts of FIGS. 2, 3A to 3E, and FIG. In this specific example, as shown in FIG. 2, the robot RB grips the workpiece WK with the hand HD attached to the tip of the arm AM, and moves the workpiece WK to the target position TP (target position / posture). , Using visual servo. In this example, the hand HD is an end effector, and the work WK is a work target. The workpiece WK is assumed to be a cylindrical object.

  As a premise, the reference image storage unit 110 stores a reference image RIM as shown in FIG. The reference image RIM is an image representing the target state of the robot RB, and shows a state where the work WK is placed at the target position TP. Note that the target position TP shown in the reference image RIM in FIG. 3A is a position corresponding to the target position TP in FIG. 2 described above.

  First, as shown in FIG. 2, the captured image reception unit 130 arranges two markers, a first marker MK1 and a second marker MK2, with respect to the hand HD of the robot RB, and performs imaging unit CM (camera ) To obtain captured images (PIM1 to PIM4) as shown in FIGS. 3B to 3E, for example (S101). Note that the first marker MK1 and the second marker MK2 may be the same marker or different markers. 3 (B) to 3 (E), for convenience of explanation, only the outline and marker of the workpiece detected in the captured image are illustrated, not the object actually reflected in the captured image. . The same applies to FIGS. 7A to 7E described later.

  In the example of FIG. 3B, as a result of performing the feature amount detection processing (S102) and marker detection processing (S103) of the workpiece WK on the acquired captured image PIM1, the contour line OL1 is obtained as the feature amount of the workpiece WK. Is detected (S104), the first marker MK1 is detected at the position MD1 in the captured image PIM1, and the second marker MK2 is detected at the position MD2.

  In this case, the positional relationship between the marker detection position (MD1 and MD2) in the captured image PIM1 and the contour line OL1 of the workpiece WK is correct to the positional relationship between the marker (MK1 and MK2) and the workpiece WK in the real space. It can be determined that it corresponds (S105). Therefore, it is determined that the feature amount of the workpiece WK has been correctly detected.

  Then, the robot RB is controlled so that the contour line OL1 of the work WK detected in the captured image PIM1 matches (approaches) the contour line OL of the work WK shown in the reference image RIM (S106). Thereafter, if the visual servo has converged, the process is terminated, and if it has not converged, the process returns to step S101 and the process is repeated (S107). In this case, visual servoing is performed without any problem.

  However, as in the example of FIG. 3B, the contour line of the workpiece WK cannot always be detected correctly. For example, FIG. 3C shows an example in which the feature quantity of the work target is not detected. In the example of FIG. 3C, the captured image PIM2 is acquired (S101), and the feature amount detection process of the workpiece WK is performed (S102), but the contour line OL2 of the workpiece WK that should be detected is not detected. (S104). On the other hand, in the captured image PIM2, the first marker MK1 is detected at the position of MD1, and the second marker MK2 is detected at the position of MD2 (S103). Therefore, it can be determined that at least the first marker MK1 and the second marker MK2 are located within the imaging range. Therefore, it can be determined that the hand HD on which the marker is pasted and the work WK held by the hand HD are also located within the imaging range (S109), and only the feature amount detection processing of the work WK has failed. Can be judged.

  Therefore, in this case, based on the marker detection result MD1 and the marker detection result MD2, the contour line OL2 of the workpiece WK is estimated (specified) at a position that should be originally detected, and based on the estimated contour line OL2 Visual servo is performed (S110). This process is an example of a process when no feature amount is detected.

  Next, FIG. 3D shows an example when the feature amount of the work target is erroneously detected. In FIG. 3D, as a result of performing the feature amount detection process of the work WK in the acquired captured image PIM3 (S101, S102), the outline OL3 of the work WK is detected (S104), and the marker detection process is performed. As a result (S103), the first marker MK1 is detected at the position of MD1, and the second marker MK2 is detected at the position of MD2. However, since the actual captured image PIM3 shows that the hand HD is holding the work WK, it is originally below the marker detection positions (MD1 and MD2) as shown in FIG. The contour line of the work WK should be detected. That is, in the example of FIG. 3D, it can be determined that another object that is not the workpiece WK is erroneously detected as the workpiece WK (S105).

  Therefore, also in this case, based on the marker detection result MD1 and the marker detection result MD2, the contour line OL3 of the workpiece WK is estimated (specified) at a position where it should be originally detected, and based on the estimated contour line OL3. Visual servo is performed (S108). This process is an example of a process when a feature amount error is detected.

  Further, FIG. 3E shows an example in which a part of the work target is out of frame. In the example shown in FIG. 3E, as a result of performing the feature amount detection processing (S101, S102) and the marker detection processing (S103) of the workpiece WK in the acquired captured image PIM4, the contour OL4 of the workpiece WK is The marker is not detected (S104) and only one of the first markers MK1 is detected at the position of MD1. In this example, it is assumed that a marker can be detected whenever a marker appears in the captured image. Therefore, in this case, it can be determined that the second marker MK2 is not reflected in the captured image PIM4, that is, the hand HD and a part of the work WK are located outside the imaging range (S109). In other words, the hand HD and a part of the work WK are out of frame.

  Therefore, visual servo cannot be continued as it is. Therefore, in this case, visual servo is performed again after moving the robot RB to a position where the entire hand HD and the work WK are reflected (S111). This process is an example of a process when a frame-out occurs.

  As described above, the result of performing the feature amount detection processing of the work target can be roughly divided into the following four cases. The first case is a case where the work target is shown in the captured image, and the result that the feature amount of the work target was correctly detected was obtained. For example, the example of FIG. 3B described above corresponds to the first case. In the first case, the visual servo can be continued without any problem thereafter (S106, S107).

  On the other hand, the second case is a case where the feature amount of the work target cannot be detected because the work target is not shown in the captured image, contrary to the first case. . That is, the work target is out of the frame. For example, the example of FIG. 3E described above corresponds to the second case. In this second case, the visual servo cannot be performed as it is, but the visual servo can be resumed by moving the robot to a state where the work target is reflected in the captured image (S111).

  The first case and the second case described above have a difference in whether or not the feature quantity of the work target can be detected, but in both cases, the feature quantity detection process itself of the work target in step S102 in FIG. 4 is performed. It is done normally and the correct result is obtained. On the other hand, the following two cases (third case and fourth case) are cases where the feature amount detection processing itself is not normally performed.

  In the third case, although the work target is shown in the captured image, it is erroneously determined that the work target is not shown, and as a result, the feature amount of the work target cannot be detected. This is the case. The example of FIG. 3C described above corresponds to the third case. For example, the third case may be reached due to flickering of the light source or extremely poor quality of the captured image.

  In the fourth case, the result is that a feature that is not the feature of the work target is detected as the feature of the work target regardless of whether the work target is reflected in the captured image. This is the case. The example of FIG. 3D described above corresponds to the fourth case. For example, when a captured object includes a peripheral object that has a shape similar to that of the work target, the feature amount of the work target may be erroneously detected as described above.

  As described above, in both cases of the third case and the fourth case, the feature amount detection processing itself of the work target is not normally performed. For this reason, the visual servo cannot be normally performed using the feature values obtained in these cases as they are. In particular, in the fourth case as shown in FIG. 3D, when the robot is operated based on the erroneously detected feature amount, there is a possibility that the robot performs an operation completely different from the expected operation.

  In summary, based on the result obtained by the feature amount detection process, the visual servo can be continued normally thereafter only in the first case shown in FIG.

  However, as described above, in the second case shown in FIG. 3 (E), there is no problem in the feature amount detection process itself, so the visual servo is moved after the robot is moved to a state where the work target is reflected in the captured image. Can be dealt with, such as resuming (S111). In addition, in the third case shown in FIG. 3C and the fourth case shown in FIG. 3D, there is a problem in the feature amount detection process itself. Servo can be continued (S108, S110).

  In other words, even if the feature quantity of the work target cannot be detected correctly, if it can be determined whether this time is the second case, or the third case or the fourth case, how can the robot be handled thereafter? It can also be determined whether control should be performed.

  In the conventional method, these four cases could not be discriminated. However, in the present embodiment, these four cases are discriminated using the result of the marker detection process, and processing suitable for each case is performed. Is possible.

  As described above, even when the feature amount of the work target is not detected or erroneously detected, visual servoing can be continuously performed. As a result, when the feature amount of the work target is not detected or is erroneously detected in the captured image, for example, it is determined that an error has occurred in the visual servo, and the robot is stopped before moving to the target position. It is possible to prevent such a situation.

  As described above, when the feature amount of the work target can be specified or estimated, the processing unit 150 performs a process of controlling the robot 300 as a visual servo so that the captured image matches or approaches the reference image. . In the example of FIGS. 3A to 3E described above, the outline of the workpiece WK specified (or estimated) in the captured image is referred to by sequentially operating the robot RB while newly acquiring the captured image. The robot RB is controlled so as to gradually match or approach the contour line of the work WK in the image.

  This makes it possible to set the position and orientation of the robot RB to the target position and orientation. That is, an image that can be captured when the robot RB is in the target state is stored as a reference image, and if the robot RB is controlled so as to capture a captured image that matches or is close to the reference image, the robot RB assumes the target state. It becomes possible to control.

  Here, the target state of the robot RB is a state in which the robot RB assumes the target position and orientation. For example, in the example of FIGS. 3A to 3E described above, the workpiece WK is set to the target position TP. It is the state which has taken the position and posture which can be put on. The visual servo will be described in more detail later.

  In addition, the reference image storage unit 110 may store the reference image itself, or may store the feature amount of the work target in the reference image. Further, the marker need not be shown in the reference image. That is, the reference image can be generated without placing a marker on the end effector 310 of the robot 300. Therefore, it is possible to perform the processing of the present embodiment using another reference image in which no marker is shown.

  Here, the captured image refers to an image obtained by capturing with the image capturing unit 200. The captured image may be an image stored in an external storage unit or an image acquired via a network.

  In the present embodiment, the robot 300 may be operated based not only on the work target but also on the feature amount of the work place (work position) of the robot 300. In the following description, work objects and work places are collectively referred to as work objects.

  Thereby, visual servoing can be performed based on at least one feature amount of the work object and the work place.

  The feature amount (feature amount information) refers to an amount (information) representing the feature of an object that is extracted (detected) when an image is analyzed in image recognition. For example, the feature amount includes the area and width of the object shown in the image, the perimeter, the contour line, the feature point, the luminance, and the like.

  Furthermore, the feature amount detection process refers to a process for detecting a feature amount (feature amount information) of an object, such as a contour line detection process or a feature point detection process.

  Here, the contour line detection process refers to a process of detecting a contour line such as a work target shown on an image and specifying contour information. An example of the contour detection process is Hough transformation. In addition, for example, a binary image in which an area having a luminance gradient greater than or equal to a given value is blacked out is generated, and a contour line is extracted by thinning the binary image. You may perform the process to do. Note that the contour information is information representing the contour of an object. The contour information is, for example, a coordinate position on the image of the start point and end point of the contour, a vector, its size, and the like.

  The feature point detection process is a process for detecting feature points of an object such as a work target shown on an image and specifying feature point information. As a method of feature point detection processing, for example, corner detection method, other general corner detection (eigenvalue, FAST feature detection), local feature descriptor represented by SIFT (Scale invariant feature transform), SURF (Speeded Up Robust Feature).

  The marker is a tangible object that can be used as a mark, a character, a figure, a symbol, a pattern, a three-dimensional shape, a combination thereof, or a combination of these and a color, and can be fixed to an object. For example, stickers, stickers, labels, and the like. It doesn't matter what the shape, color, pattern, etc. of the marker is, but it can be easily distinguished from other areas for stable detection of the marker, for example, black for white areas An image or a sticker is desirable. Specific examples include markers MK1 to MK4 as shown in FIGS. 5 (A) to 5 (D), markers MK5 to MK8 such as map symbols shown in FIG. 5 (E), and FIG. 5 (F). Examples thereof include markers MK11 to MK13 such as letters and numbers shown. Further, the marker may be a QR (Quick Response) code, an AR (Augmented Reality) marker, or the like. As an exception, the marker may be an image that can be displayed on the display unit.

  As described above, since a marker that is easier to detect than the work target is used as the marker, the marker can be recognized (detected) at a recognition rate higher than the recognition rate (detection rate) of the work target.

  The marker detection process (marker recognition process) refers to a process of detecting (recognizing) a marker appearing in an image by image processing or the like. In the marker detection process, it is sufficient that at least the presence / absence of the marker and the arrangement position can be detected.

2. Details of Processing In the following, the work WK is gripped by the hand HD attached with eight markers (MK1 to MK8) as shown in FIGS. 6A and 6B, and the work WK is moved to the target position. A specific example of processing when performing the work to be performed will be described. 6A is a diagram when the hand HD is viewed from the front, and FIG. 6B is a diagram when the hand HD is viewed from an oblique direction. It is assumed that the hand HD in this example has four fingers, and markers are attached to the base and tip of each finger. That is, the first marker MK1 to the fourth marker MK4 are affixed to the front surface of the hand HD, and the fifth marker MK5 to the eighth marker MK8 are affixed to the back surface thereof. 6A and 6B, the marker is drawn large, and a part of the marker protrudes from the finger, but actually it protrudes from the surface of the finger of the hand. It is assumed that it is affixed. Furthermore, although the first marker MK1 to the eighth marker MK8 are illustrated as the same marker, they are different from each other and are distinguishable markers.

  As described above, when moving a work target by visual servo, it is necessary to detect a feature amount of the work target in the captured image. In this example, in this case, which case of the first case to the fourth case described above applies to this case, that is, whether the feature quantity of the work target is correctly detected, non-detected, or erroneously detected. In the first place, it is determined whether the work target is out of frame.

2.1. Details of processing when feature amount is not detected First, determination processing as to whether or not the current case corresponds to the third case described above, and processing performed after it is determined that the current case is the third case (features of the work target) Details of the process when the amount is not detected will be described.

  The process of determining whether or not the current case corresponds to the third case described above is performed based on the number of markers detected in the marker detection process.

  For example, in the feature amount detection process, the processing unit 150 detects that the feature amount of the work target in the captured image is not detected, but the number of markers detected in the marker detection process is equal to or greater than a given threshold. Based on the result of the detection process, the robot is operated (visual servo is performed). That is, in this case, it is determined that it corresponds to the above-described third case, and processing when the feature amount is not detected is performed.

  Here, the given threshold value may be a value that can be set by the user, or may be a value that is determined in advance and stored in the storage unit.

  For example, when the given threshold value is 3, the feature amount of the work target is non-detected, and three or more markers are detected in the marker detection process, it is determined that it corresponds to the third case described above. To do.

  In this case, for example, it is assumed that a captured image PIM11 as illustrated in FIG. 7A is acquired and the feature amount detection process is performed. In the case shown in FIG. 7A, since the contour line OL1 (feature amount) of the work WK that is the work target is detected, it can be determined that it does not correspond to the third case. The same applies when the captured image PIM12 as shown in FIG. 7B is acquired.

  On the other hand, FIG. 7C shows a state where the captured image PIM 13 is acquired and the feature amount detection process and the marker detection process are performed. In the example of FIG. 7C, markers are detected at positions MD1 to MD4 in the captured image PIM13 even though the contour line OL3 of the workpiece WK is not detected. That is, a total of four markers are detected. Therefore, it can be determined that the example of FIG. 7C corresponds to the third case.

  In the case of the captured image PIM15 shown in FIG. 7E, the contour line OL3 of the work WK is not detected in the feature amount detection process, and only one marker is located at the position of MD2 in the marker detection process. Not detected. Therefore, it can be determined that the example of FIG. 7E does not correspond to the third case. As will be described later, FIG. 7E is an example of a second case in which a work target frame-out has occurred.

  As described above, it is possible to determine a case where the feature amount of the work target cannot be detected although the work target is reflected in the captured image. As a result, it is possible to perform processing suitable for the case where the feature amount of the work target is not detected (third case).

  Specifically, in the feature amount detection process, the processing unit 150 determines that the feature amount of the work target in the captured image is not detected or is erroneously detected (as described later), but is detected in the marker detection processing. When the number of markers that have been determined is equal to or greater than a given threshold value, the feature quantity of the work target is specified based on the result of the marker detection process, and the robot is operated based on the specified feature quantity ( Perform visual servo).

  For example, in the captured image PIM21 shown in FIG. 8A, in the feature amount detection process, the feature amount of the work target is not detected, that is, the contour line or feature point of the workpiece WK is not specified, and the marker detection process It is assumed that markers are detected at positions MD1, MD2, and MD3.

  In this case, as shown in the picked-up image PIM22 in FIG. 8B, based on the marker detection positions (MD1 to MD3), the feature amount specifying process of the workpiece WK held by the hand HD is performed. In the example of FIG. 8B, the contour line OL is specified as the feature amount of the workpiece WK. When performing a gripping operation, the approximate position of the workpiece WK with respect to the hand HD is known. For example, the workpiece WK in the gripping state is located below the hand HD and within a predetermined distance from the hand HD. The relative positional relationship between the marker and the hand HD is also known. Therefore, the contour line of the workpiece WK can be estimated based on the marker detection position.

  This makes it possible to continue visual servoing or the like even when the feature amount of the work target is not detected or is erroneously detected.

2.2. Details of processing when feature amount is erroneously detected Next, processing for determining whether or not the current case corresponds to the above-described fourth case, and processing performed after it is determined that the case is the fourth case (work target Details of the processing when the feature amount is erroneously detected will be described.

  First, the determination process of whether or not the current case corresponds to the above-described fourth case is performed based on the position and orientation relationship between the marker in the captured image and the work target.

  That is, in the feature amount detection process, the processing unit 150 detects the feature amount of the work target in the captured image, and when the number of markers detected in the marker detection process is equal to or greater than a given threshold, The position and orientation relationship between the marker and the work target is specified based on the result and the result of the feature amount detection processing, and the feature amount of the work target is erroneously detected in the feature amount detection processing based on the specified position and orientation relationship. It is determined whether or not it has been done.

  Here, in the gripping state where the end effector is gripping the work target, the work target is always located within a given distance range from the marker detection position. Therefore, in the captured image showing the gripping state, when it is determined that the work target is located within a given distance range with reference to the detection position of the marker, erroneous detection of the feature amount of the work target has occurred. If it is determined that the work target is located outside the given distance range, it is determined that an erroneous detection of the feature amount of the work target has occurred.

  For example, in the example of FIG. 7D, the outline OL4 of the work WK is detected as the feature amount of the work target in the captured image PIM14. Also, four markers are detected, and the number of detected markers is equal to or greater than a given threshold. The given threshold is 3. However, when the detection position (MD1 to MD4) of the marker in the captured image PIM14 is used as a reference, the detection position of the contour line OL4 of the workpiece is located outside the given distance range DP3. Then, it can be determined that the feature amount of the work target is erroneously detected, that is, the fourth case.

  On the other hand, in the examples of FIGS. 7A and 7B, the contour line of the workpiece WK is detected, and the number of detected markers is equal to or greater than a given threshold. Since the detected position of the contour line of the workpiece is located within a given distance range (DP1 or DP2), it can be determined that the feature amount of the work target is normally detected (not erroneously detected). 7C and 7E can be determined not to be the fourth case because the outline of the workpiece WK is not detected in the first place.

  Here, the position / posture relationship between the marker and the work target refers to the relative position / posture of the work target when the position / posture of the marker is used as a reference, for example. In addition, as in the above-described example, the position and orientation relationship between the marker and the work object is information on whether or not the work object is located within a given distance range from the detection position of the marker, and FIG. ), The overlapping area RA between the first area AR and the second area BB may be used.

  As a result, it is possible to determine when the feature amount of the work target has been erroneously detected. As a result, it is possible to perform processing suitable for the case where the feature amount of the work target is erroneously detected (fourth case). For example, as described above, based on the result of the marker detection process, it is possible to perform the process of specifying the feature quantity of the work target, and continuously perform visual servoing based on the specified feature quantity.

  Further, the position and orientation relationship between the marker and the work target in the captured image may be obtained as follows. That is, the processing unit 150 is specified by the first region specified by the plurality of markers detected by the marker detection process of the marker and the feature quantity of the work target detected by the feature quantity detection process in the captured image. Based on the detection result of the overlapping area with the second area, the position and orientation relationship between the marker and the work target may be specified, and visual servo may be performed based on the specified position and orientation relationship.

  Here, the first region is a region specified by a plurality of markers detected by the marker detection process. For example, as shown in FIG. 9A, a plurality of detected markers (MK1) To MK4) is a region identified by linking any marker. Specifically, when the first marker MK1, the second marker MK2, and the third marker MK3 are connected, the triangular area AR1 having the vertices MK1 to MK3 becomes the first area.

  The second area is an area specified by the feature quantity of the work target detected by the feature quantity detection process. For example, as shown in FIG. 9A, the second area is a bounding box (rectangular area) BB specified based on the outline OL of the work target.

  As shown in FIG. 9A, when the first area is the area AR1 and the second area is the bounding box BB, the area RA is divided into the first area AR1 and the second area. It is detected as an overlapping area with BB. In FIG. 9A, the shaded area is the overlapping area RA. This area RA represents the position and orientation relationship between the marker and the work object in FIG.

  Thus, for example, as shown in FIG. 9A, when it is determined that the overlapping area RA exists in the captured image in a state where the end effector (hand HD) is holding the work object (work), It is possible to determine that the target feature amount is not erroneously detected. On the other hand, as in the example of FIG. 7D described above, when it is determined that the first region and the second region exist in the captured image in the gripping state, but the overlapping region does not exist. It is possible to determine that the feature quantity of the work target is erroneously detected.

  In addition, when the end effector is holding the work target, the position / posture relationship between the end effector (and the marker disposed on the end effector) and the work target is between the captured images continuously acquired during the visual servo. , Often not much change. Therefore, when the position / posture relationship between the end effector (marker) and the work target changes abruptly between successively acquired images, there is a high possibility that the feature amount of the work target is erroneously detected. it is conceivable that.

  Therefore, in addition to the method described above, the following determination process may be performed. That is, by comparing the position and orientation relationship between the marker and the work target detected in the previously acquired captured image and the position and orientation relationship between the marker and the work target detected in the currently acquired captured image, the feature amount of the work target is calculated. It may be determined whether or not is erroneously detected.

  In this case, as a preliminary preparation, the processing unit 150 identifies the position / orientation relationship between the marker and the work target based on the result of the marker detection process and the result of the feature amount detection process, and Perform the registration process.

  Here, the registration process refers to a process of storing the identified marker and the position and orientation relationship of the work target in the storage unit.

  Further, the position and orientation relationship between the marker and the work target can be expressed by the position and orientation relationship between the first area and the second area described above. In that case, for example, the position and orientation relationship between the marker and the work target is stored as shown in the table of FIG. The table in FIG. 9B shows the position and orientation relationship specified in the example of FIG. In the table of FIG. 9B, the left column describes the first area detected in the captured image and each marker that is the vertex of the first area, and the right column represents the work target. Each vertex coordinate of the bounding box (second region) is described. Each vertex coordinate of the bounding box in the right column is the coordinate when the line segment connecting the vertex 1 to the vertex 2 in the first area in the left column is the X axis, and the perpendicular line from the vertex 3 to the X axis is the Y axis. Position (X, Y).

  This makes it possible to compare the position and orientation relationship between the marker and the work target in captured images acquired at different timings in a comparison process to be described later.

  Then, after the registration process, the processing unit 150 performs a marker detection process and a feature amount detection process based on the second captured image received by the captured image reception unit 130, and a marker detection process in the second captured image. And the second position and orientation relationship between the marker and the work target are specified based on the result of the above and the result of the feature amount detection process.

  Further, the processing unit 150 performs a comparison process between the second position / posture relationship and the registered position / posture relationship, and determines that the second position / posture relationship corresponds to the registered position / posture relationship. In such a case, the second position / orientation registration process is performed. That is, in this case, the second position / posture is determined in order to determine that no erroneous detection of the feature value of the work target has occurred and to refer to the second position / posture relationship specified this time in the next comparison process. Perform the relationship registration process.

  Also, the reason for performing the second position / orientation relationship registration process at this time is that the position / orientation relationship between the unregistered (unknown) work target and the marker when specifying the second position / orientation relationship. This is because it may become clear. For example, in the example of FIG. 9B, only the position and orientation relationship between the first area AR2 and the bounding box BB, that is, the position and orientation relationship between the first marker MK1 to the third marker MK3 and the workpiece WK is registered. It was assumed that it was not found (it was not known). In this case, when the second position / posture relationship is specified, for example, the position / posture relationship between the first area AR3 and the bounding box BB, that is, the position / posture relationship between the fourth marker MK4 and the workpiece WK is newly found. Etc. Such information is useful in the comparison process of the position and orientation relationship after the next time. Therefore, the registration (update) process of the second position and orientation relationship described above is performed.

  Here, when the results of the comparison processing of the two position / orientation relations are summarized, if the processing unit 150 determines that the second position / orientation relation corresponds to the registered position / orientation relation, In the amount detection processing, when it is determined that the feature amount of the work target has not been erroneously detected, and it is determined that the second position / posture relationship does not correspond to the registered position / posture relationship, the feature amount detection processing In step (b), it is determined that the feature amount of the work target is erroneously detected.

  For example, the registered position / posture relationship is a position / posture relationship as shown in FIG. 7 (A), and the second position / posture relationship specified this time is also the position / posture relationship shown in FIG. 7 (A). If it is, it is determined that the two position and orientation relationships correspond to each other, and it is determined that no erroneous detection of the feature quantity of the work target has occurred.

  Here, the second captured image refers to a captured image that is captured or acquired after the registration processing of the position and orientation relationship between the marker and the work target.

  The second position / posture relationship refers to the position / posture relationship between the marker and the work target in the second captured image.

  Further, whether or not the second position / posture relationship corresponds to the registered position / posture relationship, for example, as shown in FIG. Specified area) and the area of the difference area DA between the overlapping area RA2 (area shown by oblique lines) in the registered position and orientation relationship. For example, when the area of the difference area DA is larger than a given threshold, it is determined that the two position / orientation relationships do not correspond. On the other hand, when the area of the difference area DA is less than a given threshold value, it is determined that the two position / orientation relationships correspond to each other.

  Here, the reason why a given threshold value is provided in the comparison process between two positions and orientations will be described. Even when the robot 300 operates as expected by visual servoing, the marker and the position and orientation of the work target with respect to the imaging unit 200 change. Therefore, how the marker and the work target are reflected in the captured image also changes, and an error may occur when the position and orientation of the marker and the work target are specified based on the captured image. However, this error occurs when the position / posture relationship between the marker and the work target is specified, and does not represent a change in the position / posture relationship between the actual marker and the work target. Therefore, in order to allow this error, the given threshold value described above is provided.

  As a result, it is possible to determine a case where no erroneous detection of the feature quantity of the work target has occurred. Then, it becomes possible to update the position and orientation relationship between the marker and the work target.

  On the other hand, if it is determined in the comparison process that the two positions and orientations do not correspond to each other, it is determined that the feature amount of the work target is erroneously detected, and the feature amount of the work target is erroneously detected. If this is the case (in the fourth case), the process is performed. In other words, when it is determined in the comparison process that the second position / orientation relationship does not correspond to the registered position / orientation relationship, the processing unit 150 determines whether the second position / orientation relationship is based on the result of the marker detection process. Processing for specifying the feature quantity of the work target in the captured image is performed, and the robot is operated based on the specified feature quantity (visual servo is performed).

  For example, the registered position / posture relationship is a position / posture relationship as shown in FIG. 7 (A), but the second position / posture relationship specified this time is as shown in FIG. 7 (C). In the case of the position / orientation relationship, it is determined that an erroneous detection of the feature amount of the work target has occurred. Then, as shown in FIG. 8B described above, processing for specifying the feature amount of the work target such as specifying the contour line OL of the workpiece is performed.

  As a result, it is possible to determine that an erroneous detection of the feature quantity of the work target has occurred in the captured image, and the process when the feature quantity of the work target is erroneously detected (during the fourth case). It can be performed.

2.3. Details of processing when frame-out occurs Next, processing for determining whether or not the current case corresponds to the second case described above, and processing performed after it is determined that the case is the second case (work target is frame The details of the process when it is out will be described.

  First, the determination process of whether or not the current case corresponds to the second case described above is performed based on the number of markers detected in the marker detection process or the presence or absence of the first region.

  Specifically, the processing unit 150 determines that the work target is positioned outside the imaging range when no marker is detected in the marker detection process or when the number of detected markers is equal to or less than a given threshold. Judge that.

  For example, in the example of FIG. 7E described above, the number of detected markers is 1, which is 3 or less of a given threshold. Therefore, in the example of FIG. 7E, it can be determined that the work target is located outside the imaging range, that is, the work target is out of frame in the captured image. This corresponds to the second case described above.

  On the other hand, in all of the examples of FIGS. 7A to 7D, since three or more markers are detected, it can be determined that the work target is located within the imaging range.

  Here, the imaging range is a range in real space reflected in a captured image captured by the imaging unit 200. For example, the imaging range is a range PA shown in FIG.

  Thereby, it is possible to determine whether or not the work target is located outside the imaging range, and to perform processing suitable for this case.

  In addition, as shown in FIG. 9A, the first area described above is an area that can be identified by connecting the marker detection positions. Therefore, if three or more markers are not detected, the first area is It cannot be specified. Conversely, in the captured image obtained by arranging the markers as shown in FIGS. 6A and 6B, if the first region cannot be specified, only two or less markers are detected. It can be determined that the other marker is located outside the imaging range. In this case, there is a high possibility that the work target is also out of frame.

  Therefore, the processing unit 150 determines that the work target is located within the imaging range when the first region can be identified, and determines that the work target is located outside the imaging range when the first region cannot be identified. You may judge.

  This also makes it possible to determine whether or not the work target is located outside the imaging range, and to perform processing suitable for this case.

  Specifically, when it is determined that the work target is out of frame, that is, when it is determined that the marker detection process has failed, the processing unit 150 determines the start position of the operation (visual servo) of the robot 300. Change processing may be performed.

  For example, as shown in FIG. 10A, the hand HD holding the work WK is positioned at the point P1, and the work WK is out of frame in the captured image captured by the imaging unit CM at this time. Shall.

  In this case, as shown in FIG. 10B, the hand HD is moved to any of the points P2 to P4 set in advance within the imaging range PA, and the visual servo is started again from the moved point. . In FIG. 10B, the hand HD is moved from the point P1 to the point P2.

  Points P2 to P4 are visual servo start positions. The start positions of these visual servos are set so that the work target is always reflected in the captured image obtained by imaging the end effector (hand HD) or the work target (work WK) at these points. Note that the start position of the visual servo is not limited to three, and any number may be set.

  As a result, even when the work target is out of frame, it is possible to move the end effector to a position where the work target appears in the captured image and restart the visual servo. Therefore, even when the work target is out of the frame, the visual servo can be continued without ending, so that the installation position of the imaging unit 200 can be flexibly changed. That is, it is not always necessary to arrange the imaging unit 200 so as to reflect the entire work site of the robot 300.

2.4. Processing Flow Next, the processing flow of this embodiment will be described with reference to the flowcharts of FIGS. 11 and 12.

  FIG. 11 is a flowchart of visual servo preparation. Here, as a preliminary preparation, a plurality of reference image and visual servo start positions are set.

  First, a work object is grasped with a hand (S201), and precisely aligned with a target position (S202). This alignment is performed using a jog tool or the like.

  Then, the work target after alignment is imaged by the imaging unit, and for example, a reference image such as the RIM in FIG. 3A described above is generated (S203). It is not necessary to place a marker on the hand or the like when generating the reference image.

  Furthermore, the feature amount detection process of the work object is performed on the reference image, and the detected feature value of the work object is stored (S204).

  Next, for example, a plurality of visual servo start positions such as point P2 to point P4 in FIG. 10B described above are set and registered (S205). The above is the flow of preparation for visual servo.

  Next, the flow of processing of the visual servo of this embodiment will be described using the flowchart of FIG.

  First, before performing visual servoing, the robot holds a work object with a hand (S301). At this time, for example, as shown in FIGS. 6A and 6B, markers are arranged on the hand. Then, the hand holding the work object is moved to the initial start position of the visual servo (S302). The initial start position of this visual servo is one of the start positions registered in step S205 in FIG.

  Then, the captured images as shown in FIGS. 7A to 7E are captured (S303), the feature amount detection processing of the work object is performed on the captured image (S304), and the features of the work object are detected. It is determined whether or not an amount has been detected (S305).

  When it is determined that the feature amount of the work object has been detected, marker detection processing is performed on the captured image (S306), and it is determined whether the number of detected markers is equal to or greater than a given threshold value (S307). ). In this case, whether or not the marker detection process is successful may be determined based on the presence or absence of the first region.

  If it is determined that the number of detected markers is equal to or greater than a given threshold value, it is determined that the work object and the marker appear in an ideal state for visual servoing in the captured image. To do. That is, during visual servoing, it is determined that it is desirable to detect the work object and the marker in the captured image while maintaining the position and orientation relationship at this time. Therefore, in order to determine whether the position / orientation relationship between the work object and the marker is unchanged from the position / orientation relationship at this time, the work object represented by the feature amount of the work object detected at this time is used. The relative position and orientation relationship between the position of the object and the detection position of the marker is stored (S309). For example, it is stored as shown in the table of FIG. As described above, this is for determining whether or not the feature amount of the work object has been erroneously detected in the subsequent processing.

  On the other hand, if it is determined in step S305 that the feature amount of the work object has not been detected, or if it is determined in step S307 that the number of detected markers is less than a given threshold, imaging is performed. In the image, it is determined that the work object and the marker are not shown in an ideal state for visual servoing. Therefore, the hand is moved to the next start position among the start positions of the visual servo registered in step S205 in FIG. 11 (S308), and the process is repeated from step S303.

  After step S309, the arm movement amount (servo amount) is calculated based on the feature amount of the work object in the captured image and the feature amount of the work object in the reference image (S310). Note that the value detected in advance in step S204 in FIG. 11 is used as the feature amount of the work object in the reference image.

  If the calculated arm movement amount is equal to or less than a given threshold, it is determined that the movement of the arm is unnecessary, that is, the visual servo has converged (S311), and the process is terminated.

  On the other hand, when the calculated movement amount of the arm is larger than a given threshold value, the arm is actually operated based on the movement amount of the arm (S312).

  Furthermore, after the operation of the arm is completed, the captured image is captured again (S313), and the feature amount detection process (S314) and the marker detection process (S315) of the work object are performed on the newly captured image.

  Then, it is determined whether or not the detection of both the feature quantity and the marker of the work object is successful (S316), and if it is determined that the detection of both the feature quantity and the marker of the work object is successful, The registered relative position / posture relationship (comparison target) is compared with the relative position / posture relationship between the current work object and the marker (S317). The previously registered relative position / posture relationship is the relative position / posture relationship between the work object and the marker registered (or updated) in step S309 or step S319 described later. At this time, the presence / absence of the overlapping region between the first region and the second region described above with reference to FIG. 9A is determined, and the feature amount of the work object is determined based on the presence / absence of the overlapping region. It may be determined whether or not it is erroneously detected.

  Next, as a result of the comparison processing in step S317, when it is determined that the relative position / posture relationship specified this time corresponds to (matches) the registered relative position / posture relationship (S318), the relative position / posture relationship specified this time is determined. Based on the above, the relative position and orientation relation to be compared in the comparison process is updated (S319). Then, the process returns to step S310, and the process is repeated until the visual servo converges. This case corresponds to the first case described above.

  If it is determined in step S316 that the detection of either the feature quantity or the marker of the work object has failed, it is subsequently determined whether only the marker detection process has succeeded (S320).

  If it is determined that only the marker detection process has succeeded, it is determined that this is the above-described third case in which only the feature amount of the work target has failed to be detected. In this case, as the non-detection process described in step S110 of FIG. 4 described above, the feature amount of the work target is estimated as shown in FIG. 8B based on the detected marker position. The process is performed (S321), the process returns to step S310, and the process is repeated until the visual servo converges.

  On the other hand, if it is determined in step S320 that only the marker detection process is not successful, it is determined whether only the feature amount detection process for the work target is successful (S322). If it is determined that only the feature amount detection process of the work target has been successful, the process returns to step S310 and the process is performed again. On the other hand, if it is determined that the feature amount detection process of the work object has also failed, it is determined that the visual servo has failed and the process ends (S323). In this case, it is determined that this corresponds to the second case described above, and as a process at the time of frame out in step S111 in FIG. 4 described above, the hand is moved to the start position of another visual servo, and the visual servo is You may perform the process which redoes.

  If it is determined in step S318 that the relative position / posture relationship specified this time does not correspond to (match) the registered relative position / posture relationship as a result of the comparison processing, this corresponds to the above-described fourth case. judge. Then, as processing at the time of erroneous detection in step S108 of FIG. 4 described above, processing for estimating the feature amount of the work object is performed based on the position of the detected marker (S321), and the process returns to step S310, and visual servoing is performed. Repeat the process until converges. The above is the processing flow of the visual servo of this embodiment.

  Further, the imaging unit (camera) 200 used in the above-described embodiment includes an imaging element such as a charge-coupled device (CCD) and an optical system. The imaging unit 200 is arranged at an angle such that a detection target (work target or the end effector 310 of the robot 300) in the visual servo is within the angle of view of the imaging unit 200, for example, on a ceiling or a work table. The Then, the imaging unit 200 outputs information of the captured image to the control device 100 or the like. However, in the present embodiment, the information of the captured image is output as it is to the control device 100, but the present invention is not limited to this. For example, the imaging unit 200 can include a device (processor) used for image processing or the like.

3. Robot System and Robot Next, FIGS. 13A and 13B show a configuration example of the robot system and the robot 300 to which the control device 100 of the present embodiment is applied. In both cases of FIG. 13A and FIG. 13B, the robot 300 includes an end effector 310. The robot system includes a robot 300, a marker, and an imaging unit 200.

  The end effector 310 is a component attached to an end point of an arm in order to grip, lift, lift, attract, or process a workpiece (work object). The end effector 310 may be, for example, a hand (gripping unit), a hook, a suction cup, or the like. Further, a plurality of end effectors may be provided for one arm. The arm is a part of the robot 300 and refers to a movable part including one or more joints.

  For example, in the robot of FIG. 13A, the robot main body 300 (robot) and the control device 100 are configured separately. In this case, some or all of the functions of the control device 100 are realized by, for example, a PC (Personal Computer).

  Further, the robot according to the present embodiment is not limited to the configuration shown in FIG. 13A, and the robot body 300 and the control device 100 may be configured integrally as shown in FIG. 13B. That is, the robot 300 may include the control device 100. Specifically, as illustrated in FIG. 13B, the robot 300 includes a robot body (having an arm and an end effector 310) and a base unit portion that supports the robot body, and the control device 100 is included in the base unit portion. May be stored. The robot 300 in FIG. 13B has a structure in which wheels and the like are provided in the base unit portion so that the entire robot can move. Although FIG. 13A shows an example of a single arm type, the robot 300 may be a multi-arm type robot such as a double arm type as shown in FIG. 13B. The robot 300 may be moved manually, or may be moved by providing a motor for driving wheels and controlling the motor by the control device 100. Moreover, the control apparatus 100 is not necessarily provided in the base unit part provided under the robot 300 as shown in FIG.

  As illustrated in FIG. 14, the function of the control device 100 may be realized by a server 500 that is connected to the robot 300 via a network 400 including at least one of wired and wireless.

  Alternatively, in the present embodiment, a part of the processing of the control device of the present invention may be performed by the control device on the server 500 side. In this case, the processing is realized by distributed processing with the control device provided on the robot 300 side. The control device on the robot 300 side is realized by a terminal device 330 (control unit) installed in the robot 300, for example.

  In this case, the control device on the server 500 side performs a process assigned to the control device of the server 500 among the processes in the control device of the present invention. On the other hand, the control device provided in the robot 300 performs the process assigned to the control device of the robot 300 among the processes of the control device of the present invention. Each process of the control apparatus of the present invention may be a process assigned to the server 500 side or a process assigned to the robot 300 side.

  Thereby, for example, the server 500 having a higher processing capability than the terminal device 330 can perform processing with a large processing amount. Further, for example, the server 500 can collectively control the operations of the robots 300, and it is easy to cause a plurality of robots 300 to perform cooperative operations.

  In recent years, the production of a small number of various types of parts has been increasing. And when changing the kind of components to manufacture, it is necessary to change the operation | movement which a robot performs. With the configuration as shown in FIG. 14, it is possible to change the operation performed by the robot 300 in a batch without the server 500 performing the teaching operation on each robot of the plurality of robots 300 again.

  Furthermore, with the configuration as shown in FIG. 14, it is possible to significantly reduce the trouble of performing software update of the control device 100 as compared with the case where one control device 100 is provided for each robot 300. become.

4). Visual Servo Here, the outline of visual servo, the flow of position-based visual servo, and the flow of feature-based visual servo will be described.

  The visual servo is a kind of servo system that tracks a target by measuring a change in the position of the target as visual information and using it as feedback information. Visual servo is roughly classified into position-based visual servo and feature-based visual servo according to input information (control amount) to the servo system. In the position-based visual servo, the position information and posture information of the object are input information to the servo system, and in the feature-based visual servo, the feature amount of the image is input information to the servo system. In addition, there is a method that hybridizes the position base and the feature base. The visual servo handled in the present invention covers all these methods.

  These visual servos are common in that the input information to the servo system is obtained based on the reference image and the captured image.

4.1. Flow of Position-Based Visual Servo First, the flow of position-based visual servo is shown in the flowchart of FIG. In the position-based visual servo, first, a reference image is set (S1). Here, the reference image is also referred to as a target image or a goal image, and is an image serving as a visual servo control target, which is an image representing the target state of the robot 300. That is, the reference image is an image that represents the target position or target posture of the robot 300 or an image that represents a state where the robot 300 is located at the target position. Also, the reference image must be prepared in advance and stored in the storage unit.

  Next, a work space is imaged by the imaging unit 200, and a captured image is acquired (S2). The captured image represents the current state of the work space. When the robot 300 and the workpiece are reflected in the captured image, the captured image represents the current state of the robot 300 and the workpiece. Although a processing delay occurs depending on the performance of the imaging unit 200, it is assumed here that the captured image shows the current state even when the processing delay occurs.

  For example, FIG. 16A shows a specific example of the reference image RIM, and FIG. 16B shows a specific example of the captured image PIM. In the captured image PIM, the robot RB points the arm AM and the hand HD (or end point EP) upward, but in the reference image RIM, the robot RB bends the arm AM and brings the hand HD closer to the work WK. . Therefore, in this specific example, the robot RB is controlled so that the arm AM of the robot RB is bent and the hand HD is brought close to the work WK.

  Next, a control command is generated (S3). For example, the control command generation is performed by using homography, which is one of coordinate transformations, based on the reference image and the captured image. In this case, a homography matrix is obtained, and a speed command is generated as a control signal for the robot 300 from the homography matrix.

  Here, the control signal (control command) refers to a signal including information for controlling the robot 300. For example, the control signal includes a speed command. The speed command refers to a command method for giving a moving speed or a rotating speed of an end point or the like of the arm of the robot 300 as information for controlling each part of the robot 300.

  Then, based on the generated control signal, it is determined whether or not the control amount (here, the position and posture of the robot 300) has converged to the target value (S4). For example, in the case of using homography, when the velocity vector represented by the velocity command is 0, it can be considered that the position or orientation that is the controlled variable has reached the target state. If the speed vector is not 0, it is determined that the control amount has not converged to the target value. Here, convergence means approaching the target value as much as possible, or reaching the target value.

  If it is determined that the control amount has converged to the target value, the visual servo is terminated. On the other hand, when it is determined that the control amount has not converged to the target value, the processing unit 150 sends a control command to the robot 300 (S5).

  In the position-based visual servo, the above processing is repeated until the control amount converges to the target value.

4.2. Flow of Feature-Based Visual Servo Next, the flow of feature-based visual servo is shown in the flowchart of FIG. As with the position-based visual servo, a reference image is first set (S10), and then the work space is imaged by the imaging unit 200 to obtain a captured image (S11). When using feature-based visual servoing, it is desirable to perform feature extraction (feature amount detection processing) of the reference image and calculate the feature amount when setting the reference image. Alternatively, reference image information obtained by extracting features (feature amounts) of the reference image may be stored in the storage unit. The feature extraction of the image is performed using, for example, corner detection or a Gaussian filter.

  Next, the feature of the captured image is extracted (S12). Note that the feature extraction of the reference image is preferably performed at the time of setting the reference image, but may be performed in this step. In the feature extraction, an image feature amount is obtained as input information (control amount) to the visual servo system.

  Then, based on the feature amount of the image, it is compared whether or not the reference image matches the captured image (S13). If it is determined that the images match (S14), the visual servo is terminated. On the other hand, if it is determined that the images do not match (S14), a control command is generated (S15), and the control command is sent to the robot 300 (S16).

  In the feature-based visual servo, the above processing is repeated until the control amount converges to the target value.

  Note that a part or most of the processing of the control device, the robot system, the robot, and the like of the present embodiment may be realized by a program. In this case, a control device, a robot system, a robot, and the like according to the present embodiment are realized by a processor such as a CPU executing a program. Specifically, a program stored in the information storage medium is read, and a processor such as a CPU executes the read program. Here, the information storage medium (computer-readable medium) stores programs, data, and the like, and functions as an optical disk (DVD, CD, etc.), HDD (hard disk drive), or memory (card type). It can be realized by memory, ROM, etc. A processor such as a CPU performs various processes according to the present embodiment based on a program (data) stored in the information storage medium. That is, in the information storage medium, a program for causing a computer (an apparatus including an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing the computer to execute processing of each unit) Is memorized.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described at least once together with a different term having a broader meaning or the same meaning in the specification or the drawings can be replaced with the different term in any part of the specification or the drawings. The configurations and operations of the control device, the robot system, and the robot are not limited to those described in the present embodiment, and various modifications can be made.

100 control device, 110 reference image storage unit, 130 captured image reception unit,
150 processing unit, 200 imaging unit, 300 robot, 310 end effector,
330 terminal devices, 400 networks, 500 servers

Claims (16)

A processing unit for operating a robot having an end effector for moving a work object;
A captured image receiving unit that receives a captured image obtained by imaging the work target and the marker on the work target or the end effector by an imaging unit;
Including
The processor is
Even when the feature amount of the work object cannot be detected in the captured image, the work object and the marker are moved while the robot moves from the first position to the second position different from the first position. A control device characterized in that the robot is operated by imaging a plurality of times. In claim 1,
The processor is
Performing feature amount detection processing of the work object in the captured image,
In the feature amount detection process, even when the feature amount of the work object cannot be detected, or even when the feature amount of the work object is erroneously detected, the robot moves from the first posture to the second position. Imaging the work object and the marker multiple times while moving to a posture,
A control device that operates the robot based on a result of marker detection processing of the marker. In claim 2,
The processor is
In the feature amount detection process, the feature amount of the work object in the captured image is not detected, but when the number of markers detected in the marker detection process is equal to or greater than a given threshold,
A control apparatus that operates the robot based on a result of the marker detection process. In claim 2 or 3,
The processor is
In the feature amount detection process, when the feature amount of the work object in the captured image is detected, and the number of markers detected in the marker detection process is equal to or greater than a given threshold,
Based on the result of the marker detection process and the result of the feature amount detection process, the position / posture relationship between the marker and the work object is specified, and the feature amount detection is performed based on the specified position / posture relationship. In the processing, it is determined whether or not the feature amount of the work object is erroneously detected. In any of claims 2 to 4,
The processor is
Based on the result of the marker detection process and the result of the feature amount detection process, the position and orientation relationship between the marker and the work object is specified, and the specified position and orientation relationship is registered. Control device. In claim 5,
The processor is
After the registration process, based on the second captured image received by the captured image reception unit, the marker detection process and the feature amount detection process are performed,
Based on the result of the marker detection process in the second captured image and the result of the feature amount detection process, the second position and orientation relationship between the marker and the work object is specified,
A comparison process between the second position and orientation relationship and the registered position and orientation relationship is performed,
The control apparatus, wherein when the second position / orientation relationship is determined to correspond to the registered position / orientation relationship, the registration processing of the second position / orientation relationship is performed. In claim 6,
The processor is
In the comparison process, when it is determined that the second position / orientation relationship does not correspond to the registered position / orientation relationship,
Based on the result of the marker detection processing, the feature amount of the work object in the second captured image is specified, and the robot is operated based on the specified feature amount. Control device. In any one of Claims 2 thru | or 6.
The processor is
In the feature amount detection process, it is determined that the feature amount of the work object in the captured image has not been detected or has been erroneously detected, but the number of markers detected in the marker detection process is given. If it is greater than or equal to the threshold,
A control device that performs a process for specifying the feature amount of the work object based on a result of the marker detection process, and operates the robot based on the specified feature amount. In any of claims 2 to 8,
The processor is
In the marker detection process, when the marker is not detected, or when the number of the detected markers is equal to or less than a given threshold, it is determined that the work target is located outside the imaging range. Control device characterized. In any one of Claims 2 thru | or 9.
The processor is
A control apparatus that performs a change process of a start position of the operation of the robot when it is determined that at least one of the feature amount detection process and the marker detection process has failed. A processing unit for operating a robot having an end effector for moving a work object;
A captured image receiving unit that receives a captured image obtained by imaging the work object and the work object or a plurality of markers on the end effector by an imaging unit;
Including
The processor is
In the captured image, based on a first region specified by the plurality of markers detected by marker detection processing of a marker, and a feature amount of the work object detected by the feature amount detection processing of the work object. Identifying the position and orientation relationship between the marker and the work object based on the detection result of the overlapping region with the second region identified in this way, and operating the robot based on the identified position and orientation relationship A control device characterized by. In claim 11,
The processor is
Perform registration processing of the identified position and orientation relationship,
After the registration process, based on the second captured image received by the captured image reception unit, the marker detection process and the feature amount detection process are performed,
Based on the result of the marker detection process in the second captured image and the result of the feature amount detection process, the second position and orientation relationship between the marker and the work object is specified,
A comparison process between the second position and orientation relationship and the registered position and orientation relationship is performed,
When it is determined that the second position / posture relationship corresponds to the registered position / posture relationship, the feature amount of the work target is not erroneously detected in the feature amount detection process. And
If it is determined that the second position and orientation relationship does not correspond to the registered position and orientation relationship, the feature amount of the work object is erroneously detected in the feature amount detection process. A control device characterized by determining. In claim 11 or 12,
The processor is
If the first area can be specified, it is determined that the work object is located within the imaging range;
When the first area cannot be specified, it is determined that the work object is located outside the imaging range. A robot having an end effector for moving a work object;
A marker on the work object or the end effector;
An imaging unit for imaging the work object and the marker;
Including
The robot is
Even when the feature amount of the work object cannot be detected in the captured image obtained by capturing the work object and the marker by the imaging unit, the first posture is changed to a second posture different from the first posture. A robot system that moves the robot by imaging the work object and the marker a plurality of times while moving. Including an end effector to move the work object,
In the captured image obtained by imaging the work object and the marker on the work object or the end effector, even if the feature amount of the work object cannot be detected, the first posture is the first posture. A robot characterized in that it operates by imaging the work object and the marker a plurality of times while moving to a different second posture. A robot control method for operating a robot having an end effector for moving a work object,
Imaging the work object and the marker on the work object or the end effector;
Even when the feature amount of the work object cannot be detected in the captured image obtained by capturing the work object and the marker, the robot moves from the first position to the second position different from the first position. In between, imaging the work object and the marker a plurality of times and operating the robot;
A robot control method including:

Images