JP2014188617A - Robot control system, robot, robot control method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a robot control system, a robot, a robot control method, and a program or the like which are capable of performing positioning from a plurality of directions based on a captured image captured by a single imaging unit.SOLUTION: A robot control system 10 includes: a reference image storage section 110 which stores a reference image being an image expressing a target state of a robot 30; a processing unit 120 which outputs a control signal of the robot 30; and a robot controller 130 which controls the robot 30 based on the control signal. Then, the reference image storage section 110 stores a reference image including a reflection image of an object. Further, the processing unit 120 performs visual servo based on a captured image in which a reflecting mirror reflecting the object enters an imaging range of an imaging units 40 and the reference image.

Description

  The present invention relates to a robot control system, a robot, a robot control method, a program, and the like.

  In recent years, industrial robots have been increasingly introduced to mechanize and automate work performed by humans at production sites. However, precise positioning is a prerequisite for positioning the robot arm and hand, which is a barrier to robot introduction.

  Here, there is a visual servo as one of methods for positioning a robot arm or hand. Certain types of visual servos are useful in that they do not require precision in calibration, and are attracting attention as a technology that lowers the barrier for robot introduction.

  As an invention related to these visual servos, there is a conventional technique described in Patent Document 1.

JP 2003-305675 A

  The following problems exist in visual servoing. First, even if it seems that the position is precisely aligned from a certain direction, there may be a situation where the position is not aligned when viewed from the other direction. Further, there may be an occlusion in which the target object is not reflected in the captured image due to the posture of the robot arm or an obstacle.

  According to some aspects of the present invention, a robot control system, a robot, a robot control method, a program, and the like that can perform alignment from a plurality of directions based on a captured image captured by one imaging unit are provided. can do.

  In one embodiment of the present invention, a reference image storage unit that stores a reference image that is an image representing a target state of a robot, a processing unit that outputs a control signal of the robot, and the robot is controlled based on the control signal The reference image storage unit stores the reference image including a reflection image of the target object, and the processing unit includes a reflector that reflects the target object within an imaging range of the imaging unit. The present invention relates to a robot control system that performs visual servoing based on a captured image that enters and the reference image.

  In one aspect of the present invention, the processing unit acquires a reference image including a reflection image of the object from the reference image storage unit. On the other hand, the imaging unit captures a captured image in which the reflecting mirror that reflects the object falls within the imaging range of the imaging unit, and the processing unit acquires the captured image. Then, the processing unit performs visual servoing based on the acquired captured image and the reference image.

  Therefore, it is possible to perform alignment from a plurality of directions based on a captured image captured by one imaging unit and a reference image corresponding to the captured image.

  In the aspect of the invention, the reference image storage unit stores the reference image including the reflection image of the object, the processing unit including the reflection mirror and the object. The visual servo may be performed based on the captured image that falls within the imaging range and the reference image.

  Accordingly, it is possible to perform alignment and the like from two directions, ie, the direction from the imaging unit to the object and the direction from the reflector to the object, by performing only one visual servo.

  In the aspect of the invention, the reference image storage unit stores the reference image including two or more reflection images of the object, and the processing unit includes two or more reflectors that capture the image. The visual servo may be performed based on the captured image that falls within the imaging range of the unit and the reference image.

  Thereby, only by performing one visual servo, the three directions of the direction from the imaging unit to the target, the direction from the first reflecting mirror to the target, and the direction from the second reflecting mirror to the target are provided. Therefore, it is possible to perform alignment.

  Moreover, in 1 aspect of this invention, it has a camera control part which controls the imaging direction of the said imaging part, and the said process part is a 1st captured image imaged when the said imaging part is pointed at the said target object, A first visual servo based on the first reference image in which the object is reflected, a second captured image captured when the imaging unit is directed to the reflecting mirror, and the reflected image of the object. You may perform the 2nd visual servo based on the 2nd reference image containing.

  Thereby, since a reflecting mirror can be installed in a more free position, it becomes possible to use a more appropriate reference image.

  In the aspect of the invention, the processing unit includes a third captured image captured when the imaging unit is directed to the second reflecting mirror, and a third reflected image including the second reflected image of the object. A third visual servo based on the reference image may be performed.

  As a result, even when occlusion occurs on the object, it is possible to appropriately perform visual servoing and the like.

  In one aspect of the present invention, the reflecting mirror is a straight line perpendicular to a straight line connecting the imaging unit and the object in a work area, and the vertical line passing through the object is used as a reference. As above, it may be installed on the opposite side to the position of the imaging unit.

  Thereby, it is possible to capture a captured image in which the object is reflected on the reflecting mirror.

  In the aspect of the invention, the reflecting mirror may be attached to the arm of the robot.

  Thereby, the position of the reflecting mirror can be changed by moving the arm of the robot.

  In one embodiment of the present invention, the robot includes a first arm and a second arm, the reflecting mirror is attached to the first arm, and the processing unit includes the first arm. The visual servo that moves the second arm may be performed based on the captured image in which the reflecting mirror attached to the imaging unit falls within the imaging range of the imaging unit and the reference image.

  As a result, even when occlusion occurs during visual servoing, visual servoing can be performed flexibly.

  In the aspect of the invention, the reference image storage unit stores the reflection image of the object as the reference image, and the processing unit detects a reflector region where the reflector is reflected in the captured image. Processing may be performed, and the visual servo may be performed based on the image of the reflector region in the captured image and the reference image.

  Thereby, it is possible to facilitate the generation of the reference image.

  In the aspect of the invention, the processing unit may perform a process of detecting a marker or a frame provided on the reflector as the detection process of the reflector region.

  This makes it possible to easily detect the reflector region.

  In one aspect of the present invention, the processing unit may perform a process of controlling the robot as the visual servo so that the captured image matches or approaches the reference image.

  Thus, if a reference image showing the target posture of the robot is used, it is possible to cause the robot to take a posture that is the same as or close to the target posture.

  In the aspect of the invention, the processing unit may include a marker or a frame of the second reflecting mirror in the reflected image included in the captured image in which the first reflecting mirror that reflects the object falls within the imaging range. May be detected, and the area in which the second reflecting mirror is reflected may be masked based on the detection result.

  As a result, it is possible to prevent the visual servo from converging due to unnecessary reflection of the reflecting mirror.

  According to another aspect of the present invention, there is provided a robot control system that controls a robot so that a captured image captured by an imaging unit matches or approaches a reference image, wherein the captured image and the reference image are a reflecting mirror. This is related to a robot control system including a mirror image of an object reflected in

  Another aspect of the present invention relates to a robot including the robot control system.

  In another aspect of the present invention, a reference image that represents a target state of a robot, stores a reference image including a reflection image of an object, is obtained from an imaging unit, and a reflecting mirror that reflects the object is an imaging range of the imaging unit. The present invention relates to a robot control method that performs the visual servoing based on the captured image and the reference image that are contained therein, outputs a control signal for the robot, and controls the robot based on the control signal.

  Another aspect of the present invention relates to a program that causes a computer to function as each of the above-described units.

The system configuration example of this embodiment. 2A and 2B are configuration examples of the robot according to the present embodiment. The flowchart explaining the flow of a process of a position-based visual servo. 4A and 4B are explanatory diagrams of a reference image and a captured image. The flowchart explaining the flow of a process of a feature-based visual servo. 6A and 6B are explanatory diagrams of visual servoing using a reflecting mirror. Explanatory drawing of the arrangement position of a reflective mirror. Explanatory drawing of the detection process of the reflective mirror area | region using a marker. The flowchart explaining the flow of a process of this embodiment. FIGS. 10A and 10B are explanatory diagrams of a method for changing the imaging direction of the imaging unit. 11A and 11B are explanatory diagrams when occlusion occurs. The flowchart explaining the flow of a process of a 1st modification. Explanatory drawing of the method of installing a reflective mirror in the arm of a robot. The flowchart explaining the flow of a process of a 2nd modification.

  Hereinafter, this embodiment will be described. First, an overview of the present embodiment will be described, and then a system configuration example will be described. Then, after describing the outline of the visual servo, details of processing performed in the present embodiment, a first modification, and a second modification will be described using a flowchart and the like. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. Outline In recent years, industrial robots have been increasingly introduced in production sites to mechanize and automate work performed by humans. However, precise positioning is a prerequisite for positioning the robot arm and hand, which is a barrier to robot introduction.

  Here, there is a visual servo as one of methods for positioning a robot arm or hand. Visual servo is a technique for controlling a robot so that a difference between a reference image (goal image, target image) and a captured image (current image) becomes small. The reference image used for the visual servo is an image obtained by capturing a work target such as a workpiece in advance, and is an image representing the target state of the robot. On the other hand, the captured image is an image obtained by capturing a current state of a work, an arm, or the like when the robot is actually operated. Certain types of visual servos are useful in that they do not require precision in calibration, and are attracting attention as a technology that lowers the barrier for robot introduction.

  The following problems exist in visual servoing. First, even if it seems that the position is precisely aligned from a certain direction, there may be a situation where the position is not aligned when viewed from the other direction. This is the first problem. Therefore, it is necessary to use an image captured from a necessary and sufficient direction so that such a case does not occur. For the first problem, the following measures are generally taken, but each has a secondary problem.

  First, as a first countermeasure to the first problem, there is a method of capturing a reference image and a captured image from multiple directions using a plurality of cameras. However, in this case, there is a problem that it is necessary to secure a place for installing a plurality of cameras in the work area, and the processing is complicated and slow because images corresponding to the number of cameras are captured. There's a problem.

  Next, as a second countermeasure to the first problem, there is a method of capturing a reference image and a captured image from a plurality of postures using a camera installed on a robot arm. However, in this case, it is necessary to move the arm of the robot every time the captured image is taken, and there is a problem that only an image from a posture that can be taken by the camera installed on the robot arm is taken. There is a problem that it takes.

  Next, when performing visual servoing, there may occur an occlusion in which an object does not appear in the captured image due to the posture of the robot arm or the like. This is the second problem. Patent Document 1 described above is an invention that solves this problem, and sets a position or posture at which occlusion occurs in a prohibited area to prevent a robot arm from moving into the prohibited area.

  In order to solve the above-described problems, the robot control system according to the present embodiment efficiently captures a captured image and a reference image using a reflecting mirror. Specifically, a reflecting mirror is installed so as to fall within the imaging range of the camera in the work area, and a captured image is captured. Accordingly, it is possible to perform alignment from a plurality of directions based on the captured image captured by one imaging unit and the corresponding reference image.

2. System Configuration Example Next, a configuration example of the robot control system 10 and the robot 30 of this embodiment is shown in FIG.

  The robot control system 10 includes a reference image storage unit 110, a processing unit 120, a robot control unit 130, a camera control unit 140, and an I / F unit (input unit) 150. Note that the robot control system 10 is not limited to the configuration shown in FIG. 1, and various modifications such as omitting some of these components or adding other components are possible.

  Next, processing performed in each unit of the robot control system 10 will be described.

  First, the reference image storage unit 110 stores a reference image that is an image representing the target state of the robot 30, or serves as a work area for the processing unit 120, the robot control unit 130, and the camera control unit 140. The function can be realized by a memory such as a RAM or an HDD (hard disk drive).

  Next, the processing unit 120 performs visual servoing of the robot 30 based on a captured image obtained from the imaging unit 40 described later and a reference image acquired from the reference image storage unit 110, and outputs a control signal of the robot 30. Output.

  Then, the robot control unit 130 controls the robot 30 based on the control signal.

  In addition, the camera control unit 140 controls the imaging direction of the imaging unit 40. Note that the functions of the processing unit 120, the robot control unit 130, and the camera control unit 140 can be realized by hardware such as various processors (CPU and the like), ASIC (gate array and the like), a program, and the like.

  Furthermore, the input unit (I / F unit) 150 is an interface for receiving input from the operator to the robot control system 10 and exchanging information between the robot 30 and the imaging unit 40. When receiving an input from an operator, the input unit (I / F unit) 150 may be configured by a switch, a button, a keyboard, a mouse, or the like. When information is exchanged between the robot control system 10 and the robot 30, the input unit (I / F unit) 150 is a communication unit that performs communication via a network including at least one of wired and wireless. Also good.

  As shown in FIG. 1, the robot 30 includes a control unit 310, a first arm 320, a second arm 330, a first end effector 340, and a second end effector 350. . Further, as shown in FIGS. 2A and 2B, in the robot 30, the first end effector 340 is provided at the end point of the first arm 320, and the end point of the second arm 330 is provided at the end point of the second arm 330. A second end effector 350 is provided. Note that the robot 30 is not limited to the configurations shown in FIGS. 1, 2A, and 2B, and some of these components may be omitted or other components may be added. Various modifications are possible.

  Next, processing performed in each part of the robot 30 will be described.

  The control unit 310 controls each unit of the robot 30 (first arm 320, second arm 330, first end effector 340, second end effector 350, and the like) based on information from the robot control unit 130. I do.

  The robot 30 of this embodiment is a double-armed robot, and the robot 30 is provided with a first arm 320 and a second arm 330 different from the first arm 320. Here, the arm is a part of the robot 30 and refers to a movable part including one or more joints.

  The end effector refers to a gripping part or tool provided at the end point of the arm.

  The gripping part is a part used for gripping, lifting, lifting, or attracting a workpiece. The gripping part may be a hand, a hook, a suction cup or the like. A plurality of hands may be provided for one arm.

  The tool is a tool for performing work such as machining on a workpiece or work target, and refers to a tool such as a drill or a driver, for example. The tool may be gripped by a hand or may be provided directly at the end point of the arm. Here, the end point of the arm refers to a point at the tip of the arm and a portion that is not connected to other parts other than the hand of the robot 30.

  Further, the imaging unit 40 images the work site of the robot 30 and outputs a captured image to the processing unit 120 via the control unit 310 and the I / F unit 150. For example, the imaging unit 40 may be a visible camera or an infrared camera. Further, for example, the visible camera may be a camera using a CCD or CMOS as an image pickup device, and may be a monochrome camera or a color camera. When a monochrome camera is used, a grayscale image is captured as a captured image, and when a color camera is used, a color image is captured as a captured image. The visible camera and the infrared camera may include a device (processor) used for image processing or the like. In the present embodiment, the imaging unit 40 outputs the captured image itself, but is not limited to this. For example, the imaging unit 40 may perform part of the image processing performed in the processing unit 120. In that case, the imaging information after the image processing is performed on the captured image is output to the processing unit 120.

  Next, an example of the robot control system 10 and the robot 30 is shown in FIGS. 2 (A) and 2 (B). For example, in the robot 30 of FIG. 2A, the robot body 30 (robot) and the robot control system 10 are integrally configured. However, the robot control system 10 and the robot 30 of the present embodiment are not limited to the configuration of FIG. 2A, and the robot body 30 and the robot control system 10 are configured separately as shown in FIG. May be. Specifically, as shown in FIG. 2A, the robot 30 includes a robot body (having a first arm 320, a second arm 330, a first end effector 340, and an end effector 350) and a robot body. The base unit unit that supports the robot control system 10 may be stored in the base unit unit. The robot 30 in FIG. 2A may have a configuration in which wheels and the like are provided in the base unit portion so that the entire robot 30 can move. 2A is an example of a double-arm type, the robot 30 may be a single-arm type or a multi-arm type robot. The robot 30 may be moved manually, or may be moved by providing a motor for driving wheels and controlling the motor by the robot control system 10. Moreover, the robot control system 10 is not necessarily provided in the base unit part provided under the robot 30 as shown in FIG.

3. Overview of Visual Servo Before describing the details of the processing of this embodiment, an overview of visual servo, the flow of position-based visual servo, and the flow of feature-based visual servo will be described.

  The visual servo is a kind of servo system that tracks a target by measuring a change in the position of the target as visual information and using it as feedback information. Visual servo is roughly classified into position-based visual servo and feature-based visual servo according to input information (control amount) to the servo system. In the position-based visual servo, the position information and posture information of the object are input information to the servo system, and in the feature-based visual servo, the feature amount of the image is input information to the servo system. In addition, there is a method in which the position base and the feature base are hybridized. The visual servo handled in the present embodiment targets all these methods.

  These visual servos are common in that the input information to the servo system is obtained based on the reference image and the captured image.

3.1 Flow of Position-Based Visual Servo First, the flow of position-based visual servo is shown in the flowchart of FIG. In the position-based visual servo, first, a reference image is set (S1). Here, the reference image is also called a target image or a goal image, and is an image serving as a visual servo control target, which is an image representing the target state of the robot 30. That is, the reference image is an image representing the target position or target posture of the robot 30 or an image representing a state where the robot 30 is located at the target position. Further, the reference image must be prepared in advance and stored in the reference image storage unit 110.

  Next, a work space is imaged by the imaging unit 40, and a captured image is acquired (S2). The captured image is an image captured by the image capturing unit 40. The captured image represents the current state of the work space. When the robot 30 and the work are reflected in the captured image, the captured image represents the current state of the robot 30 and the work. Note that a processing delay occurs depending on the performance of the imaging unit 40. Here, even if a processing delay occurs, it is assumed that the current state is shown in the captured image.

  For example, FIG. 4A shows a specific example of the reference image RIM, and FIG. 4B shows a specific example of the captured image PIM. In the captured image PIM, the robot RB points the arm AM and the hand HD (or end point EP) upward, but in the reference image RIM, the robot RB bends the arm AM and brings the hand HD closer to the work WK. . Therefore, in this specific example, the robot RB is controlled so that the arm AM of the robot RB is bent and the hand HD is brought close to the work WK.

  Next, a control command is generated (S3). For example, the control command generation is performed by using homography, which is one of coordinate transformations, based on the reference image and the captured image. In this case, a homography matrix is obtained, and a speed command is generated as a control signal for the robot 30 from the homography matrix.

  Here, the control signal (control command) refers to a signal including information for controlling the robot 30. For example, the control signal includes a speed command. The speed command refers to a command method for giving a moving speed and a rotating speed of an end point of the arm of the robot 30 as information for controlling each part of the robot 30.

  Then, based on the generated control signal, it is determined whether or not the control amount (here, the position and posture of the robot 30) has converged to the target value (S4). For example, in the case of using homography, when the velocity vector represented by the velocity command is 0, it can be considered that the position or orientation that is the controlled variable has reached the target state. If the velocity vector is not 0, it is determined that the control amount has not converged to the target value. Here, convergence means approaching the target value as much as possible, or reaching the target value.

  When it is determined that the control amount has converged to the target value, the visual servo is terminated. On the other hand, when determining that the control amount has not converged to the target value, the processing unit 120 sends a control command to the robot control unit 130 (S5).

  In the position-based visual servo, the above processing is repeated until the control amount converges to the target value.

3.2 Flow of Feature-Based Visual Servo Next, the flow of feature-based visual servo is shown in the flowchart of FIG. Similar to the position-based visual servo, a reference image is first set (S10), and then the work space is imaged by the imaging unit 40 to obtain a captured image (S11). Note that when feature-based visual servoing is used, it is desirable to perform feature extraction of the reference image and calculate the feature amount when setting the reference image. Alternatively, reference image information obtained by extracting features of the reference image may be stored in the reference image storage unit 110. The feature extraction of the image is performed using, for example, corner detection or a Gaussian filter.

  Next, the feature of the captured image is extracted (S12). Note that the feature extraction of the reference image is preferably performed at the time of setting the reference image, but may be performed in this step. In the feature extraction, an image feature amount is obtained as input information (control amount) to the visual servo system.

  Then, based on the feature amount of the image, it is compared whether or not the reference image matches the captured image (S13). If it is determined that the images match (S14), the visual servo is terminated. On the other hand, if it is determined that the images do not match (S14), a control command is generated (S15), and the control command is sent to the robot controller 130 (S16).

  In the feature-based visual servo, the above processing is repeated until the control amount converges to the target value.

4). Details of Processing The robot control system 10 of this embodiment described above includes a reference image storage unit 110 that stores a reference image that is an image representing the target state of the robot 30, a processing unit 120 that outputs a control signal of the robot 30, And a robot controller 130 that controls the robot 30 based on the control signal. And the reference image memory | storage part 110 memorize | stores the reference image containing the reflective image of a target object. Further, the processing unit 120 performs visual servoing based on the captured image in which the reflecting mirror that reflects the object falls within the imaging range of the imaging unit 40 and the reference image.

  In the present embodiment, the processing unit 120 acquires a reference image including a reflection image of the object from the reference image storage unit 110.

  Here, the reflection image refers to an image in which an object appears to be reflected on a reflecting mirror. Specifically, the reflected image may be an image obtained by imaging a reflecting mirror that actually shows the object. The reflected image may be an image obtained by capturing an object from a position where a reflecting mirror is installed and performing image processing such as tilt correction on the captured image. Further, the reflection image may be an image generated by CG (Computer Graphics) or the like.

  In contrast, the imaging unit 40 captures a captured image in which the reflecting mirror that reflects the object falls within the imaging range of the imaging unit 40, and the processing unit 120 acquires the captured image. Then, the processing unit 120 performs visual servoing based on the acquired captured image and reference image.

  Here, the imaging range refers to an area in real space that is imaged by the imaging unit. Specifically, it refers to a region PA in FIG.

  In this way, the reference image and the captured image show the object viewed from at least two directions. Therefore, it is possible to perform alignment from a plurality of directions based on a captured image captured by one imaging unit and a reference image corresponding to the captured image.

  In addition, since the reflecting mirror is thinner than the imaging unit and wiring is not required, it can be installed in a narrower place (area) and in a free place. Therefore, the problem of the installation location of the imaging unit can be solved. In addition, the processing complexity can be reduced and the processing speed can be improved as compared with the case where a plurality of imaging units are provided.

  Furthermore, it is possible to image the state of the object from a plurality of directions without a timing shift.

  Further, the reference image storage unit 110 may store a reference image including a reflection image of the target object in which the target object is reflected. Then, the processing unit 120 may perform visual servoing based on the captured image in which the reflecting mirror and the target object fall within the imaging range and the reference image.

  That is, in this case, the reference image and the captured image show two surfaces, a surface facing the imaging unit of the object and a surface facing the reflector of the object. Note that, as described above, the reference image may include a reflecting mirror that reflects the object, and the reflecting image generated by CG or the like is embedded at a position corresponding to the reflecting mirror in the reference image. May be.

  Thereby, it is possible to perform alignment and the like from two directions, ie, the direction from the imaging unit 40 to the object and the direction from the reflecting mirror to the object, by performing only one visual servo.

  Further, the reference image storage unit 110 may store a reference image including two or more reflection images of the object. Then, the processing unit 120 may perform visual servoing based on the captured image in which two or more reflecting mirrors fall within the imaging range of the imaging unit 40 and the reference image.

  Specific examples are shown in FIGS. 6A and 6B. In this example, as shown in FIG. 6A, two reflecting mirrors (ML1 and ML2) are arranged behind the object OB when viewed from the imaging unit CM. Then, the imaging range PA of the imaging unit CM is set so that the object OB, the first reflecting mirror ML1, and the second reflecting mirror ML2 enter, and the work area is imaged.

  Then, a captured image PIM as shown in FIG. 6B is obtained. In the captured image PIM, the state of the object OB viewed from the imaging unit CM, the object OB reflected on the first reflecting mirror ML1, and the object OB reflected on the second reflecting mirror ML2 is within one image. Can be confirmed. In this example, visual servoing is performed based on this captured image and the corresponding reference image.

  Thus, with only one visual servo, the direction from the imaging unit 40 to the object, the direction from the first reflector to the object, and the direction from the second reflector to the object are three. It is possible to perform alignment from the direction. Therefore, it is possible to perform alignment from a plurality of directions without preparing a plurality of imaging units. Similarly, it is possible to perform alignment from a plurality of directions without attaching a camera to the arm of the robot and moving the arm without imaging an object from a plurality of directions.

  Further, the reflecting mirror needs to be arranged so that a state in which the object is reflected can be seen from the position of the imaging unit 40.

  Therefore, the reflecting mirror is a straight line perpendicular to the straight line connecting the imaging unit 40 and the object in the work area, and is opposite to the position of the imaging unit 40 with respect to the vertical straight line passing through the object. May be installed.

  This will be specifically described with reference to FIG. In FIG. 7, the first reflecting mirror ML1 and the second reflecting mirror ML2 are installed as reflecting mirrors. The straight line connecting the imaging unit CM and the object OB indicates the straight line ST, the straight line perpendicular to the straight line ST and the straight line passing through the object indicates the straight line PD. Further, the side opposite to the position of the imaging unit CM with respect to the straight line PD refers to the area SA indicated by the oblique lines in FIG. 7, and the reflecting mirror (ML1 and ML1) is located at any position within the area SA indicated by the oblique lines. It is desirable to install ML2).

  Thereby, it is possible to arrange the reflecting mirror at a position where the appearance of the object can be seen from the position of the imaging unit 40. And it becomes possible for the imaging part 40 to image the captured image in which the target object was reflected in the reflecting mirror.

  Further, the reference image storage unit 110 may store a reflection image of the object as a reference image. And the process part 120 may perform the detection process of the reflective mirror area | region in which a reflective mirror is reflected in a captured image, and may perform visual servo based on the image of the reflective mirror area | region in a captured image, and a reference image.

  For example, in the example of FIG. 6B described above, the region CIM1 in which the object is reflected and the reflector regions CIM2 and CIM3 in which the reflector is reflected may be detected from the captured image PIM. Then, the visual servo may be performed based on the detected images of the respective regions (CIM1 to CIM3) and the reference image with the reflected image of the object itself as the reference image.

  In this case, visual servo may be performed in order for each image in each area, or visual servo may be performed for each image in each area simultaneously.

  On the other hand, there is also an advantage that visual servoing can be performed without taking into consideration parts other than the area where the object is reflected and the reflector area. Therefore, convenience is high when a reference image is generated using image processing or CG.

  This eliminates the need to consider the area other than the area where the object is reflected and the reflector area, thereby facilitating the generation of a reference image.

  Moreover, the process part 120 may perform the process which detects the marker or frame provided in the reflective mirror as a detection process of a reflective mirror area | region.

  Specifically, as shown in FIG. 8, markers MK are attached to the four corners of the reflecting mirrors ML1 and ML2 that reflect the object OB, and this marker MK is detected.

  This makes it possible to easily detect the reflector region.

  Further, the processing unit 120 may perform processing for controlling the robot 30 so that the captured image matches or approaches the reference image as visual servo.

  As a result, if a reference image showing the target posture of the robot 30 is used, it is possible to cause the robot 30 to take a posture that is the same as or close to the target posture.

  In the present embodiment, a dimming mirror device capable of controlling reflection and transmission may be used as a reflecting mirror, and control may be performed so that light is reflected only when the reflecting mirror is projected.

  This makes it possible to avoid adverse effects on other image processing.

  Below, the flow of the process of this embodiment is demonstrated using the flowchart of FIG.

  First, the imaging unit captures a captured image (S101). Then, an area where the object is reflected and a reflector area are detected in the captured image (S102), and an image of the area where the object is reflected and an image of the reflector area are separated from the captured image (S103).

  Then, an image in which the object is not correctly reflected is excluded (S104). Note that the image in which the object is not correctly reflected refers to an image in which occlusion occurs, for example, as illustrated in FIG.

  Further, the image of each region and the corresponding reference image are compared to calculate an error (S105), and it is determined whether or not the error is sufficiently small (S106). Note that whether or not the error is sufficiently small is determined, for example, by comparison with a given threshold value.

  If it is determined that the error has become sufficiently small, the process ends. On the other hand, if it is determined that the error is not sufficiently small, the process returns to step S101. The above is the processing flow of this embodiment.

5. First Modified Example Next, as a first modified example, a tripod or the like that can mechanically change the direction of the imaging unit is used, and an object placed in the work area and each reflecting mirror are imaged one by one. Thus, a method of acquiring reference images and captured images from a plurality of directions used for visual servoing can be considered.

  That is, the robot control system 10 of the present embodiment may include a camera control unit 140 that controls the imaging direction of the imaging unit 40. Then, the processing unit 120 includes the first visual servo based on the first captured image captured when the imaging unit 40 is directed toward the target, the first reference image in which the target is reflected, and the imaging unit 40. You may perform the 2nd visual servo based on the 2nd captured image imaged when it turns to a reflective mirror, and the 2nd reference image containing the reflective image of a target object.

  A specific example will be described with reference to FIGS. First, as shown in FIG. 10A, the imaging unit CM images the work area so that the object OB falls within the imaging range PA, and obtains a first captured image. Then, first visual servoing is performed based on the first reference image generated (or imaged) so as to correspond to the first captured image and the first captured image.

  On the other hand, as shown in FIG. 10B, the imaging unit CM images the work area so that the reflecting mirror ML2 falls within the imaging range PA, and obtains a second captured image. Then, second visual servoing is performed based on the second reference image generated (or imaged) so as to correspond to the second captured image and the second captured image.

  The first visual servo and the second visual servo may be performed continuously as in an example described later, or only one of them may be performed.

  As in the example described above, when the imaging direction of the imaging unit 40 is fixed, a reflecting mirror must be installed within the imaging range of the imaging unit 40. On the other hand, in this example, since the reflecting mirror can be installed at a more free position, a more appropriate reference image can be used.

  In addition, the processing unit 120 is based on a third captured image that is captured when the imaging unit 40 is directed to the second reflecting mirror and a third reference image that includes the second reflected image of the object. Visual servo may be performed.

  That is, in the example of FIGS. 10A and 10B described above, the imaging unit CM images the work area so that the reflecting mirror ML1 is within the imaging range PA, and the third captured image is captured. Get. Then, the third visual servo is performed based on the third reference image generated (or imaged) so as to correspond to the third captured image and the third captured image.

  Further, as shown in FIG. 11A, when there is an obstacle HU between the object OB and the imaging unit CM, a captured image PIM as shown in FIG. 11B is captured. With such a captured image, visual servoing cannot be performed appropriately. However, as in this example, when there are a captured image obtained by imaging the object from the front and two captured images obtained by imaging the reflector, visual servoing can be appropriately performed among the obtained captured images. Visual servo can be performed using only captured images that can be determined to be possible.

  Thus, for example, visual servoing can be performed by using only an image in which an object is correctly reflected. As a result, even when occlusion occurs in the target object in the captured image, it is possible to appropriately perform visual servoing and the like.

  Hereinafter, the flow of processing of the first modification will be described with reference to the flowchart of FIG. 12 and FIGS. 10 (A) and 10 (B).

  First, as shown in FIG. 10A, the imaging unit 40 images the work area in a state where the target object is in the imaging range (S201), and the processing unit 120 extracts the feature amount of the acquired captured image. (S202).

  Next, the camera control unit 140 changes the imaging direction of the imaging unit 40 (S203). For example, as shown in FIG. 10B, the orientation of the imaging unit 40 is changed so that the reflecting mirror ML2 enters the imaging range.

  Then, the imaging unit 40 captures an image (S204), and the processing unit 120 detects a reflector region in the acquired captured image (S205), and extracts a feature amount of the detected image of the reflector region (S206). .

  Furthermore, the processing unit 120 determines whether or not the necessary and sufficient feature amount of the image of the reflector region has been extracted in performing visual servoing (S207).

  Then, when it is determined that the necessary and sufficient image feature amount of the reflector region has not been extracted, the process returns to step S203, and for example, the camera control unit 140 causes the imaging unit 40 so that the reflector ML1 enters the imaging range. The image capturing unit 40 performs image capturing (S204), and the processing unit 120 extracts the feature amount of the image in the reflector region of the captured image (S205, S206).

  On the other hand, if it is determined in step S207 that the necessary and sufficient feature value of the reflector region image has been extracted, the processing unit 120 calculates the extracted feature value and the reference image calculated in advance. The difference between the two feature values is calculated by comparing with the feature value (S208), and it is determined whether or not the difference between the two feature values is sufficiently small (S209). If it is determined that the difference between the two feature quantities is sufficiently small, the process ends.

  On the other hand, if it is determined that the difference between the two feature amounts is large, the movement direction of the arm of the robot 30 is determined based on the difference between the two feature amounts (S210). The arm is moved (S211). And it returns to the process of step S201 and the subsequent processes are repeated. The above is the process flow of the first modification.

6). Second Modification Example Next, in a second modification example, the reflecting mirror may be attached to the arm of the robot 30.

  That is, the robot 30 may have a first arm 320 and a second arm 330. Then, the reflecting mirror may be attached to the first arm 320. Further, the processing unit 120 performs visual servoing that moves the second arm 330 based on the captured image in which the reflecting mirror attached to the first arm 320 falls within the imaging range of the imaging unit 40 and the reference image. May be.

  Specifically, as illustrated in FIG. 13, the imaging unit CM images the work area so that the reflecting mirror ML attached to the first arm AM1 falls within the imaging range PA, and obtains a captured image. Then, visual servo for moving the second arm AM2 is performed based on the reference image generated (or imaged) so as to correspond to the captured image and the captured image.

  Thus, by moving the arm of the robot 30, the position of the reflecting mirror can be changed. As a result, even when occlusion occurs during visual servoing, visual servoing can be performed flexibly.

  The reflecting mirror is not limited to a plane mirror, and may be a concave mirror, for example.

  This makes it possible to pick up an image of an object from a plurality of angles with only one reflector, and even when occlusion occurs during visual servoing, it is possible to flexibly implement visual servoing, etc. .

  Further, when there are a plurality of reflecting mirrors and another reflecting mirror is reflected in one reflecting mirror, the reflecting mirror reflected may be masked.

  For example, the processing unit 120 detects a region where a marker or a frame of the second reflecting mirror appears in the reflected image included in the captured image in which the first reflecting mirror that reflects the object falls within the imaging range, and the detection result is Based on this, the area where the second reflecting mirror is reflected may be masked.

  As a result, it is possible to prevent the visual servo from converging due to unnecessary reflection of the reflecting mirror. Note that an area where an unnecessary reflecting mirror is reflected by the mask may be completely erased or only blurred. In addition, a marker or a frame provided on the reflecting mirror may be used to detect the area where the reflecting mirror is reflected. Thereby, it becomes possible to simplify the detection of unnecessary reflecting mirrors.

  Hereinafter, the processing flow of the second modification will be described with reference to the flowchart of FIG. Steps S301 to S306 correspond to steps S201 to S206 in FIG. 12, and steps S310 to S313 correspond to steps S208 to S211 in FIG.

  If it is determined in step S307 that the necessary and sufficient reflector region image feature amount has been extracted, the processing from step S310 is performed as in the case of FIG.

  On the other hand, in step S307, when it is determined that the necessary and sufficient image feature amount of the reflector region has not been extracted, the reflector is installed on the arm of the robot 30, and the number of movements of the arm is predetermined. It is determined whether or not the number is less than or equal to (S308).

  If it is determined that the reflecting mirror is not installed on the arm of the robot 30 or that the number of movements of the arm is greater than the predetermined number, the process returns to step S303.

  On the other hand, if it is determined that the reflecting mirror is installed on the arm of the robot 30 and the number of movements of the arm is equal to or less than the predetermined number, the arm to which the reflecting mirror is attached is moved to The position is changed (S309). Then, the process returns to step S304. The above is the processing flow of the second modification.

  Note that the robot control system 10 and the robot 30 according to the present embodiment may realize part or most of the processing by a program. In this case, the robot control system 10 and the robot 30 according to the present embodiment are realized by a processor such as a CPU executing a program. Specifically, a program stored in the information storage medium is read, and a processor such as a CPU executes the read program. Here, the information storage medium (computer-readable medium) stores programs, data, and the like, and functions as an optical disk (DVD, CD, etc.), HDD (hard disk drive), or memory (card type). It can be realized by memory, ROM, etc. A processor such as a CPU performs various processes according to the present embodiment based on a program (data) stored in the information storage medium. That is, in the information storage medium, a program for causing a computer (an apparatus including an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing the computer to execute processing of each unit) Is memorized.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described at least once together with a different term having a broader meaning or the same meaning in the specification or the drawings can be replaced with the different term in any part of the specification or the drawings. Further, the configuration and operation of the robot control system and the robot are not limited to those described in this embodiment, and various modifications can be made.

10 robot control system, 30 robot, 40 imaging unit,
110 reference image storage unit, 120 processing unit, 130 robot control unit,
140 camera control unit, 150 I / F unit (input unit), 310 control unit,
320 first arm, 330 second arm, 340 first end effector,
350 Second end effector

Claims (16)

A reference image storage unit that stores a reference image that is an image representing the target state of the robot;
A processing unit for outputting a control signal of the robot;
A robot control unit for controlling the robot based on the control signal;
Including
The reference image storage unit
Storing the reference image including a reflection image of the object;
The processor is
A robot control system, wherein visual servoing is performed based on a captured image in which a reflecting mirror that reflects the object falls within an imaging range of an imaging unit and the reference image. In claim 1,
The reference image storage unit
The object is reflected, the reference image including the reflected image of the object is stored,
The processor is
The robot control system, wherein the visual servoing is performed based on the captured image in which the reflecting mirror and the target object fall within the imaging range and the reference image. In claim 1 or 2,
The reference image storage unit
Storing the reference image including two or more reflection images of the object;
The processor is
2. The robot control system according to claim 1, wherein the visual servoing is performed based on the captured image in which two or more of the reflecting mirrors fall within the imaging range of the imaging unit and the reference image. In claim 1,
A camera control unit that controls an imaging direction of the imaging unit;
The processor is
A first visual servo based on a first captured image captured when the imaging unit is directed toward the object, and a first reference image in which the object is reflected;
Performing a second visual servo based on a second captured image captured when the imaging unit is directed toward the reflecting mirror and a second reference image including the reflected image of the object. Robot control system. In claim 4,
The processor is
Performing a third visual servo based on a third captured image captured when the imaging unit is directed to a second reflecting mirror and a third reference image including a second reflected image of the object. Robot control system characterized by In any one of Claims 1 thru | or 5,
The reflector is
In the work area, the straight line is perpendicular to the straight line connecting the imaging unit and the object, and is set on the opposite side of the position of the imaging unit with respect to the vertical straight line passing through the object. A robot control system characterized by that. In any one of Claims 1 thru | or 6.
The reflector is
A robot control system, wherein the robot control system is attached to the arm of the robot. In any one of Claims 1 thru | or 6.
The robot is
Having a first arm and a second arm;
The reflector is
Attached to the first arm;
The processor is
Performing the visual servo for moving the second arm based on the captured image in which the reflecting mirror attached to the first arm falls within the imaging range of the imaging unit and the reference image; Characteristic robot control system. In any one of Claims 1 thru | or 8.
The reference image storage unit
Storing the reflected image of the object as the reference image;
The processor is
Robot control characterized by performing a detection process of a reflecting mirror area in which the reflecting mirror is reflected in the captured image, and performing the visual servo based on the image of the reflecting mirror area in the captured image and the reference image. system. In claim 9,
The processor is
The robot control system characterized by performing the process which detects the marker or frame provided in the said reflective mirror as the said detection process of the said reflective mirror area | region. In any one of Claims 1 thru | or 10.
The processor is
As the visual servo, a process for controlling the robot so that the captured image matches or approaches the reference image is performed. In any one of Claims 1 thru | or 11,
The processor is
In the reflected image included in the captured image in which the first reflecting mirror that reflects the object falls within the imaging range, a region where the marker or frame of the second reflecting mirror is reflected is detected, and based on the detection result, A robot control system characterized by masking an area where two reflecting mirrors are reflected. A robot control system that controls a robot so that a captured image captured by an imaging unit matches or approaches a reference image,
The captured image and the reference image are:
A robot control system comprising a mirror image of an object reflected on a reflecting mirror.   A robot comprising the robot control system according to claim 1. Represents the target state of the robot, stores a reference image including a reflection image of the object,
Based on the captured image obtained from the imaging unit and reflecting the mirror reflecting the object within the imaging range of the imaging unit, and the reference image, the visual servo is performed,
Outputting a control signal of the robot;
A robot control method, comprising: controlling the robot based on the control signal. A reference image storage unit that stores a reference image that is an image representing the target state of the robot;
A processing unit for outputting a control signal of the robot;
As a robot controller that controls the robot based on the control signal,
Make the computer work,
The reference image storage unit
Storing the reference image including a reflection image of the object;
The processor is
A program that performs visual servoing based on a captured image in which a reflecting mirror that reflects the object falls within an imaging range of an imaging unit and the reference image.

Images