JP5962394B2 - Calibration apparatus and imaging apparatus calibration method - Google Patents

Description

  The present invention relates to calibration of an imaging apparatus.

  When operating a robot that moves or attaches a workpiece, the imaging device (for example, a digital video camera) captures an area including the workpiece, identifies the position of the workpiece from the obtained image, and based on the identified position A technique for operating a robot arm is used. As described above, in the configuration in which the position of the object is specified based on the image obtained by the imaging device, the physical position of the object (world coordinate system) and the position of the object in the captured image (camera coordinate system) are determined in advance. It is necessary to perform a process (a so-called calibration) for obtaining a correction value (conversion matrix in consideration of rotation, translation, lens distortion, etc.) for conversion.

  Such calibration is performed by, for example, capturing a plurality of images in which the relative positional relationship between the calibration board and the imaging device in which white and black squares are alternately drawn vertically and horizontally is different from each other. This is executed based on the plurality of images. Patent Document 1 discloses a method of performing calibration by fixing a calibration board and moving a camera mounted on a robot to pick up a plurality of images.

JP-A-8-82505

  In the above-described calibration, in order to reduce the conversion error using the correction value and improve the calibration accuracy, a plurality of images in which the calibration board is captured at different positions in the image are used as a plurality of captured images. In particular, it is preferable to use a plurality of images in which the calibration board is scattered throughout the imaging region.

  However, it is not easy to capture an image suitable for calibration. For example, in the method described in Patent Document 1, a robot is taught a camera position for capturing an image suitable for calibration. What is the “suitable image” depends on the user's experience. In many cases, it is difficult for a user or the like who does not have sufficient experience to teach smoothly, so the user is forced to bear a large work load. In addition, this often occurs depending on the user's skill level and equipment conditions, but if the calibration board itself is placed out of the predetermined position, the camera is moved to the predetermined position. The desired image could not be obtained even if the image was taken. For example, in a configuration in which the camera is fixed and the user moves the calibration board to obtain a plurality of captured images, the user may arrange the calibration board in order to obtain an image suitable for calibration. However, there is a problem that it takes a long time to make the calibration accuracy equal to or higher than a predetermined value. For this reason, a technique capable of improving the calibration accuracy of the imaging apparatus in a short period of time has been desired.

  SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms.

  (1) According to one form of this invention, a calibration apparatus is provided. This calibration device is a calibration device for calibrating an imaging device arranged in a predetermined positional relationship with an arm of a robot, and the imaging device images a calibration board having feature points. For each of the obtained plurality of captured images, a similarity calculation unit that calculates a similarity to each of a plurality of master images in which the calibration board appears at different positions; and based on the calculated similarity A classification unit that classifies a plurality of captured images into a plurality of groups associated with each master image; in each group of the plurality of groups, one representative image having the similarity equal to or greater than a predetermined value is extracted and extracted; And the calibrating of the feature points based on a representative image group including a plurality of the representative images. A calibration processing unit that determines a correction value for converting a physical position on the calibration board and a position of the feature point in the captured image; and an error in the conversion due to the determined correction value is equal to or less than a predetermined value A determination unit that determines whether or not the number of captured images is related to the number of the captured images classified into each of the plurality of groups when the error is determined to be larger than the predetermined value. A display unit for displaying.

  According to the calibration device of this aspect, when the conversion error due to the correction value determined by the calibration processing unit is larger than the predetermined value, the imaging related to the number of captured images classified into each of the plurality of groups. Since the number-related information is displayed, the user can be made to understand which group has a small number of captured images. Therefore, it is possible to prompt the user to capture an image that can be classified into a group with a small number of captured images using the imaging device. If an image is added in a group with a small number of captured images, since there are few images (existing images) that already belong to the group, the newly added captured image (added image) may be different from the existing images. Get higher. And since existing images will already be picked up for improvement on the premise that the conversion error due to the correction value is larger than the predetermined value, of course, than the existing images There is a high possibility of an image having a high similarity to the master image. Therefore, it can be said that the additional image has a higher effect on the “conversion error due to the correction value” than the existing image. For this reason, calibration accuracy can be improved in a short period of time.

  (2) In the calibration device according to the above aspect, the number-of-captured information may include information that can identify a group having the smallest number of the classified captured images among the plurality of groups. According to the calibration device of this embodiment, the user can easily understand which group the captured image is classified into the smallest number. Therefore, it is possible to make the user understand what kind of image is obtained so that the calibration board can obtain a plurality of images that are scattered throughout the imaging region. In addition, it is possible to prompt the user to acquire an image that can be classified into a group that may not include a captured image having a high degree of similarity with the master image.

  (3) In the calibration device according to the above aspect, the number-of-capture-related information may include information indicating the number of the captured images classified into each of the plurality of groups. According to the calibration device of this aspect, the user can easily understand the number of captured images classified into each group. Therefore, it is possible to make the user understand what kind of image is obtained so that the calibration board can obtain a plurality of images that are scattered throughout the imaging region. In addition, it is possible to prompt the user to acquire an image that can be classified into a group that may not include a captured image having a high degree of similarity with the master image.

  (4) The calibration apparatus according to the above aspect further includes a statistical processing unit that obtains a statistical characteristic of the similarity calculated for each captured image classified into the group for each group, and the display unit May display statistical characteristic related information related to the statistical characteristic when the error is larger than the predetermined value. According to this type of calibration apparatus, the user can easily understand the statistical characteristics of the similarity for each group. Therefore, for example, among the captured images with the highest similarity in each group, the user can understand the group of the captured images with the lowest similarity and the group with the lowest average similarity with the master image, The user can be prompted to obtain images that can be classified into these groups.

(5) The calibration apparatus of the above embodiment, the statistical properties is the highest value among the similarities calculated for each of the classified captured images to each group, the statistical characteristic-related information, Information that can identify the group having the lowest maximum value among the plurality of groups may be included. According to the calibration device of this aspect, the user can easily understand the group having the lowest similarity value among the groups. Therefore, it is possible to prompt the user to acquire images that can be classified into such groups.

(6) In the calibration apparatus of the above embodiment, the statistical characteristic is an average value of the similarities calculated for each of the classified captured images to each group, the statistical characteristic-related information, the Information that can identify a group having the lowest average value among a plurality of groups may be included. According to the calibration device of this aspect, the user can easily understand the group having the lowest average value of the similarity among the groups. Therefore, it is possible to prompt the user to acquire images that can be classified into such groups.

  (7) In the calibration device according to the above aspect, the display unit may display the error. According to this form of the calibration apparatus, the user can easily understand whether the error increases or decreases each time a captured image is added. Therefore, when adjusting the arrangement position of the calibration board in order to obtain additional captured images, the user can understand how the error is reduced when the adjustment is made. Therefore, the calibration accuracy can be improved in a short period.

(8) In the calibration device according to the above aspect, the imaging device continuously acquires the captured image, and the calibration processing unit is obtained each time the captured image is acquired by the imaging unit. The representative image group that minimizes the error among the plurality of captured images may be determined. According to the calibration device of this embodiment, it is not necessary to acquire a large number of captured images in advance, and the work burden on the user can be reduced. In addition, each time a captured image is acquired, a representative image group with the smallest error is determined, so that calibration can be completed simply by obtaining as many captured images as the number of groups. Can improve between.

  The present invention can be realized in various forms other than the apparatus. For example, the present invention can be realized in the form of a calibration method for an imaging apparatus, a computer program for realizing the method, a non-temporary recording medium on which the computer program is recorded, and the like.

1 is an explanatory diagram showing an external configuration of a robot system 10 including a calibration device as one embodiment of the present invention. 2 is a block diagram showing an internal configuration of the robot system 10. FIG. It is a flowchart which shows the procedure of the camera calibration process in 1st Embodiment. 6 is an explanatory diagram illustrating an example of a master image stored in a master image storage unit 284. FIG. It is explanatory drawing which shows an example of the information displayed in step S150. It is a flowchart which shows the procedure of the camera calibration process in 2nd Embodiment. It is explanatory drawing which shows an example of the information displayed on the management terminal 400 in step S105a. It is explanatory drawing which shows an example of the information displayed in step S150a. FIG. 10 is an explanatory diagram illustrating an example of information displayed on a management terminal 400 in step S150 of camera calibration processing according to Modification 1. 10 is a flowchart showing a procedure of camera calibration processing in Modification 2. 10 is a flowchart illustrating a procedure of camera calibration processing in Modification 3. It is explanatory drawing which shows an example of the information displayed in step S150b of the modification 3. 10 is a flowchart showing a procedure of camera calibration processing in Modification 4. It is explanatory drawing which shows an example of the information displayed in step S150c of the modification 4. It is a block diagram which shows the internal structure of the robot system 10a of the modification 5. It is a block diagram which shows the internal structure of the robot system 10b of the modification 6. FIG.

A. First embodiment:
A1. System configuration:
FIG. 1 is an explanatory diagram showing an external configuration of a robot system 10 including a calibration device as an embodiment of the present invention. The robot system 10 includes an articulated industrial robot body 100, a control device 200, a teaching device 300, and a management terminal 400. The robot body 100, the teaching device 300, and the management terminal 400 are each connected to the control device 200 by a predetermined cable.

  The teaching device 300 is a dedicated device for teaching the operation of the robot main body 100 to the control device 200. The management terminal 400 is a device for performing various settings for the control device 200, and can be configured by, for example, a personal computer.

  The robot body 100 includes a base part 101, a shoulder part 102, a lower arm 103, an upper arm 104, a wrist 105, a flange part 106, an end effector 107, and a camera 150. The base unit 101 is fixed to equipment in the factory (for example, a machine desk or a floor surface). The shoulder portion 102 is connected to the base portion 101 via a joint shaft that can turn in the horizontal direction. The lower arm 103 has a lower end connected to the shoulder portion 102 via a joint shaft that can pivot in the vertical direction. The upper arm 104 has a substantially central portion connected to the tip of the lower arm 103 via a joint shaft that can pivot in the vertical direction. The wrist 105 is connected to the tip of the upper arm 104 via a joint shaft that can pivot in the vertical direction. The flange portion 106 is rotatable and is provided at the tip of the wrist 105. The end effector 107 is attached to the flange portion 106 and can grip a workpiece (not shown). The camera 150 is installed on the wrist 105 so that the lens optical axis is parallel to the longitudinal direction of the end effector 107 (the axial direction of the wrist 105).

  In the robot system 10, the calibration board CB placed on the work surface S <b> 1 of the worktable 500 on which a work can be placed is imaged, and camera calibration processing described later is executed based on the obtained image. As shown in FIG. 1, the calibration board CB has a pattern in which white and black squares are alternately arranged vertically and horizontally on the surface. In a captured image including such a calibration board CB, points (four corners of the square) where the same color squares touch each other can be extracted as feature points of the calibration board CB.

  FIG. 2 is a block diagram showing an internal configuration of the robot system 10. The robot body 100 includes a servo motor 130 configured as a three-phase AC brushless DC motor, a speed reducer 120 that decelerates the rotation of the servo motor 130, and a joint shaft that is driven by the servo motor 130 via the speed reducer 120. 110. Each of the arms 103 and 104, the wrist 105, and the flange portion 106 are mechanically connected to the joint shaft 110.

  The control device 200 includes an inverter 210, a drive circuit 220, a CPU (Central Processing Unit) 230, a storage unit 280, and an input / output interface unit 290. The inverter 210 is electrically connected to the drive circuit 220 and the servo motor 130 of the robot body 100, and supplies three-phase AC power to the servo motor 130 based on the drive signal from the drive circuit 220. The drive circuit 220 performs switching control of the inverter 210 in accordance with an instruction from the CPU 230.

  The CPU 230 executes a control program stored in the storage unit 280, thereby providing a calibration processing unit 231, a user interface (UI) control unit 232, an imaging control unit 233, an image processing unit 234, and a drive control unit 235. Function. The calibration processing unit 231 performs a camera calibration process to be described later. The camera calibration means processing for obtaining a correction value for converting the physical position (world coordinates) of an object to be imaged by the camera 150 and the position of the object (image coordinate system) in the captured image. To do. Such a correction value means a transformation matrix that takes into account distortion and the like of a lens included in the camera 150 in addition to rotation and translation. The robot system 10 identifies the physical position of the workpiece based on the captured image obtained by imaging the workpiece with the camera 150 and the correction value, and executes gripping and movement of the workpiece.

  The user interface control unit 232 creates and displays various menu screens on the teaching device 300 or the management terminal 400, and interprets various instructions input from the teaching device 300 or the management terminal 400. The imaging control unit 233 instructs the camera 150 to perform imaging according to various parameters for imaging (for example, an aperture value and a shutter speed). The image processing unit 234 adjusts brightness, sharpness, and the like for a captured image obtained by imaging with the camera 150. Further, the image processing unit 234 analyzes the obtained captured image, calculates similarity with a master image described later, and extracts feature points in the captured image. The drive control unit 235 outputs a signal for driving the servo motor 130 to the drive circuit 220 based on the teaching data stored in advance in the storage unit 280.

  The storage unit 280 includes a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. The storage unit 280 stores the above-described control program, a captured image storage unit 282, a master image storage unit 284, and a correction value storage unit 286. And. The captured image storage unit 282 stores an image obtained by imaging with the camera 150. The master image storage unit 284 stores in advance a master image used in a camera calibration process to be described later. The correction value storage unit 286 stores correction values calculated in camera calibration processing described later. The input / output interface unit 290 includes various interface groups for connecting the camera 150, the teaching apparatus 300, and the management terminal 400 to the CPU 230, respectively.

  The camera 150 includes an imaging unit 152, a control unit 154, and a connection interface unit 156. The imaging unit 152 includes an imaging device such as a CCD (Charge Coupled Device) and an imaging lens system that forms an optical image of a subject on an imaging surface, and acquires the image data by converting the optical image into an electrical signal. . The control unit 154 controls the imaging unit in accordance with an instruction from the imaging control unit 233, and executes imaging with a predetermined shutter speed and aperture value. The connection interface unit 156 has an interface for connecting to the control device 200. As such an interface, for example, USB (Universal Serial Bus) or Ethernet (registered trademark) can be adopted.

  In the robot system 10 having the above-described configuration, the calibration accuracy can be improved in a short period of time by executing a camera calibration process described later. In the robot system 10, the position of the camera 150 is fixed and the calibration board CB is moved to acquire a plurality of images in which the calibration board CB is captured at different positions in the imaging region. The camera calibration is executed using the images.

  The calibration processing unit 231, the image processing unit 234, and the management terminal 400 described above correspond to the calibration device in the claims. The calibration processing unit 231 corresponds to a calibration processing unit, a similarity calculation unit, a classification unit, a determination unit, and a statistical processing unit in the claims. The management terminal 400 (the display unit included in the management terminal 400) corresponds to the display unit in the claims.

A2. Camera calibration process:
FIG. 3 is a flowchart showing the procedure of the camera calibration process in the first embodiment. When the user selects and executes the camera calibration process menu on the management terminal 400, the camera calibration process is started.

  The calibration processing unit 231 controls the user interface control unit 232 to display on the management terminal 400 a message for instructing a predetermined number or more of images (step S105). The user operates the management terminal 400 and instructs the camera 150 to capture an image while placing the calibration board CB at an arbitrary position on the work surface S1 of the work table 500. At this time, the user can also arrange the calibration board CB so as to be inclined in the vertical direction on the work surface S1. An arbitrary number can be set as the predetermined number to be imaged. In the following description, it is assumed that 60 images are set. In the control device 200, when receiving an imaging instruction from the management terminal 400, the imaging control unit 233 instructs the camera 150 to perform imaging under a predetermined imaging condition. In the camera 150, when receiving an imaging instruction from the imaging control unit 233, the control unit 154 controls the imaging unit 152 to perform imaging.

  The calibration processing unit 231 waits until receiving a notification of completion of imaging from the management terminal 400 (step S110). When the imaging of a predetermined number or more is completed, the user operates the management terminal 400 to notify the control device 200 of the completion of imaging. When the calibration processing unit 231 receives a notification of completion of imaging (step S110: YES), the calibration processing unit 231 controls the image processing unit 234 to store each of the plurality of master images stored in the master image storage unit 284 for each captured image. Is calculated (step S115).

  FIG. 4 is an explanatory diagram illustrating an example of a master image stored in the master image storage unit 284. As shown in FIG. 4, in the first embodiment, four master images FA, FB, FC, and FD are stored in the master image storage unit 284. In FIG. 4, in each of the master images FA, FB, FC, and FD, broken lines indicating the vertical and horizontal centers of the images are shown for convenience of explanation.

  The master images FA, FB, FC, and FD are image groups in which the calibration accuracy is equal to or higher than a predetermined value (error is equal to or lower than a predetermined value) when camera calibration is performed based on these four images. The master images FA, FB, FC, and FD can be set by extracting a combination in which the user performs camera calibration by experiments in advance and the calibration accuracy is equal to or greater than a predetermined value. The calibration accuracy will be described later.

  As shown in FIG. 4, in each of the master images FA, FB, FC, and FD, the image CBf of the calibration board CB is shown in different areas in the imaging area. Specifically, in the master image FA, the image CBf is shown in the upper right area. In the master image FB, the image CBf is shown in the lower right area. In the master image FC, the image CBf is shown in the upper left area. In the master image FD, the image CBf is shown in the lower left area. In this way, the calibration accuracy can be improved by using a plurality of images in which the image CBf is scattered throughout the imaging region. Further, if an image similar to such a master image can be obtained by imaging, the calibration accuracy can be improved when camera calibration is performed based on the obtained image.

  The similarity can be calculated, for example, by calculating a difference between luminance values of pixels at the same position between the captured image and the master image and calculating the difference as a total value. In this case, the greater the difference total value, the lower the similarity, and the smaller the difference difference, the higher the similarity.

  The calibration processing unit 231 classifies each captured image into a master image group with the highest similarity based on the similarity obtained in step S115 (step S120). As shown in FIG. 4, the group to which the group of images evaluated as having the highest similarity of the master image FA is hereinafter referred to as “group A”. Similarly, the group to which the image group evaluated that the master image FB has the highest similarity belongs to “Group B”, and the group to which the image group to which the master image FC is evaluated to have the highest similarity belongs to “Group C”. ”And the group to which the image group evaluated as having the highest similarity in the master image FD is referred to as“ group D ”hereinafter.

  The calibration processing unit 231 specifies a captured image with the highest similarity in each group classified in step S120 (step S125).

  The calibration processing unit 231 performs camera calibration using the captured image group specified in step S125 (a total of four images for each group) (step S130).

  Camera calibration can be performed as follows, for example. First, for each contact point (hereinafter referred to as an “intersection point”) of the same color in the calibration board CB, a position (coordinate) in the world coordinate system is specified, and a position in the captured image of each intersection point (a position in the image coordinate system) ). Next, by solving a plurality of expressions obtained by applying the position (coordinates) of each specified intersection point in the world coordinate system and the position in the captured image to the following expression 1, a rotation matrix and a translation vector ( Correction value).

  T = A · [R | t] · S (1)

  In the above formula 1, S represents a position in the world coordinate system. T represents a position in the image coordinate system. In Equation 1, A represents an internal parameter of the camera, R represents a rotation matrix between the camera coordinate system and the world coordinates, and t represents a translation vector. The camera internal parameter means a determinant indicating a correspondence relationship between the camera coordinate system having the pinhole position of the camera 150 as an origin and the image coordinate system. The internal parameters of the camera are known determinants set by parameters such as the focal length of the camera, the size of the image sensor, and the distortion coefficient of the lens.

  The calibration processing unit 231 obtains an error when conversion between the world coordinate system and the image coordinate system is performed using the correction value obtained in step S130 (step S135). Specifically, the position (coordinate in the world coordinate system) obtained by performing the inverse transformation of Equation 1 based on the position of the feature point in the captured image specified in the camera calibration described above, and the camera calibration described above. Are compared with the positions of feature points in the world coordinate system specified in (coordinates in the world coordinate system), and errors in these positions (distance between the positions) are obtained. The smaller the error, the higher the accuracy of camera calibration.

  The calibration processing unit 231 determines whether or not the error obtained in step S135 is equal to or smaller than a predetermined value (step S140). If it is determined that the error is equal to or smaller than the predetermined value, the camera calibration process is performed. finish. When the camera calibration process is completed, the correction value obtained in step S130 is used when the robot body 100 is subsequently operated.

  If it is determined in step S140 described above that the error is larger than the predetermined value (step S140: NO), the calibration processing unit 231 specifies the number of images belonging to each group (step S145). Note that an arbitrary value can be set as the “predetermined value” in step S140, for example, 1 mm.

  The calibration processing unit 231 controls the user interface control unit 232 to display, on the management terminal 400, information indicating a group with the smallest number of images to which the calibration processing unit 231 belongs and a message for prompting acquisition of an image belonging to the group ( Step S150).

  FIG. 5 is an explanatory diagram illustrating an example of information displayed in step S150. For example, when the group with the smallest number of images belonging to is group A, when step S150 is executed, a window W1 shown in FIG. 5 is displayed. In the first embodiment, the master image of each group is displayed as “information indicating the group having the smallest number of images to which it belongs” in step S150. Therefore, as shown in FIG. 5, the master image FA is displayed in the window W1. In the window W1, a message “Please take 10 images similar to the following master image” is displayed. In order to obtain an image similar to the master image FA, the user who has seen such a message places the calibration board CB on the upper right of the work table 500 and takes 10 images. Note that a message that prompts an arbitrary number of images, not limited to ten, may be employed.

  After step S150 is executed, the process returns to step S110. Therefore, this time, steps S115 to S150 described above are executed based on a group of captured images including a total of 10 captured images (70 captured images in total). In this manner, an image is added and camera calibration is executed until it is determined in step S140 that the error is equal to or smaller than the predetermined value. If it is determined in step S140 that the error is equal to or smaller than the predetermined value, camera calibration is performed. The process ends.

  In the robot system 10 according to the first embodiment described above, the management terminal 400 displays information indicating a group to which the number of captured images belonging to the least number and a message prompting acquisition of an image belonging to the group are displayed. Therefore, an image can be added to the group having the smallest number of captured images belonging to the user. When an image is added to the group with the smallest number of captured images belonging to it, since there are few images (existing images) that already belong to the group, the newly added image (added image) There is a high possibility of differences. And in the existing image, on the premise of the state that “the error of conversion by the correction value is larger than the predetermined value”, the additional image is taken with the aim of improving the calibration accuracy. Of course, there is a high possibility that the image has a higher similarity to the master image than the existing image. Therefore, by adding an image, the calibration accuracy can be improved as compared with the case where an existing image is used.

B. Second embodiment:
FIG. 6 is a flowchart illustrating a procedure of camera calibration processing in the second embodiment. The robot system according to the second embodiment performs step S105a instead of step S105, step S146 instead of step S145, and step S150a instead of step S150 in the camera calibration process. In this respect, unlike the robot system 10 according to the first embodiment, the apparatus configuration and other procedures in the camera calibration process are the same as those of the robot system 10 according to the first embodiment. The device configuration of the robot system of the second embodiment is the same as the device configuration of the robot system 10 of the first embodiment.

  In the camera calibration process of the first embodiment, a relatively large number of captured images are obtained in advance (step S105), and a captured image is additionally obtained when the calibration accuracy is low (error is large). On the other hand, in the camera calibration process of the second embodiment, captured images are obtained in advance (relative to the number of groups), and images are taken one by one when the calibration accuracy is low (the error is large). Then, camera calibration (step S130) is executed.

  Specifically, the calibration processing unit 231 controls the user interface control unit 232 to send a master image and a message instructing to obtain one captured image for each group to the management terminal 400. It is displayed (step S105a).

  FIG. 7 is an explanatory diagram illustrating an example of information displayed on the management terminal 400 in step S105a. As shown in FIG. 7, the same master image as the master image shown in FIG. 4 is displayed in the window W2 displayed on the management terminal 400, and “one image similar to each master image is displayed one by one. Please take a picture. "Message is displayed. Therefore, the user can capture a total of four captured images similar to each master image by viewing such information.

  After step S105a is executed, steps S110 to S140 described above are executed. When Steps S110 to S140 are executed first, camera calibration (Step S130) is executed using the four images taken after Step S105a.

  If it is determined in step S140 that the error obtained in step S135 is greater than the predetermined value, the calibration processing unit 231 captures the image having the lowest similarity among the captured images having the highest similarity in each group. A group to which the image belongs is specified (step S146).

  The calibration processing unit 231 controls the user interface control unit 232 to display information indicating the group specified in step S146 and a message prompting acquisition of an image belonging to the group on the management terminal 400 (step S150a). .

  FIG. 8 is an explanatory diagram illustrating an example of information displayed in step S150a. For example, when the group to which the captured image with the lowest similarity belongs is group A among the captured images with the highest similarity in each group, the window W1a displayed on the management terminal 400 is displayed in FIG. The master image FA is displayed in the same manner as the window W1 shown. Further, unlike the window W1, the window W1a displays a message instructing (prompting) to take “one” image similar to the master image FA. Therefore, after viewing the window W1a, the user adjusts the arrangement position of the calibration board CB and acquires one image similar to the master image FA (highly likely to be classified into the group A).

  After step S150a is executed, the process returns to step S110, and steps S110 to S150a are executed again.

  The robot system of the second embodiment having the above configuration has the same effects as the robot system 10 of the first embodiment. In addition, in the camera calibration processing of the second embodiment, when the camera calibration is performed using the four captured images obtained first, and the calibration accuracy is equal to or less than a predetermined value (conversion error is predetermined). If it is larger than the value, camera calibration is executed every time one captured image is added (step S130). Therefore, it is not necessary to acquire a large number of captured images in advance, and the camera calibration process can be completed by only obtaining the minimum number of captured images for the number of groups. Can be reduced.

C. Variations:
C1. Modification 1:
FIG. 9 is an explanatory diagram showing an example of information displayed on the management terminal 400 in step S150 of the camera calibration process in the first modification. In the first embodiment described above, in step S150 of the camera calibration process, the master image is displayed as information indicating the group to which the number of captured images belonging is the smallest. However, the present invention is not limited to this. . For example, as shown in a window W3 in FIG. 9, the master images FA, FB, FC, and FD and the number of captured images belonging to each master image group can be displayed. Specifically, as shown in FIG. 8, the number of captured images “8” belonging to the group A can be displayed in the vicinity of the master image FA. Similarly, the number of captured images “12” belonging to group B in the vicinity of the master image FB, the number of captured images “25” belonging to group C in the vicinity of the master image FC, and the group in the vicinity of the master image FD are grouped. The number of captured images “15” belonging to D can be displayed respectively. In the window W3, a message “Please take 10 images of the group with the smallest number of images” is displayed. By displaying such a window W3, the user understands that the number of captured images belonging to group A is smaller than that of other groups, and that it is necessary to capture images belonging to group A. Can be made.

C2. Modification 2:
FIG. 10 is a flowchart illustrating a procedure of camera calibration processing according to the second modification. The camera calibration process of the second modified example is different from the camera calibration process of the first embodiment shown in FIG. 3 in that step S146 is executed instead of step S145 and step S150a is executed instead of step S150. Differently, other procedures are the same as those in the first embodiment. Further, the device configuration of the robot system in Modification 2 is the same as the device configuration of the robot system 10 of the first embodiment.

  Step S146 is the same as step S146 of the camera calibration process of the second embodiment shown in FIG. Similarly, step S150a is the same as step S150a of the camera calibration process of the second embodiment shown in FIG.

  In the robot system according to the second modification having the above-described configuration, information indicating the group and acquisition of an image belonging to the group of the captured image with the lowest similarity among the captured images with the highest similarity in each group. Is displayed on the management terminal 400, it is possible to prompt an increase in the number of images belonging to the group. For this reason, possibility that the similarity of the captured image with the highest similarity in such a group will increase can be increased, and the calibration accuracy can be improved. In the second modification, as in the first modification, the master images FA, FB, FC, and FD are displayed, and the similarity of the captured image having the highest similarity in each of the groups A to D is determined as each master image FA. , FB, FC, FD can be displayed in association with each other.

C3. Modification 3:
FIG. 11 is a flowchart illustrating a procedure of camera calibration processing according to the third modification. The camera calibration process of Modification 3 is different from the camera calibration process of the first embodiment shown in FIG. 3 in that step S150b is executed instead of step S150, and other procedures are the same as those of the first embodiment. is there. Further, the device configuration of the robot system in Modification 2 is the same as the device configuration of the robot system 10 of the first embodiment.

  In the camera calibration process of the third modification, after step S145 is executed, the calibration processing unit 231 controls the user interface control unit 232 to indicate information indicating the group with the smallest number of images to which the user belongs, and the group. A message that prompts the user to acquire an image belonging to the message and the error obtained in step S135 are displayed on the management terminal 400 (step S150b).

  FIG. 12 is an explanatory diagram illustrating an example of information displayed in step S150b of the third modification. As shown in FIG. 12, the window W4 displayed on the management terminal 400 is different from the window W1 shown in FIG. 5 in that an error display area Ar is provided, and the other configuration is the same as the window W1. . The error display area Ar is an area for displaying the error obtained in step S135. In the example of FIG. 5, “1.5 mm” is displayed as the error.

  The robot system of Modification 3 having the above configuration has the same effects as the robot system 10 of the first embodiment. In addition, since the management terminal 400 displays the error obtained in step S135, it is possible to let the user understand how much the calibration accuracy is. Therefore, when the user adjusts the position of the calibration board CB, the user can understand whether the calibration accuracy is improved or decreased, and the numerical value (error) displayed in the error display area Ar is calibrated. It can be used as a guide when shifting the position of the action board CB.

C4. Modification 4:
FIG. 13 is a flowchart illustrating a procedure of camera calibration processing according to the fourth modification. The camera calibration process of Modification 4 differs from the camera calibration process of the first embodiment shown in FIG. 3 in that step S147 is added and executed, and step S150c is executed instead of step S150. Other procedures are the same as those in the first embodiment. Further, the device configuration of the robot system in Modification 4 is the same as the device configuration of the robot system 10 of the first embodiment.

  In the camera calibration process of the modified example 4, after the above-described step S145 is executed, the calibration processing unit 231 obtains the average similarity of each group based on the similarity of each captured image calculated in step S115 (step S115). S147). The calibration processing unit 231 controls the user interface control unit 232 to display a histogram of the number of captured images, the highest similarity, and the average similarity of each group on the management terminal 400 (step S150c).

  FIG. 14 is an explanatory diagram illustrating an example of information displayed in step S150c of the fourth modification. As shown in FIG. 14, in the window W5 displayed on the management terminal 400, a histogram of the number of captured images, a histogram of the highest similarity, and a histogram of the average similarity are displayed. In FIG. 14, the unit of similarity (maximum similarity and average similarity) is a percentage. As described above, since the histogram of the number of captured images, the histogram of the highest similarity, and the histogram of the average similarity are displayed on the management terminal 400, the user belongs to any of the groups. It can be easily understood whether the number of captured images is the smallest, which group has the lowest similarity, and which group has the lowest average similarity. Therefore, for example, the user can improve the calibration accuracy by acquiring an image belonging to the group having the lowest highest similarity.

C5. Modification 5:
FIG. 15 is a block diagram illustrating an internal configuration of the robot system 10a according to the fifth modification. In the robot system 10 of the first embodiment, all the functional units that realize the camera calibration process except the camera 150 are included in the control device 200. However, in the robot system 10a of the fifth modification, these functional units are provided. , The control device and the management terminal share.

  Specifically, in the control device 200a of Modification 5, the CPU 230 does not function as the calibration processing unit 231, the user interface control unit 232, the imaging control unit 233, and the image processing unit 234, and the drive control unit 235 Instead of the control device 200 of the first embodiment, the other configuration is the same as that of the control device 200 in that it functions as the drive sub-control unit 236. The drive sub-control unit 236 supplies a signal received from a management terminal 400a described later to the drive circuit 220.

  The management terminal 400a according to the modified example 5 includes a computer main body 401, a display 460, and a keyboard 470. The computer main body 401 includes an input / output interface unit 410, a CPU 420, a hard disk 430, a RAM 440, and a ROM 450.

  The input / output interface unit 410 includes various connection interface groups, and is connected to the input / output interface unit 290 of the control device 200a, the connection interface unit 156 of the camera 150, the display 460, the keyboard 470, and the CPU 420. The CPU 420 reads out a control program stored in the ROM 450, develops it in the RAM 440 and executes it, thereby executing a calibration processing unit 421, an imaging control unit 422, a drive main control unit 423, a user interface control unit 424, and image processing. It functions as the unit 425. Since the calibration processing unit 421 is the same as the calibration processing unit 231 in the first embodiment, a description thereof will be omitted. Similarly, the imaging control unit 422 is the imaging control unit 233 in the first embodiment, the user interface control unit 424 is the user interface control unit 232 in the first embodiment, and the image processing unit 425 is the image processing unit in the first embodiment. 234 and the description thereof is omitted. The drive main control unit 423 transmits a signal for driving the servo motor 130 to the drive sub-control unit 236 based on the teaching data stored in advance in the ROM 450.

  The hard disk 430 includes a captured image storage unit 431, a master image storage unit 432, and a correction value storage unit 433. Since the captured image storage unit 431 is the same as the captured image storage unit 282 in the first embodiment, the description thereof is omitted. Similarly, the master image storage unit 432 is the same as the master image storage unit 284 in the first embodiment, and the correction value storage unit 433 is the same as the correction value storage unit 286 in the first embodiment, and thus description thereof is omitted.

  The robot system 10a of Modification 5 having the above configuration has the same effects as the robot system 10 of the first embodiment.

C6. Modification 6:
FIG. 16 is a block diagram illustrating an internal configuration of the robot system 10b according to the sixth modification. The robot system 10b according to the modified example 6 does not have the management terminal 400, has a camera 150a instead of the camera 150, and omits some functional units included in the control device 200. Unlike the robot system 10 of the first embodiment, other configurations and camera calibration processing procedures are the same as those of the robot system 10 of the first embodiment.

  The control device 200a of Modification 6 is the same as the control device 200 of Modification 5 shown in FIG.

  The camera 150a of Modification 6 includes a CPU 160, a storage unit 170, an operation unit 180, and a display unit 182, and the connection destination of the connection interface unit 156 is the CPU 160 instead of the control unit 154. Unlike the camera 150 of the first embodiment, other configurations are the same as those of the camera 150 of the first embodiment.

  The CPU 160 functions as a calibration processing unit 161, an imaging control unit 162, a drive main control unit 163, a user interface control unit 164, and an image processing unit 165 by executing a control program stored in the storage unit 170. . Since the calibration processing unit 161 is the same as the calibration processing unit 231 in the first embodiment, a description thereof will be omitted. Similarly, the imaging control unit 162 is the imaging control unit 233 in the first embodiment, the driving main control unit 163 is the driving main control unit 423 in the second embodiment, and the user interface control unit 164 is the user interface in the first embodiment. Since the control unit 232 and the image processing unit 165 are the same as the imaging control unit 233 in the first embodiment, description thereof will be omitted.

  The storage unit 170 includes a RAM, a ROM, and the like. The storage unit 170 stores the above-described control program, and includes a captured image storage unit 171, a master image storage unit 172, and a correction value storage unit 173. Since the captured image storage unit 171 is the same as the captured image storage unit 282 in the first embodiment, the description thereof is omitted. Similarly, the master image storage unit 172 is the same as the master image storage unit 284 of the first example, and the correction value storage unit 173 is the same as the correction value storage unit 286 of the first embodiment, and thus description thereof is omitted.

  The operation unit 180 is connected to the CPU 160. The operation unit 180 has various operation buttons, and notifies the CPU 160 of instructions input via the operation buttons. The display unit 182 is connected to the CPU 160 and displays various menu screens and various windows generated by the user interface control unit 164.

  The robot system 10b of Modification 6 having the above configuration has the same effects as the robot system 10 of the first embodiment. In addition, since the management terminal 400 can be omitted, the entire system can be reduced in size.

C7. Modification 7:
In each embodiment and each modification, the number of captured images used for camera calibration is four. However, the number is not limited to four, and may be any plural number. For example, the imaging area of the camera 150 can be divided into six, and a captured image in which the calibration board CB is captured in each of the divided areas can be used for camera calibration. In this configuration, six master images are prepared, and the captured images are classified into six groups.

C8. Modification 8:
In each embodiment and each modification excluding the modification 4, the “information indicating a group” displayed on the management terminal 400 includes the master image, but the master image can be omitted. In this case, for example, a group name can be displayed as information indicating the group.

C9. Modification 9:
In step S150 of the first embodiment, the management terminal 400 displays information indicating a group with the smallest number of captured images and a message for prompting image acquisition. However, instead of the management terminal 400, the teaching apparatus 300 is displayed. Can also be displayed. Further, it is also possible to employ a configuration in which a display unit is provided in the control device 200 and information indicating a group with the smallest number of captured images and a message for prompting image acquisition are displayed on the display unit.

C10. Modification 10:
In the first embodiment, in step S150 of the camera calibration process, information indicating the group having the smallest number of images belonging to the group is displayed. Instead, the group having the smallest number of images belonging to the second group is displayed. Thereafter, information indicating a small number of arbitrary groups can be displayed. For example, it is possible to display information indicating the group with the fewest number of images and the second smallest group. In this configuration, two images that are more similar to the master image can be used in the next camera calibration, so that the calibration accuracy can be improved in a shorter time.

C11. Modification 11:
In each embodiment, the correction value (step S130) and the conversion error (step S135) are obtained every time a new captured image is newly obtained, but the present invention is not limited to this. For example, the following procedure can be executed in place of steps S130, S135, and S140. First, it is determined whether or not the similarity of the captured image specified in step S125 is greater than or equal to a predetermined value. If the similarity is greater than or equal to a predetermined value, the camera calibration process is terminated, and the similarity is greater than the predetermined value. If the value is lower, the above-described steps S145 and S150 can be executed. According to such a configuration, calculation of the correction value and the error can be omitted, so that the period required for the camera calibration process can be shortened.

C12. Modification 12:
In each embodiment, the cameras 150 and 150a are installed on the wrist 105, but the present invention is not limited to this. For example, instead of the wrist 105, the upper arm 104, the lower arm 103, the shoulder portion 102, the flange portion 106, or the like can be arranged. That is, in general, an imaging device arranged at a position that has a predetermined positional relationship with the arm of the robot main body 100 can be employed as the imaging device in the present invention. The “arm” in the claims has a broad meaning including the lower arm 103, the upper arm 104, the wrist 105, and the flange portion 106 in each embodiment.

C13. Modification 13:
A part of the configuration realized by software in each embodiment may be replaced with hardware. On the contrary, a part of the configuration realized by hardware may be replaced with software.

  The present invention is not limited to the above-described embodiments and modifications, and can be realized with various configurations without departing from the spirit thereof. For example, the technical features in the embodiments and the modifications corresponding to the technical features in each embodiment described in the summary section of the invention are to solve some or all of the above-described problems, or In order to achieve part or all of the effects, replacement or combination can be performed as appropriate. Further, if the technical feature is not described as essential in the present specification, it can be deleted as appropriate.

DESCRIPTION OF SYMBOLS 10, 10a, 10b ... Robot system 100 ... Robot main body 101 ... Base part 102 ... Shoulder part 103 ... Lower arm 104 ... Upper arm 105 ... Wrist 106 ... Flange part 107 ... End effector 110 ... Joint axis 120 ... Reduction gear 130 ... Servo Motor 150, 150a ... Camera 152 ... Imaging unit 154 ... Control unit 156 ... Connection interface unit 160 ... CPU
161: Calibration processing unit 162 ... Imaging control unit 163 ... Drive main control unit 164 ... User interface control unit 165 ... Image processing unit 170 ... Storage unit 171 ... Captured image storage unit 172 ... Master image storage unit 173 ... Correction value storage unit 180 ... Operation unit 182 ... Display unit 200, 200a ... Control device 210 ... Inverter 220 ... Drive circuit 230 ... CPU
231 ... Calibration processing unit 232 ... User interface control unit 233 ... Imaging control unit 234 ... Image processing unit 235 ... Drive control unit 236 ... Drive sub-control unit 280 ... Storage unit 282 ... Captured image storage unit 284 ... Master image storage unit 286 ... Correction value storage section 290 ... Input / output interface section 300 ... Teaching device 400, 400a ... Management terminal 401 ... Computer main body 410 ... Input / output interface section 420 ... CPU
421 ... Calibration processing unit 422 ... Imaging control unit 423 ... Drive main control unit 424 ... User interface control unit 430 ... Hard disk 431 ... Captured image storage unit 432 ... Master image storage unit 433 ... Correction value storage unit 460 ... Display 470 ... Keyboard 500 ... Workbench S1 ... Work surface W1, W1a, W2, W3, W4, W5 ... Window FA, FB, FC, FD ... Master image CB ... Calibration board Ar ... Error display area CBf ... Image of calibration board

Claims (9)

A calibration device for calibrating an imaging device arranged in a predetermined positional relationship with an arm of a robot,
For each of a plurality of captured images obtained by imaging the calibration board having feature points by the imaging device, a similarity is calculated for each of a plurality of master images in which the calibration board appears at different positions. A similarity calculator;
A classification unit that classifies the plurality of captured images into a plurality of groups associated with each master image based on the calculated similarity;
In each group of the plurality of groups, one representative image having a similarity equal to or greater than a predetermined value is extracted, and the feature point in the calibration board is based on a representative image group including the extracted plurality of representative images. A calibration processing unit that determines a correction value for converting a physical position and a position of the feature point in the captured image;
A determination unit that determines whether an error of the conversion by the determined correction value is equal to or less than a predetermined value;
A display unit that displays imaging number related information related to the number of the captured images classified into each of the plurality of groups when the error is determined to be larger than the predetermined value;
A calibration device comprising: The calibration device according to claim 1,
The number-of-capturing number related information is a calibration device including information that can identify a group having the smallest number of classified captured images among the plurality of groups. In the calibration apparatus according to claim 1 or 2,
The number-of-capturing number related information is a calibration device including information indicating the number of the captured images classified into each of the plurality of groups. The calibration device according to any one of claims 1 to 3, further comprising:
For each group, a statistical processing unit for obtaining a statistical characteristic of the similarity calculated for each of the captured images classified into the group,
The said display part is a calibration apparatus which displays the statistical characteristic relevant information relevant to the said statistical characteristic, when the said error is larger than the said predetermined value. The calibration device according to claim 4,
The statistical characteristic is the highest value among the similarities calculated for each of the captured images classified into each group,
The statistical characteristic-related information, among the plurality of groups, the maximum value including information capable of identifying the lowest group, the calibration apparatus. The calibration device according to claim 4,
The statistical characteristic is an average value of the similarity calculated for each captured image classified into each group,
The statistical characteristic-related information, among the plurality of groups, the average value including information capable of identifying the lowest group, the calibration apparatus. In the calibration apparatus as described in any one of Claim 1- Claim 6,
The said display part is a calibration apparatus which displays the said error. In the calibration apparatus according to any one of claims 1 to 7,
The imaging device continuously acquires the captured image,
The calibration apparatus, wherein the calibration processing unit determines the representative image group having the smallest error among the plurality of obtained captured images each time the captured image is acquired by the imaging unit. A calibration method for an imaging apparatus arranged in a predetermined positional relationship with an arm of a robot,
(A) For each of a plurality of captured images obtained by imaging the calibration board having feature points by the imaging device, the similarity to each of a plurality of master images in which the calibration board is located at different positions Calculating
(B) classifying the plurality of captured images into a plurality of groups associated with each master image based on the calculated similarity;
(C) In each group of the plurality of groups, one representative image having a similarity equal to or greater than a predetermined value is extracted, and the calibration of the feature points is performed based on a representative image group including the extracted plurality of representative images. Determining a correction value for converting a physical position on the motion board and a position of the feature point in the captured image;
(D) determining whether an error in the conversion due to the determined correction value is equal to or less than a predetermined value;
(E) when it is determined that the error is larger than the predetermined value, displaying imaging number related information related to the number of the captured images classified into each of the plurality of groups;
An imaging apparatus calibration method comprising:

Images