JP2022006427A - Imaging system construction device - Google Patents

Abstract

To provide a device that automatically generates, on a virtual space, a specification and an arrangement of an imaging device and a parameter necessary for camera/robot calibration, and automatically construct an optimal imaging system by performing an evaluation using a real device.SOLUTION: An imaging system construction device comprises: an imaging system planning unit that plans specifications and arrangements of an imaging device satisfying a system requirement of an imaging system and a reference object and parameters for calibrating the imaging device and a robot; an imaging system evaluation unit that evaluates accuracy of calibrating, using a real device consisted of the robot, the reference object and the imaging device, which is planned by the imaging system planning unit; and a requirement feedback part that when the accuracy in calibrating evaluated by the imaging system evaluation unit does not satisfy the system requirement, feeds back an additional requirement necessary for satisfying the system requirement to the imaging system planning unit to enable the calibration to be re-executed or the real device to be restructured.SELECTED DRAWING: Figure 1

Description

The present invention relates to an imaging system construction device that assists in the construction of an imaging system used for estimating the surrounding environment of an autonomous robot.

Needs for autonomous robots that automate various tasks that handle some kind of work (work object), such as picking work in the logistics field and assembly work in the industrial field, which have been performed by humans in the past in order to solve the labor shortage in recent years. Is increasing. In order for the autonomous robot to perform appropriate autonomous work, it is necessary to construct an imaging system that can recognize the surrounding environment such as the posture and work of the autonomous robot with high accuracy.

To build a high-precision imaging system, select the type and specifications of the imaging device so that the attitude of the autonomous robot estimated based on the captured image matches the actual attitude of the autonomous robot, or for the autonomous robot. It is necessary to carry out calibration (hereinafter also referred to as "calibration") to determine the relative arrangement (position, posture) of the image pickup device and to optimize the parameters used for actual posture estimation from the captured image. .. For this calibration, for example, a target marker with a known shape is attached on an autonomous robot, the robot changed to various postures is photographed by an image pickup device, and a plurality of pairs of captured images and the postures of the robot are compared. , There is a method of estimating the relative position and posture between the image pickup device and the autonomous robot.

However, with the above calibration method, there is a concern that the man-hours will increase when constructing the imaging system. For example, since the work contents of the robot and the conditions of the surrounding environment differ depending on the system introduction destination, a huge amount of man-hours are required to select an imaging method such as selection and arrangement of an imaging device depending on the conditions. In addition, there is a risk that the man-hours for adjusting parameters such as the posture group of the robot for realizing highly accurate calibration will be enormous. Therefore, in recent years, much attention has been paid to mechanisms and prototypes for reducing the man-hours for constructing an imaging system.

For example, in the abstract of Patent Document 1, as a solution for "facilitating the setting for performing calibration", "the robot control system is based on the image captured by the image pickup unit, and the image pickup unit is concerned. The robot's A command generation unit that generates a command to position the action unit, a calibration execution unit that executes calibration to calculate the correspondence relationship, and a reference object associated with the robot's action unit in calibration are arranged. It includes a setting reception unit that accepts the setting of the calibration area, which is the area to be used. " That is, in Patent Document 1, a simulation environment is constructed from the information of the robot photographed by the image pickup device, and the posture group of the robot used for calibration is selected or automatically generated on the GUI to reduce the man-hours for calibration. It is reducing.

Japanese Unexamined Patent Publication No. 2019-217571

As described above, according to Patent Document 1, calibration can be performed while reducing the man-hours for determining parameters related to the posture group of the robot.

However, Patent Document 1 does not consider the accuracy of success or failure of the work after calibration, nor does it consider reducing the work man-hours related to the selection of the imaging method.

Therefore, an object of the present invention is to utilize a real device including a robot, an image pickup device, and a target marker, and a virtual device on a simulator to select and arrange an image pickup device that can guarantee the success of work, and the posture of the robot at the time of calibration. By automatically formulating the parameters related to the group and feeding back the additional requirements for advancing the imaging system from the evaluation of the calibration by the actual device to the simulator, the calibration is automatically performed again in the virtual device, or It is an object of the present invention to provide an image pickup system construction device that makes it possible to present an instruction for modifying an image pickup method to an operator and facilitates the construction of an optimum image pickup system.

In order to solve the above problems, the image pickup system construction device of the present invention is an image pickup system construction device that assists in the construction of an image pickup system used for estimating the surrounding environment of a robot, and the image pickup system has a three-dimensional position with the robot. The image pickup system construction device includes an image pickup device for capturing a reference object for acquisition, and a base / camera conversion matrix used for conversion between the base coordinate system of the robot and the camera coordinate system of the image pickup device. An imaging system planning unit that plans the specifications and arrangement of the imaging device and the reference object that meet the system requirements of the system, and calibration parameters of the imaging device and the robot, and the robot planned by the imaging system planning unit. An image pickup system evaluation unit that evaluates the accuracy of calibration by a real device including the reference object and the image pickup device, and if the accuracy of the calibration evaluated by the image pickup system evaluation unit does not satisfy the system requirements, the system. It is provided with a requirement feedback unit that enables re-execution of the calibration or reconstruction of the actual device by feeding back the additional requirements necessary for satisfying the requirements to the imaging system planning unit. ..

According to the image pickup system construction device of the present invention, an image pickup system that meets customer requirements by automatically formulating the type and arrangement of the image pickup device and calibration parameters by utilizing a real device and a virtual device on a simulator. The man-hours required for construction can be reduced.

Functional block diagram showing an imaging system construction device in one embodiment Diagram showing the hardware configuration of the controller Schematic diagram of a real device Diagram showing the coordinate system and coordinate transformation of a real device Diagram showing an example of the input screen of the requirement input device Diagram showing an example of the input screen of the requirement input device Diagram showing the configuration of the actual equipment planning unit The figure which shows the structure of the virtual calibration evaluation part Flow chart showing the processing of the calibration data acquisition unit The figure which shows the structure of the calibration evaluation part Flow chart showing the processing of the calibration accuracy estimation unit Flow chart showing the processing of the error estimation unit Flowchart showing the processing of the evaluation operation execution unit Diagram showing the configuration of the requirements feedback section The figure which shows the structure of the cause estimation part of lack of accuracy Flowchart showing the processing of the additional requirement generator The figure which shows the structure of the modification which changed the arrangement of a real device

Hereinafter, an embodiment of the image pickup system construction apparatus of the present invention will be described in detail with reference to the drawings as appropriate.

(overall structure)
FIG. 1 is a functional block diagram showing the configuration of the image pickup system construction device 2 of this embodiment and the input / output of the actual device 3 and the requirement input device 4. The image pickup system construction device 2 is for capturing an image for estimating the posture of the robot 31, which is necessary for automating the picking work by the robot arm (hereinafter, referred to as "robot 31"), which is one aspect of the autonomous robot. It is a device that assists in the construction of the imaging system of. The image pickup system in the present invention is a system defined by _a combination of the specifications and arrangement (relative position, relative posture) of the image pickup device 33 described later and the base / camera conversion matrix M1 described later.

(Control device 1)
First, the hardware configuration of the control device 1 that embodies the image pickup system construction device 2 will be described with reference to FIG. As shown in the figure, the control device 1 connects a CPU 11 that controls the entire system, a bus 12 that transmits commands of the CPU 11, a ROM 13, a RAM 14, a storage device 15, a network interface 16, and an image pickup device 33 that will be described later. It is a general computer having an image pickup interface 17 for performing screen output, a screen display interface 18 for performing screen output, and an input interface 19 for performing external input. Further, a robot control device 16a for controlling and operating the robot 31 is connected to the network interface 16.

Further, in the storage device 15, a program 15a for generating a virtual device on the simulator and executing calibration (calibration), an operating system 15b, and a real device 3 are used as virtual devices on the simulator. A 3D model 15c or the like for generation is stored. By processing the program 15a loaded in the RAM 14 by the CPU 11, each function of the image pickup system construction device 2 is realized.

(Image pickup system construction device 2)
Next, an outline of each function of the image pickup system construction device 2 realized by the control device 1 will be described. As shown in FIG. 1, the functions of the image pickup system construction device 2 are roughly classified into an image pickup system planning unit 2A, an image pickup system evaluation unit 2B, and a requirement feedback unit 29.

The image pickup system planning unit 2A is composed of a requirement reception unit 20, a real device planning unit 21, a virtual calibration evaluation unit 22, a calibration parameter generation unit 23, and a calibration parameter 24. The requirement reception unit 20 outputs the system requirements input from the requirement input device 4 to the actual device planning unit 21 and the virtual calibration evaluation unit 22. The actual device planning unit 21 plans the specifications and arrangement (position, posture) of the actual device 3 that satisfies the system requirements by using the simulator. The virtual calibration evaluation unit 22 generates a virtual device on the simulator from the information of the specifications and arrangement of the real device 3 planned by the real device planning unit 21, and uses the virtual device to generate the calibration data necessary for executing the calibration. Evaluation is performed by virtually generating it and virtually executing calibration. The calibration parameter generation unit 23 generates the calibration parameter 24, which is a posture group of the robot 31, based on the calibration data generated by the virtual calibration evaluation unit 22.

The image pickup system evaluation unit 2B is composed of a calibration data acquisition unit 25, a calibration data 26, a calibration execution unit 27, and a calibration evaluation unit 28. The calibration data acquisition unit 25 controls the actual device 3 based on the calibration parameter 24, and collects the calibration data 26 including the encoder value which is the angle information of each joint of the robot 31 and the image pickup data by the image pickup device 33. The calibration execution unit 27 obtains a transformation matrix of relative positions and postures of the robot 31 and the image pickup device 33 by executing calibration using the calibration data 26. The calibration evaluation unit 28 obtains the accuracy of calibration.

The requirement feedback unit 29 determines whether the calibration accuracy estimated by the configuration evaluation unit 28 meets the system requirements, and if it does not meet the system requirements, it is necessary to reconstruct the imaging system so as to meet the system requirements. The additional requirements are fed back to the requirement reception unit 20.

Hereinafter, after explaining the actual device 3 and the requirement input unit 4, the actual device planning unit 21, the virtual calibration evaluation unit 22, the calibration data acquisition unit 25, the calibration evaluation unit 28, and the requirement feedback unit 29 of the imaging system construction device 2 are described in detail. To explain to.

(Reality device 3)
FIG. 3 is a schematic diagram of the actual device 3. As shown here, the reality device 3 accommodates a robot 31 which is an articulated robot arm, a target marker 32 which is a reference object installed on the robot 31, and a part of the robot 31 and its surroundings in an angle of view. It has an image pickup device 33 and. Hereinafter, each will be described in detail.

In this embodiment, a 6-axis robot arm is used as the robot 31. This 6-axis robot arm has a base portion 31a, a first link 31b, a first joint 31c, a second link 31d, a second joint 31e, a third link 31f, and a third joint 31g. Since the arm 31h having 3 degrees of freedom is attached before the 3rd joint 31g, the robot 31 has 6 degrees of freedom in total including the 1st joint 31c, the 2nd joint 31e, and the 3rd joint 31g. , Can take any posture. Although the installation position of the robot 31 is fixed to the floor or the base in this embodiment, the degree of freedom can be increased by using a short axis slider or the like, and work targeting a wider space can be executed. ..

The target marker 32 is a flat jig printed with a pattern having known dimensions, and the three-dimensional position of the target marker 32 can be specified by the captured image of the image pickup apparatus 33. If the three-dimensional position can be specified by the captured image of the image pickup device 33, the target marker 32 does not need to be a flat jig, and a spherical jig or a jig having a special shape can be used. .. The attachment position of the target marker 32 is attached to the tip of the arm 31h in this embodiment, but the attachment position is not limited to this example as long as it is on the robot 31. However, in general, the robot 31 has a high degree of freedom, and it is desirable to attach it to the tip of an arm capable of photographing the target marker 32 in a space where picking work is performed.

The imaging method of the image pickup apparatus 33 needs to be capable of three-dimensional measurement in order to measure the shape of the work that is the picking object. Although a stereo camera is used in this embodiment, a Time of Flight (ToF) type camera or an active stereo type camera may be used. Although the image pickup device 33 can be installed at any place such as a ceiling or a wall, it is desirable to install the image pickup device 33 so that the work can be seen within a range where three-dimensional measurement can be performed accurately.

(Calibration of the coordinate system and coordinate transformation of the actual device 3)
FIG. 4 is a diagram showing the relationship between the coordinate system of the actual device 3 and the coordinate transformation. As shown here, the coordinate system of the actual device 3 includes a base coordinate system C1 whose origin is the base of the robot 31, an arm coordinate system _C2 whose origin is the arm tip of the robot 31, and _a target marker 32. There is a marker coordinate system C3 whose origin is the center, and a camera coordinate system C _{4 whose} origin is the optical center of the image pickup apparatus 33.

The conversion matrix used for the coordinate conversion of the actual device 3 includes the base / camera conversion matrix M ₁ used for the conversion of the base coordinate system C ₁ and the camera coordinate system C ₄ , the marker coordinate system C ₃ and the camera coordinate system C ₄ . The marker / camera conversion matrix M ₂ used for the conversion of, the arm / base conversion table column M ₃ used for the conversion of the arm coordinate system C ₂ and the base coordinate system C ₁ , and the arm coordinate system C ₂ and the marker coordinate system C ₃ . There is an arm marker transformation matrix _M4 used for transformation.

For example, when the rotation matrix of the base / camera transformation matrix M ₁ is R _cr and the parallel traveling matrix is t _cr , any point C (x _c , y _c , z _c ) in the camera coordinate system C ₄ is expressed by the following equation. ₁ can be converted into a point R (x _r , y _r , z _r ) of the base coordinate system C1.

The coordinate transformation matrix of the real device 3 is generally acquired by the following method. The base / camera transformation matrix M ₁ is acquired by calibration. The marker-camera transformation matrix M ₂ is acquired by photographing the target marker 32 with the image pickup apparatus 33 and analyzing the obtained image. The arm / base transformation matrix M ₃ is acquired by using the 3D model 15c of the robot 31 and the joint angle information. The arm marker transformation matrix M ₄ is acquired by unknown or by the design value of the jig for attaching the target marker 32 to the robot 31.

In this embodiment, the origin of each coordinate system, the direction of the coordinate system, and the coordinate transformation are set as described above, but the present invention is not limited to this example.

(Requirement input device 4)
In the requirement input device 4, the worker inputs customer requirements such as work contents and work type. As a result, the image pickup system construction device 2 can acquire system requirements such as measurement accuracy, calibration accuracy, cost, operating range of the robot 31, and a space in which the image pickup device 33 can be attached, which are necessary for the image pickup system.

5A and 5B are screen examples of the screen display unit of the requirement input device 4. Here, a general notebook PC is used as the requirement input device 4. The worker in charge of the customer requirement investigation can easily investigate the customer requirement by conducting a hearing according to the contents displayed on the screen at the customer's site. Hereinafter, the details of the requirement input device 4 will be described.

As an example of customer requirements to be input to the requirement input device 4, as shown in the input screen 41 of FIG. 5A, the type of robot 31 to be used, the type of work content such as depalletization and piece picking, the type of work such as cardboard and wooden box, and the work. There are size and shape of the robot, budget and target picking work time. By outputting these customer requirements to the requirement reception unit 20 of the image pickup system construction device 2 via the requirement input device 4, the requirement reception unit 20 outputs the system requirements necessary for the image pickup system to the actual device planning unit 21 or virtual. It can be output to the calibration evaluation unit 22. For example, if the work is a plurality of large cardboard boxes loaded on a luggage rack, it is necessary to select an image pickup device 33 having a wide angle of view. Is output. In addition, as a customer requirement, by outputting to the requirement reception unit 20 that the work content is picking and the end effector is a suction hand, the areas of the suction surface of the suction hand and the gripping surface of the cardboard are compared, and it is necessary for gripping. The system requirements for measurement accuracy and calibration accuracy are output to the actual device planning unit 21 and the virtual calibration evaluation unit 22.

Further, as shown in the input screen 42 of FIG. 5A, the customer requirement for input includes a space in which the image pickup device 33 can be installed in the factory where the robot 31 is introduced. By creating a three-dimensional space based on the design information of the factory, installing one or more robots, and specifying the space where the image pickup device 33 can be installed around it, the system requirements of the mounting range of the image pickup device 33 can be specified. It can be output to the actual device planning unit 21. The mounting range may be a space designation such as a simple rectangular parallelepiped, or a complicated shape.

Further, as shown in the input screen 43 of FIG. 5B, installation objects and obstacles in the factory or in the vicinity of the robot 31 may be registered as customer requirements. Further, the equipment is not limited to the current equipment, and may be equipment that is expected to be installed in the future. By inputting this customer requirement into the requirement reception unit 20, it is possible to prevent the work or the robot 31 from appearing in the image taken by the image pickup apparatus 33 due to an installed object or an obstacle in advance. Further, when there is an area or an object that cannot be photographed from the viewpoint of information confidentiality, the system requirements regarding the imaging range can be output to the actual device planning unit 21 by similarly registering the object as an object that cannot be photographed.

Further, as in the input screen 44 of FIG. 5B, the position of a light source such as a lighting or a window may be input as a customer requirement. Some of the image pickup devices 33 capable of three-dimensional measurement cannot be accurately measured due to the light emitted by lighting or the like. By registering the light source as a customer requirement, in a space where the lighting is strong, select a model that is strong against the illumination light as the image pickup device 33, and set the position and mounting angle of the image pickup device 33 so that the reflected light does not enter. It is possible to output the system requirements to be performed to the actual device planning unit 21.

(Reality Equipment Planning Department 21)
The actual device planning unit 21 determines the specifications and arrangement of the image pickup device 33 and the target marker 32 among the actual devices 3 on the simulator based on the system requirements input from the requirement input device 4.

FIG. 6 is a block configuration diagram of the actual device planning unit 21. As shown here, the actual device planning unit 21 includes a requirement database 21a, an image pickup device database 21b, an image pickup device selection unit 21c, an image pickup device placement determination unit 21d, a marker selection unit 21f, and a marker placement determination unit 21g. And the hardware arrangement information 21h. Hereinafter, they will be described in sequence.

The requirement database 21a is a system requirement output from the requirement reception unit 20, such as measurement accuracy and calibration accuracy required for picking work, cost, picking work time, operating range of the robot 31, and space in which the image pickup device 33 can be attached. To hold. The image pickup device database 21b stores information such as a shooting method, an angle of view, an operating distance, accuracy, a resolution, design information, and a protection grade for each of the plurality of image pickup devices 33. Based on the requirement database 21a and the image pickup device database 21b, the image pickup device selection unit 21c selects an image pickup device 33 that satisfies the requirements of price and picking work time and has the measurement accuracy required for picking in an attachable space. The image pickup device arrangement determination unit 21d is located in a position or posture in which the image pickup device 33 can be mounted by utilizing existing equipment, or a mounting position and posture in which the selected image pickup device 33 can be measured most accurately. To decide.

The marker selection unit 21f selects a target marker that can be accurately detected when the captured image of the selected image pickup apparatus 33 is analyzed from the marker database 21e. The marker placement determination unit 21g determines the mounting position of the target marker 32 based on the shape and working range of the robot 31, the surrounding environment, and the like.

The hardware arrangement information 21h holds the types and arrangements of the selected image pickup apparatus 33 and the target marker 32.

Further, the actual device configuration database that holds the configuration of the actual device 3 in the past project is added to the actual device planning unit 21, and the above-mentioned image pickup device selection unit 21c, image pickup device arrangement determination unit 21d, marker selection unit 21f, and so on. The configuration of the actual device 3 may be determined by connecting to the marker arrangement determination unit 21g.

In this embodiment, the specifications and arrangement information of the image pickup device 33 and the target marker 32 are automatically determined. However, a plurality of candidates satisfying the system requirements for the type and arrangement of the image pickup device 33 and the target marker 32 are stored in the user. A real device presentation unit for comparison and examination by presenting and a real device determination unit to be reflected in the hardware arrangement information 21h after the determination may be provided.

(Virtual Calibration Evaluation Unit 22)
The virtual calibration evaluation unit 22 virtually generates the actual device 3 on the simulator, virtually generates calibration data based on the algorithm, and performs calibration using the data. As a result, the accuracy is sufficient. Evaluate if there is.

FIG. 7 is a block configuration diagram of the virtual calibration evaluation unit 22. As shown here, the virtual calibration evaluation unit 22 includes a virtual device generation unit 22a, a calibration method determination unit 22b, a generation logic database 22c, a calibration parameter generation unit 22d, a virtual calibration parameter 22e, and virtual calibration. It has a data generation unit 22f, a virtual calibration data 22g, a virtual calibration execution unit 22h, a calibration accuracy evaluation unit 22i, and the above-mentioned 3D model 15c. Hereinafter, they will be described in sequence.

The virtual device generation unit 22a virtually generates the hardware arrangement information 21h output by the real device planning unit 21 on the simulator using the 3D model 15c stored in the storage device 15. The calibration method determination unit 22b selects a calibration method from the calibration accuracy required by the system requirements. The calibration parameter generation unit 22d defines an algorithm for generating the command value of the posture group from the generation logic database 22c that stores a plurality of algorithms for generating the command value of the posture group of the robot 31 used for calibration, and the robot 31 is based on the algorithm. The virtual calibration parameter 22e, which is the command value of the posture group of, is generated.

The virtual calibration data generation unit 22f generates virtual calibration data 22g including posture values and shooting data when the robot 31 on the simulator is moved based on the virtual calibration parameter 22e.

The virtual calibration execution unit 22h executes calibration from the virtual calibration data 22g, and estimates the base _/ camera conversion matrix M1. The calibration accuracy evaluation unit 22i compares the estimated base / camera transformation matrix M ₁ with the true value at the time of simulator design, and evaluates whether the calibration accuracy satisfies the system requirements.

A calibration method DB that holds a plurality of calibration methods for each use case may be created and used by the calibration method determination unit 22b to determine the calibration method. Similarly, the calibration method determination unit 22b may present a plurality of calibration methods on the screen and allow the user to select them.

Further, the generation logic database 22c stores a plurality of logics for generating the command value of the posture group of the robot 31 used for calibration, but this is used as the posture group DB and the command value of the posture group itself is used as the calibration parameter generation unit 22d. You may get it at.

Further, the virtual calibration data generation unit 22f and the virtual calibration execution unit 22h are used for the combination of the plurality of calibration methods and the algorithm for generating the command value of the posture group of the robot 31 from the calibration method determination unit 22b and the calibration parameter generation unit 22d, respectively. , The calibration accuracy evaluation unit 22i performs the processing, and by presenting the calibration accuracy of each combination and the time required for calibration to the user, the calibration method satisfying the system requirements and the command value of the posture group of the robot 31. You may choose a set of generation algorithms for.

(Proofreading data acquisition unit 25)
FIG. 8 is a flowchart showing the processing of the calibration data acquisition unit 25. This process is a process of acquiring the calibration data 26 including the posture value of the robot 31 required for calibration and a plurality of sets of images captured by the image pickup device 33. In this embodiment, the set of the robot 31 and the image is acquired. Shall be acquired in N sets.

First, in S1, the total number of steps N and the command value of the posture group of the robot 31 are acquired from the calibration parameter 24, and in S2, the number of steps i is set to 1.

Next, in S3, the posture of the robot 31 is controlled by the command value of the i-th (first) posture group, and in S4, the robot 31 is photographed by the image pickup device 33.

In S5, it is determined whether the target marker 32 can be detected by analyzing the shooting data. Originally, it is desirable that only the command value of the robot posture that can be detected by the target marker 32 is applied on the simulator by the virtual calibration evaluation unit 22, but it cannot be detected due to the difference between the simulator and the actual environment. May be.

If it is determined in S5 that the target marker 32 cannot be detected (No in the figure), the current posture is stored in the inappropriate posture database 25a in S6, and then the number of steps i is increased by 1 in S7. Return to S3.

On the other hand, when it is determined in S5 that the target marker 32 can be detected (Yes in the figure), the marker / camera transformation matrix _M2 is estimated by analyzing the shooting data in S8, and the image and the image in S9. The marker / camera transformation matrix M ₂ and the base / arm transformation matrix M ₃ are stored in the calibration data 26.

Then, in S10, it is determined whether all the steps have been completed. If all the steps have not been completed (No in the figure), the number of steps is increased by 1 in S7, and then the process returns to S3. On the other hand, when all the steps are completed (Yes in the figure), the processing of the calibration data acquisition unit 25 is completed.

(Proofreading evaluation unit 28)
The calibration evaluation unit 28 estimates the calibration accuracy from the log data of the calibration result obtained by the calibration execution unit 27 performing the calibration with the actual device 3.

FIG. 9 shows a block configuration diagram of the calibration evaluation unit 28. As shown here, the calibration evaluation unit 28 includes a calibration result 28a, a calibration evaluation 28b, a calibration accuracy estimation unit 28c, an error estimation unit 28d, and an evaluation operation execution unit 28e.

In the calibration result 28a, as a result of the calibration of the calibration execution unit 27, a log of the base / camera conversion matrix M ₁ , the image used for the calibration, the arm / base conversion matrix M ₃ which is the posture value of the robot 31, and the like. Includes data. Further, the calibration evaluation 28b maintains the accuracy of calibration.

The calibration accuracy estimation unit 28c estimates the accuracy of the base _/ camera transformation matrix M1 estimated by calibration using the calibration result 28a. The error estimation unit 28d estimates the magnitude of the error of the base _/ camera transformation matrix M1. The evaluation operation execution unit 28e generates and evaluates an operation for evaluating the accuracy of calibration.

The calibration evaluation unit 28 implements one or a plurality of calibration accuracy estimation unit 28c, error estimation unit 28d, and evaluation operation execution unit 28e after performing calibration for newly constructing an imaging system. Hereinafter, the calibration accuracy estimation unit 28c, the error estimation unit 28d, and the evaluation operation 28e will be described in detail.

<< Calibration accuracy estimation unit 28c >>
FIG. 10 is a flowchart showing an example of calibration accuracy estimation processing by the calibration accuracy estimation unit 28c.

First, in S11, the calibration accuracy estimation unit 28c acquires the base / camera conversion matrix M ₁ and the arm / marker transformation matrix M ₄ estimated by calibration from the calibration result 28a. Further, in S12, the arm / base conversion matrix M ₃ and the marker / camera conversion matrix M ₂ of each step collected by the calibration data acquisition unit 25 are acquired from the calibration data 26.

After that, in S13, the base / camera transformation matrix M ₁ obtained by calibration, the arm / base transformation matrix M ₃ of each step, and the marker / camera transformation matrix M ₂ are used, and the arm / marker transformation matrix in each step is used. Calculate _M4 . Then, in S14, the variance of the arm _- marker transformation matrix M4 in all steps of i = 1 to N is calculated and output as an evaluation value.

In S13 above, the rotation matrix of the base / camera conversion matrix M ₁ is R _cr , the parallel traveling matrix is t _cr , the rotation matrix of the marker / camera conversion matrix M ₂ at the i-step is R _mci , and the parallel traveling matrix is t _mci , i. Let R _ari be the rotation matrix of the arm / base conversion matrix _M3 of the step _, and _tari be the _{parallel progression} matrix. Is calculated as

In this way, the variance of the arm-marker transformation matrix M4 in all steps was calculated in S14, but since the value of the arm-marker transformation matrix M4 does not originally fluctuate for each step, _each parameter depends _on the magnitude of the variance value. It is possible to evaluate the estimation accuracy of.

<< Error estimation unit 28d >>
FIG. 11 is a flowchart showing an example of error estimation processing by the error estimation unit 28d.

First, in S15, the parameters D (x, _y , z, r _x , ry, r _z ) of each axis of the base / camera transformation matrix M ₁ estimated by the calibration are acquired from the calibration result 28a. Further, in S16, the evaluation function f obtained when the calibration is executed is acquired from the calibration result 28a.

After that, in S17, the second-order differential value of the evaluation function f is calculated for each axis, and in S18, the sensitivity of the evaluation function f is estimated from the second-order differential value in each axial direction. Finally, in S19, the accuracy of each axis is evaluated and output by the sensitivity of the evaluation function f.

Here, an example of acquiring the second derivative value of the evaluation function f on the x-axis when dx is a minute value in S18 is shown below.

In Equation 3, the parameter D (x, y, z) of the position component is evaluated, but another index such as the parameter D (r _x , _{ry, r z )} of the posture component may be used as the evaluation index. ..

<< Evaluation operation execution unit 28e >>
FIG. 12 is a flowchart showing an example of a process in which the evaluation operation execution unit 28e operates the actual device 3 to obtain the accuracy of calibration.

First, in S20, the target marker 32 on the robot 31 is moved to an arbitrary point A such as on the optical axis of the image pickup device 33 by using the calibration result 28a. Next, in S21, the image pickup device 33 is made to take an image of the target marker 32. After that, in S22, the marker-camera transformation matrix _M2 is estimated by analyzing the imaging data, and in S23, the accuracy is calculated by comparing the estimation result with the true value of the point A.

It should be noted that the current image obtained by evaluating the deviation by bringing the hand of the robot 31 into contact with an object in space or by photographing the target marker 32 with the image pickup device 33 and the calibration result 28a are referred to. Whether the actual device 3 is operated to evaluate the accuracy, such as generating an ideal image in the posture of the robot 31 and evaluating the difference, or operating the robot 31 to evaluate whether it can be moved to an arbitrary position. For example, the processing content of the evaluation operation execution unit 28e is not limited to the example of FIG.

(Requirement feedback section 29)
The requirement feedback unit 29 has a function of switching the logic for generating a command value of the calibration posture group used in the virtual calibration evaluation unit 22 when the calibration accuracy does not meet the system requirements, or a command of an additional posture. A function to output additional requirements to generate values, a function to output changes in the specifications and arrangement of the image pickup device 33 to the actual device planning unit 21 as additional requirements, and during picking work using the image pickup system built by the customer. It has a function to determine whether calibration needs to be performed when the work fails, and to output additional requirements to perform calibration again if necessary.

FIG. 13 is a functional block diagram showing an example of the requirement feedback unit 29. As shown here, the requirement feedback unit 29 includes an accuracy satisfaction determination unit 29a, a virtual device change requirement generation unit 29b, an accuracy shortage cause estimation unit 29c, an additional requirement generation unit 29d, and a robot work execution unit 29f. It is equipped with a recalibration implementation evaluation unit 29e.

The accuracy sufficiency determination unit 29a determines whether the calibration accuracy estimated by the calibration accuracy evaluation unit 22i satisfies the system requirements.

When the calibration accuracy meets the system requirements, the virtual device change requirement generation unit 29b corrects the arrangement of the virtual device on the simulator based on the value of the calibration data 26 obtained by calibrating with the actual machine, and calibrates. An instruction is generated to delete the posture registered in the inappropriate posture database 25a from the calibration parameter 24 which is the command value of the posture group of the robot 31 for the robot 31.

The accuracy estimation unit 29c estimates the cause when the calibration accuracy does not meet the system requirements. The additional requirement generation unit 29d generates additional requirements necessary for satisfying the system requirements for the simulator due to the cause of insufficient accuracy.

After the robot work execution unit 29f completes the construction of the image pickup system once, the robot 31 of the actual device 3 is used to carry out the work content required by the customer. The recalibration implementation evaluation unit 29e determines whether it is necessary to perform calibration again when the work content fails in the robot work execution unit 29f, and if necessary, performs calibration in the requirement reception unit 20. Generate additional requirements and feed them back to the simulator.

Hereinafter, the accuracy shortage cause estimation unit 29c, the additional requirement generation unit 29d, and the recalibration implementation evaluation unit 29e will be described in detail.

<< Insufficient accuracy cause estimation unit 29c >>
FIG. 14 shows a functional block diagram of the accuracy shortage cause estimation unit 29c. As shown here, the accuracy shortage cause estimation unit 29c includes a real device arrangement verification unit 29c1, a calibration parameter verification unit 29c2, and a real device specification verification unit 29c3.

The actual device arrangement verification unit 29c1 verifies whether or not the accuracy is insufficient due to the arrangement of the actual device 3. Specifically, when there is a discrepancy between the arrangement of the actual device 3 by the actual device planning unit 21 and the arrangement of the actual device 3 by the calibration data 26, it is estimated that the cause is the arrangement of the actual device 3. Further, by comparing the imaging data acquired by the imaging device 33 with the virtual imaging data generated by the virtual calibration evaluation unit 22, it is confirmed whether the arrangement of the actual device 3 generated by the simulator is significantly different from the actual arrangement. You may.

The calibration parameter verification unit 29c2 verifies whether the accuracy is insufficient due to the calibration parameter 24. Specifically, when there is a posture of the robot 31 in which the target marker 32 cannot be detected in the inappropriate posture database 25a, the virtual calibration data 22g in which the posture of the inappropriate database 25a is deleted is virtually calibrated for accuracy. Confirm. As a result, if the accuracy drops by a certain value or more, it is presumed that the cause is the calibration parameter 24, which is the posture group of the robot 31.

The actual device specification verification unit 29c3 verifies whether or not the accuracy is insufficient due to a failure or malfunction of the actual device 3. Specifically, if the number of inappropriate posture databases 25a occupies the majority and the target marker 32 cannot be detected at all, the image pickup device 33 is out of order or the lens of the image pickup device 33 is dirty. Or, it is determined that the target marker 32 is damaged or the actual device is installed in a place completely different from the assumption. In order to verify the failure or dirt of the image pickup device 33, three-dimensional measurement is performed from the image pickup device 33, and if valid three-dimensional information cannot be obtained at the center of the visual field, it is presumed that the image pickup device 33 is the cause. When valid three-dimensional information is obtained at the center of the visual field, it is presumed that the target marker 32 is damaged and the actual device 3 is misaligned.

<< Additional requirement generation unit 29d >>
FIG. 15 is a flowchart showing the processing of the additional requirement generation unit 29d. As shown here, first, in S24, it is confirmed whether the cause of the lack of accuracy estimated by the accuracy estimation unit 29c is a misplacement of the actual device 3. Then, if the cause is a misplacement of the real device 3 (Yes in the figure), the process proceeds to S25, and an additional requirement is presented to the user to match the layout of the real device 3 with the layout selected by the real device planning unit 21. To generate. On the other hand, if the cause is not the arrangement of the actual device 3 (No in the figure), the process proceeds to S26, and it is confirmed whether the cause of the lack of accuracy estimated by the accuracy estimation unit 29c is a defect of the calibration parameter 24.

Then, if the calibration parameter 24 is not the cause, the process proceeds to S27, and the determination result of whether the three-dimensional measurement is possible with the image pickup apparatus 33 is confirmed. If it is measurable (Yes in the figure), the process proceeds to S28 to generate an additional requirement for confirming whether the target marker 32 is damaged or the actual device 3 is arranged correctly. On the other hand, if it is not measurable (No in the figure), the process proceeds to S29 to generate an additional requirement for confirming a failure of the image pickup apparatus 33 or a stain on the lens.

On the other hand, if the cause is the calibration parameter 24, the process proceeds to S30 to determine whether the number of inappropriate postures is smaller than the number of all postures. Then, if the number is small (Yes in the figure), the process proceeds to S31, the inappropriate posture is deleted, the image pickup device 33 confirms whether there are obstacles or the like in the vicinity, and the robot is located around the inappropriate posture and in a space where there are no obstacles. Generate additional requirements to generate command values for 31 attitudes. On the other hand, if the number of inappropriate postures is not small (No in the figure), the process proceeds to S32, and an additional requirement for changing the algorithm for generating the command value of the posture group is generated.

Then, when any of the processes of S25, S28, S29, S31, and S32 is completed, the process of FIG. 15 is completed.

<< Recalibration Implementation Evaluation Unit 29e >>
Subsequently, the recalibration implementation evaluation unit 29e will be described. After the image pickup system is constructed, the recalibration execution evaluation unit 29e is called by the robot work execution unit 29f when the picking operation fails, and determines whether or not the calibration needs to be executed again. In this embodiment, by causing the actual device 3 to perform an operation of evaluating the calibration accuracy as in the evaluation operation execution unit 28e, it is determined whether or not there is a deviation in the positional relationship between the robot 31 and the image pickup device 33, or the calibration result is determined. Based on this, it is determined whether the image taken by the virtual device on the simulator and the image taken by the real device are different, or if there is a difference between the imaged data at the time of failure and the past imaged data, calibration is performed again. Generate additional requirements. Whether or not the images are dissociated can be determined by machine learning or the like.

With the above configuration, by feeding back the evaluation of the imaging system by the actual device 3 to the simulator, the logic for generating the command value of the calibration posture group is switched, or the additional requirement for generating the command value of the additional posture is output. By doing so, it is possible to re-calibrate or present the user to correct the imaging method. As an example, while the robot 31 is performing the work content of the customer requirement, the image pickup device 33 or the work is intentionally shifted to cause the work to fail, so that calibration is performed again or the image pickup method is modified. Can be confirmed.

(Specific example of operation of image pickup system construction device 2)
Next, as a specific example of the operation of the image pickup system construction device 2 of this embodiment, an operation of feeding back additional requirements to the simulator will be shown. First, the customer requirements are input from the requirement input device 4 by the worker, and the real device planning unit 21 and the virtual calibration evaluation unit 22 are implemented from the system requirements estimated based on the customer requirements by the requirement reception unit 20. After that, the calibration parameter generation unit 23 generates command values for the posture groups of the 20 robots 31, and the calibration data acquisition unit 25 performs processing.

However, when an unknown obstacle occurs around the robot 31 during calibration work due to a case where the customer's environment changes dynamically, there are some cases where the robot 31 cannot move to the command value. The command value at this time is stored in the inappropriate posture database 25a. In this case, it is assumed that four postures are registered in the inappropriate posture database 25a. Subsequently, the calibration is executed by the calibration execution unit 27 using the calibration data 26 normally acquired by the calibration data acquisition unit 25. The calibration evaluation unit 28 estimates the accuracy of calibration, and the requirement feedback unit 29 determines whether the accuracy satisfies the system requirements by the accuracy satisfaction determination unit 29a.

At this time, if the calibration accuracy does not satisfy the system requirements, the accuracy shortage cause estimation unit 29c is executed. In the accuracy shortage cause estimation unit 29c, the actual device placement verification unit 29c1 is first executed. The real device arrangement verification unit 29c1 compares the shooting data by the real device 3 with the virtual shooting data by the virtual device. If it is determined by this comparison that the cause is not the actual device 3, the verification process by the calibration parameter verification unit 29c2 is executed. The calibration parameter verification unit 29c2 refers to the inappropriate posture database 25a and virtually executes calibration according to the command value of the posture group of the robot 31 excluding the inappropriate posture, as in the virtual calibration evaluation unit 22.

As a result, if the calibration by the posture group excluding the four inappropriate postures does not satisfy the system requirements, the additional requirement generation unit 29d is called because it is necessary to change the calibration parameter 24. Since the calibration parameter 24 is the cause, the additional requirement generation unit 29d verifies the number of inappropriate postures, and as in this example, only a small number of the postures among all the command values are regarded as inappropriate postures, and the inappropriate posture database 25a. When it is stored in, the command value in the inappropriate posture database 25a is deleted from the calibration parameter 24, and the space with the obstacle is estimated by confirming whether there is an obstacle or the like in the vicinity by the image pickup device 33. Then, an additional requirement is generated so as to regenerate the command value of the posture of the robot 31 around the inappropriate posture and in a space where there is no obstacle, and the feedback is fed back to the requirement reception unit 20 to be calibrated by the virtual calibration evaluation unit 22. The algorithm for generating the parameters of the motion is changed, and the calibration with the calibration parameter 24 generated again is executed. The above processing completes the construction of an imaging device that meets the system requirements.

With the above image pickup system construction device 2, the robot at the time of selection / arrangement and calibration of the image pickup device 33 that can guarantee the success of the picking work by utilizing the actual device 3 including the robot 31, the target marker 32, and the image pickup device 33 and the simulator. By automatically formulating the parameters related to the 31 posture groups and feeding back the additional requirements for upgrading the imaging system from the evaluation of the calibration by the actual device to the simulator, the calibration is automatically executed again, or It is possible to present an instruction to modify the imaging method to the operator, and it is possible to easily construct an optimum imaging system.

Further, by inputting customer requirements such as work content and work type from the requirement input device 4, it is possible to mount the measurement accuracy, calibration accuracy, cost, operating range of the robot 31 and the image pickup device 33 required for the image pickup system. System requirements including various spaces can be estimated.

Further, the real device planning unit 21 can reduce the man-hours required for determining the specifications and arrangement of the real device that satisfies the input system requirements by the simulator having the three-dimensional information and the performance information of the real device.

Further, the virtual calibration evaluation unit 22 can generate a virtual device on the simulator based on the specifications and arrangement of the actual device determined by the actual device planning unit 21. In addition, the command value of the posture group of the robot 31 at the time of calibration is generated based on the command value generation algorithm on the simulator, calibration is performed by the virtual device, and the relative position between the robot and the sensor based on the calibration result. The accuracy of calibration can be confirmed by comparing the estimation parameters of the posture with the true values at the time of virtual device generation.

With the simulator including the actual device planning unit 21 and the virtual calibration evaluation unit 22, the specifications and arrangement of the actual device that can guarantee the success of the picking work, and the determination of the command value of the posture group of the robot 31 for calibration are determined. The man-hours required to construct the including imaging system can be reduced.

Further, the calibration evaluation unit 28 can evaluate the estimation accuracy when the calibration is performed by the actual device.

Originally, it is desirable to meet the system requirements with an imaging system constructed based on the simulator, but there is a risk that the calibration accuracy will decrease due to the difference between the simulator and the actual situation. Therefore, the requirement feedback unit 29 has a function of determining whether the calibration evaluation by the actual device 3 satisfies the system requirement, and if the calibration evaluation does not satisfy the system requirement, estimates the cause. If the cause is the command value of the calibration posture group, the simulator is switched with the algorithm for generating the command value of the calibration posture group, or an additional requirement for generating the command value of the posture of the additional robot 31 is output. By doing so, it is possible to execute the calibration again. In addition, when the cause is a real device, the user can check the failure of the real device, improper placement, etc. by outputting the change of the specification and arrangement of the image pickup device 33 to the simulator as an additional requirement and displaying it on the simulator. Is possible. In addition, if the customer fails to perform the picking work using the built imaging system, it is determined whether calibration needs to be performed, and if necessary, additional requirements are output to perform it again. By doing so, it has a function to execute calibration again.

As described above, the image pickup system construction apparatus of this embodiment can reduce the man-hours required for constructing an image pickup system that meets customer requirements.

(Modification 1)
As shown in FIG. 16, the image pickup device 33 may be attached to the arm 31h of the robot 31, and the image pickup system may be constructed using the target markers installed in the vicinity as a real device. Further, an image pickup device mounting method selection unit may be added to the actual device planning unit 21 to select whether to install the image pickup device 33 on the robot or in the vicinity.

1: Control device 2: Imaging system construction device 2A: Imaging system planning unit 20: Requirements reception unit 21: Real equipment planning unit 22: Virtual calibration evaluation unit 23: Calibration parameter generation unit 24: Calibration parameter 2B: Imaging system evaluation Unit 25: Calibration data acquisition unit 26: Calibration data 27: Calibration execution unit 28: Calibration evaluation unit 29: Requirements feedback unit 3: Actual device 31 Robot arm 32 Target marker 33 Imaging device 4: Requirements input device

Claims (10)

An imaging system construction device that assists in the construction of an imaging system used to estimate the surrounding environment of a robot.
The imaging system is
An image pickup device that captures a robot and a reference object for acquiring a three-dimensional position,
The base coordinate system of the robot and the base / camera conversion matrix used for conversion of the camera coordinate system of the image pickup apparatus are included.
The image pickup system construction device is
An imaging system planning unit that plans the specifications and arrangement of the imaging device and the reference object that satisfy the system requirements of the imaging system, and calibration parameters of the imaging device and the robot.
An imaging system evaluation unit that evaluates the accuracy of calibration by a real device including the robot, the reference object, and the imaging device, which is planned by the imaging system planning unit.
If the accuracy of the calibration evaluated by the imaging system evaluation unit does not meet the system requirements, the additional requirements necessary to meet the system requirements are fed back to the imaging system planning unit to re-calibrate the calibration. A requirement feedback unit that enables execution or reconstruction of the actual device,
An imaging system construction device characterized by being equipped with. In the image pickup system construction apparatus according to claim 1,
The imaging system planning unit is
An image pickup system construction device comprising a real device planning unit that plans specifications and arrangement of the image pickup device that satisfies the system requirements based on a database that holds performance information of the image pickup device. In the image pickup system construction apparatus according to claim 2,
The reality equipment planning department
An image pickup device selection unit that determines the specifications of the image pickup device based on the system requirements, and
An image pickup device arrangement determination unit that determines the arrangement of the image pickup device, and
A marker selection unit that selects the reference material, and
A marker placement determination unit that determines the mounting position of the reference object on the robot, and
An imaging system construction device characterized by being equipped with. In the image pickup system construction apparatus according to claim 1,
The imaging system planning unit is
Based on the database that holds the three-dimensional information of the actual device, a virtual device is generated on the simulator, calibration data is generated by operating the virtual device, and accuracy is evaluated by performing calibration. An imaging system construction device characterized by having a calibration evaluation unit. In the image pickup system construction apparatus according to claim 4,
The virtual calibration evaluation unit
A virtual device generation unit that generates a virtual device on the simulator based on the three-dimensional model of the real device,
A calibration method determination unit that determines the calibration method based on the system requirements,
A calibration parameter generator that generates calibration parameters for the posture of the robot used for calibration,
A virtual calibration data generation unit that operates the virtual device based on a command value and virtually generates calibration data,
A virtual calibration execution unit that virtually calibrates,
A calibration accuracy evaluation unit that evaluates whether the difference between the calibration result and the true value at the time of simulator generation meets the system requirements.
An imaging system construction device characterized by being equipped with. In the image pickup system construction apparatus according to claim 5.
The calibration parameter generation unit is
An imaging system construction device comprising a generation logic database having a plurality of algorithms for generating command values of the posture group of the robot. In the image pickup system construction apparatus according to claim 5.
The imaging system evaluation unit is
A calibration data acquisition unit that collects calibration data by operating the actual device based on the calibration parameters.
A calibration execution unit that executes calibration using the collected calibration data,
A calibration evaluation unit that evaluates the accuracy of calibration using the calibration results,
An imaging system construction device characterized by being equipped with. In the image pickup system construction apparatus according to claim 7.
The calibration evaluation unit
A calibration accuracy estimation unit that evaluates the accuracy of calibration based on the log data of the calibration result,
An error estimation unit that estimates the error based on the log data of the calibration result,
An evaluation operation execution unit that evaluates accuracy by generating and executing an operation for evaluation using the actual device, and
An imaging system construction device characterized by being equipped with. In the image pickup system construction apparatus according to claim 1,
The requirement feedback section is
An accuracy sufficiency judgment unit that determines whether the calibration accuracy meets the system requirements,
Insufficient accuracy cause estimation unit that identifies the cause when accuracy is insufficient,
An additional requirement generator that generates the additional requirements that the imaging system needs to meet the system requirements due to the identified cause,
An imaging system construction device characterized by being equipped with. In the image pickup system construction apparatus according to claim 1,
The requirement feedback section is
A robot work execution unit that causes the actual device to perform work content based on customer requirements,
A recalibration implementation evaluation unit that determines whether recalibration is necessary when the work fails and, if necessary, sends additional requirements for calibration to the imaging system planning unit.
An imaging system construction device characterized by being equipped with.

Images