JP6458912B1 - Position control device and position control method - Google Patents

Abstract

  In the case of including alignment with insertion for two objects, a route determination is performed that indicates a control amount for insertion based on the image acquired from the imaging unit 201 and the value of the force sensor 801 and learns from the result of alignment The periodic control amount adjustment value is output based on the unit 802, the periodic control amount set for each control period in order to reach the control amount, and the control amount adapted to the external force based on the value of the force sensor 801. Since the control parameter adjustment unit 1401 is provided, the robot arm 100 is operated by a control amount obtained by adding the trajectory generation control and the compliant motion control using the force sensor 801 until learning is converged in a normal reinforcement learning model. Although trial and error is necessary and may damage the environment, the present invention allows safe trials even in the early stages of learning. It is possible.

Description

  The present invention relates to a position control device and a position control method.

  When constructing a production system in which an assembly operation is performed by a robot arm, a teaching operation by a human hand called teaching is generally performed. However, in this teaching, since the robot repeatedly performs the operation only on the stored position, it may not be able to cope with an error caused by manufacturing or mounting. Therefore, if a position correction technique capable of absorbing this individual error can be developed, improvement in productivity can be expected, and the scene where the robot plays an active role also increases.

  Even in the current technology, there is a technology for performing position correction until just before connector insertion work using a camera image (Patent Document 1). Further, if a plurality of devices such as a force sensor and a stereo camera are used, position errors relating to assembly (insertion, work holding, etc.) can be absorbed. However, in order to determine the position correction amount, it is necessary to explicitly calculate the amounts such as the center coordinates of the gripped connector and the center coordinates of the connector on the insertion side from the image information as in the same reference. This calculation depends on the shape of the connector and must be set by the designer for each connector used. In addition, if 3D information can be obtained from a distance camera or the like, this calculation is relatively easy. However, since it is necessary to develop an image processing algorithm for each connector in order to obtain 2D image information, a lot of design costs are required. It will take.

In addition, there are techniques called deep learning and deep reinforcement learning as a technique for the robot to learn by itself and acquire appropriate behavior. However, in order to acquire appropriate actions through these learnings, it is usually necessary to collect a large amount of appropriate learning data. Also, when collecting data using techniques such as reinforcement learning, it is necessary to experience the same scene over and over again, requiring a huge number of trials, and performance cannot be guaranteed for unexperienced scenes. . Therefore, it is necessary to collect learning data for various scenes uniformly, which takes a lot of work.
For example, there is a method for obtaining an optimum route in one successful trial as in Patent Document 2, but it is not possible to collect data that can be used for deep learning or deep reinforcement learning, and it is necessary to perform learning for a considerable number of times. There is.

WO98-017444 JP 2005-125475 A

  When performing alignment with insertion for two objects, there is usually an independent function for indicating the position for learning and a function for operating a servo motor or the like for controlling the position of the robot. Therefore, since the load applied to the object is not considered, there is a problem that an excessive load is applied to the object depending on the amount of displacement applied for learning.

  The present invention has been made in order to solve the above-described problems, and learning while preventing an excessive load from being applied even if the function for learning and the function for controlling the position of the robot are different. The purpose is to collect data.

Position control apparatus according to the present invention, when aligning with the insert from one of mono two things to the other things, a shall be acquired from the imaging unit, is gripped by the gripping portion of the robot arm while the goods and the other image and objects have been photographed in the position instructs the value of the force sensor that measures the load on the gripper, the control amount of the position of the robot arm for insertion on the basis of the results for the combined, the path determination section that learns a control amount of the position of the robot arm positioning is successful from the value of the image and force sensor, in the course of learning, control of the position of the robot arm from the route deciding section which receives the amount and period control amount set for each first control cycle of the control unit to reach the control amount of the position of the robot arm, the force sensor corresponding to one control period A control amount that is adapted to the external force based on a control parameter adjusting section for outputting a periodic control amount adjustment value to the control unit on the basis of, having a.

  According to this invention, even if the function for learning and the function for controlling the position of the robot are different, learning data can be collected while preventing an excessive load from being applied to the object.

The figure in which the robot arm 100, the male side connector 110, and the female side connector 120 in Embodiment 1 are arrange | positioned. FIG. 3 is a functional configuration diagram of the position control device in the first embodiment. FIG. 3 is a hardware configuration diagram of the position control device according to the first embodiment. 5 is a flowchart in position control of the position control device in the first embodiment. The example of the figure which shows the insertion start position which the monocular camera 102 in Embodiment 1 image | photographed, the camera image in the periphery vicinity, and control amount. The figure which shows the example of the neural network in Embodiment 1, and the learning rule of a neural network. 5 is a flowchart using a plurality of networks in the neural network according to the first embodiment. FIG. 6 is a functional configuration diagram of a position control device in a second embodiment. The hardware block diagram of the position control apparatus in Embodiment 2. FIG. The figure which shows the mode of the trial of fitting with the male side connector 110 and the female side connector 120 in Embodiment 2. FIG. 10 is a flowchart in route learning of the position control device in the second embodiment. 10 is a flowchart in route learning of the position control device in the third embodiment. The figure which shows the example of the neural network in Embodiment 3, and the learning rule of a neural network. FIG. 6 is a functional configuration diagram of a position control device in a fourth embodiment. 10 is a flowchart in route learning of the position control device in the fourth embodiment.

Embodiment 1 FIG.
Embodiments of the present invention will be described below.

  In the first embodiment, a robot arm that learns the insertion position of each connector and assembles on the production line and a position control method thereof will be described.

  The configuration will be described. FIG. 1 is a diagram in which a robot arm 100, a male connector 110, and a female connector 120 according to the first embodiment are arranged. The robot arm 100 is provided with a grip portion 101 for gripping the male connector 110, and a monocular camera 102 is attached to the robot arm 100 at a position where the grip portion can be seen. The position of the monocular camera 102 is such that when the grip portion 101 at the tip of the robot arm 100 grips the male connector 110, the tip of the gripped male connector 110 and the female connector 120 on the side to be inserted can be seen. Install in.

FIG. 2 is a functional configuration diagram of the position control device according to the first embodiment.
2, the functions of the monocular camera 102 in FIG. 1, an imaging unit 201 that captures an image, a control parameter generation unit 202 that generates a control amount of the position of the robot arm 100 using the captured image, and a position The control unit 203 for controlling the current / voltage value with respect to the drive unit 204 of the robot arm 100 using the control amount and the position of the robot arm 100 are changed based on the current / voltage value output from the control unit 203. The drive unit 204 is configured.

The control parameter generation unit 202 is a function of the monocular camera 102. When an image is acquired from the imaging unit 201 that captures an image, the control parameter generation unit 202 controls the position (X, Y, Z, Ax, Ay, Az) value of the robot arm 100. The amount (ΔX, ΔY, ΔZ, ΔAx, ΔAy, ΔAz) is determined, and the control amount is output to the control unit 203. (X, Y, and Z are robot arm positions, and Ax, Ay, and Az are posture angles of the robot arm 100)
The control unit 203 controls the driving unit 204 based on the control amounts (ΔX, ΔY, ΔZ, ΔAx, ΔAy, ΔAz) with respect to the received position (X, Y, Z, Ax, Ay, Az) values of the robot arm 100. Determine and control the current and voltage values for each device.
The drive unit 204 operates at the current / voltage value for each device received from the control unit 203, so that the robot arm 100 moves to a position of (X + ΔX, Y + ΔY, Z + ΔZ, Ax + ΔAx, Ay + ΔAy, Az + ΔAz).

FIG. 3 is a hardware configuration diagram of the position control device according to the first embodiment.
The monocular camera 102 is communicably connected to the processor 302 and the memory 303 via the input / output interface 301 regardless of wired wireless communication. The input / output interface 301, the processor 302, and the memory 303 constitute the function of the control parameter generation unit 202 in FIG. The input / output interface 301 is also communicably connected to a control circuit 304 corresponding to the control unit 203 regardless of wired wireless communication. The control circuit 304 is also electrically connected to the motor 305. The motor 305 corresponds to the drive unit 204 in FIG. 2 and is configured as a component for controlling the position of each device. In this embodiment, the motor 305 is used as the hardware corresponding to the drive unit 204, but hardware that can control the position may be used. Therefore, the monocular camera 201 and the input / output interface 301, and the input / output interface 301 and the control circuit 304 may be configured separately.

Next, the operation will be described.
FIG. 4 is a flowchart in position control of the position control apparatus in the first embodiment.
First, in step S <b> 101, the grip portion 101 of the robot arm 100 grips the male connector 110. The position and orientation of the male connector 110 are registered in advance on the control unit 203 side in FIG. 2 and operated based on a control program registered in advance on the control unit 203 side.

Next, in step S102, the robot arm 100 is brought close to the vicinity of the insertion position of the female connector 120. The approximate position and orientation of the female connector 110 are registered in advance on the control unit 203 side in FIG. 2, and the position of the male connector 110 is determined based on the control program registered in advance on the control unit 203 side. Be operated.
Next, in step S103, the control parameter generation unit 202 instructs the imaging unit 201 of the monocular camera 102 to capture an image, and the monocular camera 103 includes the male connector 110 held by the holding unit 101, An image in which both the female connector 120 as the insertion destination is shown is captured.

  Next, in step S104, the control parameter generation unit 202 acquires an image from the imaging unit 201, and determines control amounts (ΔX, ΔY, ΔZ, ΔAx, ΔAy, ΔAz). Regarding the determination of the control amount, the control parameter generation unit 202 uses the processor 302 and the memory 303 of FIG. 3 as hardware and calculates the control amount (ΔX, ΔY, ΔZ, ΔAx, ΔAy, ΔAz) using a neural network. To do. A control amount calculation method using a neural network will be described later.

Next, in step S105, the control unit 203 acquires the control amounts (ΔX, ΔY, ΔZ, ΔAx, ΔAy, ΔAz) output from the control parameter generation unit 202, and sets a predetermined threshold and control amount. Compare all ingredients. If all the components of the control amount are equal to or less than the threshold value, the process proceeds to step S107, and the control unit 203 controls the drive unit 204 to insert the male connector 110 into the female connector 120.
If any component of the control amount is larger than the threshold value, in step S106, the control unit 203 uses the control amount (ΔX, ΔY, ΔZ, ΔAx, ΔAy, ΔAz) output by the control parameter generation unit 202 to drive the drive unit. 204 is controlled, and the process returns to step S103.

Next, the control amount calculation method using the neural network in step S104 of FIG. 4 will be described.
Before calculating the control amount using a neural network, as a preliminary preparation, the neural network can calculate the amount of movement from the input image to the fitting success. Collect. For example, the male connector 110 is gripped by the grip portion 101 of the robot arm 100 with respect to the male connector 110 and the female connector 120 in a fitted state whose positions are known. Then, the grip unit 101 is moved to a known insertion direction while moving to the insertion start position, and a plurality of images are acquired by the monocular camera 102. Further, the insertion start position is set as a control amount (0, 0, 0, 0, 0, 0), not only the movement amount from the fitting state to the insertion start, but also the peripheral movement amount, that is, the control amount ( ΔX, ΔY, ΔZ, ΔAx, ΔAy, ΔAz) and corresponding images are also acquired.
FIG. 5 is an example of a diagram illustrating an insertion start position taken by the monocular camera 102 according to Embodiment 1, a camera image near the periphery, and a control amount.

Then, based on a learning rule of a general neural network, using a plurality of sets of movement amounts from the fitting state to the insertion start position and images of the insertion start position and surrounding positions in the monocular camera 102 (eg, probability) Gradient method).
There are various forms such as CNN and RNN in the neural network, but the present invention does not depend on the form, and any form can be used.

FIG. 6 is a diagram illustrating an example of the neural network and the learning rule of the neural network according to the first embodiment.
An image obtained from the monocular camera 102 (for example, luminance and color difference values of each pixel) is input to the input layer, and control amounts (ΔX, ΔY, ΔZ, ΔAx, ΔAy, ΔAz) are output to the output layer. .
In the learning process of the neural network, the parameters of the intermediate layer are optimized so that the output value of the output layer obtained from the input image through the intermediate layer approximates the control amount stored in the image set. . The approximation method includes a stochastic gradient method.

Therefore, as shown in FIG. 5, not only the amount of movement from the fitted state to the start of insertion, but also the movement around it and the corresponding images are acquired and learned, so that more accurate learning is possible. It can be carried out.
5 shows a case where the position of the male connector 110 is fixed with respect to the monocular camera 102 and the position of only the female connector 120 is changed, but in reality, the gripping portion 101 of the robot arm 100 is shown. However, the male connector 110 is not gripped at an accurate position, and the male connector 110 may be misaligned due to individual differences or the like. Therefore, by acquiring and learning a set of a plurality of control amounts and images at the insertion start position and the position in the vicinity thereof when the male connector 110 is displaced from an accurate position in the learning process, the male connector 110 is acquired. And learning that can cope with individual differences between the female connector 120 and the female connector 120 are performed.

  However, it should be noted here that control amounts (ΔX, ΔY, ΔZ, ΔAx, ΔAy, ΔAz) are calculated by excluding the amount of movement from the fitting state position to the insertion start position at the time of shooting. The amount of movement from the position to the fitted state position needs to be stored separately for use in step S107 in FIG. Further, since the coordinates are obtained as the coordinate system of the monocular camera, the control unit 203 needs to control the robot arm 100 after converting the coordinate system of the monocular camera when the coordinate system of the entire robot arm 100 is different. .

  In this embodiment, since the monocular camera is fixed to the robot arm 100, the coordinate system where the female connector 120 is placed and the coordinate system of the monocular camera 102 are different. Therefore, if the monocular camera 102 is in the same coordinate system as the position of the female connector 120, conversion from the coordinate system of the monocular camera 102 to the coordinate system of the robot arm 100 is not necessary.

Next, details of the operation of FIG. 4 and an operation example will be described.
In step S101, the robot arm 100 grips the male connector 110 according to the operation registered in advance in order to grip the male connector 110, and in step S102, the female connector 120 is moved almost up.

  At this time, the position immediately before gripping of the male connector 110 being gripped is not always constant. There may be a case where a subtle error always occurs due to a subtle operation deviation of the machine for setting the position of the male connector 110. Similarly, the female connector 120 may have some error.

  Therefore, in step S103, an image in which both the male connector 110 and the female connector 120 are captured in the image captured by the imaging unit 201 of the monocular camera 102 attached to the robot arm 100 as shown in FIG. 5 is acquired. Is important. Since the position of the monocular camera 102 with respect to the robot arm 100 is always fixed, the relative position information of the male connector 110 and the female connector 120 is reflected in this image.

  In step S104, control amounts (ΔX, ΔY, ΔZ, ΔAx, ΔAy, ΔAz) are calculated by the control parameter generation unit 202 having a neural network as shown in FIG. . However, depending on whether or not learning is possible, the control amount output by the control parameter generation unit 202 may not operate to the insertion start position. In that case, by repeating the loop of steps S103 to S106 a plurality of times, the control parameter generation unit 202 repeatedly calculates so as to be equal to or less than the threshold value shown in step S105, and the control unit 203 and the drive unit 204 control to control the robot arm 100. In some cases, the position is controlled.

  The threshold value shown in S105 is determined by the required accuracy of the male connector 110 and the female connector 120 to be fitted. For example, when the fitting with the connector is loose and the accuracy of the connector characteristic is not necessary so much, the threshold value can be set large. In the opposite case, the threshold value is set small. In general, in the case of a manufacturing process, an error that can be tolerated in production is often defined, and this value can be used.

  Also, depending on whether or not learning is possible, assuming that the control amount output from the control parameter generation unit 202 cannot operate to the insertion start position, a plurality of insertion start positions may be set. If the insertion start position is set without taking a sufficient distance between the male connector 110 and the female connector 120, the male connector 110 and the female connector 120 come into contact with each other before the insertion starts, and either one is damaged. There is also the risk of doing so. In this case, for example, the clearance between the male connector 110 and the female connector 120 is 5 mm first, 20 mm next, and 10 mm next, depending on the number of loops between step S103 to step S106 in FIG. An insertion start position may be set.

In the present embodiment, the description has been given using the connector, but the application of this technique is not limited to the fitting of the connector. For example, the present invention can be applied to the case where an IC is mounted on a substrate. In particular, when a capacitor having a large dimensional error is inserted into a hole in the substrate, the same method can be used.
Further, the present invention is not necessarily limited to the insertion into the substrate, and can be used for general position control for obtaining the control amount from the relationship between the image and the control amount. In the present invention, there is a merit that each individual difference at the time of aligning an object and an object can be absorbed by learning a relationship between an image and a control amount using a neural network.

  Therefore, in the first embodiment, the image capturing unit 201 that captures an image in which two objects are present, and information on the captured two object images are input to the input layer of the neural network, and the positional relationship between the two objects is determined. A control parameter generation unit 202 that outputs a control amount of a position for control as an output layer of the neural network, and a current or voltage for controlling the positional relationship between the two objects using the output control amount of the position And a drive unit 204 that moves one position of the two things using a current or voltage for controlling the positional relation between the two things. Alternatively, even if there is an error in the positional relationship between the two objects, there is an effect that the alignment can be performed using only a monocular camera.

Although the embodiment using one neural network has been described this time, it is necessary to use a plurality of neural networks as necessary. This is because when the input is an image and the output is a numerical value as in this case, there is a limit to the approximation accuracy of the numerical value, and an error of several percent may occur depending on the situation. Depending on the amount from the position near the insertion start position in step 2 in FIG. 4 to the insertion start position, the determination in step S105 may always be No and the operation may not be completed. In such a case, a plurality of networks are used as shown in FIG.
FIG. 7 is a flowchart using a plurality of networks in the neural network according to the first embodiment. The detailed step of FIG.4 S104 is shown. The plurality of parameters are included in the control parameter generation unit of FIG.

In step S701, the control parameter generation unit 202 selects which network to use based on the input image.
When the number of loops is the first time or the obtained control amount is 25 mm or more, the neural network 1 is selected and the process proceeds to step S702. If the control amount obtained after the second loop is 5 mm or more and less than 25 mm, the neural network 2 is selected and the process proceeds to step S703. Further, when the control amount obtained after the second loop is less than 5 mm, the neural network 3 is selected and the process proceeds to step S704. A control amount is calculated using the neural network selected in steps S702 to S704.
For example, each neural network has been learned according to the distance or control amount between the male connector 110 and the female connector 120, and the neural network 3 in the figure has learned data within an error range of ± 1 mm and ± 1 degree. The neural network 2 changes the range of data to be learned step by step from learning data in the range of ± 1 to ± 10 mm and ± 1 to ± 5 degrees. Here, it is more efficient not to overlap the range of images used in each neural network.
Further, although three examples are shown in FIG. 7, the number of networks is not particularly limited. When such a method is used, it is necessary to prepare the discrimination function in step S701 for determining which network is used as a “network selection switch”.
This network selection switch can also be constituted by a neural network. In this case, the input image to the input layer and the output of the output layer are network numbers. The image data uses a pair of images and network numbers used in all networks.

Although an example using a plurality of neural networks has been described using a connector, application of this technique is not limited to connector fitting. For example, the present invention can be applied to the case where an IC is mounted on a substrate. In particular, when a capacitor having a large dimensional error is inserted into a hole in the substrate, the same method can be used.
Further, the example using a plurality of neural networks is not necessarily limited to the insertion into the board, but can be used for the entire position control for obtaining the control amount from the relationship between the image and the control amount. In the present invention, by learning the relationship between the image and the control amount using a neural network, there is a merit that each individual difference when aligning a thing and a thing can be absorbed, and more accurately. Control amount can be calculated.

  Therefore, the image capturing unit 201 that captures an image in which two things are present, and information on the captured two thing images are input to the input layer of the neural network, and position information for controlling the positional relationship between the two things is input. A control parameter generation unit 202 that outputs a control amount as an output layer of a neural network, a control unit 203 that controls a current or voltage for controlling the positional relationship between two things using the output control amount of the position, A drive unit 204 is provided for moving one position of the two thing positional relationships using a current or voltage for controlling the positional relationship between the two things, and the control parameter generating unit 202 is one of a plurality of neural networks. Because it is configured to select, even if there is an individual difference of individual items or an error in the positional relationship between two items, it is more There is an effect that can be performed well every time.

Embodiment 2. FIG.
In the first embodiment, the male connector 110 is gripped by the grip portion 101 of the robot arm 100 with respect to the male connector 110 and the female connector 120 in a fitted state whose positions are known. And while moving the holding | grip part 101 to the known pulling direction, it moved to the insertion start position, and the multiple image was acquired with the monocular camera 102. FIG. In the second embodiment, a case where the fitting position of the male connector 110 and the female connector 120 is unknown will be described.

A technique called reinforcement learning has been studied as a prior study of techniques for robots to learn themselves and acquire appropriate behavior. In this method, the robot performs various actions on a trial and error basis, optimizing the behavior as a result while memorizing the behavior with good results, but a large number of trials are required to optimize the behavior. It is said.
As a technique for reducing the number of trials, a framework called “on policy” is generally used in reinforcement learning. However, it is difficult to apply this framework to teaching of a robot arm because it is necessary to devise various measures specialized for the robot arm and control signals, and it has not been put into practical use.
In the second embodiment, the robot as in the first embodiment performs various operations on a trial and error basis, and memorizes the actions that have given good results while reducing the number of trials for optimizing the actions as a result. The form which can be done is demonstrated.

The system configuration will be described. Parts that are not particularly described are the same as those in the first embodiment.
Although the overall hardware configuration is the same as that of FIG. 1 of the first embodiment, a force sensor 801 (not shown in FIG. 1) for measuring the load applied to the gripping unit 101 is added to the robot arm 100. Is different.

FIG. 8 is a functional configuration diagram of the position control device according to the second embodiment. The difference from FIG. 2 is that a force sensor 801 and a route determination unit 802 are added, and the route determination unit 802 includes a critical unit 803, an actor unit 804, an evaluation unit 805, and a route setting unit 806. Yes.
FIG. 9 is a hardware configuration diagram of the position control device according to the second embodiment. The only difference from FIG. 3 is that the force sensor 801 is electrically or communicably connected to the input / output interface 301. In addition, the input / output interface 301, the processor 302, and the memory 303 constitute the function of the control parameter generation unit 202 in FIG. 8 and also the function of the route determination unit 802. Therefore, the force sensor 801, the monocular camera 201, and the input / output interface 301, and the input / output interface 301 and the control circuit 304 may be configured separately.

Next, details of FIG. 8 will be described.
The force sensor 801 measures a load applied to the grip portion 101 of the robot arm 100, and can measure a force value when the male connector 110 and the female connector 120 in FIG. It is.
Critic part 803 and Actor part 804 are the same as Critic part and Actor part, where S3 and S4 are the conventional reinforcement learning.
Here, a conventional reinforcement learning method will be described. In this embodiment, a model called Actor-Critic model is used in reinforcement learning (reference: reinforcement learning: RSSutton and AGBarto published in December 2000). The Actor unit 804 and the Critic unit 803 acquire the state of the environment through the imaging unit 201 and the force sensor 801. The Actor unit 804 is a function that receives the environmental state I acquired using the sensor device and outputs a control amount A to the robot controller. The Critic unit 803 is a mechanism for the Actor unit 804 to appropriately learn the output A with respect to the input I so that the fitting to the Actor unit 804 is appropriately successful.
Hereinafter, the conventional reinforcement learning method will be described.

  In reinforcement learning, an amount called a reward R is defined, and the Actor unit 804 can acquire an action A that maximizes the R. As an example, assuming that the learning operation is the fitting between the male connector 110 and the female connector 120 as shown in the first embodiment, R = 1 when the fitting is successful, R = 0 when the fitting is successful, and so on. Defined. The action A indicates a movement correction amount from the current position (X, Y, Z, Ax, Ay, Az) this time, and A = (ΔX, ΔY, ΔZ, ΔAx, ΔAy, ΔAz). Here, X, Y, and Z indicate position coordinates with the center of the robot as the origin, and Ax, Ay, and Az indicate rotation amounts about the X, Y, and Z axes, respectively. The movement correction amount is a control amount from the fitting start position for first trying the fitting of the male connector 110 from the current point. The observation of the environmental state, that is, the trial result is obtained from the image from the imaging unit 201 and the value of the force sensor 801.

In reinforcement learning, a function called a state value function V (I) is learned by the critical part 803. Here, at time t = 1 (for example, at the start of a fitting trial), action A (1) is taken in state I (1), and time t = 2 (for example, the second time after the end of the first fitting trial). It is assumed that the environment changes to I (2) at the time of (before the start of fitting) and a reward amount R (2) (first fitting trial result) is obtained. Various update formulas can be considered, but the following is given as an example.
The update formula for V (I) is defined below.

ここで、δは予測誤差、αは学習係数であり0 〜 1までの正の実数、γは割引率であり0〜 1までの正の実数である。
Actor部８０４は入力をI、出力をA(I)とし以下の通り、A(I)が更新される。
δ>0の時

Here, δ is a prediction error, α is a learning coefficient and is a positive real number from 0 to 1, and γ is a discount rate and is a positive real number from 0 to 1.
In the Actor unit 804, the input is I and the output is A (I), and A (I) is updated as follows.
When δ> 0

δ≦0の時

When δ ≦ 0

ここで、σは出力の標準偏差の値を示し、Actorは状態Iにおいて、A(I)に平均0、分散をσ²とした分布を持つ乱数を加算する。すなわち、試行の結果いかんにかかわらず、ランダムに２回目の移動補正量が決定されるようなものである。
なお、上記の更新式を一例として用いているが、Actor-Criticモデルも様々な更新式があり、上記にとらわれず一般的に使用されているモデルであれば変更が可能である。

Here, σ represents the value of the standard deviation of the output, and in the state I, the Actor adds a random number having a distribution with an average of 0 and a variance of σ ² to A (I). That is, the second movement correction amount is determined randomly regardless of the result of the trial.
Although the above update formula is used as an example, the Actor-Critic model also has various update formulas and can be changed as long as it is a commonly used model without being limited to the above.

However, the Actor unit 804 learns an appropriate action in each state in the above configuration, but moves as in Embodiment 1 when learning is completed. During learning, the recommended action at the time of learning is calculated and passed from the route setting unit 806. Therefore, at the time of learning, the control unit 203 receives the movement signal from the route setting unit 806 as it is, and the control unit 203 drives the driving unit 204. Will be controlled.
In other words, in the conventional model of Actor-Critic, R = 1 is defined when the mating is successful, and R = 0 when the mating is successful. Until the success, the movement correction amount used for the trial is randomly given, and therefore the movement correction amount for the next trial according to the degree of failure of the trial is not determined. This is the same result because not only the conventional model of Actor-Critic but also other reinforcement learning models such as Q-Learning, only the success and failure of the fitting are evaluated. In the present embodiment of the present invention, a process for evaluating the degree of failure and determining a movement correction amount for the next trial will be described.

The evaluation unit 805 generates a function for performing evaluation at each fitting trial.
FIG. 10 is a diagram showing a state of trial of fitting between the male connector 110 and the female connector 120 in the second embodiment.
For example, assume that an image as shown in FIG. 10A is obtained as a result of the trial. This attempt failed because the fitting position of the connector is greatly displaced. At this time, the degree of success is measured and digitized to obtain an evaluation value indicating the degree of success. As a numerical method, for example, as shown in FIG. 10B, there is a method of calculating the connector surface area (number of pixels) on the insertion destination side in the image. In this method, when the insertion failure of the male connector 110 and the female connector 120 is detected by the force sensor 801 of the robot arm 100, only the surface of the female connector 120 fitting surface is coated with a color different from the other backgrounds. Or by sticking a sticker, the data acquisition and calculation from an image become easier. Further, although the method described so far is a case where the number of cameras is one, it is also possible to shoot a plurality of cameras side by side and combine the results using the captured images. In addition to the connector surface area, the same can be evaluated by obtaining the number of pixels in the two-dimensional direction (for example, the X and Y directions).

The route setting unit 806 is divided into two steps as processing.
In the first step, the evaluation result processed by the evaluation unit 805 and the movement that the robot has moved to are learned. When the robot movement correction amount is A and the evaluation value indicating the degree of success processed by the evaluation unit 805 is E, the path setting unit 806 prepares and approximates a function having A as an input and E as an output. . An example of a function is the RBF (Radial Basis Function) network. RBF is known as a function that can easily approximate various unknown functions.
For example, the kth input

に対して出力f(x)は、以下のように定義される。

The output f (x) is defined as follows.

ここで、σは標準偏差、μはRBFの中心を意味する。

Here, σ means standard deviation, and μ means the center of RBF.

The data learned by RBF is not a single data, but all data from the start of trial to the latest data. For example, currently, N data are prepared for the N-th trial. It is necessary to determine the above W = (w_1,... W_J) by learning, and various methods can be considered for the determination, but the following RBF complementation is given as an example.

とした時に

When

にて、学習が完了する。

The learning is completed.

After the approximation by RBF interpolation, the minimum value is obtained by the RBF network by a general optimization method such as steepest descent method or PSO (Particle Swam Optimization). This minimum value is input to the next Actor unit 804 as the next recommended value.
In short, the above example will be explained in detail. The surface area and the number of pixels in the two-dimensional direction with respect to the movement correction amount at the time of failure are evaluated as the evaluation values, and the optimal solution is obtained using the values of the arrangement in time series. Is. More simply, the movement correction amount that is moved at a constant rate in the direction of decreasing the number of pixels in the two-dimensional direction may be obtained.

Next, an operation flow is shown in FIG.
FIG. 11 is a flowchart in route learning of the position control device according to the second embodiment.
First, in step S <b> 1101, the grip portion 101 of the robot arm 100 grips the male connector 110. The position and orientation of the male connector 110 are registered in advance on the control unit 203 side in FIG. 8 and operated based on a control program registered in advance on the control unit 203 side.

  Next, in step S1102, the robot arm 100 is brought close to the vicinity of the insertion position of the female connector 120. The approximate position and orientation of the female connector 110 are registered in advance on the control unit 203 side in FIG. 8, and the position of the male connector 110 is determined based on the control program registered in advance on the control unit 203 side. Be operated. The steps so far are the same as steps S101 to S102 in the flowchart of FIG. 4 in the first embodiment.

  Next, in step S1103, the route determination unit 802 instructs the imaging unit 201 of the monocular camera 102 to capture an image, and the monocular camera 102 includes the male connector 110 held by the holding unit 101, An image in which both the female connector 120 as the insertion destination is shown is captured. Further, the route determination unit 802 instructs the control unit 203 and the monocular camera 102 to capture an image near the current position, and is moved by the drive unit 204 based on a plurality of movement values instructed to the control unit 203. In this position, the monocular camera captures an image in which both the male connector 110 and the female connector 120 as the insertion destination are shown.

Next, in step S1104, the Actor unit 804 of the route determination unit 802 gives a control amount for performing the fitting to the control unit 203 and moves the robot arm 100 by the driving unit 204, and the male connector 110, An attempt is made to fit the female connector 120 as the insertion destination.
Next, in step S1105, when the connectors contact each other while the robot arm 100 is being moved by the drive unit 204, the route determination unit displays the value of the force sensor 801 and the image from the monocular camera 102 for each unit amount of movement. The evaluation unit 805 and the critical unit 803 of 802 memorize.

In step S1106, the evaluation unit 805 and the critical unit 803 confirm whether the fitting has been successful.
Usually, the mating is not successful at this point. Therefore, in step S1108, the evaluation unit 805 evaluates the success degree by the method described with reference to FIG. 10 and gives an evaluation value indicating the success degree with respect to the alignment to the route setting unit 806.
In step S1109, the route setting unit 806 performs learning using the method described above, and the route setting unit 806 gives the next recommended value to the Actor unit 804, and the Critic unit 803 obtains the amount according to the reward amount. The Actor unit 804 receives the received value. In step S1110, the Actor unit 804 adds the value calculated according to the reward amount output from the Critic unit 803 and the next recommended value output from the route setting unit 806 to determine the movement correction amount. In addition, in this step, when there is a sufficient effect only by using the next recommended value output by the route setting unit 806, it is not necessary to add the value obtained according to the reward amount output by the Critic unit 803. Needless to say. Further, the Actor unit 804 sets an addition ratio between the value calculated according to the reward amount output from the Critic unit 803 and the next recommended value output from the route setting unit 806 in order to determine the movement correction amount. You may change according to.

Thereafter, in step S <b> 1111, the Actor unit 804 moves the grip unit 101 of the robot arm 100 by giving the movement correction amount to the control unit 203.
Thereafter, the process returns to step 1103 again, an image is taken at the position moved by the movement correction amount, and the fitting operation is performed. Repeat until successful.
When the fitting is successful, in step S1107, after successful fitting, learning of the actor part 804 and the critical part 803 is performed for I from steps S1102 to S1106 when the fitting is successful. Finally, the route determination unit 802 gives the learned neural network data to the control parameter generation unit 202, thereby enabling the operation in the first embodiment.

  In step S1107, the Actor unit 804 and the Critic unit 803 learn about I when the fitting is successful. However, the data of all the trials from the disclosure of the fitting trial to the success is used. The unit 803 may learn. In that case, in the first embodiment, the case where a plurality of neural networks are formed according to the control amount is described, but if the position of the successful fitting is known, the control is performed using the distance to the successful fitting. It becomes possible to simultaneously form a plurality of appropriate neural networks according to the magnitude of the quantity.

The reinforcement learning module is described based on the Actor-Critic model, but other reinforcement learning models such as Q-Learning may be used.
Although the RBF network is given as the function approximation, other function approximation methods (linear, quadratic function, etc.) may be used.
As an evaluation method, the method of changing the color of the surface of the connector has been described. However, the shift amount between the connectors and the like may be used as the evaluation method by other image processing techniques.

Further, as described in the first embodiment and the present embodiment, the application of this technique is not limited to the fitting of the connector. For example, the present invention can be applied when an IC is mounted on a substrate, and even when a capacitor or the like having a large dimensional error is inserted into a hole in the substrate, the same method is effective.
Further, the present invention is not necessarily limited to the insertion into the substrate, and can be used for general position control for obtaining the control amount from the relationship between the image and the control amount. In the present invention, by learning the relationship between the image and the control amount using a neural network, there is a merit that each individual difference when aligning a thing and a thing can be absorbed, and more accurately. Control amount can be calculated.

  Therefore, in the present embodiment, when the Actor-Critic model is used to learn the control amount, the Actor unit 804 uses the value obtained by the Critic unit 803 according to the reward amount, and the route setting unit 806 uses the evaluation value. The normal Actor-Critic model requires a lot of trial and error until alignment is successful. The invention can greatly reduce the number of alignment trials.

In the present embodiment, it has been described that the number of alignment trials is reduced by evaluating an image from the imaging unit 201 when the alignment fails, but the value of the force sensor 801 at the time of the alignment trial is described. Even if is used, the number of trials can be reduced. For example, in the alignment including the fitting of the connector or the insertion of two objects, when the value of the force sensor 801 exceeds a certain threshold value when the failure occurs, the position of the two objects is completed or the insertion is completed. In general, the Actor unit 804 determines whether or not it is in a position. In that case, a. If fitting or insertion is in progress when the threshold is reached, b. Although the fitting and insertion have been completed, the value of the force sensor 801 during the fitting or insertion may show a certain value.
a. In this case, there is a method of learning both the value of the force sensor 801 and an image, and details can be implemented by using the method described in the third embodiment.
b. In this case, the method described in Embodiment 3 can be used as a method of learning only with the value of the force sensor 801. Alternatively, in the definition of reward R in the Actor-Critic model, when the maximum load applied during mating or insertion is F and A is a positive constant, R = (1 -A / F), Failure When R = 0 is defined, the same effect can be achieved.

Embodiment 3 FIG.
In the present embodiment, a method for efficiently collecting data in a learning process performed after successful alignment in the second embodiment will be described. Therefore, unless otherwise specified, it is the same as in the second embodiment. That is, FIG. 8 is a functional configuration diagram of the position control device according to the third embodiment, and FIG. 9 is a hardware configuration diagram.

  In operation, a method of collecting learning data more efficiently during the operation of step S1107 in FIG. 11 in the second embodiment will be described below.

FIG. 12 shows a flowchart in route learning of the position control device in the third embodiment.
First, in step S1201, when the male connector 110 and the female connector 120 are successfully fitted in step S1107 of FIG. 11, the path setting unit 806 sets the variables as i = 0, j = 1, k = 1, and is initialized. Turn into. The variable i is the number of subsequent learnings of the robot arm 100, the variable k is the number of learnings after the male connector 110 and the female connector 120 are disengaged, and the variable j is the number of loops in the flowchart of FIG. It is.

  Next, in step S1202, the path setting unit 806 gives the movement amount to the control unit 203 via the Actor unit 804 so as to return 1 mm from the movement amount given for fitting in step S1104 in FIG. Then, the robot arm 100 is moved by the drive unit 204. Then, 1 is added to the variable i. Here, an instruction to return 1 mm from the movement amount is given, but it is not necessarily limited to 1 mm, and may be a unit amount such as 0.5 mm or 2 mm.

Next, in step S1203, the route setting unit 806 stores the coordinates at that time as O (i) (at this time i = 1).
In step S1204, the path setting unit 806 determines control amounts (ΔX, ΔY, ΔZ, ΔAx, ΔAy, ΔAz) at random with O (i) as the center, and controls the control unit 203 via the Actor unit 804. The robot arm 100 is moved by the driving unit 204. At this time, the maximum amount of the control amount can be arbitrarily set as long as it can be moved.

  Next, in step S1205, at the position after the movement in step S1204, the Actor unit 804 collects the values of the force sensor 801 corresponding to the movement amounts (ΔX, ΔY, ΔZ, ΔAx, ΔAy, ΔAz). In S1206, the critical part 803 and the actor part 804 multiply the movement amount by −1 (−ΔX, −ΔY, −ΔZ, −ΔAx, −ΔAy, −ΔAz) and hold the male connector 110. The sensor value of the force sensor 801 that measures force is recorded as learning data.

In step S1207, the path setting unit 806 determines whether the collected data number has reached the specified number J. If the number of data is insufficient, 1 is added to the variable j in step S1208, and the process returns to step S1204. The control amount (ΔX, ΔY, ΔZ, ΔAx, ΔAy, ΔAz) is changed by a random number to obtain data, and the specified number J S1204 to S1207 are repeated until the number of data is accumulated.
When the prescribed number of data is accumulated, in step S1209, the path setting unit 806 sets the variable j to 1, and then confirms whether the male connector 110 and the female connector 120 are disengaged in step S1210. To do.

If not, the process returns to step S1202 via step S1211.
In step S1211, the path setting unit 806 gives a control amount to the control unit 203 via the Actor unit 804 so as to return the coordinates of the robot arm 100 to the coordinates O (i) before giving the control amount, and drives the driving unit 204. To move the robot arm 100.
Thereafter, the loop from step S1202 to step S1210 is returned to the process of returning 1 mm or a unit amount from the given control amount until the male connector 110 and the female connector 120 are disengaged. The process of collecting the data of the force sensor 801 by giving a control amount around the position is repeated. If the male connector 110 and the female connector 120 are disengaged, the process proceeds to step S1212.

  In step S1212, the path setting unit 806 sets the variable i to I (I is an integer larger than the value of i when it is determined that the male connector 110 and the female connector 120 are disengaged). Then, a control amount is given to the control unit 203 via the Actor unit 804 so as to return, for example, 10 mm (this may be another value) from the movement amount given for the fitting, and the robot arm 100 is moved by the driving unit 204. Move.

Next, in step S1213, the route setting unit 806 stores the coordinate position of the robot arm 100 moved in step S1212 as the center position O (i + k).
Next, in step S1214, the route setting unit 806 determines control amounts (ΔX, ΔY, ΔZ, ΔAx, ΔAy, ΔAz) again at random with the center position O (i + k) as the center. A control amount is given to the control unit 203 via the actor unit 804, and the robot arm 100 is moved by the drive unit 204.

In step S1215, the Critic unit 803 and the Actor unit 804 are images captured by the imaging unit 201 of the monocular camera 102 at the position of the robot arm 100 after being moved by the control amounts (ΔX, ΔY, ΔZ, ΔAx, ΔAy, ΔAz). To get.
In step S1216, the Critic part 803 and the Actor part 804 record the image as one learning data by multiplying the movement amount by -1 (-ΔX, -ΔY, -ΔZ, -ΔAx, -ΔAy, -ΔAz). .

In step S1217, the path setting unit 806 determines whether the collected data number has reached the specified number J. If the number of data is not sufficient, 1 is added to the variable j in step S1212, and the process returns to step S1214. The control amount (ΔX, ΔY, ΔZ, ΔAx, ΔAy, ΔAz) is changed by a random number to obtain data, and the specified number J S1214 to S1217 are repeated until individual data are accumulated.
In addition, the maximum value of the control amount (ΔX, ΔY, ΔZ, ΔAx, ΔAy, ΔAz) in S1204 and the random value of the control amount in S1204 can take different values.
The learning data acquired by the above method performs learning of the Actor unit 804 and the Critic unit 803.

FIG. 13 is a diagram illustrating an example of a neural network and a learning rule of the neural network according to the third embodiment.
In the first and second embodiments, the learning method using the data of the force sensor 801 is not described. In the first and second embodiments, the input layer is only an image, whereas in the third embodiment, the value of the force sensor 801 may be input to the input layer instead of the image. The force sensor 801 may have three values (force and moment in two directions) or six values (three directions and moment in three directions). Control amounts (ΔX, ΔY, ΔZ, ΔAx, ΔAy, ΔAz) are output from the output layer. When the male connector 110 and the female connector 120 are not fitted, the image and the value of the force sensor 801 are simultaneously input to the input layer.
In the learning process of the neural network, the output value of the output layer obtained through the intermediate layer from the input image and the value of the force sensor 801 approximates the control amount stored as a set with the value of the image and force sensor 801. In order to achieve this, the parameters of the intermediate layer are optimized and learned.
Finally, the route determination unit 802 gives the learned neural network data to the control parameter generation unit 202, thereby enabling the operation in the first embodiment.

In this embodiment, the robot arm 100 is moved slightly to the periphery while learning from the movement for fitting the male connector 110 and the female connector 120 little by little. The description has been made on the assumption that sufficient learning is not possible depending on the pixel amount of the image of the monocular camera 102 until the match is lost.
However, if the image of the monocular camera 102 is sufficiently high-definition and can be sufficiently learned even if the robot arm 100 is moved slightly to the periphery, it may be learned only with the image of the monocular camera 102, Even when the male connector 110 and the female connector 120 are fitted, both the image of the monocular camera 102 and the value of the force sensor 801 may be used.

  Furthermore, in the first and second embodiments, cases where a plurality of neural networks are used are described. Also in the present embodiment, for example, a state in which the male connector 110 and the female connector 120 are fitted and a case in which the male connector 110 and the female connector 120 are not fitted is used as a neural network. You may distinguish. As described above, when the male connector 110 and the female connector 120 are mated, only the force sensor 801 is formed as an input layer, and when it is out of the mating, the input layer is formed only by an image. However, it is possible to perform learning with higher accuracy, and even when learning is performed only with an image, by distinguishing between cases where the fitting is not performed, it is possible to perform learning with high accuracy because the configuration of the image is different.

As described in the first and second embodiments, even in the present embodiment, the application of this technique is not limited to the fitting of the connector. For example, the present invention can be applied when an IC is mounted on a substrate, and even when a capacitor having a large dimensional error is inserted into a hole in the substrate, the same method can be used.
Further, the present invention is not necessarily limited to the insertion into the substrate, and can be used for general position control for obtaining the control amount from the relationship between the image and the control amount. In the present invention, by learning the relationship between the image and the control amount using a neural network, there is a merit that each individual difference when aligning a thing and a thing can be absorbed, and more accurately. Control amount can be calculated.

  Therefore, in the present embodiment, in the case of including alignment with insertion for two objects, in order to learn the control amount, when extracting from the insertion state, it is moved on the path from the insertion state and its periphery. The route setting unit 806 for instructing the control amount, the output layer of the moved position, and the moved position and the value of the force sensor 801 to learn the value of the force sensor 801 of the moved position as the input layer Since the Actor unit 804 to be acquired is provided, learning data can be efficiently collected.

Embodiment 4 FIG.
In the present embodiment, a method for performing safe control in the learning process (particularly in the initial learning stage) in the second embodiment will be described. The hardware configuration diagram of the position control device in the fourth embodiment is the same as that in the second embodiment shown in FIG.

  FIG. 14 is a functional configuration diagram of the position control device according to the fourth embodiment. The difference from FIG. 8 is that a control parameter adjustment unit 1401 is added, and the control parameter adjustment unit 1401 includes a trajectory generation unit 1402, a coordinate conversion unit 1403, a gravity correction unit 1404, a compliant motion control unit 1405, a synthesis. Part 1406.

  The configuration of the route determination unit 802 is the same as that of the second embodiment, and an Actor-Critic model is described as a reinforcement learning module, and a configuration including an evaluation unit 805 and a route setting unit 806 is described as a module for evaluating the degree of success. Alternatively, other reinforcement learning models such as DDPG may be used. Further, from the point of the contents described in the present embodiment, even if the evaluation unit 805 or the route setting unit 806 is not provided, even if the function for learning is different from the function for controlling the position of the robot, it is excessive to the thing. Learning data can be collected while preventing a heavy load.

Next, details of FIG. 14 will be described.
In the second embodiment, the control amount generated by the control parameter generation unit 202 is output to the control unit 203 to determine and control the current / voltage value for each device constituting the drive unit 204. In particular, in the initial learning process, the control amount becomes inappropriate, and the drive unit 204 may stop in error, or the surrounding environment such as the robot arm, the male connector 110 and the female connector 120 may be damaged. Moreover, even if the strength of the male connector 110 and the female connector 120 is weaker than expected and the control amount is set sufficiently small in the learning process, the surrounding environment such as the male connector 110 and the female connector 120 is damaged. there is a possibility. This is a factor that may occur because the control amount setting side and the control side based on the control amount are independent. Here, in the present embodiment, a mechanism is introduced that does not overload the surrounding environment even during the learning process.

The trajectory generation unit 1402 that performs trajectory generation acquires the control amount (ΔX, ΔY, ΔZ, ΔAx, ΔAy, ΔAz) generated by the control parameter generation unit 202 as a target position, and adjusts the speed and acceleration to be smooth. The cycle control amount (ΔX ′, ΔY ′, ΔZ ′, ΔAx ′, ΔAy ′, ΔAz ′) is output in accordance with the control cycle of the robot arm 100, that is, the control cycle of the control unit 203. Control amounts (ΔX, ΔY, ΔZ, ΔAx, ΔAy, ΔAz) as target positions are defined as control amounts that reach a plurality of cycles as control cycles, whereas periodic control amounts (ΔX ′, ΔY ′) , ΔZ ′, ΔAx ′, ΔAy ′, ΔAz ′) are basically control amounts that are set for each cycle in order to reach the control amount, and are control amounts that consider the load on the surrounding environment, and are assumed The force sensor 801 can also cope with the detected load. A method for adjusting the period control amount will be described later.
The force sensor 801 measures a load applied to the grip portion 101 of the robot arm 100, and can measure a force value when the male connector 110 and the female connector 120 in FIG. It is. In Embodiment 2, there is a possibility that an excessive force is applied to the surrounding environment due to the operation output in the trial at the initial stage of learning, and the surrounding environment such as the robot arm 100, the male connector 110 and the female connector 120 may be damaged. There is. Therefore, in the fourth embodiment, the compliant motion control unit 1405 is arranged at the subsequent stage of the control parameter generation unit 202 and operated according to the external force acquired by the force sensor 801, so that the robot arm 100 and the male connector 110 to prevent excessive force from being applied to the surrounding environment such as the female connector 110 and the female connector 120. Thereby, the trial required for learning can be performed safely.
The force sensor 801 may have three values (force and moment in two directions) or six values (three directions and moment in three directions). The values of the force sensor 801 in the six cases can be expressed as (Fx, Fy, Fz, Tx, Ty, Tz). However, since the coordinates are obtained as the coordinate system of the force sensor, the coordinate conversion unit 1403 determines the value of the force sensor 801 for the entire robot arm 100 when the coordinate system of the force sensor and the coordinate system of the entire robot arm 100 are different. Has a function to convert to a coordinate system.
The value measured by the force sensor 801 is affected by gravity. The gravity correction unit 1404 has a function of removing the influence of gravity from the value measured by the force sensor 801.
The compliant motion control unit 1405 acquires the value of the force sensor 801 corrected by the coordinate conversion unit 1403 and the gravity correction unit 1404. According to the physical law, a control amount adapted to the external force detected from the force sensor 801 is output. A control amount adjustment method adapted to the external force will be described later.
The combining unit 1406 combines the control amount that is the output of the trajectory generation unit 1402 and the control amount that is the output of the compliant motion control unit 1405, and outputs them to the control unit 203. The synthesis method uses addition of the control amount that is the output of the trajectory generation unit 1402 and the control amount that is the output of the compliant motion control unit 1405. Alternatively, an addition ratio may be set and weighted addition according to the addition ratio may be performed.

The trajectory generation unit 1402 calculates a periodic control amount in a control cycle unit so as not to exceed at least one of these limitations, given the control cycle of the robot arm 100, the maximum speed, the maximum acceleration, and the maximum jerk. .
For example, non-patent literature (KROGER, Torsten; PADIAL, Jose. Simple and robust visual servo control of robot arms using an on-line trajectory generator. In: Robotics and Automation (ICRA), 2012 IEEE International Conference on. IEEE, 2012. p. 4862-4869.), there is a method to calculate the periodic control amount so that all of the following conditions are satisfied.
The following constants are given constants corresponding to the specifications of the robot arm 100.

Tcycle: Cycle Vmax: Maximum speed Amax: Maximum acceleration Jmax: Maximum jerk (jerk)

ここで、ｘｉ、ｖｉ、αｉ、ｊｉは以下を表す変数である。
ｘｉ：ステップｉにおける現在位置
ｖｉ：ステップｉにおける現在速度
αｉ：ステップｉにおける現在加速度
ｊｉ：ステップｉにおける現在加加速度（ジャーク）

Here, xi, vi, αi, and ji are variables representing the following.
xi: current position at step i vi: current speed αi at step i: current acceleration at step i ji: current jerk at step i (jerk)

  A control amount adjustment method adapted to the external force in the compliant motion control unit 1405 described above will be described. In the compliant motion control, a subordinate operation is calculated from external force information given a coefficient representing the stability and rigidity of the environment. For example, the dependent motion Δx (t) when the external force is f (t) can be calculated by solving the following differential equation.

ここで、m, d, kは環境の安定性および剛性を表す係数である。
m: 重力定数
d: 抵抗
k: ばね定数

Here, m, d, and k are coefficients representing the stability and rigidity of the environment.
m: gravity constant
d: resistance
k: spring constant

Next, an operation flow is shown in FIG.
FIG. 15 is a flowchart in route learning of the position control device according to the fourth embodiment.
First, in step S1501, the grip portion 101 of the robot arm 100 grips the male connector 110. The position and orientation of the male connector 110 are registered in advance on the control unit 203 side in FIG. 14 and operated based on a control program registered in advance on the control unit 203 side.

  Next, in step S1502, the robot arm 100 is brought close to the vicinity of the insertion position of the female connector 120. The approximate position and orientation of the female connector 110 are registered in advance on the control unit 203 side in FIG. 14, and the position of the male connector 110 is determined based on the control program registered in advance on the control unit 203 side. Be operated. The steps so far are the same as steps S101 to S102 in the flowchart of FIG. 4 in the first embodiment.

Next, in step S1503, the route determination unit 802 instructs the imaging unit 201 of the monocular camera 102 to capture an image, and the monocular camera 102 includes the male connector 110 held by the holding unit 101, An image in which both the female connector 120 as the insertion destination is shown is captured. The evaluation unit 805 and the critical unit 803 of the route determination unit 802 store the image from the monocular camera 102.
In step S1504, the route determination unit 802 instructs the force sensor 801 to acquire an external force, and the force sensor 801 acquires the external force at the current position. At the same time, the evaluation unit 805 and the critical unit 803 of the route determination unit 802 store the value of the force sensor 801.

  Next, in step S <b> 1505, the Actor unit 804 of the route determination unit 802 calculates a control amount for performing fitting and supplies the calculated amount to the trajectory generation unit 1402.

  Next, in step S1506, a new control amount adjusted so that the velocity / acceleration is smoothed by the trajectory generation unit 1402 is calculated. More specifically, a periodic control amount that is a target position for each control period is calculated for xi that satisfies a given constant, Tcycle, Vmax, Amax, and Jmax corresponding to the specification of the robot arm 100 described above.

  In step S1507, the coordinate conversion unit 1403 converts the value of the force sensor 801 acquired in step S1503 into the coordinate system of the entire robot arm 100.

  Next, in step S1508, the gravity correction unit 1404 removes the influence of gravity from the value of the force sensor 801 coordinate-converted in step S1506, and gives it to the compliant motion control unit 1405.

  Next, in step S1509, the compliant motion control unit 1405 calculates a control amount adapted to the external force from the value of the force sensor 801 corrected for gravity in step S1508, and supplies the calculated control amount to the synthesis unit 1406. For the control amount adapted to the external force, for example, the value calculated above is calculated so that the value of the force sensor 801 becomes small.

  Next, in step S1510, the synthesizing unit 1406 synthesizes the periodic control amount calculated in step S1506 and the compliant motion control amount calculated in step S1509 by adding or weighting and adding the periodic control amount adjustment value. To the control unit 203.

Next, in step S1511, the robot arm 100 is moved by the drive unit 204 to try to insert a connector. The control parameter adjustment unit 1401 checks whether or not the periodic control amount adjustment value has reached the control amount generated by the control parameter generation unit 202. If not, the process returns to step S1504. Therefore, the operations from step S1504 to step S1511 can be repeated every control cycle.
Further, even when the male connector 110 and the female connector 120 contact each other before reaching the control amount generated by the control parameter generation unit 202, it is detected every control cycle that the value of the force sensor 801 increases. In addition, since feedback control is performed as the periodic control amount adjustment value, it is possible to reduce the possibility of destroying the surrounding environment even in the initial learning stage.

  Next, in step S1512, the evaluation unit 805 and the critical unit 803 confirm whether the fitting has been successful, and at the same time, based on the values of the monocular camera 102 and force sensor 801 stored in step S1503 and the control parameter values calculated in S1504. , The neural network parameters of the Actor unit 804 and the Critic unit 803 are updated.

If the fitting is not successful in step S1513, the position of the robot arm 100 is not moved and the process returns to step S1503 to perform the next trial.
Although not shown in FIG. 15 before returning to step S1503, evaluation is performed using the evaluation method used in step S1108 in FIG. 11 of the second embodiment, step S1109, and the second embodiment shown in FIG. By returning to step S1503, the same effect as in the second embodiment can be obtained.

In step S1513, if the combination is successful, the combination task itself ends. When further learning of the actor part 804 and the critical part 803 is continued, it is possible to improve the learning accuracy by returning to step S1501 and retrying from the grasping of the male connector 110.

  Although an Actor-Critic model has been described as a reinforcement learning module, other reinforcement learning models such as DDPG may be used.

Further, as described in the first embodiment and the present embodiment, the application of this technique is not limited to the fitting of the connector. For example, the present invention can be applied when an IC is mounted on a substrate, and even when a capacitor or the like having a large dimensional error is inserted into a hole in the substrate, the same method is effective.
Further, the present invention is not necessarily limited to the insertion into the substrate, and can be used for general position control for obtaining the control amount from the relationship between the image and the control amount. In the present invention, by learning the relationship between the image and the control amount using a neural network, there is a merit that each individual difference when aligning a thing and a thing can be absorbed, and more accurately. Control amount can be calculated.

  Therefore, in the present embodiment, in the case of including alignment with insertion for two objects, the control amount for insertion based on the image acquired from the imaging unit 201 and the value of the force sensor 801 is indicated and the position is set. Based on the route determination unit 802 that learns from the result of the combination, the periodic control amount that is set for each control period to reach the control amount, and the control amount that is adapted to the external force based on the value of the force sensor 801 Since the control parameter adjustment unit 1401 for outputting the periodic control amount adjustment value is provided, the robot arm 100 is operated by the control amount obtained by adding the trajectory generation control and the compliant motion control using the force sensor 801, and normal reinforcement learning is performed. The model requires trial and error until learning converges and may damage the environment. Even early in it is possible to perform a safe trial.

In the present embodiment, the description is limited to the function of the control parameter adjustment unit 1401. However, even if the function of the control parameter adjustment unit 1401 is added to the contents described in the second and third embodiments. The second and third embodiments can be operated, and the learning speed can be improved safely.

100: Robot arm,
101: gripping part,
102: Monocular camera 110: Male connector 120: Female connector 201: Imaging unit 202: Control parameter generation unit 203: Control unit 204: Drive unit 301: Input / output interface 302: Processor,
303: memory,
304: Control circuit,
305: Motor,
801: Force sensor 802: Route determination unit 803: Critic unit 804: Actor unit 805: Evaluation unit 806: Path setting unit 1401: Control parameter adjustment unit 1402: Trajectory generation unit 1403: Coordinate conversion unit 1404: Gravity correction unit 1405: Compliant motion control unit 1406: composition unit

Claims (8)

When aligning with insertion from one mono two things to the other things, a shall be acquired from the imaging unit, while said gripped by the gripper of the robotic arm of mono- and of the other and images and objects have been photographed, the result of the alignment instructs the value of the force sensor that measures the load on the gripping portion, the control amount of the position of the robot arm for insertion on the basis of, A route determination unit that learns a control amount of the position of the robot arm from which the alignment succeeds from the value of the image and the force sensor ;
In the course of the learning,
The control amount of the position of the robot arm from the path determination unit is received, and the periodic control amount set for each control period of the control unit to reach the control amount of the position of the robot arm; a control amount that is adapted to the external force based on the value of the force sensor corresponding to the control period, a control parameter adjusting section for outputting a periodic control amount adjustment value to the control unit on the basis of,
A position control device.   The periodic control amount is set for each cycle in consideration of any of maximum speed, maximum acceleration, and maximum jerk to reach the control amount, and the control amount adapted to the external force is the force sense. The position control device according to claim 1, wherein the position control device is determined according to a value obtained by excluding a gravity component from a sensor value. With a control unit for controlling the current or voltage for controlling the positional relationship of the two mono using pre SL cycle control amount adjustment value, a current or voltage for controlling the two positional relationship mono A driving unit that moves one position of the positional relationship between the two objects, and the force sensor acquires a force applied when the positional relationship between the two objects is maintained. The position control device described. The route determination unit includes: a route setting unit that instructs a movement amount to move to and around the route from the insertion state when extracting from the insertion state; an output layer of the moved position data; An Actor unit for acquiring the value of the moved position and the value of the force sensor in order to learn the value of the force sensor as an input layer;
The position control device according to claim 1, comprising: The position control apparatus according to claim 4 , wherein the actor unit performs learning from the input layer and the output layer using an Actor-Critic model. The Actor section learns a plurality of neural networks, and one of the plurality of neural networks uses the position data in which the positional relationship between the two objects is inserted for learning, and the other data The position control device according to claim 5 , wherein data of a position in which the positional relationship between two objects is not inserted is used for learning. In the Actor unit, the value of the force sensor is used for the data of the position where the positional relationship between the two objects is inserted, and the data for the position where the positional relationship of the two objects is not inserted, The position control device according to claim 6 , wherein image data is used. A method for controlling the position of two objects,
In the case of performing alignment with insertion of one of the two objects from the other object to the other object, an image of the one object and the other object gripped by the grip portion of the robot arm, and the grip outputs the value of the force sensor that measures the load on the part, the control amount of the position of the robot arm for insertion on the basis of,
In the course of learning,
The control amount of the position of the robot arm is received, the periodic control amount of the control unit that can reach the control amount of the position of the robot arm in one control cycle, and the force sensor corresponding to the one control cycle A control amount adapted to an external force based on the value , and a periodic control amount adjustment value based on the control amount is output to the control unit ,
A position control method for two things, wherein the control amount of the position of the robot arm that is successfully aligned is learned from the image and the value of the force sensor from the result of the alignment.

Images