JP2019214112A - Machine learning device, and robot system equipped with the same - Google Patents

Abstract

To provide a machine learning device capable of learning the motion of a robot that enables holding of a workpiece, on which holding by a hand part is determined to be impossible because a holding space is not secured.SOLUTION: The motion of a robot (2) equipped in a robot system (1) is controlled on the basis of a learning result of a machine learning device (5). The machine learning device (5) includes a determination part (7) and a learning part (63). The determination part (7) determines whether a workpiece that becomes a candidate for next holding by a hand part (26) is a holding impossible workpiece for which a holding space is not secured. The learning part (63) learns a displacement technique for enabling the holding impossible workpiece to be displaced to secure a holding space, and learns an action pattern of the robot (2) using the displacement technique.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a machine learning device that learns an operation of a robot that takes out a work in a piled state, and a robot system including the machine learning device.

  2. Description of the Related Art As a system for taking out a work from a container accommodating a plurality of works in a stacked state, a robot system for taking out a work by a robot having a hand unit is known (see Patent Document 1). The robot system disclosed in Patent Document 1 includes a machine learning device that learns a robot take-out operation. When the machine learning device takes out the workpiece from the container based on the teacher data that associates the movement of the robot corresponding to the three-dimensional map of the workpiece measured by the three-dimensional measuring instrument and the determination result of the success or failure of the workpiece removal. Learning the behavior of robots.

  When the operation of taking out the work from the container is repeated, the hand unit may not be able to hold the hand unit with respect to the next candidate work. For example, when the work is arranged in a state close to the inner surface of the container, or when a plurality of works are arranged in a state close to each other, a holding space for enabling the holding by the hand unit is provided. The work is not secured, and the work in such a situation cannot be held by the hand unit.

  For example, Patent Document 2 discloses a technique for enabling a hand in which a holding space is not secured to be held by a hand unit. In the technique disclosed in Patent Literature 2, the work in the container is disturbed by the hand unit. However, since the work in the container is randomly disturbed by the hand, the holding space may not be sufficiently secured and the holding by the hand may not be possible.

JP 2017-64910 A JP 2011-115930 A

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a robot capable of holding a workpiece that cannot be held by a hand unit without holding space. An object of the present invention is to provide a machine learning device capable of learning an operation and a robot system including the same.

  A machine learning device according to one aspect of the present invention is a device for learning an operation of a robot including a hand unit that removes a plurality of works by holding the works from a container that accommodates the works in a stacked state. This machine learning device, before or when the hand unit holds one work in the container, recognizes the accommodation state of each work in the container, and the next holding candidate by the hand unit A determination unit that determines whether a work to be held is a non-holdable work in which a holding space for enabling the holding by the hand unit is not secured around, and the non-holdable work so that the holding space is ensured. A learning unit that learns a displacement method capable of displacing the robot, and learns an action pattern of the robot using the displacement method; and an action pattern of the robot that is based on a learning result of the learning unit. And an action determining unit that determines the work as an action pattern for enabling the work to be held by the hand unit.

  According to this machine learning device, when the determination unit determines that the work to be held next by the hand unit is a work that cannot be held, the learning unit determines the work that cannot be held so that a holding space is secured. A displacement method that can be displaced is learned, and a behavior pattern of the robot using the displacement method is learned. Thus, the learning unit can learn the behavior pattern of the robot using a predetermined displacement method that enables the holding of the work that cannot be held by the hand unit. Then, the action determining unit determines an action pattern of the robot based on a learning result of the learning unit as an action pattern for enabling the hand unit to hold the work that cannot be held. When the robot operates in accordance with this behavior pattern, a holding space for enabling the holding by the hand unit is secured around the work that cannot be held by the hand unit. It becomes possible. Therefore, it is possible to avoid stopping the operation of the robot due to the existence of the non-holdable work as much as possible, and it is possible to continue the operation of taking out the work from the container by the hand unit.

  In the machine learning device described above, the displacement method includes a plurality of methods different in a method of displacing the non-holdable work, and the learning unit learns an action pattern of the robot in which the plurality of methods are combined. It may be.

  Further, in the machine learning device described above, the displacement method is a method of displacing the non-retainable work by moving the hand unit in a state where the held one work is in contact with the non-retainable work. The behavior element that defines the behavior pattern of the robot that the learning unit learns is an element that determines a contact position of the one work with the non-holdable work and a movement trajectory of the hand unit. A configuration including elements may be included.

  In the machine learning device described above, the displacement method is a method of displacing the work that cannot be held by moving the hand unit while holding the container, and the learning unit learns the robot. The action element that defines the action pattern includes an element that determines a holding position where the hand unit holds the container, an element that determines a movement locus of the hand unit, and an element that determines a moving speed of the hand unit. May be included.

  A robot system according to another aspect of the present invention includes a robot having a hand unit that removes a plurality of works by holding the works from a container that accommodates the works in a bulk state, and learns the operation of the robot. The apparatus includes the machine learning device described above, and a control device that controls an operation of the robot based on a learning result of the machine learning device.

  According to this robot system, there is provided the above-mentioned machine learning device capable of learning an action pattern of a robot which enables the holding of a workpiece which cannot be held by the hand unit. Therefore, the robot can be prevented from being stopped due to the existence of the non-holdable work as much as possible, and can continue the operation of removing the work from the container by the hand unit.

  As described above, according to the present invention, there is provided a machine learning device capable of learning an operation of a robot capable of holding a work that cannot be held by a hand unit because a holding space is not secured, and a machine learning device including the machine learning device. Robot system can be provided.

It is a block diagram which shows the structure of the robot system which concerns on one Embodiment of this invention. It is a figure which shows an example of the robot with which a robot system is equipped. It is a figure for demonstrating operation | movement of the state observation part of the machine learning apparatus with which a robot system is equipped. It is a figure for demonstrating operation | movement of the action observation part of a machine learning apparatus. It is a figure for demonstrating the action element which prescribes | regulates the action pattern of a robot. It is a figure for explaining a displacement method for displacing a work which cannot be held. It is a figure for explaining the 1st example of the displacement operation which displaces the work which cannot be held. It is a figure for demonstrating operation | movement of the displacement amount observation part of a machine learning apparatus. It is a figure for explaining learning result information generated by a learning part in the displacement operation of the 1st example. It is a flowchart which shows operation | movement of the machine learning apparatus regarding the displacement operation | movement of a 1st example. It is a figure for explaining the modification of the action pattern of the robot in the displacement operation of the 1st example. It is a figure for explaining the 2nd example of the displacement operation which displaces the work which cannot be held. It is a figure for explaining learning result information generated by a learning part in the displacement operation of the 2nd example. It is a flowchart which shows operation | movement of the machine learning apparatus regarding the displacement operation | movement of the 2nd example. It is a figure for explaining learning result information generated by a learning part in the displacement operation of the 3rd example. It is a flowchart which shows operation | movement of the machine learning apparatus regarding the displacement operation | movement of the 3rd example.

[Entire configuration of robot system]
FIG. 1 is a block diagram showing a configuration of a robot system 1 according to an embodiment of the present invention. The robot system 1 includes a robot 2, an imaging device 3, a control device 4, and a machine learning device 5. In the robot system 1, the machine learning device 5 learns the operation of the robot 2 based on the image data output from the imaging device 3, and the control device 4 controls the operation of the robot 2 based on the learning result.

  First, the robot 2 will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of the robot 2 provided in the robot system 1. The robot 2 is a robot for taking out the workpiece W from a container CN that accommodates a plurality of workpieces W in a stacked state. The container CN is formed in a bottomed cylindrical shape whose upper side is open. The robot 2 takes out the workpiece W through the opening on the upper side of the container CN.

  The robot 2 is not particularly limited as long as it has a hand unit capable of taking out the workpiece W from the container CN. For example, the robot 2 is a vertical articulated robot, a horizontal articulated robot, or a double-armed robot. An articulated robot can be employed. Hereinafter, the configuration of the robot 2 will be described using the 6-axis vertical articulated robot shown in FIG. 2 as an example. Note that the number of axes in the vertical articulated robot is not limited to six, and may be any number of axes. The robot 2 includes a base part 21, a body part 22, a first arm 23, a second arm 24, a wrist part 25, and a hand part 26.

  The base 21 is a box fixedly installed on a floor, a table, or the like, and accommodating a drive motor (not shown) therein. The body portion 22 is disposed on the upper surface of the base portion 21 so as to be rotatable in both forward and reverse directions around the first axis 2A extending in the vertical direction (vertical direction). The first arm 23 is an arm member having a predetermined length, and one end portion in the longitudinal direction thereof is attached to the trunk portion 22 via a second shaft 2B extending in the horizontal direction. The first arm 23 can rotate in both forward and reverse directions around the second axis 2B.

  The second arm 24 includes an arm base 241 and an arm part 242. The arm base 241 is a base portion of the second arm 24, and is attached to the other end portion in the longitudinal direction of the first arm 23 via a third shaft 2C extending parallel to the second shaft 2B and extending in the horizontal direction. Yes. The arm base 241 can rotate in both forward and reverse directions around the third axis 2C. The arm portion 242 is an arm member having a predetermined length, and one end portion in the longitudinal direction thereof is attached to the arm base 241 via a fourth shaft 2D perpendicular to the third shaft 2C. The arm portion 242 can rotate in both forward and reverse directions around the fourth axis 2D.

  The wrist 25 is attached to the other end in the longitudinal direction of the arm 242 via a fifth shaft 2E extending in a horizontal direction in parallel with the second shaft 2B and the third shaft 2C. The wrist portion 25 can rotate in both forward and reverse directions around the fifth axis 2E.

  The hand portion 26 is a portion for taking out the work W from the container CN in the robot 2 and is attached to the wrist portion 25 via a sixth axis 2F perpendicular to the fifth axis 2E. The hand part 26 can rotate in both forward and reverse directions around the sixth axis 2F. The hand part 26 is not particularly limited as long as it can hold the workpiece W in the container CN. For example, the hand part 26 may have a structure including a plurality of claw parts that hold and hold the workpiece W. And the structure provided with the electromagnet or negative-pressure generator which generate | occur | produces an attractive force with respect to the workpiece | work W may be sufficient. In the present embodiment, the hand unit 26 has a structure including a plurality of claws 261, and takes out the work W in the container CN by holding (gripping) the work W in the container CN.

  Next, the imaging device 3 images the entire inside of the container CN from above so that all of the plurality of works W accommodated in the container CN fall within the field of view, and outputs image data including position information of the work W. It is a device to do. In the present embodiment, the imaging device 3 is a three-dimensional measuring instrument such as a three-dimensional visual sensor including a camera 31 and an image processing unit 32 as shown in FIG. The camera 31 captures an image of the entire inside of the container CN from above, and acquires an image including each image region of the plurality of works W accommodated in the container CN. The image processing unit 32 generates image data including three-dimensional position information of each workpiece W by performing image processing on the image acquired by the camera 31. The three-dimensional position information of each workpiece is represented by, for example, coordinate values (X, Y, Z) using an XYZ orthogonal coordinate system. The XYZ orthogonal coordinate system is a coordinate system in which coordinate axes are arranged such that a plane including the X axis and the Y axis (XY plane) is horizontal and the Z axis is perpendicular to the XY plane. The image data output from the imaging device 3 is input to a displacement observation unit 64 and a determination unit 7 provided in the machine learning device 5 described later.

  Next, the control device 4 controls the operation of the imaging device 3 while controlling the operation of the robot 2. The control device 4 controls the operation of the robot 2 based on information generated by an action determining unit 9 provided in the machine learning device 5 described later.

[Configuration of machine learning device]
Next, the machine learning device 5 will be described. As illustrated in FIG. 1, the machine learning device 5 includes a learning processing unit 6 that performs a learning process of learning the operation of the robot 2 (machine learning), a determination unit 7, a storage unit 8, an action determination unit 9, Is provided. The learning method executed by the machine learning device 5 is not particularly limited, and for example, “supervised learning”, “unsupervised learning”, “reinforcement learning”, and the like can be employed. In the present embodiment, a Q learning method as reinforcement learning is employed as a learning method in the machine learning device 5. Q-learning is a method of learning a high-value behavior that rewards the behavior of the robot 2 when the continuous motion of the robot 2 is divided into a plurality of states and the states are sequentially shifted. Further, Q learning as reinforcement learning executed by the machine learning device 5 can be realized by using, for example, a neural network. A neural network has a configuration that mimics the structure of a human brain, and is configured by stacking multiple logic circuits that mimic the function of neurons (neurons) in the human brain.

<About the learning processor>
The learning processing unit 6 is a part that executes a learning process for learning the operation of the robot 2. The learning processing unit 6 may execute the learning process when the robot 2 is performing the production operation, or may execute the learning process separately from the production operation of the robot 2. The production operation of the robot 2 is a continuous operation of the robot 2 in which the work W is taken out of the container CN by the hand unit 26 and the taken out work W is placed on the pallet PL (see FIG. 3 described later). It is. The learning processing unit 6 includes a state observation unit 61, a behavior observation unit 62, a learning unit 63, and a displacement amount observation unit 64.

(About the state observation unit)
FIG. 3 is a diagram for explaining the operation of the state observation unit 61. Note that FIG. 3 shows a state in which the robot 2 and the container CN are viewed from above, and three works W1, W2, and W3 are stacked in the container CN. The robot 2 performs a continuous operation of taking out one work W3 from the container CN by the hand unit 26 and placing the taken work W3 on the pallet PL. The state observation unit 61 divides the continuous operation of the robot 2 into a plurality of states, and observes the state of the robot 2.

  Although the number of states of the robot 2 observed by the state observation unit 61 is not particularly limited, FIG. 3 shows four states S1, S2, S3, and S4. In the state S1, the postures of the trunk portion 22, the first arm 23, the second arm 24, the wrist portion 25, and the hand portion 26 are set so that the hand portion 26 of the robot 2 is disposed at a predetermined position above the pallet PL. The state of the adjusted robot 2 is shown. The state S2 is a state immediately before holding the work W3 to be held (gripped) in the container CN by the claw part 261 of the hand part 26, and the hand part 26 is arranged at a predetermined position directly above the work W3. This is the state of the robot 2 in which the postures of the trunk 22, the first arm 23, the second arm 24, the wrist 25, and the hand 26 are adjusted. The state S3 is such that the claws 261 of the hand unit 26 hold the work W3 to be held in the container CN, the body 22, the first arm 23, the second arm 24, the wrist 25, and the hand unit 26. This is the state of the robot 2 whose posture has been adjusted. In the state S4, the postures of the body 22, the first arm 23, the second arm 24, the wrist 25, and the hand 26 are set such that the work W3 held by the claw 261 of the hand 26 is placed on the pallet PL. This is the state of the robot 2 adjusted. The robot 2 takes out one work W3 from the container CN by the hand unit 26 by continuously shifting the state in the order of the state S1, the state S2, the state S3, and the state S4, and places the taken out work W3 on the pallet PL. Place.

  The state of the robot 2 is defined by state variables (ΔX, ΔY, ΔZ, p, d). The state variables (ΔX, ΔY, ΔZ, p, d) are variables that change each time the state of the robot 2 is shifted.

  The state variable “ΔX” refers to an X coordinate value related to the position in the container CN of the work W3 to be held by the claw 261 of the hand unit 26 in a XYZ orthogonal coordinate system as a reference value (hereinafter referred to as “X reference value”). ), And represents a difference between the X coordinate value (hereinafter, referred to as “hand X value”) of the position of the hand unit 26 with respect to the X reference value. The state variable “ΔY” refers to a Y coordinate value related to the position in the container CN of the work W3 to be held by the claw portion 261 of the hand unit 26 in a XYZ orthogonal coordinate system as a reference value (hereinafter, “Y reference value”). ), And represents a difference between the Y coordinate value (hereinafter, referred to as “hand Y value”) of the position of the hand unit 26 with respect to the Y reference value. The state variable “ΔZ” refers to a Z coordinate value related to the position in the container CN of the work W3 to be held by the claw 261 of the hand unit 26 in a XYZ orthogonal coordinate system as a reference value (hereinafter referred to as “Z reference value”). ), And represents the difference between the Z coordinate value (hereinafter, referred to as “hand Z value”) of the position of the hand unit 26 with respect to the Z reference value. The state variable “p” indicates whether or not the claw portion 261 of the hand unit 26 holds the work W3. The state variable “p” is “1” when the claw 261 of the hand unit 26 holds the work W3, and is “1” when the claw 261 of the hand unit 26 does not hold the work W3. "0: Zero" is set. As for the state variable “d”, a holding space for enabling the holding by the claw 261 is secured around the work that is the next holding candidate for one work W3 by the claw 261 of the hand unit 26. It indicates whether or not. The state variable "d" is set to "1" when a holding space is secured around the next candidate work, and is set to "0: zero" when a holding space is not secured. Is done.

  In the example shown in FIG. 3, when the state of the robot 2 is the state S1, the hand unit 26 is separated from the container CN in the X-axis, Y-axis, and Z-axis directions, The work W3 is not held by the part 261 and a holding space for the claw part 261 is secured around the work W1, W2. Therefore, in the state variables (ΔX, ΔY, ΔZ, p, d) that define the state S1 of the robot 2, “ΔX”, “ΔY”, and “ΔZ” are predetermined values “XA”, “YA”, respectively. , “ZA”, “p” indicates “0: zero”, and “d” indicates “1”.

  In the example illustrated in FIG. 3, when the state of the robot 2 is the state S2, the hand unit 26 is not separated from the container CN in the X-axis and Y-axis directions but is not separated in the Z-axis direction. The work W3 is not held by the claw portion 261 and the holding space for the claw portion 261 is secured around the works W1 and W2. Therefore, in the state variables (ΔX, ΔY, ΔZ, p, d) that define the state S2 of the robot 2, “ΔX” and “ΔY” indicate “0; zero”, respectively, and “ΔZ” indicates a predetermined value. The value “ZA” is indicated, “p” indicates “0; zero”, and “d” indicates “1”.

  In the example illustrated in FIG. 3, when the state of the robot 2 is the state S3, the hand unit 26 is not separated from the container CN in the X-axis, the Y-axis, and the Z-axis. The work W3 is held by the claws 261 and a space for holding the work W3 is secured around the works W1 and W2. For this reason, in the state variables (ΔX, ΔY, ΔZ, p, d) that define the state S3 of the robot 2, “ΔX”, “ΔY” and “ΔZ” indicate “0; Indicates “1”, and “d” indicates “1”.

  In the example shown in FIG. 3, when the state of the robot 2 is the state S4, the hand unit 26 is separated from the container CN in the X-axis, Y-axis, and Z-axis directions, The work W3 is held by the portion 261 and a holding space for the claw portion 261 is secured around the works W1 and W2. Therefore, in the state variables (ΔX, ΔY, ΔZ, p, d) that define the state S4 of the robot 2, “ΔX”, “ΔY”, and “ΔZ” are predetermined values “XA”, “YA”, respectively. And “ZA”, “p” indicates “1”, and “d” indicates “1”.

  The state observation unit 61 determines whether the state of the robot 2 is the state S1, the state S2, the state S3, or the state based on the state variables (ΔX, ΔY, ΔZ, p, d) that change each time the state of the robot 2 is shifted. The state of S4 can be recognized. When the state of the robot 2 is any one of the state S1, the state S2, and the state S3, the posture of the trunk unit 22, the first arm 23, the second arm 24, the wrist unit 25, the hand unit 26, and the like Due to the difference, there are a plurality of sub-states. When the state of the robot 2 is any of the state S1, the state S2, and the state S3, the state observation unit 61 also observes the sub state. As for the state S4 indicating the state of the final target of the robot 2 in which the work W3 held by the claw part 261 of the hand part 26 is placed on the pallet PL, there are sub-states such as the state S1, the state S2, and the state S3. do not do.

(About the behavior observation section)
FIG. 4 is a diagram for explaining the operation of the behavior observation unit 62. Note that FIG. 4 shows that the state of the robot 2 includes a plurality of sub-states “S1-1, S1-2,... S1-n” in the state S1, and a plurality of sub-states in the state S2. , S2-2,... S2-n are present, and the state S3 includes a plurality of sub-states "S3-1, S3-2,. Is present.

  The behavior observation unit 62 observes a behavior pattern of the robot 2 when the state of the robot 2 is shifted. More specifically, the behavior observing unit 62 regards the behavior pattern of the robot 2, the behavior pattern when the state of the robot 2 is shifted from the state S1 to the state S2, the behavior pattern when the state is shifted from the state S2 to the state S3, The behavior pattern when the state S3 is shifted to the state S4 is observed. There are a plurality of action patterns that the robot 2 can take when the state is shifted, depending on the number of sub-states in each of the states S1, S2, and S3 (action A1, action A2,... Action An). When the state of the robot 2 is shifted from the state S3 to the state S4, a take-out operation is performed in which one work W in the container CN is taken out from the container CN while being held by the claw 261 of the hand unit 26. I have.

  The behavior elements that define the behavior pattern of the robot 2 observed by the behavior observation unit 62 include the grip angle θ, the grip position HP, the rotation angle β1 and the rotation speed pattern on the first axis 2A, and the second pattern shown in FIG. Rotation angle β2 and rotation speed pattern on axis 2B, rotation angle β3 and rotation speed pattern on third axis 2C, rotation angle β4 and rotation speed pattern on fourth axis 2D, rotation angle β5 and rotation speed pattern on fifth axis 2E, The rotation angle β6 and the rotation speed pattern on the sixth axis 2F are included. As described above, in the robot 2 composed of the vertical articulated robot, the number of axes is not limited to six, but is arbitrary. For this reason, the rotation angle and the rotation speed pattern in each axis included in the action element that defines the action pattern of the robot 2 are in accordance with the number of axes.

  The grip angle θ is an angle formed by two claw portions 261 for holding (gripping) the work W in the hand portion 26 (see FIG. 2). The gripping position HP indicates a position where the one work W is held (gripped) by the claw 261 when the hand unit 26 takes out one work W. The rotation angle β1 on the first axis 2A represents the rotation angle of the trunk 22 around the first axis 2A when the state of the robot 2 is shifted. Since the barrel 22 can rotate in both forward and reverse directions around the first axis 2A, the rotation angle β1 is indicated by a “positive; plus” rotation angle when the barrel 22 rotates in the forward direction. When the lens rotates in the reverse direction, it is indicated by a rotation angle of “negative; minus”. The rotation angle β2 on the second axis 2B represents the rotation angle around the second axis 2B of the first arm 23 when the state of the robot 2 is shifted. Since the first arm 23 is rotatable in both the forward and reverse directions about the second axis 2B, the rotation angle β2 is indicated by a “positive; plus” rotation angle when the first arm 23 rotates in the forward direction. When the one arm 23 rotates in the reverse direction, the rotation angle is indicated by “negative; minus”. The rotation angle β3 on the third axis 2C represents the rotation angle around the third axis 2C of the arm base 241 when the state of the robot 2 is shifted. Since the arm base 241 can rotate in both forward and reverse directions around the third axis 2C, the rotation angle β3 is indicated by a “positive; plus” rotation angle when the arm base 241 rotates in the forward direction. When the lens rotates in the reverse direction, it is indicated by a rotation angle of “negative; minus”.

  The rotation angle β4 on the fourth axis 2D represents the rotation angle of the arm 242 about the fourth axis 2D when the state of the robot 2 is shifted. Since the arm portion 242 can rotate in both forward and reverse directions around the fourth axis 2D, the rotation angle β4 is indicated by a “positive; plus” rotation angle when the arm portion 242 rotates in the forward direction. When the lens rotates in the reverse direction, it is indicated by a rotation angle of “negative; minus”. A rotation angle β5 on the fifth axis 2E represents a rotation angle around the fifth axis 2E of the wrist 25 when the state of the robot 2 is shifted. Since the wrist portion 25 can rotate in both forward and reverse directions around the fifth axis 2E, the rotation angle β5 is indicated by a “positive; plus” rotation angle when the wrist portion 25 rotates in the forward direction. When the lens rotates in the reverse direction, it is indicated by a rotation angle of “negative; minus”. The rotation angle β6 on the sixth axis 2F represents the rotation angle around the sixth axis 2F of the hand unit 26 when the state of the robot 2 is shifted. Since the hand portion 26 can rotate in both forward and reverse directions around the sixth axis 2F, the rotation angle β6 is indicated by a “positive; plus” rotation angle when the hand portion 26 rotates in the forward direction. When the lens rotates in the reverse direction, it is indicated by a rotation angle of “negative; minus”.

  The rotation speed pattern on each of the axes 2A to 2F indicates a rotation speed pattern around each axis, and is divided into a first pattern, a second pattern, and a third pattern shown in FIG. The first pattern of the rotational speed includes two ascending regions in which the rotational speed increases linearly with the passage of time, and a descending region in which the rotational speed decreases linearly with the passage of time from the end of the upward region. It consists of two areas. The second pattern of the rotational speed includes an ascending region where the rotational speed increases linearly with the passage of time, a constant velocity region where the rotational speed is constant at a constant time from the end of the ascending region, and a constant speed. It consists of three regions: a descending region where the rotational speed decreases linearly with the passage of time from the end of the region. The third pattern of the rotational speed includes two ascending areas in which the rotational speed rises in a curve with time and a descending area in which the rotational speed falls in a curve with time from the end of the ascending area. It consists of two areas.

  The behavior observation unit 62 can recognize a behavior pattern when the state of the robot 2 is shifted based on each behavior element.

  The optimum behavior pattern of the robot 2 when the state of the robot 2 is shifted from the state S1 to the state S2 and the optimum behavior pattern of the robot 2 when the state of the robot 2 is changed from the state S2 to the state S3 are described later in a learning unit. 63 has already been learned. Further, regarding the optimal behavior pattern of the robot 2 when the state is shifted from the state S3 to the state S4 when the state variables (ΔX, ΔY, ΔZ, p, d) are (0, 0, 0, 1, 1). Have already been learned by the learning unit 63 described later. That is, regarding the behavior pattern of the robot 2 when the state of the robot 2 is shifted from the state S3 to the state S4, the state variable “d” is “1”, and the next holding candidate by the claw 261 of the hand unit 26 is set. The behavior pattern in the case where the holding space for the claw portion 261 is secured around the work to be formed has already been learned. The behavior pattern of the robot 2 already learned by the learning unit 63 is stored in the storage unit 8.

  The existing behavior pattern stored in the storage unit 8 is read from the storage unit 8 by the behavior determination unit 9 described later, and is output to the control device 4. The control device 4 to which the existing behavior pattern is input can control the operation of the robot 2 based on the existing behavior pattern. Under the control of the control device 4, the robot 2 executes a continuous production operation in which the hand W takes out the work W from the container CN and places the taken out work W on the pallet PL.

  On the other hand, regarding the optimal behavior pattern of the robot 2 when shifting from the state S3 to the state S4 when the state variables (ΔX, ΔY, ΔZ, p, d) are (0, 0, 0, 1, 0). Is subjected to reinforcement learning by a learning unit 63 described later. That is, regarding the behavior pattern of the robot 2 when the state of the robot 2 is shifted from the state S3 to the state S4, the state variable “d” is “0”, and the next holding candidate by the claw 261 of the hand unit 26 is set. The behavior pattern when the holding space of the claw portion 261 is not secured around the work to be performed is subjected to the reinforcement learning by the learning section 63.

  The determination unit 7 determines whether or not the work to be held next by the claw 261 of the hand unit 26 is a work that cannot be held by the claw 261 because a holding space is not secured around the work. Is determined. Before or when the hand unit 26 holds one work W in the container CN by the claw unit 261, the determination unit 7 determines that a work that is the next holding candidate for one work W is a work that cannot be held. It is determined whether or not there is. Before or when the hand unit 26 holds one work W in the container CN by the claw unit 261, a reference image is obtained by an imaging operation of the camera 31 in the imaging device 3, and an image corresponding to the reference image of the image processing unit 32 is obtained. Reference image data is generated by the processing. The reference image data is image data including three-dimensional position information regarding a work to be a next holding candidate. The determination unit 7 recognizes the accommodation state of each work in the container CN based on the reference image data output from the imaging device 3, and determines whether or not the work to be the next holding candidate is a work that cannot be held. judge.

  The determination unit 7 determines whether the next work candidate to be held is arranged close to the inner surface of the container CN so that the claw unit 261 of the hand unit 26 cannot be inserted, or if a plurality of works In the case where the work is arranged in the proximity state, it is determined that the holding space for enabling the holding by the claw portion 261 is not secured, and it is determined that the work cannot be held. When the determination unit 7 determines that the next work to be held is a work that cannot be held, the state variables (ΔX, ΔY, ΔZ, p, d) when the state of the robot 2 is the state S3 are ( 0,0,0,1,0).

  When the state of the robot 2 is the state S3 and the determination unit 7 determines that the work to be the next holding candidate is a work that cannot be held, the robot 2 performs a predetermined operation before shifting from the state S3 to the state S4. A displacement operation for displacing the non-holdable work is performed by using the displacement method of (1). Examples of the displacement method used when the robot 2 performs the displacement operation for displacing the non-holdable work include first to seventh methods shown in FIG. Further, a combination of a plurality of methods selected from the first to seventh methods may be used as the displacement method. In FIG. 6, when one work W3 in the container CN is held by the claw portion 261 of the hand portion 26, no holding space is secured around the work W1, W2, and the work W1, W2 is The work cannot be held.

  The first method is to move one work W3 held by the claw portion 261 toward the pallet PL by moving the hand portion 26 in a state where the one work W3 is in contact with the non-holdable work W2. This is a displacement method for displacing the non-holdable work W2. By the displacement operation of the robot 2 using the first method, a holding space for enabling the holding by the claw portion 261 is secured around the work W2 where the holding of the hand portion 26 by the claw portion 261 is impossible. Thus, the work W2 can be held by the claw portion 261.

  In the second method, after one work W3 held by the claw portion 261 is placed on the pallet PL, the hand portion 26 moves while holding the container CN by the claw portion 261 to respond to the movement of the container CN. This is a displacement method for displacing the work W2 that cannot be held. By the displacement operation of the robot 2 using the second method, a holding space for enabling the holding by the claw portion 261 is secured around the work W2 where the holding of the hand portion 26 by the claw portion 261 is impossible. Thus, the work W2 can be held by the claw portion 261.

  In the third method, after one work W3 held by the claw portion 261 is placed on the pallet PL, the hand portion 26 moves while the claw portion 261 is in contact with the work W2 that cannot be held, thereby holding the work W3. This is a displacement method for displacing the impossible work W2. By the displacement operation using the third method by the robot 2, a holding space for enabling the holding by the claw portion 261 is secured around the work W2 where the holding of the hand portion 26 by the claw portion 261 is impossible. Thus, the work W2 can be held by the claw portion 261.

  In the fourth method, after one work W3 held by the claw portion 261 is placed on the pallet PL, a work WS different from the work W1, W2, and W3 and taken out of another container is held by the claw portion 261. This is a displacement method in which the hand unit 26 moves while the work WS is in contact with the non-holdable work W2, thereby displacing the non-holdable work W2. By the displacement operation using the fourth method by the robot 2, a holding space for enabling the holding by the claw 261 is secured around the work W2 where the holding by the claw 261 of the hand unit 26 is disabled. Thus, the work W2 can be held by the claw portion 261.

  In the fifth method, after one work W3 held by the claw portion 261 is placed on the pallet PL, the special jig JG is held by the claw portion 261 and the hand portion 26 holds the special jig JG on the work W2 that cannot hold the special jig JG. This is a displacement technique for displacing the non-holdable work W2 by moving in a state where the workpiece W2 is in contact with the workpiece. By the displacement operation using the fifth method by the robot 2, a holding space for enabling the holding by the claw portion 261 is secured around the work W2 where the holding of the hand portion 26 by the claw portion 261 is impossible. Thus, the work W2 can be held by the claw portion 261.

  In the sixth method, after one work W3 held by the claw portion 261 is placed on the pallet PL, the nozzle NZ capable of injecting gas such as air is held by the claw portion 261 and the work cannot hold the gas from the nozzle NZ. This is a displacement method for displacing the non-holdable work W2 by moving the hand unit 26 in a state of being ejected toward W2. By the displacement operation of the robot 2 using the sixth technique, a holding space for enabling the holding by the claw 261 is secured around the work W2, which is not allowed to be held by the claw 261 of the hand unit 26. Thus, the work W2 can be held by the claw portion 261.

  The seventh method is a displacement method of displacing the non-retainable works W1 and W2 by breaking the non-retainable works W1 and W2 in the take-out operation of taking out one work W3 held by the claw portion 261. The seventh method is effective when, for example, non-holdable works W1 and W2 are arranged on one work W3. By the displacement operation of the robot 2 using the seventh method, a holding space for enabling the holding by the claw 261 is provided around the workpieces W1 and W2, which cannot be held by the claw 261 of the hand unit 26. Thus, the workpieces W1 and W2 can be held by the claw portions 261.

  When the determination unit 7 determines that the work to be the next holding candidate is a non-holdable work, the behavior observation unit 62 determines the action pattern in the displacement operation of the robot 2 that displaces the non-holdable work using the above-described displacement method. Observe also. The behavior observation unit 62 can recognize the behavior pattern in the displacement operation of the robot 2 based on each behavior element shown in FIG.

(Displacement observation unit)
In the displacement amount observation unit 64, the determination unit 7 has determined that the work to be the next holding candidate is a non-holdable work, and the robot 2 has performed a displacement operation of displacing the non-holdable work using a predetermined displacement method. Sometimes, the work displacement of the work that cannot be held is observed. The displacement amount observation unit 64 observes the work displacement amount of the non-holdable work based on each image data output from the imaging device 3 before and after the displacement operation by the robot 2.

  More specifically, the displacement amount observing unit 64 is the image data before the displacing operation by the robot 2, the reference image data referred to when the judging unit 7 judges the presence or absence of the work that cannot be held, The work displacement amount of the non-holdable work is observed based on the image data after the displacement operation. The displacement amount observation unit 64 calculates each coordinate value in the three-dimensional position information of the non-holdable work included in the reference image data, and each coordinate value in the three-dimensional position information of the work included in the image data after the displacement operation by the robot 2. By calculating the difference, the work displacement amount of the work that cannot be held is observed. Details of the operation of the displacement amount observation unit 64 will be described later.

(About the learning department)
The learning unit 63 learns an optimal behavior pattern of the robot 2 when the state of the robot 2 is shifted. Further, the learning unit 63 displaces the non-holding work so that a holding space is secured when the determination unit 7 determines that the work to be the next holding candidate by the hand unit 26 is the non-holding work. In addition to learning an optimal displacement method capable of performing the same, the behavior pattern of the robot 2 using the displacement method is learned.

  As described above, the learning unit 63 determines the optimum behavior pattern of the robot 2 when the state of the robot 2 shifts from the state S1 to the state S2 and the robot 2 when the state of the robot 2 shifts from the state S2 to the state S3. The optimal behavior pattern has already been learned. In addition, the learning unit 63 determines that the behavior pattern of the robot 2 when the state of the robot 2 is shifted from the state S3 to the state S4 includes a claw section around the work to be held next by the claw section 261 of the hand section 26. The behavior pattern when the holding space according to H.261 is secured has already been learned. The behavior pattern of the robot 2 that has already been learned by the learning unit 63 is stored in the storage unit 8. Hereinafter, learning of the behavior pattern of the robot 2 in the displacement operation of displacing the non-holdable work using the predetermined displacement method when the state of the robot 2 is the state S3 will be described in detail.

  The learning unit 63 compares the behavior pattern of the robot 2 observed by the behavior observation unit 62 when the non-holdable work is displaced using a predetermined displacement method with the work of the non-holdable work observed by the displacement amount observation unit 64. Learning is performed in association with the displacement amount. The learning unit 63 is based on the teacher data that associates the behavior pattern of the robot 2 with the amount of work displacement, and based on the teacher data, an optimal displacement method for displacing the non-holdable work so that a holding space can be secured, and the behavior pattern of the robot 2 To learn.

  As shown in FIG. 1, the learning unit 63 includes a reward setting unit 631 and a value function updating unit 632.

  The reward setting unit 631 sets a reward R (see FIG. 9 described later) according to the amount of work displacement of the work that cannot be held, for the behavior pattern in the displacement operation of the robot 2 observed by the behavior observation unit 62. The reward setting unit 631 may set the reward R stepwise according to the work displacement amount of the work that cannot be held. For example, the reward setting unit 631 sets the first value R1 (for example, “100”) for the behavior pattern of the robot 2 in which the work displacement amount of the non-holdable work is equal to or larger than a predetermined threshold value WDT (see FIG. 9 described later). ) Is given. In addition, the reward setting unit 631 determines that, for the behavior pattern of the robot 2 in which the work displacement amount of the non-holdable work is equal to or more than (threshold value WDT × 0.5) and less than the threshold value WDT, a first value R1 smaller than the first value R1. A reward R of a value R2 of 2 (for example, “10”) is given. In addition, the reward setting unit 631 sets a third value R3 smaller than the second value R2 for an action pattern of the robot 2 in which the work displacement amount of the non-holdable work is less than (threshold value WDT × 0.5). (For example, “0: zero”).

  The threshold value WDT is, for example, a value obtained by multiplying the thickness of the claw portion 261 of the hand portion 26 by a coefficient (eg, “1.2”) of “1” or more. That is, the threshold value WDT is set to a value slightly larger than a holding space corresponding to the thickness of the claw portion 261 of the hand portion 26 for enabling the claw portion 261 to hold the work.

  The value function updating unit 632 updates the value function that defines the value Q (s, a) of the behavior pattern of the robot 2 according to the reward R set by the reward setting unit 631. The value function updating unit 632 updates the value function using an update expression of the value Q (s, a) represented by the following expression (1).

  In the above equation (1), “s” represents the state of the robot 2 (state S3), and “a” represents the behavior of the robot 2 according to the behavior pattern. The state of the robot 2 shifts from the state “s” (state S3) to the state “s ′” (state after the displacement operation) by the action “a”. R (s, a) represents the reward R obtained by the transition of the state.

  In the above equation (1), the term to which “max” has been added represents “γ” as the value Q (s ′, a ′) when the highest value action “a ′” is selected in the state “s ′”. Multiplied by “Γ” is a parameter called an attenuation factor, and is in a range of 0 <γ ≦ 1 (for example, 0.9). “Α” is a parameter called a learning rate, and is in a range of 0 <α ≦ 1 (for example, 0.1). “Ε” is a parameter called a correction coefficient, and is in a range of 0 <ε ≦ 1. The correction coefficient ε is calculated by the learning unit 63, which will be described later in detail. In the updating equation of the value Q (s, a) shown in the above equation (1), “ε” is set to “ε = 1” until the learning unit 63 calculates the correction coefficient ε.

  The above equation (1) calculates the value Q (s, a) of the action “a” in the state “s” based on the reward R (s, a) set by the reward setting unit 631 for the action “a”. Represents an update expression for updating. In other words, the above equation (1) is such that the value Q (s ′, a ′) of the action “a ′” in the state “s ′” is greater than the value Q (s, a) of the action “a” in the state “s”. And the reward R (s, a) is larger, the value Q (s, a) is increased, and on the contrary, the value Q (s, a) is decreased. In other words, the value function updating unit 632 updates the value function using the update expression represented by the above expression (1), thereby updating the value Q (s, a) of the certain action “a” in the certain state “s”. ) To the reward R set for the action “a” and the value Q (s ′, a ′) of the best action “a ′” in the next state “s ′” by the action “a”. I try to get closer.

  Here, although the details will be described later, with reference to the first example of the displacement operation shown in FIGS. 7 to 11, by the displacement operation by the action “a” (the action A1 in FIG. 7) of the robot 2, When the work that cannot be held is displaced so that the holding space can be secured, the state of the robot 2 shifts from the state “s” (state S3) to the state “s ′” (state S31 in FIG. 7). The state variables (ΔX, ΔY, ΔZ, p, d) in this state “s ′” (state S31) are (0, 0, 0, 1, 1). That is, since the state variable “d” is “1” and the holding space by the claw portion 261 is secured around the work to be held next by the hand portion 26, the robot 2 to which the next transition is made Is the state S4. Therefore, the action A1 ′ (FIG. 7) that is the action “a ′” that shifts from the state “s ′” (state S31) to the state S4 is selected, and the value Q (s ′, a) when shifting to the state S4 is selected. ') Will be the highest.

  On the other hand, when the non-holdable work is displaced by the displacement operation due to the action “a” of the robot 2 (action A2 in FIG. 7) but the holding space is not secured, the state of the robot 2 changes to the state “s” ( The state shifts from the state S3) to the state “s ′” (the state S32 in FIG. 7). The state variables (ΔX, ΔY, ΔZ, p, d) in this state “s ′” (state S32) are (0, 0, 0, 1, 0). In this case, the state variable “d” is “0”, and a holding space is not secured around the work to be the next holding candidate by the hand unit 26. The state becomes the state S3 again, and the displacement operation is retried. Therefore, the action A2 ′ (FIG. 7) that is the action “a ′” that shifts from the state “s ′” (state S32) to the state S3 is selected, and the value Q (s ′, a) when shifting to the state S3 is selected. ') Is lower than the value at the time of shifting to the state S4.

  Further, when the non-holdable work hardly displaces due to the displacement operation by the action “a” (action A3 in FIG. 7) of the robot 2, the state of the robot 2 changes from the state “s” (state S3) to the state “s”. '"(State S33 in FIG. 7). The state variables (ΔX, ΔY, ΔZ, p, d) in the state “s ′” (state S33) are (0, 0, 0, 1, 0). In this case, the state variable “d” is “0”, and a holding space is not secured around the work to be the next holding candidate by the hand unit 26. The state becomes the state S3 again, and the displacement operation is retried. Therefore, the action A3 ′ (FIG. 7) that is the action “a ′” that shifts from the state “s ′” (state S33) to the state S3 is selected, and the value Q (s ′, a) when shifting to the state S3 is selected. ') Is similarly low.

  The second example of the displacement operation shown in FIGS. 12 to 14 described later and the third example of the displacement operation shown in FIGS. 15 and 16 are also similar to the case of the first example of the above-described displacement operation. The updating formula of the value Q (s, a) shown in the above formula (1) can be applied.

  The learning unit 63 generates learning result information indicating a learning result of an action pattern of the robot 2 in a displacement operation of displacing a work that cannot be held using a predetermined displacement method. The learning result information generated by the learning unit 63 is stored in the storage unit 8. The learning unit 63 may learn the behavior pattern of the robot 2 in the displacement operation of displacing the non-holdable work when the robot 2 is performing the production operation, or may perform the production of the robot 2. Learning may be performed separately from the operation.

<About the action decision section>
The action determining unit 9 reads the action pattern of the robot 2 stored in the storage unit 8 from the storage unit 8 to determine the action pattern of the robot 2 at the time of the state transition. The action determining unit 9 outputs the action pattern of the robot 2 read from the storage unit 8 to the control device 4. More specifically, the action determining unit 9 reads out the above-described existing action pattern from the storage unit 8 and outputs it to the control device 4. The control device 4 to which the existing behavior pattern has been input controls the operation of the robot 2 based on the existing behavior pattern. Under the control of the control device 4, the robot 2 executes a continuous production operation in which the hand W takes out the work W from the container CN and places the taken out work W on the pallet PL.

  Further, at the time of executing the displacement operation for displacing the non-holdable work, the action determining unit 9 refers to the learning result information stored in the storage unit 8 and representing the learning result of the learning unit 63. For example, when an action pattern capable of securing a holding space is registered in the learning result information, the action determining unit 9 sets the action pattern to enable the non-holdable work to be held by the claw 261. To be determined. The action determining unit 9 determines an action pattern of the robot 2 at the time of performing the displacement operation by reading out an action pattern registered in the learning result information stored in the storage unit 8 and capable of securing a holding space. I do. The action determining unit 9 outputs the action pattern at the time of the displacement operation of the robot 2 read from the storage unit 8 to the control device 4. The control device 4 to which the behavior pattern at the time of the displacement operation is input controls the operation of the robot 2 based on the behavior pattern. Under the control of the control device 4, the robot 2 displaces the non-holdable work such that a space for holding the claw portion 261 is secured around the robot.

  When a holding space is secured around the work that cannot be held, the work can be held by the claw portion 261. Therefore, after the holding space is secured around the non-holding work, the action determining unit 9 reads out the above-described existing action pattern from the storage unit 8 and outputs it to the control device 4. Accordingly, under the control of the control device 4, the robot 2 executes a continuous production operation of taking out the work W from the container CN by the hand unit 26 and placing the taken out work W on the pallet PL.

[Specific example of displacement operation for displacing a workpiece that cannot be held]
As described above, when the state of the robot 2 is the state S3 and one work is held by the claw part 261 of the hand unit 26, it is determined that the work to be the next holding candidate is a work that cannot be held. When the determination is made by the unit 7, a displacement operation using a predetermined displacement method for displacing the non-holdable work so that a holding space is secured is executed. Hereinafter, the displacement operation for displacing the non-holdable work will be described in detail with reference to a specific example.

<About the first example of the displacement operation>
A first example of the displacement operation will be described with reference to FIGS. FIG. 7 is a diagram illustrating a first example of a displacement operation for displacing a non-holdable work. FIG. 8 is a diagram for explaining the operation of the displacement amount observation unit 64. FIG. 9 is a diagram for explaining the learning result information JH1 generated by the learning unit 63 in the displacement operation of the first example. FIG. 10 is a flowchart illustrating the operation of the machine learning device 5 regarding the displacement operation of the first example.

  The state observing unit 61 determines that the state of the robot 2 has been shifted from the state S2 to the state S3 based on the state variables (ΔX, ΔY, ΔZ, p, d) that change each time the state of the robot 2 is shifted. Is observed (step a1 in FIG. 10). When the state of the robot 2 is shifted from the state S2 to the state S3, that is, when one work is held by the claw part 261 of the hand unit 26, the determination unit 7 outputs the reference image output from the imaging device 3. Data is acquired (step a2 in FIG. 10). The reference image data is image data including three-dimensional position information regarding a work to be a next holding candidate. The determination unit 7 recognizes the accommodation state of each work in the container CN based on the reference image data, and determines whether the next work as a candidate for holding is a work that cannot be held (step a3 in FIG. 10). ).

  If the determination unit 7 determines that the work to be the next holding candidate is not a work that cannot be held, the state variables (ΔX, ΔY, ΔZ, p, d) when the state of the robot 2 is the state S3 are determined. (0, 0, 0, 1, 1). In this case, the action determining unit 9 reads an existing action pattern for shifting from the state S3 to the state S4 from the storage unit 8 and outputs the pattern to the control device 4. The control device 4 to which the existing behavior pattern has been input controls the operation of the robot 2 based on the existing behavior pattern. Under the control of the control device 4, the robot 2 executes a take-out operation of taking out one work held by the claw portion 261 from the container CN (step a5 in FIG. 10).

  On the other hand, when the determination unit 7 determines that the work to be the next holding candidate is a work that cannot be held, as shown in FIG. 7, the state variable (ΔX, ΔY, ΔZ, p, d) are set to (0, 0, 0, 1, 0). In the example illustrated in FIG. 7, when one work W3 in the container CN is held by the claw portion 261 of the hand unit 26, no holding space is secured around the works W1 and W2, and the work W1 and W2 are The work cannot be held. For this reason, the state variable “d” is “0”, and the holding space of the claw 261 is not secured around the workpieces W1 and W2 to be held next by the claw 261 of the hand unit 26. It is shown. In the example shown in FIG. 7, the non-retainable work W1 is disposed close to the inner surface of the container CN, and the non-retainable work W2 is disposed adjacent to the non-retainable work W1 on the side of the non-retainable work W1. I have. Therefore, a holding space is not secured around the non-holding works W1 and W2.

  When the determination unit 7 determines that the work W1 or W2 that is the next holding candidate is a work that cannot be held, the claw unit 261 holds at least one of the work W1 and the work W2 that is the work that cannot be held. A displacement operation is performed to displace so that a space is secured around. In the first example of the displacement operation, as shown in FIG. 7, the robot 2 displaces the work W2 as the non-holdable work by the displacement operation based on the action pattern using the first method. In the first method, as described above, when moving one work W3 held by the claw portion 261 toward the pallet PL, the hand unit 26 causes the one work W3 to contact the non-holdable work W2. This is a displacement method of displacing the non-holdable work W2 by moving in a state where the workpiece W2 is held.

  The behavior observing unit 62 observes the behavior pattern of the robot 2 using the first method (Step a4 in FIG. 10). In the example illustrated in FIG. 7, three types of behavior patterns of the behavior A1, the behavior A2, and the behavior A3 are illustrated as the behavior patterns of the robot 2 using the first technique. The action A1 is that the hand unit 26 moves the non-holdable works W1 and W2 in a state where the tip of one work W3 held by the claw 261 is in contact with one longitudinal end surface of the non-holdable work W2 (abutting position CP). This is an action pattern that moves in a direction obliquely away from the non-holdable work W1 in the parallel direction (movement trajectory MT). The action A2 has the same action pattern as the action A1 except that the inclination degree of the movement trajectory MT in the parallel direction of the non-holdable works W1 and W2 is different from the action A1. In the action A3, the hand unit 26 moves in the state where the tip of one work W3 held by the claw 261 abuts on the side surface of the non-holdable work W2 (abutting position CP) and the parallel direction of the non-holdable works W1 and W2. This is an action pattern that moves (movement trajectory MT) in a direction perpendicular to the direction, that is, a direction along the side surface of the non-holdable work W2.

  The behavior elements that define the behavior pattern of the robot 2 observed by the behavior observation unit 62 include the grip angle θ, the grip position HP, the rotation angle β1 and the rotation speed pattern on the first axis 2A shown in FIG. Rotation angle β2 and rotation speed pattern on second shaft 2B, rotation angle β3 and rotation speed pattern on third shaft 2C, rotation angle β4 and rotation speed pattern on fourth shaft 2D, rotation angle β5 and rotation speed on fifth shaft 2E The pattern, the rotation angle β6 on the sixth axis 2F, and the rotation speed pattern are included. Each action element shown in FIG. 5 is an element that determines the contact position CP of one work W3 held by the claw portion 261 with respect to the non-holdable work W2 in the action pattern of the robot 2 using the first method. , The moving trajectory MT of the hand unit 26.

  When the displacement operation based on the behavior pattern using the first method is completed, the displacement amount observation unit 64 acquires the image data after the displacement operation output from the imaging device 3 (step a6 in FIG. 10). The image data after the displacement operation is image data including three-dimensional position information on the non-holdable works W1 and W2 after being displaced by the behavior pattern of the robot 2 using the first method. The displacement amount observing section 64 is image data before the displacing operation by the robot 2, the reference image data which the judging section 7 refers to when judging the existence of the non-holdable works W <b> 1, W <b> 2, and after the displacing operation by the robot 2. The work displacement amount of the non-holdable works W1 and W2 is observed based on the image data in (1) (step a7 in FIG. 10).

  In the example illustrated in FIG. 8, before the displacement operation by the robot 2 and when one work W3 is held by the claw portion 261 of the hand unit 26, the reference image GS is formed by the imaging operation of the camera 31 in the imaging device 3. The acquired reference image data GDS is generated by the image processing unit 32 performing image processing on the reference image GS. The reference image GS includes an image area GW1 corresponding to the non-holdable work W1 and an image area GW2 corresponding to the non-holdable work W2. The reference image data GDS includes information on coordinate values (X1, Y1, Z1) as three-dimensional position information of the non-holdable work W1 and coordinate values (X2, Y2) as three-dimensional position information of the non-holdable work W2. , Z2).

  In the example illustrated in FIG. 8, after the displacement operation by the robot 2, the first image G <b> 1, the second image G <b> 2, and the third image G <b> 3 are acquired by the imaging operation of the camera 31 in the imaging device 3. The first image data GD1, the second image data GD2, and the third image data GD3 are generated by performing image processing on the images G1, G2, and G3, respectively.

  The first image G1 and the first image data GD1 show an image and image data after the displacement operation of the robot 2 based on the behavior pattern A1 (the behavior A1 in FIG. 7) using the first method. The first image G1 includes an image area GW1 corresponding to the non-holdable work W1 and an image area GW2 corresponding to the non-holdable work W2 for the non-holdable works W1 and W2 after the displacement operation of the robot 2 based on the action pattern A1. It is included. The first image data GD1 includes information on coordinate values (X11, Y11, Z11) as three-dimensional position information of the non-holdable work W1 and coordinate values (X21, Y21, Z21).

  The second image G2 and the second image data GD2 show an image and image data after the displacement operation of the robot 2 based on the action pattern A2 (the action A2 in FIG. 7) using the first method. The second image G2 includes an image area GW1 corresponding to the non-holdable work W1 and an image area GW2 corresponding to the non-holdable work W2 for the non-holdable works W1 and W2 after the displacement operation of the robot 2 based on the action pattern A2. It is included. The second image data GD2 includes information on the coordinate values (X12, Y12, Z12) as the three-dimensional position information of the non-holdable work W1, and the coordinate values (X22, Y22, Z22).

  The third image G3 and the third image data GD3 show the image and the image data after the displacement operation of the robot 2 based on the behavior pattern A3 (the behavior A3 in FIG. 7) using the first method. The third image G3 includes an image area GW1 corresponding to the non-holdable work W1 and an image area GW2 corresponding to the non-holdable work W1 for the non-holdable works W1 and W2 after the displacement operation of the robot 2 based on the action pattern A3. It is included. Further, the third image data GD3 includes information on the coordinate values (X13, Y13, Z13) as the three-dimensional position information of the non-retainable work W1 and the coordinate values (X23, Y23, Z23).

  The displacement amount observing unit 64 is configured to perform the displacement operation of the robot 2 based on the action pattern A1 using the first method based on the reference image data GDS and the first image data GD1, and the non-holding work W1, The first work displacement WD1 representing the displacement of W2 in the container CN is observed. The first work displacement amount WD1 includes a work displacement amount (XD11, YD11, ZD11) of the non-holdable work W1 and a work displacement amount (XD21, YD21, ZD21) of the non-holdable work W2. In the work displacement amount of the non-holdable work W1, “XD11” is the X coordinate value “X1” in the three-dimensional position information of the non-holdable work W1 included in the reference image data GDS, and the non-holdable work included in the first image data GD1. The difference from the X coordinate value “X11” in the three-dimensional position information of W1 is shown. In addition, “YD11” in the work displacement amount of the non-holdable work W1 is the Y coordinate value “Y1” in the three-dimensional position information of the non-holdable work W1 included in the reference image data GDS and the hold included in the first image data GD1. The difference from the Y coordinate value “Y11” in the three-dimensional position information of the unacceptable work W1 is shown. In addition, “ZD11” in the work displacement amount of the non-holdable work W1 is the Z coordinate value “Z1” in the three-dimensional position information of the non-holdable work W1 included in the reference image data GDS and the hold included in the first image data GD1. The difference from the Z coordinate value “Z11” in the three-dimensional position information of the unacceptable work W1 is shown.

  Similarly, “XD21” in the work displacement amount of the non-holdable work W2 is included in the X-coordinate value “X2” in the three-dimensional position information of the non-holdable work W2 included in the reference image data GDS, and in the first image data GD1. The difference from the X coordinate value “X21” in the three-dimensional position information of the work W2 that cannot be held is shown. In the work displacement amount of the non-holdable work W2, “YD21” is the Y coordinate value “Y2” in the three-dimensional position information of the non-holdable work W2 included in the reference image data GDS, and is stored in the first image data GD1. The difference from the Y coordinate value “Y21” in the three-dimensional position information of the unacceptable work W2 is shown. Further, in the work displacement amount of the non-holdable work W2, “ZD21” is the Z coordinate value “Z2” in the three-dimensional position information of the non-holdable work W2 included in the reference image data GDS and the hold included in the first image data GD1. The difference from the Z coordinate value “Z21” in the three-dimensional position information of the unacceptable work W2 is shown.

  As is clear from the comparison between the reference image GS and the first image G1, the position of the non-holdable work W1 after the displacement operation of the robot 2 based on the action pattern A1 using the first method is compared with the position before the displacement operation. However, the position of the non-holdable work W2 has changed to such an extent that a holding space for the claw portion 261 is secured. Therefore, each value of the work displacement amount (XD11, YD11, ZD11) of the non-holdable work W1 included in the first work displacement amount WD1 observed by the displacement amount observation unit 64 indicates a value close to “0; zero”. However, each value of the work displacement amount (XD21, YD21, ZD21) of the non-holdable work W2 indicates a value corresponding to the displacement of the non-holdable work W2.

  When the non-holdable work W2 is displaced by the displacement operation of the robot 2 based on the action pattern A1 using the first method so that the holding space can be secured, as shown in FIG. 7, the state of the robot 2 is changed to the state S31 ( The state variables ([Delta] X, [Delta] Y, [Delta] Z, p, d) when the state is the state after the displacement operation are (0, 0, 0, 1, 1). In the example shown in FIG. 7, a holding space is secured around the non-holdable work W2 by the displacement operation of the robot 2 based on the action pattern A1, and the work W2 can be held. Therefore, the state variable “d” is “1”, which indicates that the holding space for the claw portion 261 is secured around the work W2.

  In addition, the displacement amount observing unit 64 performs the non-holding work when the robot 2 performs the displacement operation based on the action pattern A2 using the first method based on the reference image data GDS and the second image data GD2. The second work displacement WD2 representing the displacement of W1 and W2 in the container CN is observed. The second work displacement amount WD2 includes a work displacement amount (XD12, YD12, ZD12) of the non-holdable work W1 and a work displacement amount (XD22, YD22, ZD22) of the non-holdable work W2. In the work displacement amount of the non-holdable work W1, “XD12” is the X coordinate value “X1” in the three-dimensional position information of the non-holdable work W1 included in the reference image data GDS, and the non-holdable work included in the second image data GD2. The difference from the X coordinate value “X12” in the three-dimensional position information of W1 is shown. In addition, “YD12” in the work displacement amount of the non-holdable work W1 is the Y coordinate value “Y1” in the three-dimensional position information of the non-holdable work W1 included in the reference image data GDS and the hold included in the second image data GD2. The difference from the Y coordinate value “Y12” in the three-dimensional position information of the unacceptable work W1 is shown. Further, in the work displacement amount of the non-holdable work W1, “ZD12” is the Z coordinate value “Z1” in the three-dimensional position information of the non-holdable work W1 included in the reference image data GDS and the hold included in the second image data GD2. The difference from the Z coordinate value “Z12” in the three-dimensional position information of the unacceptable work W1 is shown.

  Similarly, “XD22” in the work displacement amount of the non-holdable work W2 is included in the X coordinate value “X2” in the three-dimensional position information of the non-holdable work W2 included in the reference image data GDS, and in the second image data GD2. The difference from the X coordinate value “X22” in the three-dimensional position information of the work W2 that cannot be held is shown. Further, in the work displacement amount of the non-holdable work W2, “YD22” is the Y coordinate value “Y2” in the three-dimensional position information of the non-holdable work W2 included in the reference image data GDS and the hold included in the second image data GD2. The difference from the Y coordinate value “Y22” in the three-dimensional position information of the unacceptable work W2 is shown. Further, in the work displacement amount of the non-holdable work W2, “ZD22” is the Z coordinate value “Z2” in the three-dimensional position information of the non-holdable work W2 included in the reference image data GDS and the hold included in the second image data GD2. The difference from the Z coordinate value “Z22” in the three-dimensional position information of the unacceptable work W2 is shown.

  As is clear from the comparison between the reference image GS and the second image G2, the position of the non-holdable work W1 after the displacement operation of the robot 2 based on the action pattern A2 using the first method is compared with the position before the displacement operation. However, the position of the non-holding work W2 changes in a range smaller than the holding space. Therefore, each value of the work displacement amount (XD12, YD12, ZD12) of the non-holdable work W1 included in the second work displacement amount WD2 observed by the displacement amount observation unit 64 indicates a value close to “0; zero”. However, each value of the work displacement amount (XD22, YD22, ZD22) of the non-retainable work W2 indicates a value corresponding to the displacement of the non-retainable work W2.

  When the non-holdable work W2 is displaced in a range smaller than the holding space by the displacement operation of the robot 2 based on the action pattern A2 using the first method, as shown in FIG. 7, the state of the robot 2 is changed to the state S32 ( The state variables ([Delta] X, [Delta] Y, [Delta] Z, p, d) in the state after the displacement operation are (0, 0, 0, 1, 0). In the example shown in FIG. 7, although the work W2 that cannot be held is displaced by the displacement operation of the robot 2 based on the action pattern A2, a holding space is not secured around the work W2, and the work W2 cannot be held. . Therefore, the state variable “d” is “0”, which indicates that the holding space for the claw portion 261 is not secured around the work W2.

  In addition, the displacement amount observing unit 64 performs the non-holding work when the displacement operation of the robot 2 based on the action pattern A3 using the first method is performed based on the reference image data GDS and the third image data GD3. The third work displacement WD3 representing the displacement of W1 and W2 in the container CN is observed. The third work displacement amount WD3 includes a work displacement amount (XD13, YD13, ZD13) of the non-holdable work W1 and a work displacement amount (XD23, YD23, ZD23) of the non-holdable work W2. In the work displacement amount of the non-holdable work W1, “XD13” is the X coordinate value “X1” in the three-dimensional position information of the non-holdable work W1 included in the reference image data GDS, and the non-holdable work included in the third image data GD3. The difference from the X coordinate value “X13” in the three-dimensional position information of W1 is shown. Further, in the work displacement amount of the non-holdable work W1, “YD13” is the Y coordinate value “Y1” in the three-dimensional position information of the non-holdable work W1 included in the reference image data GDS and the hold included in the third image data GD3. The difference from the Y coordinate value “Y13” in the three-dimensional position information of the unacceptable work W1 is shown. Further, in the work displacement amount of the non-holdable work W1, “ZD13” is the Z coordinate value “Z1” in the three-dimensional position information of the non-holdable work W1 included in the reference image data GDS, and the hold included in the third image data GD3. The difference from the Z coordinate value “Z13” in the three-dimensional position information of the unacceptable work W1 is shown.

  Similarly, in the work displacement amount of the non-holdable work W2, “XD23” is included in the X coordinate value “X2” in the three-dimensional position information of the non-holdable work W2 included in the reference image data GDS, and in the third image data GD3. The difference from the X coordinate value “X23” in the three-dimensional position information of the work W2 that cannot be held is shown. Further, “YD23” in the work displacement amount of the non-holdable work W2 is the Y coordinate value “Y2” in the three-dimensional position information of the non-holdable work W2 included in the reference image data GDS, and the hold included in the third image data GD3. The difference from the Y coordinate value “Y23” in the three-dimensional position information of the unacceptable work W2 is shown. Further, in the work displacement amount of the non-holdable work W2, “ZD23” is the Z coordinate value “Z2” in the three-dimensional position information of the non-holdable work W2 included in the reference image data GDS and the hold included in the third image data GD3. The difference from the Z coordinate value “Z23” in the three-dimensional position information of the unacceptable work W2 is shown.

  As is clear from the comparison between the reference image GS and the third image G3, the positions of the non-holdable works W1 and W2 after the displacement operation of the robot 2 based on the action pattern A3 using the first method are the positions before the displacement operation. There is hardly any change compared to. Therefore, each value of the work displacement amount (XD13, YD13, ZD13) of the non-holdable work W1 included in the third work displacement amount WD3 observed by the displacement amount observation unit 64 and the work displacement amount of the non-holdable work W2 ( Each value of XD23, YD23, ZD23) indicates a value close to "0; zero".

  When the displacement operation of the robot 2 based on the action pattern A3 using the first method is executed, as shown in FIG. 7, the state variable (the state after the displacement operation) when the state of the robot 2 is the state S33 (the state after the displacement operation) ΔX, ΔY, ΔZ, p, d) are set to (0, 0, 0, 1, 0). In the example shown in FIG. 7, the work W2 that cannot be held is hardly displaced by the displacement operation of the robot 2 based on the action pattern A3, and the work W2 cannot be held because the holding space is not secured around the work W2. It is. Therefore, the state variable “d” is “0”, which indicates that the holding space for the claw portion 261 is not secured around the work W2.

  When the displacement amount of the non-holdable works W1 and W2 is observed by the displacement amount observation unit 64, the reward setting unit 631 of the learning unit 63 sets at least one of the non-holdable works W1 and W2 (the non-holdable work W2). ) Is determined to be equal to or more than (threshold value WDT × 0.5) (step a8 in FIG. 10). Further, the reward setting unit 631 determines whether or not the work displacement amount of the non-holdable work W2 is equal to or larger than the threshold value WDT (step a9 in FIG. 10). The reward setting unit 631 sets the first value R1 (for example, “100”) for the behavior pattern (the behavior A1 in FIG. 7) of the robot 2 in which the work displacement amount of the non-holdable work W2 is equal to or larger than the predetermined threshold WDT. (Step a10 in FIG. 10). The reward setting unit 631 sets the first movement pattern (behavior A2 in FIG. 7) of the robot 2 in which the work displacement amount of the non-holdable work W2 is equal to or more than (threshold value WDT × 0.5) and less than the threshold value WDT. A reward R of a second value R2 (for example, “10”) smaller than the value R1 is given (step a15 in FIG. 10). The reward setting unit 631 determines that the action pattern of the robot 2 (action A3 in FIG. 7) in which the work displacement amount of the non-holdable work W2 is less than (threshold value WDT × 0.5) is smaller than the second value R2. A reward R of a small third value R3 (for example, “0: zero”) is given (step a14 in FIG. 10).

  Next, the value function updating unit 632 of the learning unit 63 updates the value function defining the value Q (s, a) of the behavior pattern of the robot 2 by using the above-mentioned equation (1) (FIG. 10). Steps a11 and a16).

  Each time the value function is updated by the value function updating unit 632, the learning unit 63 uses the first method to learn the behavior result of the robot 2 in the displacement operation of displacing the non-holdable work W2 by using the learning result information JH1 ( 9) is generated. The learning result information JH1 generated by the learning unit 63 is stored in the storage unit 8. The learning result information JH1 is, for example, information in which displacement method information J11, reference image data information J12, action pattern information J13, work displacement amount information J14, and reward information J15 are associated with each other. The displacement method information J11 is information indicating the displacement method used at the time of the displacement operation of the robot 2. The reference image data information J12 is information representing the reference image data GDS that the determination unit 7 referred to when determining whether or not there was a work that could not be held. The behavior pattern information J13 is information representing a behavior pattern of the robot 2 observed by the behavior observation unit 62 at the time of the displacement operation of the robot 2, and includes a behavior element that defines the behavior pattern. The work displacement amount information J14 is information representing the work displacement amount of the non-holdable work observed by the displacement amount observation unit 64 during the displacement operation of the robot 2. The reward information J15 is information representing the reward R set by the reward setting unit 631 for the behavior pattern of the robot 2 observed by the behavior observation unit 62.

  In the learning result information JH1 illustrated in FIG. 9, the first method (displacement method information J11) is applied to a work that cannot be held in an arrangement state corresponding to the reference image data GDS represented by the reference image data information J12. This shows that the displacement operation of the robot 2 based on the used action patterns A1, A2, and A3 (action pattern information J13) has been executed. In the behavior pattern A1 using the first method, the work displacement amount WD1 of the work that cannot be held becomes equal to or larger than the threshold value WDT (work displacement amount information J14), and the reward R (reward information J15) of the first value R1 (= 100) is obtained. ) Is given. That is, the learning unit 63 uses the behavior pattern A1 using the first method as the optimal behavior pattern of the robot 2 for displacing the work that cannot be held in the arrangement state corresponding to the reference image data GDS so that the holding space can be secured. You have learned. Referring to FIG. 7, the learning unit 63 cannot hold the tip of one work W3 held by the claw 261 by analyzing each action element that defines the action pattern A1 using the first method. If the hand portion 26 is moved to a position (contact position CP) where the hand W is brought into contact with the work W2 (movement locus MT), the work W2 that cannot be held can be displaced so that a holding space can be secured. To learn. In addition, the learning unit 63 has learned that the behavior patterns A2 and A3 using the first method are behavior patterns that cannot secure a holding space around the non-holdable work.

  When the learning unit 63 recognizes the behavior pattern to which the reward R of the first value R1 (= 100) has been given, that is, the behavior pattern in which the non-holdable work is displaced so that the holding space is secured. To end. In the example illustrated in FIG. 9, the learning unit 63 ends the learning process when recognizing the behavior pattern A1 using the first method, to which the reward R of the first value R1 (= 100) has been given. As described above, when the displacement operation of the robot 2 based on the action pattern provided with the reward R of the first value R1 (= 100) is performed, the holding space is secured around the non-holdable work, and The holding by the claw portion 261 becomes possible. Therefore, after the holding space is secured around the non-holdable work, the action determining unit 9 reads out the above-described existing action pattern from the storage unit 8 to determine the behavior pattern of the robot 2 with respect to the work for which the holding space is secured. Is determined (step a12 in FIG. 10), and the determined action pattern is output to the control device 4 (step a13 in FIG. 10). Thereby, under the control of the control device 4, the robot 2 executes a continuous production operation of taking out the work with the holding space secured from the container CN by the hand unit 26 and placing the taken out work on the pallet PL. I do.

  On the other hand, an action pattern in which a reward R of the second value R2 (= 10) or the third value R3 (= 0) is given, that is, an action pattern that does not lead to securing a holding space around a non-holdable work. When learning is performed, the learning unit 63 determines whether the number of times of the learning process has reached a predetermined number of times of learning (step a17 in FIG. 10). When the learning unit 63 repeats the recognition of the behavior pattern to which the reward R of the second value R2 (= 10) or the third value R3 (= 0) has been given, and reaches a predetermined number of times of learning, the learning unit 63 determines that a holding space cannot be secured around the non-holding work, and outputs the work holding non-holding information (step a18 in FIG. 10). The work holding impossible information is information indicating that the holding of the work by the claw part 261 of the hand part 26 is impossible. When the learning unit 63 outputs the work holding impossible information, the one work held by the claw unit 261 is placed on the pallet PL when the displacement operation of the robot 2 is performed based on the behavior pattern using the first method. After the placing operation is performed, the production operation of the robot 2 is interrupted. When the production operation of the robot 2 is interrupted, the worker checks the accommodation state of the work in the container CN, and performs a measure such as moving the work which is assumed to be impossible to be held by the hand unit 26. Just do it.

  The learning result information JH1 representing the current learning result generated by the learning unit 63 is referred to at the time of performing the displacement operation of the robot 2 from the next time. For example, it is assumed that learning result information JH1 in which an action pattern to which a reward R of a first value R1 (= 100) is given is stored in the storage unit 8. When the determination unit 7 determines that there is a non-holdable work having the same or similar arrangement status as the arrangement status represented by the reference image data GDS registered in the learning result information JH1 stored in the storage unit 8, The learning process regarding the displacement operation by the learning unit 63 is omitted. In this case, the behavior determining unit 9 reads the behavior pattern provided with the reward R of the first value R1 (= 100) registered in the learning result information JH1 stored in the storage unit 8 to perform the displacement. The action pattern of the robot 2 at the time of executing the action is determined. The action determining unit 9 outputs the action pattern at the time of the displacement operation of the robot 2 read from the storage unit 8 to the control device 4. The control device 4 to which the behavior pattern at the time of the displacement operation is input controls the operation of the robot 2 based on the behavior pattern. Under the control of the control device 4, the robot 2 displaces the work that cannot be held such that a space for holding the claw portion 261 is secured around the robot.

  As described above, when the determination unit 7 determines that the work to be held next by the claw unit 261 of the hand unit 26 is a work that cannot be held, the learning unit 63 ensures that the holding space is secured. The behavior pattern of the robot 2 using the first method, which can displace the work that cannot be held, is learned. Thereby, the learning unit 63 can learn the optimal behavior pattern of the robot 2 using the first method, which enables the holding of the work that cannot be held by the claw unit 261 of the hand unit 26. . Then, in the next production operation of the robot 2, the action determining unit 9 sets the action given the reward R of the first value R1 (= 100) registered in the learning result information JH1 generated by the learning unit 63. The pattern is determined as an action pattern for enabling the work that cannot be held by the hand unit 26 to be held. When the robot 2 performs the displacement operation in accordance with this behavior pattern, a holding space for enabling the holding by the claw portion 261 is secured around the work that cannot be held by the claw portion 261 of the hand unit 26. Thus, the work can be held by the claw portion 261. Therefore, it is possible to avoid stopping the operation of the robot 2 due to the existence of the non-holdable work as much as possible, and the operation of taking out the work from the container CN by the hand unit 26 can be continued.

  In the displacement operation of the robot 2 based on the behavior pattern using the first method, the behavior pattern of the robot 2 is not limited to the example illustrated in FIG. You may. FIG. 11 is a diagram for explaining a modification of the behavior pattern of the robot 2 in the displacement operation of the first example.

  In the example illustrated in FIG. 11, when one work W3 in the container CN is held by the claw portion 261 of the hand unit 26, no holding space is secured around the works W1 and W2, and the work W1 and W2 are The work cannot be held. Therefore, the state variables (ΔX, ΔY, ΔZ, p, d) when the state of the robot 2 is the state S3 are set to (0, 0, 0, 1, 0). In the example shown in FIG. 11, similarly to the example shown in FIG. 7, the non-holdable work W1 is disposed close to the inner surface of the container CN, and the non-holdable work W2 is not held on the side of the non-holdable work W1. It is arranged close to the work W1. Therefore, a holding space is not secured around the non-holding works W1 and W2.

  As an action pattern of the robot 2 using the first method, an action A4 illustrated in FIG. 11 is a non-holding work in which the tip of one work W3 held by the claw portion 261 is arranged close to the inner surface of the container CN. This is an action pattern in which the hand unit 26 moves (movement trajectory MT) in a state where the hand unit 26 is in contact with one longitudinal end surface of W1 (contact position CP). In the action A4, the hand unit 26 moves in the parallel direction of the non-holdable works W1 and W2 so that the contact position CP at the tip of one work W3 changes from the non-holdable work W1 to the non-holdable work W2 during the movement. It moves in a direction obliquely away from the non-holdable work W1 (movement locus MT).

  When the displacement operation of the robot 2 based on the behavior pattern A4 (behavior A4) using the first method is executed, it is possible to displace both of the work W1 and the work W2 that cannot be held, and at least hold the work. The impossible work W2 can be displaced to such an extent that a holding space is secured.

  The reward setting unit 631 displaces the plurality of non-holdable works W1 and W2 as described above and displaces at least one non-holdable work W2 to such an extent that a holding space is secured. A reward R having a value larger than the value R1 (= 100) may be given.

<About the second example of the displacement operation>
A second example of the displacement operation will be described with reference to FIGS. FIG. 12 is a diagram for explaining a second example of the displacement operation for displacing the non-holdable work. FIG. 13 is a diagram illustrating the learning result information JH2 generated by the learning unit 63 in the displacement operation of the second example. FIG. 14 is a flowchart illustrating the operation of the machine learning device 5 regarding the displacement operation of the second example. The displacement operation of the second example is a displacement operation of the robot 2 based on an action pattern using the second method.

  The state observing unit 61 determines that the state of the robot 2 has been shifted from the state S2 to the state S3 based on the state variables (ΔX, ΔY, ΔZ, p, d) that change each time the state of the robot 2 is shifted. Is observed (step b1 in FIG. 14). When the state of the robot 2 is shifted from the state S2 to the state S3, the determination unit 7 acquires the reference image data output from the imaging device 3 (Step b2 in FIG. 14). The determination unit 7 recognizes the accommodation state of each work in the container CN based on the reference image data, and determines whether or not the work to be the next holding candidate is a work that cannot be held (step b3 in FIG. 14). ).

  When the determination unit 7 determines that the work to be the next holding candidate is not a work that cannot be held, the state variables (ΔX, ΔY, ΔZ, p, d) when the state of the robot 2 is the state S3 are determined. (0, 0, 0, 1, 1). In this case, the action determining unit 9 reads an existing action pattern for shifting from the state S3 to the state S4 from the storage unit 8 and outputs the pattern to the control device 4. The control device 4 to which the existing behavior pattern has been input controls the operation of the robot 2 based on the existing behavior pattern. Under the control of the control device 4, the robot 2 executes a take-out operation of taking out one work held by the claw portion 261 from the container CN (step b5 in FIG. 14).

  On the other hand, when the determination unit 7 determines that the work to be the next holding candidate is a work that cannot be held, as shown in FIG. 12, the state variable (ΔX, ΔY, ΔZ, p, d) are set to (0, 0, 0, 1, 0). In the example illustrated in FIG. 12, when one work W3 in the container CN is held by the claw 261 of the hand unit 26, no holding space is secured around the works W1 and W2, and the work W1 and W2 are The work cannot be held. In the example illustrated in FIG. 12, similarly to the example illustrated in FIG. 7, the non-retainable work W1 is disposed close to the inner surface of the container CN, and the non-retainable work W2 is non-retainable on the side of the non-retainable work W1. It is arranged close to the work W1. Therefore, a holding space is not secured around the non-holding works W1 and W2.

  When the determination unit 7 determines that the work W1 or W2 that is the next holding candidate is a work that cannot be held, the claw unit 261 holds at least one of the work W1 and the work W2 that is the work that cannot be held. A displacement operation is performed to displace so that a space is secured around. In the second example of the displacement operation, as shown in FIG. 12, the robot 2 displaces the work that cannot be held by the displacement operation based on the behavior pattern using the second method. Note that, as described above, the second method is such that, after placing one work W3 held by the claw portion 261 on the pallet PL, the hand portion 26 moves while holding the container CN by the claw portion 261. This is a displacement method for displacing the non-holdable work in accordance with the movement of the container CN.

  The behavior observation unit 62 observes the behavior pattern of the robot 2 using the second technique (Step b4 in FIG. 14). In the example illustrated in FIG. 12, three types of action patterns of the action A1, the action A2, and the action A3 are illustrated as the action patterns of the robot 2 using the second technique. The action A1 is a predetermined movement in a direction in which the hand unit 26 holds the container CN by the claw unit 261 and inclines so as to approach the non-holdable works W1 and W2 with respect to the parallel direction of the non-holdable works W1 and W2. This is an action pattern that moves in a speed pattern (movement locus MT). The action A2 is the same as the action A1, except that the acceleration of the hand unit 26 when moving is slower than the action A1 and the moving speed pattern is different. The action A3 is a predetermined movement in a direction in which the hand unit 26 holds the container CN by the claw unit 261 and is inclined away from the non-holdable works W1 and W2 with respect to the parallel direction of the non-holdable works W1 and W2. This is an action pattern that moves in a speed pattern (movement locus MT).

  The behavior elements that define the behavior pattern of the robot 2 observed by the behavior observation unit 62 include the grip angle θ, the grip position HP, the rotation angle β1 and the rotation speed pattern on the first axis 2A shown in FIG. Rotation angle β2 and rotation speed pattern on second shaft 2B, rotation angle β3 and rotation speed pattern on third shaft 2C, rotation angle β4 and rotation speed pattern on fourth shaft 2D, rotation angle β5 and rotation speed on fifth shaft 2E The pattern, the rotation angle β6 on the sixth axis 2F, and the rotation speed pattern are included. Each of the action elements shown in FIG. 5 is an element that determines the holding position where the claw 261 holds the container CN in the action pattern of the robot 2 using the second method, and determines the movement trajectory MT of the hand unit 26. This is an element that determines the moving speed pattern of the hand unit 26.

  When the displacement operation based on the action pattern using the second method is completed, the displacement amount observation unit 64 acquires the image data after the displacement operation output from the imaging device 3 (step b6 in FIG. 14). The image data after the displacement operation is image data including three-dimensional position information on the non-holdable works W1 and W2 after being displaced by the action pattern of the robot 2 using the second method. The displacement amount observing section 64 is image data before the displacing operation by the robot 2, the reference image data which the judging section 7 refers to when judging the existence of the non-holdable works W <b> 1, W <b> 2, and after the displacing operation by the robot 2. The amount of work displacement of the non-holdable works W1 and W2 is observed based on the image data in (1) (step b7 in FIG. 14).

  In the example shown in FIG. 12, after the displacement operation of the robot 2 based on the action pattern A1 using the second method, both the work that cannot be held W1 and the work that cannot be held W2 are displaced, and at least the work W2 that cannot be held is held. It is displaced to the extent that space is secured. When at least the non-holdable work W2 is displaced by the displacement operation of the robot 2 based on the action pattern A1 using the second method so that a holding space can be secured, the state of the robot 2 is changed to the state S31 (state after the displacement operation). , The state variables (ΔX, ΔY, ΔZ, p, d) are set to (0, 0, 0, 1, 1).

  In the example shown in FIG. 12, after the displacement operation of the robot 2 based on the action pattern A2 using the second method, the non-holdable work W1 is hardly displaced, but the non-holdable work W2 is smaller than the holding space. It is displaced in the range. Since the non-holdable work W2 is displaced by the displacement operation of the robot 2 based on the action pattern A2 using the second method, but is displaced in a range smaller than the holding space, the state of the robot 2 is changed to the state S32 (after the displacement operation). State variable (ΔX, ΔY, ΔZ, p, d) is (0, 0, 0, 1, 0).

  Further, in the example shown in FIG. 12, after the displacement operation of the robot 2 based on the behavior pattern A3 using the second method, both the non-holdable work W1 and the non-holdable work W2 hardly displace. Due to the displacement operation of the robot 2 based on the action pattern A3 using the second method, both the non-holdable work W1 and the non-holdable work W2 are hardly displaced, and no holding space is secured around them. When the state of the robot 2 is the state S33 (state after the displacement operation), the state variables (ΔX, ΔY, ΔZ, p, d) are set to (0, 0, 0, 1, 0).

  When the displacement amount of the non-holdable works W1 and W2 is observed by the displacement amount observation unit 64, the reward setting unit 631 of the learning unit 63 sets at least one of the non-holdable works W1 and W2 (the non-holdable work W2). ) Is determined to be equal to or more than (threshold value WDT × 0.5) (step b8 in FIG. 14). Further, the reward setting unit 631 determines whether or not the work displacement amount of the non-holdable work W2 is equal to or larger than the threshold value WDT (step b9 in FIG. 14). The reward setting unit 631 sets the first value R1 (for example, “100”) for the behavior pattern (the behavior A1 in FIG. 12) of the robot 2 in which the work displacement amount of the non-holdable work W2 is equal to or more than the predetermined threshold WDT. (Step b10 in FIG. 14). The reward setting unit 631 sets the first movement pattern (the movement A2 in FIG. 12) of the robot 2 in which the work displacement amount of the non-holdable work W2 is equal to or more than (threshold value WDT × 0.5) and less than the threshold value WDT. A reward R of a second value R2 (for example, “10”) smaller than the value R1 is given (step b15 in FIG. 14). The reward setting unit 631 determines that the action pattern (action A3 in FIG. 12) of the robot 2 in which the work displacement amount of the non-holdable work W2 is less than (threshold value WDT × 0.5) is smaller than the second value R2. A reward R of a small third value R3 (for example, “0: zero”) is given (step b14 in FIG. 14).

  Next, the value function updating unit 632 of the learning unit 63 updates the value function defining the value Q (s, a) of the behavior pattern of the robot 2 using the updating expression of the above expression (1) (FIG. 14). Steps b11 and b16).

  Each time the value function is updated by the value function updating unit 632, the learning unit 63 uses the second method to learn the learning result information JH2 () representing the learning result of the behavior pattern of the robot 2 in the displacement operation of displacing the non-holdable work W2. 13) is generated. The learning result information JH2 generated by the learning unit 63 is stored in the storage unit 8. Like the learning result information JH1 shown in FIG. 9 described above, the learning result information JH2 includes, for example, displacement method information J21, reference image data information J22, action pattern information J23, work displacement amount information J24, and reward information. J25 is associated information.

  In the learning result information JH2 illustrated in FIG. 13, the second method (displacement method information J21) is applied to a work that cannot be held in an arrangement state corresponding to the reference image data GDS represented by the reference image data information J22. This shows that the displacement operation of the robot 2 based on the used action patterns A1, A2, and A3 (action pattern information J23) has been performed. In the behavior pattern A1 using the second technique, the work displacement amount WD1 of the work that cannot be held becomes equal to or larger than the threshold value WDT (work displacement amount information J24), and the reward R (reward information J25) of the first value R1 (= 100) ) Is given. In other words, the learning unit 63 uses the behavior pattern A1 using the second method as the optimal behavior pattern of the robot 2 for displacing the work that cannot be held in the arrangement state corresponding to the reference image data GDS so that the holding space can be secured. You have learned. Referring to FIG. 12, the learning unit 63 holds which position of the container CN by the claw unit 261 by analyzing each action element that defines the action pattern A1 using the second method (holding position). ), Learns in what direction and in what kind of moving speed pattern (moving locus MT) the hand unit 26 can displace the non-holdable work W2 so that a holding space can be secured. In addition, the learning unit 63 has learned that the behavior patterns A2 and A3 using the second method are behavior patterns that do not lead to securing a holding space around the work that cannot be held.

  When the learning unit 63 recognizes the behavior pattern to which the reward R of the first value R1 (= 100) has been given, that is, the behavior pattern in which the non-holdable work is displaced so that the holding space is secured. To end. In the example illustrated in FIG. 13, the learning unit 63 ends the learning process when recognizing the behavior pattern A1 using the second method, to which the reward R of the first value R1 (= 100) has been given. As described above, when the displacement operation of the robot 2 based on the action pattern provided with the reward R of the first value R1 (= 100) is performed, the holding space is secured around the non-holdable work, and The holding by the claw portion 261 becomes possible. Therefore, after the holding space is secured around the non-holdable work, the action determining unit 9 reads out the above-described existing action pattern from the storage unit 8 to determine the behavior pattern of the robot 2 with respect to the work for which the holding space is secured. Is determined (step b12 in FIG. 14), and the determined action pattern is output to the control device 4 (step b13 in FIG. 14). Thereby, under the control of the control device 4, the robot 2 executes a continuous production operation of taking out the work with the holding space secured from the container CN by the hand unit 26 and placing the taken out work on the pallet PL. I do.

  On the other hand, an action pattern in which a reward R of the second value R2 (= 10) or the third value R3 (= 0) is given, that is, an action pattern that does not lead to securing a holding space around a non-holdable work. When learning is performed, the learning unit 63 determines whether the number of times of the learning process has reached a predetermined number of times of learning (step b17 in FIG. 14). When the learning unit 63 repeats the recognition of the behavior pattern to which the reward R of the second value R2 (= 10) or the third value R3 (= 0) has been given, and reaches a predetermined number of times of learning, the learning unit 63 determines that a holding space cannot be secured around the non-holding work, and outputs the work holding non-holding information (step b18 in FIG. 14). When the learning unit 63 outputs the work holding impossible information, when the displacement operation of the robot 2 based on the action pattern using the second method is performed, the holding of the container CN by the claw unit 261 is released, and then the movement of the robot 2 is stopped. Production operation is interrupted. When the production operation of the robot 2 is interrupted, the worker checks the accommodation state of the work in the container CN, and performs a measure such as moving the work which is assumed to be impossible to be held by the hand unit 26. Just do it.

  The learning result information JH2 representing the current learning result generated by the learning unit 63 is referred to at the time of executing the displacement operation of the robot 2 from the next time. For example, it is assumed that learning result information JH2 in which an action pattern provided with a reward R of a first value R1 (= 100) is stored in the storage unit 8. When the determination unit 7 determines that there is a non-holdable work having the same or similar layout status as the layout status represented by the reference image data GDS registered in the learning result information JH2 stored in the storage unit 8, The learning process regarding the displacement operation by the learning unit 63 is omitted. In this case, the behavior determining unit 9 reads the behavior pattern provided with the reward R of the first value R1 (= 100) registered in the learning result information JH2 stored in the storage unit 8, thereby performing the displacement. The action pattern of the robot 2 at the time of executing the action is determined. The action determining unit 9 outputs the action pattern at the time of the displacement operation of the robot 2 read from the storage unit 8 to the control device 4. The control device 4 to which the behavior pattern at the time of the displacement operation is input controls the operation of the robot 2 based on the behavior pattern. Under the control of the control device 4, the robot 2 displaces the work that cannot be held such that a space for holding the claw portion 261 is secured around the robot.

  As described above, when the determination unit 7 determines that the work to be held next by the claw unit 261 of the hand unit 26 is a work that cannot be held, the learning unit 63 ensures that the holding space is secured. Then, the behavior pattern of the robot 2 using the second method, which can displace the work that cannot be held, is learned. Accordingly, the learning unit 63 can learn the optimal behavior pattern of the robot 2 using the second method, which enables the holding of the work that cannot be held by the claw unit 261 of the hand unit 26. . Then, in the next production operation of the robot 2, the action determining unit 9 sets the action in which the reward R of the first value R1 (= 100) registered in the learning result information JH2 generated by the learning unit 63 is given. The pattern is determined as an action pattern for enabling the work that cannot be held by the hand unit 26 to be held. When the robot 2 performs the displacement operation in accordance with this behavior pattern, a holding space for enabling the holding by the claw portion 261 is secured around the work that cannot be held by the claw portion 261 of the hand unit 26. Thus, the work can be held by the claw portion 261. Therefore, it is possible to avoid stopping the operation of the robot 2 due to the existence of the non-holdable work as much as possible, and the operation of taking out the work from the container CN by the hand unit 26 can be continued.

<About the third example of the displacement operation>
A third example of the displacement operation will be described with reference to FIGS. FIG. 15 is a diagram illustrating the learning result information JH3 generated by the learning unit 63 in the displacement operation of the third example. FIG. 16 is a flowchart illustrating the operation of the machine learning device 5 regarding the displacement operation of the third example. In the third example, the machine learning device 5 learns an optimal behavior pattern for displacing the non-retainable work so as to secure a holding space while switching the displacement method for displacing the non-retainable work.

  The type, number, and trial order of the displacement methods tried in the displacement operation of the robot 2 according to the third example are not particularly limited. The learning unit 63 presets the type, number, and trial order of the displacement methods. Hereinafter, in the displacement operation of the robot 2, the trial order of the displacement method is set to the order of the third method, the fourth method, the second method, the fifth method, and the sixth method illustrated in FIG. Will be described.

  The state observing unit 61 determines that the state of the robot 2 has been shifted from the state S2 to the state S3 based on the state variables (ΔX, ΔY, ΔZ, p, d) that change each time the state of the robot 2 is shifted. Is observed (step c1 in FIG. 16). When the state of the robot 2 is shifted from the state S2 to the state S3, the determination unit 7 acquires the reference image data output from the imaging device 3 (Step c2 in FIG. 16). The determination unit 7 recognizes the state of accommodation of each work in the container CN based on the reference image data, and determines whether the next work as a candidate for holding is a work that cannot be held (step c3 in FIG. 16). ).

  If the determination unit 7 determines that the work to be the next holding candidate is not a work that cannot be held, the state variables (ΔX, ΔY, ΔZ, p, d) when the state of the robot 2 is the state S3 are determined. (0, 0, 0, 1, 1). In this case, the action determining unit 9 reads an existing action pattern for shifting from the state S3 to the state S4 from the storage unit 8 and outputs the pattern to the control device 4. The control device 4 to which the existing behavior pattern has been input controls the operation of the robot 2 based on the existing behavior pattern. Under the control of the control device 4, the robot 2 executes a take-out operation of taking out one work held by the claw portion 261 from the container CN (step c5 in FIG. 16).

  On the other hand, when the determination unit 7 determines that the work to be the next holding candidate is a work that cannot be held, the state variables (ΔX, ΔY, ΔZ, p, d) when the state of the robot 2 is the state S3. ) Is (0,0,0,1,0).

  When the determination unit 7 determines that the work to be the next holding candidate is a work that cannot be held, a displacement operation is performed to displace the work that cannot be held so that a space for holding the claw 261 is secured around the work. In the third example of the displacement operation, the robot 2 first performs a trial of displacing the non-holdable work by the displacement operation based on the action pattern using the third method. In the third method, as described above, after one work held by the claw portion 261 is placed on the pallet PL, the hand portion 26 moves while the claw portion 261 is in contact with the non-holdable work. This is a displacement method for displacing the work that cannot be held.

  The behavior observation unit 62 observes the behavior pattern of the robot 2 using the third technique (Step c4 in FIG. 16). When the displacement operation based on the action pattern using the third method is completed, the displacement amount observation unit 64 acquires the image data after the displacement operation, which is output from the imaging device 3 (step c6 in FIG. 16). The image data after the displacement operation is image data including three-dimensional position information on the non-holdable work that has been displaced by the behavior pattern of the robot 2 using the third method. The displacement amount observing section 64 is image data before the displacing operation by the robot 2, reference image data to be referred to when the judging section 7 judges the presence or absence of a work that cannot be held, and image data after the displacing operation by the robot 2. Based on the above, the work displacement amount of the work that cannot be held is observed (step c7 in FIG. 16).

  When the displacement amount of the non-holdable work is observed by the displacement amount observation unit 64, the reward setting unit 631 of the learning unit 63 determines whether the work displacement amount of the non-holdable work is equal to or more than (threshold WDT × 0.5). Is determined (step c8 in FIG. 16). Further, the reward setting unit 631 determines whether or not the work displacement amount of the work that cannot be held is equal to or larger than the threshold value WDT (step c9 in FIG. 16). The reward setting unit 631 gives the reward R of the first value R1 (for example, “100”) to the behavior pattern of the robot 2 in which the amount of work displacement of the work that cannot be held is equal to or larger than the predetermined threshold WDT (FIG. 16). Step c10). The reward setting unit 631 sets the second value smaller than the first value R1 for the behavior pattern of the robot 2 in which the work displacement amount of the non-holdable work is equal to or more than (threshold value WDT × 0.5) and less than the threshold value WDT. A reward R of a value R2 (for example, “10”) is given (step c15 in FIG. 16). For a behavior pattern of the robot 2 in which the work displacement amount of the non-holdable work is less than (threshold value WDT × 0.5), the reward setting unit 631 sets a third value R3 (for example, smaller than the second value R2). A reward R of “0: zero” is given (step c14 in FIG. 16).

  Next, the value function updating unit 632 of the learning unit 63 updates the value function defining the value Q (s, a) of the behavior pattern of the robot 2 using the updating expression of the above expression (1) (FIG. 16). Steps c11 and c16).

  When a reward R of a third value R3 (= 0) is given to the behavior pattern using the third method, the learning unit 63 sets the reference number ( For example, “20”), it is determined whether or not the reward R of the third value R3 (= 0) is continuously given (step c17 in FIG. 16). When the reward R of the third value R3 (= 0) is given continuously for the reference number of times (= 20) to the action pattern using the third method, the learning unit 63 cannot hold the third method. It is determined that the displacement method has a low degree of appropriateness for displacing the work, and the displacement method is switched from the third method to the fourth method in the next trial order (step c18 in FIG. 16). In the fourth method, as described above, after one work held by the claw 261 is placed on the pallet PL, the work WS taken out from another container is held by the claw 261 and the hand 26 This is a displacement method for displacing the non-holdable work by moving the work WS in contact with the non-holdable work.

  The learning unit 63 determines whether the number of times of the learning process has reached the predetermined number of times of learning (step c19 in FIG. 16). If the number of times of learning has not reached the predetermined number of times of learning, the fourth unit switched in step c18. The learning process on the behavior pattern using the method is returned to step c4, and is performed in the same manner as in the case of the third method. On the other hand, when the predetermined number of times of learning has been reached, the learning unit 63 determines that it is not possible to secure a holding space around the work that cannot be held by the displacement method tried so far, and outputs work holding impossible information (FIG. 16 Step c20). When the learning section 63 outputs the work holding impossible information, the production operation of the robot 2 is interrupted. When the production operation of the robot 2 is interrupted, the worker checks the accommodation state of the work in the container CN, and performs a measure such as moving the work which is assumed to be impossible to be held by the hand unit 26. Just do it.

  Hereinafter, the description will be continued assuming that the displacement technique has been tried in the order of the fourth technique, the second technique, the fifth technique, and the sixth technique within the range of the predetermined number of times of learning.

  The reward R of the third value R3 (= 0) was given to the behavior pattern using the above-described third technique continuously for the reference number of times (= 20). Therefore, the learning unit 63 has determined that the third technique is a displacement technique having a low degree of appropriateness for displacing the non-holdable work. For the behavior pattern using the fourth method that has been tried next to the third method, the reward R of the third value R3 (= 0) is continuously given “19” times smaller than the reference number, It is assumed that the reward R of the second value R2 (= 10) has been given “1” times. In this case, although the learning unit 63 has a slightly higher degree of appropriateness to displace the non-holdable work in the fourth method than in the third method, the reward R of the first value R1 (= 100) is not given. Therefore, it is determined that the work that cannot be held has been displaced so that the holding space can be secured. Therefore, the learning unit 63 switches the displacement method from the fourth method to the second method in the next trial order. As described above, in the second method, as described above, after one work held by the claw portion 261 is placed on the pallet PL, the hand portion 26 moves while holding the container CN by the claw portion 261. This is a displacement method for displacing a non-holdable work according to the movement of the container CN.

  For the behavior pattern using the second method that has been tried next to the fourth method, the reward R of the third value R3 (= 0) is continuously given “15” times smaller than the reference number, Thereafter, it is assumed that the reward R of the first value R1 (= 100) has been given. In this case, the learning unit 63 determines that the holding space is secured around the non-holdable work when the reward R of the first value R1 (= 100) is given, and ends the learning process.

  When the displacement operation of the robot 2 based on the action pattern using the second method, in which the reward R of the first value R1 (= 100) is given, a holding space is secured around the non-holdable work, The work can be held by the claw portion 261. Therefore, after the holding space is secured around the non-holdable work, the action determining unit 9 reads out the above-described existing action pattern from the storage unit 8 to determine the behavior pattern of the robot 2 with respect to the work for which the holding space is secured. Is determined (step c12 in FIG. 16), and the determined action pattern is output to the control device 4 (step c13 in FIG. 16). Thereby, under the control of the control device 4, the robot 2 executes a continuous production operation of taking out the work with the holding space secured from the container CN by the hand unit 26 and placing the taken out work on the pallet PL. I do.

  As described above, since the reward R of the first value R1 (= 100) is given to the action pattern using the second technique, the learning unit 63 ends the learning process. However, in the behavior pattern using the second method, the number of times the reward R of the third value R3 (= 0) is given is not “0; Thus, a reward R of a third value R3 (= 0) is given. For this reason, the learning unit 63 uses the second method, which is optimal as a displacement method for displacing the work that cannot be held in the arrangement status that has been tried this time and is represented by the reference image data acquired by the determination unit 7. Judge that it is not a method. Therefore, when the determination unit 7 determines that there is an unretainable work having the same or similar placement status as the placement status tested this time, the learning unit 63 performs the fifth trial of the next trial rank with the second trial. A learning process related to the used behavior pattern is executed. In the fifth method, as described above, after one work held by the claw 261 is placed on the pallet PL, the dedicated jig JG is held by the claw 261 and the hand 26 is This is a displacement method in which the non-holdable work is displaced by moving in a state of contact with the non-holdable work.

  For the behavior pattern using the fifth method, the number of times the reward R of the third value R3 (= 0) is given is “0; zero”, and the reward of the first value R1 (= 100) is Let R be given. In this case, the learning unit 63 determines that the fifth technique is the optimal technique as the displacement technique for displacing the work that cannot be held in the arrangement status tried this time, and the reward R of the first value R1 (= 100). When the is given, the learning process ends. The learning unit 63 has determined that the fifth technique is the optimal technique as the displacement technique for displacing the work that cannot be held in the arrangement situation that has been tried this time, and is therefore set in the next trial order with respect to the fifth technique. For the sixth method, no trial was performed.

  The learning unit 63 generates learning result information JH3 (FIG. 15) representing a learning result of the behavior pattern of the robot 2 in the displacement operation of displacing the non-holdable work while switching the displacement method. The learning result information JH3 generated by the learning unit 63 is stored in the storage unit 8. The learning result information JH3 is, for example, information in which reference image data information J31, displacement technique information J32, reward information J33, consecutive zero reward information J34, and correction coefficient information J35 are associated with each other.

  The reference image data information J31 is information representing the reference image data GDS that the determination unit 7 refers to when determining whether there is a work that cannot be held. The displacement method information J32 is information representing the displacement method set by the learning unit 63. In the example illustrated in FIG. 15, the third technique, the fourth technique, the second technique, the fifth technique, and the sixth technique are registered as the displacement technique information J32 in the order of the trial order. The reward information J33 is information indicating the reward R set by the reward setting unit 631 for the behavior pattern of the robot 2 observed by the behavior observation unit 62. In the example shown in FIG. 15, a reward R of a third value R3 (= 0) is given to the third method, and a reward R of a second value R2 (= 10) is given to the fourth method. It is shown that the reward R of the first value R1 (= 100) was given to the second technique and the fifth technique. Since the trial has not been performed for the sixth method, “not performed” indicating that fact has been registered.

  The reward zero continuous count information J34 is provided with a reward R of a third value R3 (= 0) in a trial of the displacement operation of the robot 2 based on the action pattern using the displacement technique represented by the displacement technique information J32. This is information indicating the number of consecutive times. In the example illustrated in FIG. 15, the number of consecutive times that the reward R of the third value R3 (= 0) is given is “20” which is the same as the reference number in the third method, and is “19” in the fourth method. Yes, the second method is “15”, and the fifth method is “0”. As for the sixth method, “not performed” indicating that the trial has not been performed is registered.

  The correction coefficient information J35 is information indicating a correction coefficient ε which is an index of the appropriateness of displacing the non-holdable work in the displacement method represented by the displacement method information J32. The correction coefficient ε represented by the correction coefficient information J35 is based on the reference number “M” serving as a reference for the number of trials of each displacement method, and the third value R3 (= The learning unit 63 calculates the correction coefficient ε = (M−K) / M based on the number of consecutive times “K” to which the reward R of 0) is given. The correction coefficient ε increases as the number of consecutive times “K” in which the reward R of the third value R3 (= 0) is given is smaller. That is, as the correction coefficient ε increases, the degree of appropriateness for displacing the non-holdable work increases. In the example shown in FIG. 15, the correction coefficient ε is “0” in the third method, “0.05” in the fourth method, “0.25” in the second method, and “5” in the fifth method. It is shown to be “1”. As for the sixth method, “not performed” indicating that the trial has not been performed is registered. The correction coefficient ε calculated by the learning unit 63 is reflected in “ε” of the updating equation of the value Q (s, a) shown in the above equation (1). Note that “ε” in the updating equation of the value Q (s, a) shown in the above equation (1) is “ε = 1” until the correction coefficient ε is calculated by the learning unit 63.

  The learning result information JH3 generated by the learning unit 63 is referred to at the time of executing the displacement operation of the robot 2 from the next time. When the determination unit 7 determines that there is a non-holdable work having the same or similar arrangement status as the arrangement status represented by the reference image data GDS registered in the learning result information JH3 stored in the storage unit 8, The learning unit 63 refers to the reward information J33 and the correction coefficient information J35, and determines whether a new learning process is necessary. Specifically, the learning unit 63 multiplies the compensation R represented by the compensation information J33 by the modification coefficient ε represented by the modification coefficient information J35 for each displacement technique represented by the displacement technique information J32. , And the corrected reward value (corresponding to “εR (s, a)” in equation (1)) is calculated. The corrected reward value is a value obtained by multiplying the reward R by a correction coefficient ε, which is an index of the appropriateness for displacing the work that cannot be held, and can be said to be a reward that takes into account the appropriateness given for each displacement method.

  The learning unit 63 recognizes, based on the learning result information JH3, a displacement technique in which the corrected reward value is the same as the reward R of the first value R1 (= 100), that is, the first reward value R1 (= 100). When the reward R is given and the displacement method with the modification coefficient ε of “1” is recognized (the fifth method in FIG. 15 is equivalent), it is determined that the displacement method is the optimal method for displacing the work that cannot be held. Judge, and the learning process is omitted. In this case, the action determining unit 9 determines the action pattern using the fifth technique registered in the learning result information JH3 stored in the storage unit 8 as the action pattern of the robot 2 at the time of performing the displacement operation. The action determining unit 9 outputs the determined action pattern using the fifth technique to the control device 4. The control device 4 to which the behavior pattern at the time of the displacement operation is input controls the operation of the robot 2 based on the behavior pattern. Under the control of the control device 4, the robot 2 displaces the work that cannot be held such that a space for holding the claw portion 261 is secured around the robot.

  In addition, when the reward R of the first value R1 (= 100) is given and the displacement method with the modification coefficient ε of “1” is recognized (the fifth method in FIG. 15 is equivalent), the learning process is performed as described above. May be omitted, but when the number of trials of the recognized method is small, robustness can be improved by starting the learning process again. When the number of trials of the recognized method is small, the learning unit 63 determines the displacement method as a candidate of the optimal method for displacing the non-holdable work (optimal candidate method: the fifth method in the example of FIG. 15). It is determined that there is, and the learning process is executed again for the optimal candidate method. When executing the learning process again for the optimal candidate method, the learning unit 63 sets the number of trials in advance, and is given a reward R of the first value R1 (= 100) of the set number of trials. The correction coefficient ε may be calculated by evaluating at the ratio of the number of times of the correction. For example, the number of trials of the optimal candidate method is set to “three times”, and the determination unit 7 determines that there is a non-holdable work having the same or similar arrangement state as the arrangement state represented by the reference image data GDS. When it is determined, the learning unit 63 selects the optimal candidate method as the displacement method for displacing the non-holdable work each time, and corrects according to the number of times the reward R of the first value R1 (= 100) is given. Calculate the coefficient ε. The learning unit 63 trials the optimal candidate method “3 times”, and sets the modification coefficient ε to “1” when a reward R of the first value R1 (= 100) is given to all “3 times”. Then, it is determined that the displacement method that has been regarded as the optimal candidate method is the optimal method for displacing the work that cannot be held.

  Further, once it is determined that a certain displacement technique is the optimal technique for displacing the non-holdable work, for example, as in the above-described fifth technique in FIG. 15, the learning result information JH3 stored in the storage unit 8 is obtained. When the determination unit 7 determines that the non-holdable work having the same or similar arrangement status as the arrangement status represented by the reference image data GDS registered in the learning unit 63 is determined, the learning unit 63 performs the learning process illustrated in FIG. May be repeatedly executed. In this case, as the displacement technique to be tried first in step c4 in FIG. 16, a technique once determined to be the optimal technique like the fifth technique in FIG. 15 may be used. If the reward R of the first value R1 (= 100) is not given, another displacement method such as the sixth method in FIG. 15 may be tried. That is, the learning unit 63 may allow a change in the method determined to be optimal.

  In the case where the determination unit 7 determines that there is a work that cannot be held in an arrangement state that is significantly different from the arrangement state represented by the reference image data GDS registered in the learning result information JH3 stored in the storage unit 8, The learning unit 63 trials the displacement operation of the robot 2 based on the behavior pattern using each displacement method according to the trial order set in advance, and learns the behavior pattern.

  As described above, when the determination unit 7 determines that the work to be held next by the claw unit 261 of the hand unit 26 is a work that cannot be held, the learning unit 63 ensures that the holding space is secured. In addition to learning an optimal displacement method capable of displacing a workpiece that cannot be held, a behavior pattern of the robot 2 using the displacement method is learned. As a result, the learning unit 63 can learn the optimal behavior pattern of the robot 2 using the optimal displacement method, which enables the holding of the work that cannot be held by the claw unit 261 of the hand unit 26. it can. Then, in the next production operation of the robot 2, the action determination unit 9 uses the hand unit 26 to convert the action pattern using the optimal displacement method based on the learning result information JH 3 generated by the learning unit 63 into a non-holdable work. It is determined as an action pattern for enabling the holding. When the robot 2 performs the displacement operation in accordance with this behavior pattern, a holding space for enabling the holding by the claw portion 261 is secured around the work that cannot be held by the claw portion 261 of the hand unit 26. Thus, the work can be held by the claw portion 261. Therefore, it is possible to avoid stopping the operation of the robot 2 due to the existence of the non-holdable work as much as possible, and the operation of taking out the work from the container CN by the hand unit 26 can be continued.

  In the above description, an attempt is made to displace a non-holdable work by a displacement operation based on an action pattern using one displacement method. However, a displacement operation based on an action pattern obtained by combining a plurality of methods is attempted. You may. In this case, for example, a trial of a displacement operation based on an action pattern in which the fifth technique using the dedicated jig JG and the second technique for moving the container CN are considered.

  Further, for example, the work after being placed on the pallet PL is inspected for any damage caused by the displacement operation of the robot 2 and the like, and a reward R considering the inspection result is added to the displacement operation of the robot 2. You may make it give to the corresponding action pattern. In this case, for example, for an action pattern using a displacement method that displaces a work that cannot be held so that a holding space can be secured and that does not generate a scratch or the like, the first value R1 (= 100) is set. What is necessary is just to give the reward R of the value which added the predetermined value (for example, "1").

Reference Signs List 1 robot system 2 robot 26 hand unit 3 imaging device 4 control device 5 machine learning device 6 learning processing unit 61 state observation unit 62 action observation unit 63 learning unit 64 displacement amount observation unit 7 determination unit 8 storage unit 9 action determination unit

Claims (5)

A machine learning device that learns the operation of a robot including a hand unit that takes out a plurality of works by holding the works from a container that accommodates the works in a bulk state,
Before or when the hand unit holds one work in the container, the hand unit recognizes the accommodation state of each work in the container, and the next work candidate to be held by the hand unit is the hand unit. A determination unit that determines whether or not a holding space for enabling holding by the unit is a non-holdable work that is not secured around;
A learning unit that learns a displacement method capable of displacing the non-holdable work so that the holding space is secured, and learns an action pattern of the robot using the displacement method.
A machine learning device comprising: a behavior determining unit that determines a behavior pattern of the robot based on a learning result of the learning unit as a behavior pattern for enabling the holding unit to hold the work that cannot be held. The displacement method includes a plurality of different methods for displacing the non-holdable work,
The machine learning device according to claim 1, wherein the learning unit learns an action pattern of the robot in which the plurality of methods are combined. The displacement method is a method of displacing the non-retainable work by moving the hand unit in a state where the held one work is in contact with the non-retainable work,
The action elements that define the action pattern of the robot that the learning unit learns include an element that determines a contact position of the one work with the non-holdable work and an element that determines a movement locus of the hand unit. The machine learning device according to claim 1, which is included. The displacement method is a method of displacing the non-holdable work by moving the hand unit while holding the container,
The behavior element that defines the behavior pattern of the robot that is learned by the learning unit includes an element that determines a holding position where the hand unit holds the container, an element that determines a movement trajectory of the hand unit, The machine learning device according to claim 1, further comprising: an element that determines a moving speed of the unit. A robot having a hand unit that takes out a plurality of works by holding the works from a container that accommodates the works in a piled state,
The machine learning device according to any one of claims 1 to 4, which learns an operation of the robot,
And a control device that controls the operation of the robot based on a learning result of the machine learning device.

Images