JP2018200537A - Learning device, learning control method, and its program - Google Patents

Abstract

To provide a technique for reducing time required for a learning device to achieve the purposes of learning without through manual handling.SOLUTION: A learning device that learns the control of movements related to a predetermined task includes: a learning data reception unit for receiving learning data including the purposes of learning; a neural network for executing learning on the basis of the learning data; and an output unit for outputting the results of learning by the neural network. The neural network executes; first learning for achieving initial stages of the purposes of learning; second learning for learning control to reach a state where movements related to learning cannot be continued, based on the result of the first learning; and third learning for achieving the purposes of learning by elimination of the control for reaching the state where the movements cannot be continued, based on the result of the second learning.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a learning device, a learning control method, and a program thereof.

  Conventionally, research on artificial intelligence technology such as a neural network (hereinafter referred to as “AI technology”) has been widely performed (see, for example, Patent Document 1). In particular, with the rise of AI technology called Deep Learning, recognition technology for objects using images, for example, has rapidly improved the recognition rate in recent years, and the level of image classification exceeds human recognition rate. Is reaching. Deep learning technology is not only image recognition, but also voice recognition, personal authentication, behavior prediction, sentence summarization, automatic translation, monitoring, automatic driving, failure prediction, sensor data analysis, music genre determination, content generation, Applications in security systems and other broad fields are expected.

  In machine learning such as deep learning, a machine can perform learning to acquire a predetermined ability. At this time, the learning device that performs machine learning repeatedly performs the learning operation until a predetermined ability is acquired.

  For example, Patent Literature 1 discloses a robot learning control method. In the learning control method described in Patent Document 1, a robot driving unit is based on an error generated between a target trajectory that is a target of robot operation set in advance by a person and an actual trajectory when the robot actually operates. Correct the input value supplied to.

JP-A-6-289918

  In learning devices that control actuators based on a large number of sensor information, such as control of automobile engines and driving, or chemical plants, the control method and output of sensor information influence each other, so a control method is acquired. To do this, it is necessary to perform more complicated learning. Therefore, it is not easy for a learning apparatus that performs such complicated learning to set a control amount target value in advance as in Patent Document 1. On the other hand, when the learning apparatus performs learning without setting a target value, it is necessary to repeat a large number of try errors, which is inefficient.

  Therefore, an object of the present invention is to provide a technique for shortening the time required for the learning device to achieve the learning purpose without involving human hands.

  A learning device according to an aspect of the present invention is a learning device that learns control of an operation related to a predetermined task, and a learning data receiving unit that receives learning data including a learning purpose, and learning based on the learning data. A neural network to be executed, and an output unit for outputting a learning result by the neural network. The neural network executes the first learning for achieving the initial stage of the learning purpose, and is based on the result of the first learning. Then, the second learning is performed to learn the control that reaches the state where the learning operation cannot be continued, and the learning purpose is achieved by excluding the control reaching the state where the learning cannot be continued based on the result of the second learning. 3rd learning for performing is performed.

  According to the above-described configuration, before the third learning for achieving the learning purpose, the control is learned so that the operation related to the learning cannot be continued. As a result, the learning can be achieved in a shorter period of time because it is possible to perform the learning by excluding the control in which the apparatus itself cannot be continued without being given a condition for limiting the control operation by a person. it can.

  The output unit may output a result of the second learning. According to this aspect, the learning result of the control leading to the state where the continuation is impossible can be utilized also in other learning apparatuses.

  The learning device is a learning device that learns control of a series of operations related to a predetermined task. The learning device divides the task into a plurality of scenes, and performs each of the divided scenes in the scene in the series of operations. The neural network may further include a classifying unit that identifies a partial action to be performed, and the neural network may perform the second learning and the third learning for each partial action.

  According to this aspect, the learning apparatus can classify the motions related to learning into partial motions that are smaller units depending on the scene, and can learn for each classified partial motion. As a result, the learning purpose can be achieved in a shorter period of time.

  An automatic travel control learning device according to one aspect of the present invention is an automatic travel control learning device that learns control regarding a series of operations related to automatic travel of a vehicle that circulates a predetermined course, and the course is determined within a predetermined time. A learning data receiving unit that receives learning data including a learning purpose intended to circulate the number of times, a neural network that performs learning based on the learning data, and an output unit that outputs a learning result by the neural network, And the neural network performs a first learning for achieving the ability to make one round of the course, and learns a control for reaching a state in which the operation related to the learning cannot be continued based on a result of the first learning. 3rd for performing learning and excluding the control which will be in the state which becomes impossible to continue based on the result of the said 2nd learning, and achieving a learning objective To run the learning.

  The robot control learning device according to one aspect of the present invention is a robot control learning device that learns control about a series of operations related to a task of gripping a predetermined workpiece and stacking it on a placement place corresponding to the shape of the workpiece. A learning data receiving unit for receiving learning data including a learning purpose for the purpose of accumulating a predetermined number of workpieces in a predetermined place within a predetermined time, and a neural network for performing learning based on the learning data And an output unit that outputs a learning result by the neural network, and the neural network executes the first learning for achieving the loading of the work in the previous place, and the result of the first learning Based on the above, the second learning is performed to learn the control that leads to the state where the operation related to learning cannot be continued, and based on the result of the second learning Executes a third learning for achieving the learning object by excluding the control to the state becomes impossible to continue.

  A learning method according to one aspect of the present invention is a learning method for learning control of an operation related to a predetermined task, which is executed by a computer including a control unit, and the control unit stores learning data including a learning purpose. The step of receiving, the step of executing learning based on the learning data, and the step of outputting the learning result by the step of executing learning, and the step of executing learning achieve the initial stage of learning purpose First learning is performed, and based on the result of the first learning, second learning is performed to learn control for reaching a state where the operation related to learning cannot be continued, and based on the result of the second learning A learning method including a step of performing a third learning for achieving a learning purpose by excluding a control that leads to a state where it is impossible to continue.

  A program according to an aspect of the present invention executes a procedure for receiving learning data including a learning purpose in a computer that learns control of an operation related to a predetermined task, a procedure for executing learning based on the learning data, and learning. The procedure for outputting the learning result according to the procedure for performing the learning and executing the learning is to execute the first learning for achieving the initial stage of the learning purpose, and to perform the learning based on the result of the first learning. The second learning for learning the control that reaches the state where the operation cannot be continued is performed, and the control for reaching the state where the operation cannot be continued is excluded based on the result of the second learning to achieve the learning purpose. 3. A program including a procedure for performing learning.

  An apparatus according to an aspect of the present invention is an apparatus that executes a predetermined task, and includes a first sensor that senses information necessary for the operation of the apparatus to execute the task, an actuator, and an apparatus using the actuator. A second sensor that senses a change in the state of the sensor, a control unit that controls the actuator based on sensor values output from the first sensor and the second sensor, and a storage unit that stores a learning result performed by the learning device The control unit determines a control amount according to the sensor value output from the first sensor and the second sensor based on the learning result stored in the storage unit.

  ADVANTAGE OF THE INVENTION According to this invention, the technique for shortening the time required for a learning apparatus to achieve the learning objective can be provided without a human hand.

It is a block diagram which shows schematic structure of the learning apparatus in 1st Embodiment. It is a schematic diagram which shows the course which the vehicle controlled by the learning apparatus in 1st Embodiment drive | works automatically. It is a flowchart which shows the outline of a process of the learning apparatus in 1st Embodiment. It is a block diagram which shows the detailed structure of the learning apparatus in 1st Embodiment. It is a flowchart which shows the detail of a process of the learning apparatus in 1st Embodiment. It is a flowchart which shows the detail of a process of the learning apparatus in 1st Embodiment. It is a flowchart which shows the detail of a process of the learning apparatus in 1st Embodiment. It is a flowchart which shows the detail of a process of the learning apparatus in 1st Embodiment. It is a figure which shows an example of the hardware constitutions of the learning apparatus in 1st Embodiment. It is a block diagram which shows schematic structure of the learning apparatus in 2nd Embodiment.

[First embodiment]
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In addition, the same code | symbol is attached | subjected to the same element and the overlapping description is abbreviate | omitted. Further, the following embodiments are exemplifications for explaining the present invention, and are not intended to limit the present invention only to the embodiments. Furthermore, the present invention can be variously modified without departing from the gist thereof.

<1. System overview>
An overview of the system in this embodiment will be described with reference to FIGS. 1 to 3.
FIG. 1 is a block diagram illustrating a schematic configuration of a learning device 1 according to the present embodiment. The learning device 1 learns a predetermined task. The learning device 1 according to the present embodiment is mounted on an automatic travel control vehicle (hereinafter also simply referred to as “vehicle”) 90 as an example, and controls the vehicle 90 to automatically travel a predetermined course (see FIG. 2). To learn. At this time, learning data is given to the learning device 1 from an operator, for example. The learning data is data including the following learning objectives and learning requirements, for example.

(Learning purpose)
・ Go 10 courses within a predetermined time and finish.
(Learning requirements)
・ Do not go out of the course ・ Circling direction is clockwise ・ Goal ・ In the initial stage level, “Go around the course once”

  Note that a task is an operation related to learning (the “operation related to learning” in the present embodiment is various controls necessary for automatic traveling of the vehicle 90. Note that the operation is performed by the vehicle 90 by the various controls. In this embodiment, it is to go around the course. The learning purpose is a level that the task should achieve. In the present embodiment, as described above, “goal is achieved by making 10 laps within a predetermined time”. Then, in this embodiment, it is considered that the learning requirement is that the task can be performed in the learning at the initial stage level.

  Further, in the following description, the learning apparatus 1 is described as being configured by a computer such as a PC (Personal Computer) or a server apparatus, but is not limited to this. For example, an arbitrary set having a processor, RAM, and ROM May be realized by an embedded device. Further, the configuration implemented in each device is not limited to the configuration realized by software. Arbitrary configurations included in each device may be realized by hardware. For example, the neural network 22 to be described later may be configured by an electronic circuit such as a custom LSI (Large-Scale Integration) or FPGA (Field-Programmable Gate Array).

  As illustrated in FIG. 1, the learning device 1 includes a control unit 10, a machine learning unit 20, an action classification unit 30, and a storage unit 40.

  In the vehicle 90, the control unit 10 is connected to a control sensor 91, an actuator 92, and a state detection sensor 93 provided outside the learning device 1. The control unit 10 controls the actuator 92 in accordance with the outputs from the control sensor 91 and the state detection sensor 93 to perform the automatic traveling of the vehicle 90.

  The control sensor 91 is a sensor group for performing automatic travel control of the vehicle 90. For example, the control sensor 91 includes an in-vehicle camera, a vehicle outside obstacle detection sensor such as a laser, a road surface state detection sensor, and the like. On the other hand, the state detection sensor 93 is a sensor group that detects the control state of the vehicle 90 that is traveling automatically. For example, the state detection sensor 93 includes a vibration sensor, a noise sensor, a fuel consumption detection sensor, a vehicle speed sensor, an acceleration sensor, a yaw rate sensor, and the like.

  The actuator 92 is controlled by the control unit 10 to cause the vehicle 90 to automatically travel. The actuator 92 includes, for example, an accelerator actuator, a brake actuator, and a steering actuator. The accelerator actuator controls the driving force of the vehicle by controlling the throttle opening in accordance with a control signal from the control unit 10. The brake actuator controls the braking force on the wheels of the vehicle by controlling the operation amount of the brake pedal in accordance with a control signal from the control unit 10. The steering actuator controls the steering action of the vehicle by controlling the driving of the steering assist motor of the electric power steering system in accordance with a control signal from the control unit 10.

  Next, a procedure for the learning apparatus 1 to perform learning will be roughly described with reference to FIG. Details of the processing of each step will be described later. FIG. 3 is a flowchart illustrating an outline of a processing flow when the learning device 1 performs learning. First, as an initial learning stage (S1), learning is performed for the purpose of being able to perform a task (that is, to be able to perform an operation that satisfies the learning requirements of the initial stage). In the learning device 1 according to the present embodiment, as an initial stage, “go around the course and make a goal” is given as a learning requirement.

  When the purpose of the initial stage level is cleared, the operation classification (S2) is performed next. At this stage, by analyzing the learning content performed in the initial learning stage of S1, the task is divided into a plurality of parts based on predetermined parameters (hereinafter, the divided task is also referred to as “scene”), and is divided. In each of the performed scenes, a process of specifying an operation (hereinafter, also referred to as “partial operation”) performed in the scene among a series of operations related to the task is performed. Predetermined parameters for dividing a task include, for example, the amount of movement displacement during task learning, the environment in which the operation associated with task learning is executed (elapsed time from the start of the task, Position). In the present embodiment, the position from the start location of the task (environment in which an operation related to task learning is executed) is used as the predetermined parameter. That is, in the present embodiment, the learning device 1 divides a task into scenes based on positions on the course, and performs a series of operations related to learning based on actions performed in units of courses corresponding to the divided scenes. Classified into scenes. Learning can be performed more efficiently by performing learning in units of partial motion classified according to the scene. In this embodiment, learning efficiency may mean, for example, shortening the time required from the start of learning to the achievement of the learning purpose.

  When the actions are classified, as a next step, learning of control for reaching the learning continuation impossible state (S3) is performed for each classified partial action. Here, the state in which learning cannot be continued refers to a state in which the task cannot be continued. For example, when the learning in the learning device 1 is control of a predetermined device, it means a case where the operation of the predetermined device to be controlled stops or a case where the predetermined device fails and becomes inoperable. In this embodiment, the state in which learning cannot be continued is, for example, a state in which the user goes out of course, crashes on a wall or the like, cannot move, or fails. By learning in advance the control leading to the state where learning cannot be continued, learning can be performed while avoiding the state where learning cannot be continued when learning optimal control in a later step. This makes it possible to perform learning more efficiently.

  In the final learning stage (S4), learning optimization is performed. At this stage, learning that performs optimally from the start to the end of the motion is performed after combining the partial motions classified and learned for each scene. In the present embodiment, as the final stage of learning, learning is performed in which a goal is made by making 10 laps within a predetermined time.

<2. Detailed processing>
Next, with reference to FIG. 4 to FIG. 8, the details of the processing of the learning device 1 in each step will be described. FIG. 4 is a block diagram showing a detailed configuration of the learning apparatus 1 according to the present embodiment. As shown in FIG. 4, the machine learning unit 20 includes a learning data input / output unit 21, a neural network 22, and a learning result output unit 23. The operation classification unit 30 includes a control data extraction unit 31 and an operation classification result extraction unit 32.
Below, the detail of the process of each part is demonstrated for every step of FIG.

(2-1. Early learning stage)
FIG. 5 is a flowchart showing a detailed processing flow in the initial learning stage of S1 shown in FIG. First, in the initial stage of learning (first learning), the learning data input / output unit 21 receives learning data (S101). The learning data is data including the learning purpose and the learning requirements described above, for example.

  In the next step (S102), machine learning is performed. In this embodiment, since the conditions for restricting individual control operations are not designated in advance, the learning device 1 learns the control operations themselves. Specifically, the control unit 10 operates by setting a random control amount for the actuator 92. At this time, since the vehicle 90 cannot naturally travel along the course, the vehicle 90 travels randomly while going out of the course. The control unit 10 reads outputs (hereinafter also referred to as “sensor values”) from the control sensor 91 and the state detection sensor 93 for the control amount given at random, and stores these data (control amount and sensor value). Store in the unit 40. The neural network 22 refers to the storage unit 40, reads the stored control amount and sensor value, and learns the control operation adapted to the learning requirement by deep learning (S102).

  The learning requirement is set to “go around the course and make a goal” as the purpose of the initial stage level. Accordingly, the learning device 1 determines that the machine learning has reached the initial stage level when it is determined that the goal has been made by making one round of the course based on the output from the control sensor 91, for example (S103: Y). , To finish the initial learning.

(2-2. Classification of operation)
FIG. 6 is a flowchart showing a detailed processing flow in the operation classification of S2 shown in FIG. First, in performing the operation classification process, the control data extraction unit 31 obtains the sensor value of the control sensor 91 and the control amount of the actuator 92 and the sensor value of the state detection sensor 93 at the end of the learning initial stage. Extracted from the storage unit 40 (S201). The control data extraction unit 31 inputs the extracted values as learning data to the neural network 22.

  Next, the neural network 22 performs machine learning based on the learning data input by the control data extraction unit 31 (S202). At this time, the neural network 22 classifies the orbiting operation into scenes divided into a predetermined number.

A more detailed description will be given of the classification processing into scenes of the orbiting operation by the neural network 22. The neural network 22 classifies the circular motion into scenes based on the scene vector and the motion vector. The scene vector represents a scene of a task performed by the vehicle 90. The scene vector is acquired from, for example, sensor values output from the control sensor 91 (for example, the position (or distance) from the start point and the direction from the start point). As an example, assuming x and y coordinates with the starting point as the origin, the scene vector at the point l can be represented by (l _x , _y ).

On the other hand, the motion vector represents the control state of the traveling vehicle 90. The motion vector is acquired from, for example, a sensor value (for example, velocity, acceleration, angular velocity, angular acceleration, etc.) output by the state detection sensor 93. As an example, the motion vector at a certain point l is represented by (v _l , a _l ) using the velocity v and acceleration a at the point l.

The neural network 22 divides the task into scenes based on the scene vector (l _x , _y ), and the action to be learned in the scene for each divided scene based on the motion vector (v _l , a _l ). Learn classification. Thereby, the learning apparatus 1 can learn the partial motion according to the scene by determining which scene it is in. As an example, the neural network 22 grasps acceleration / deceleration / direction change of the operation of the vehicle 90 by focusing on the change point of the motion vector in addition to the position represented by the scene vector, and based on the change point, a series of Can be classified into actions according to the scene. Further, for example, the neural network 22 can learn the classification of the action based on the similarity of the action vectors.

In the example of the course shown in FIG. 2, the task is divided into scenes corresponding to five courses a to o. The partial operations classified into each scene are as follows, for example.
Scene A: First straight partial operation (for example, control of deceleration timing, traveling position, etc. when approaching the next first corner)
Scene A: First corner partial movement (for example, control of steering operation at the corner, acceleration timing when entering the second straight, etc.)
Scene C: Second straight partial operation (for example, control of deceleration timing, travel position, etc. when approaching the next second corner)
Scene D: Second corner partial movement (for example, steering operation at the corner, control of acceleration timing when entering the third straight, etc.)
Scene O: Third straight partial movement (for example, control of acceleration or the like when entering the first straight)

  Note that the neural network 22 is preferably capable of rearranging the divided scenes in the order of progress.

  The action classification result extraction unit 32 extracts the classification of partial actions learned by the neural network 22 and stores it in the storage unit 40 (S203).

(2-3. Learning of control leading to a state where learning cannot be continued)
FIG. 7 is a flowchart showing a detailed processing flow in the learning of control (second learning) in S3 shown in FIG. First, the learning data input / output unit 21 refers to the storage unit 40, selects one of the partial operations classified in the process of S2, and controls the actuator 92 necessary for the partial operation. To extract. Further, the learning data input / output unit 21 executes control with the control amount extracted by referring to the storage unit 40, and as a result, whether or not the learning data input / output unit 21 has reached the state where learning cannot be continued is determined based on, for example, the output from the state detection sensor 93. Judgment. The learning data input / output unit 21 reads out the extracted control amount and, as a result, information on whether or not the learning continuation incapable state has been reached as learning data, and gives it to the neural network 22 as learning data. The neural network 22 performs learning by deep learning based on the given learning data (S301).

  At this time, the learning result output unit 23 can output a learning result of control that leads to a state where learning cannot be continued. As a result, the neural network 22 can receive additional control as learning data, for example, from another learning device 1 ′ having the same configuration as learning data (S 302). As a result, more efficient learning can be performed. Efficient learning refers to learning in which the time required from the start of learning to the achievement of the learning purpose is short, for example. Note that the process of S302 is not an essential process.

  The learning device 1 performs the process of S301 (and S302) for all classified partial operations (S303).

  Although not essential, the learning device 1 can learn again through a series of operations after learning the control to reach the learning continuation impossible state for all classified partial operations (S304). This makes it possible to perform faster circulation control.

  As described above, the learning device 1 according to the present embodiment can learn about the classified partial operations by avoiding the control in the subsequent learning by first learning the control that reaches the learning continuation impossible state. become. Thereby, more efficient learning can be performed.

(2-4. Optimization learning)
FIG. 8 is a flowchart showing a detailed processing flow in the optimization learning (third learning) in S4 shown in FIG. In the optimization learning, the learning purpose given as learning data at the start of learning is performed by optimizing the learning performed in the steps up to S3 (in this embodiment, “10 courses are made within a predetermined time. Learning to achieve “goal”. In the optimization learning, learning is performed excluding the control that has been learned in S3 and reaches the state where learning cannot be continued. At this time, the learning data input / output unit 21 refers to the storage unit 40 and extracts the learning data (set by the operator) input in the initial learning stage (S1 in FIG. 3). The learning data input / output unit 21 further refers to the storage unit 40 and extracts the state of the neural network 22 after learning the control to reach the learning continuation impossible state. The learning data input / output unit 21 sets the extracted data in the control unit 10.

  The control unit 10 outputs a control amount for the actuator 92 based on the set data described above, and acquires sensor values of the control sensor 91 and the state detection sensor 93 for this. The control unit 10 causes the storage unit 40 to store the given control amount and the sensor value output thereto.

  The neural network 22 reads the control amount and the sensor value stored by the control unit 10 in the above processing, and performs learning by deep learning (S401). As a result, the neural network 22 learns the control leading to the state where the learning cannot be continued, and more efficiently performs the control operation adapted to the learning requirement from the start to the end of the operation (that is, from the start of the course to the goal). Can learn well. The process of S401 is repeatedly performed until the entire learning is optimized (S402). The result of optimization learning is extracted by the learning result output unit 23 and stored in the storage unit 40. Thereby, in the optimization learning, it is possible to perform the learning by excluding the control that leads to the state where the learning cannot be continued.

  As described above, according to the learning device 1 according to the present embodiment, the learning device 1 itself can perform learning by classifying the operation related to learning into partial operations. Since individual optimization can be achieved for each operation classified by this, learning can be performed more efficiently (that is, in a shorter period of time). Furthermore, according to the learning device 1 according to the present embodiment, when learning the partial motion, first, the control to reach the state where the learning cannot be continued is learned. As a result, it is possible for a person to efficiently learn without setting detailed conditions for each operation in advance.

(Hardware configuration)
With reference to FIG. 9, an example of a hardware configuration when the learning device 1 described above is realized by a computer 800 will be described. The configuration of each device can be realized by dividing it into a plurality of devices.

  As shown in FIG. 9, a computer 800 includes a processor 801, a memory 803, a storage device 805, an input interface unit (input I / F unit) 807, a data interface unit (data I / F unit) 809, a communication interface unit (communication). I / F unit) 811 and a display device 813.

  The processor 801 controls various processes in the computer 800 by executing a program stored in the memory 803. For example, when the processor 801 executes a program stored in the memory 803, the control unit 10, the machine learning unit 20, the action classification unit 30, and the like of the learning device 1 can be realized.

  The memory 803 is a storage medium such as a RAM (Random Access Memory). The memory 803 temporarily stores a program code of a program executed by the processor 801 and data necessary for executing the program.

  The storage device 805 is a non-volatile storage medium such as an auxiliary storage device such as a hard disk drive (HDD) or a solid state drive, or a flash memory. The storage device 805 stores an operating system and various programs for realizing the above-described configurations. Such programs and data are referred to by the processor 801 by being loaded into the memory 803 as necessary. For example, the storage unit 40 described above is realized by the storage device 805.

  The input I / F unit 807 is a device for receiving an input from the administrator. Specific examples of the input I / F unit 807 include a keyboard, a mouse, a touch panel, various sensors, and a wearable device. The input I / F unit 807 may be connected to the computer 800 via an interface such as a USB (Universal Serial Bus).

  A data I / F unit 809 is a device for inputting data from the outside of the computer 800. Specific examples of the data I / F unit 809 include a drive device for reading data stored in various storage media. The data I / F unit 809 may be provided outside the computer 800. In this case, the data I / F unit 809 is connected to the computer 800 via an interface such as a USB.

  The communication I / F unit 811 is a device for performing data communication with the external device of the computer 800 via the Internet N by wire or wireless. The communication I / F unit 811 may be provided outside the computer 800. In that case, the communication I / F unit 811 is connected to the computer 800 via an interface such as a USB.

  The display device 813 is a device for displaying various information. Specific examples of the display device 813 include a liquid crystal display, an organic EL (Electro-Luminescence) display, and a wearable device display. The display device 813 may be provided outside the computer 800. In that case, the display device 813 is connected to the computer 800 via, for example, a display cable.

[Second Embodiment]
In the first embodiment, an example in which the learning device 1 is used for the automatic travel control vehicle 90 has been described. However, the device to which the learning device 1 is applied is not limited to the example shown in the first embodiment, and can be applied to various devices. In the present embodiment, an example applied to control of a robot having a task of performing a pick-and-place operation will be described. In the second embodiment, the difference from the first embodiment will be mainly described.

  First, the difference between the system configuration according to the present embodiment and the first embodiment will be described with reference to FIG. The configuration of the learning device 1 is the same as that of the first embodiment. On the other hand, regarding the configuration outside the learning device 1, in the present embodiment, the control sensor 91 'is configured by a sensor group for performing a pick-and-place operation. Specifically, it includes a workpiece detection sensor (image sensor), a robot gripping force sensor, and the like. Further, the control sensor 91 ′ has an image recognition algorithm and can recognize the shape of the workpiece to be gripped. Other configurations outside the learning device 1 are the same as those in the first embodiment.

Next, the difference between learning according to the present embodiment and learning according to the first embodiment will be described.
The pick and place operation that is a task according to the present embodiment refers to an operation performed in the following procedure.
1. Recognize and grip the workpiece shape.
2. Lift the gripped work.
3. The lifted work is moved to a predetermined position according to the work shape.
4). Each workpiece shape is stacked in a cylinder.

  In the robot control learning according to the present embodiment, the learning objectives and learning requirements given are as follows.

(Learning purpose)
・ Work shape within a specified time by pick-and-place operation from a container in which workpieces with three different shapes (for example, cylindrical workpiece, quadrangular prism workpiece and triangular prism workpiece) are stacked. Ten workpieces are stacked in a cylinder (circular, square, triangular) having an inlet corresponding to the above.
(Learning requirements)
-Do not place workpieces in any other position-Stack 10 workpieces in each cylinder for each workpiece shape-At the initial level, "load one workpiece in a cylinder of an appropriate workpiece shape"

  In the present embodiment, the task is to stack the workpieces in a cylinder corresponding to the shape. In the present embodiment, the pick-and-place operation to be learned is that, in the first embodiment, the task is divided into scenes based on the course on which the vehicle 90 travels, and the partial operations are classified based on the scenes. In the same procedure, the pick-and-place operation may be classified into partial operations according to the scene. For example, in the present embodiment, the task corresponds to the scene according to the operation of gripping the workpiece, the scene according to the operation of transporting the workpiece, and the operation of stacking the workpieces based on the amount of movement displacement during learning of the task. Divided into scenes. The pick-and-place operation is classified as a partial operation according to the divided scene.

In the present embodiment, the state where learning cannot be continued refers to, for example, a state where the workpiece does not enter the cylinder. Therefore, the control learned in the learning stage of the control to reach the state where learning cannot be continued is, for example, as follows.
・ Wrong mounting location (The shape of the workpiece and the shape of the cylinder entrance are different)
・ Incorrect stacking direction of workpieces (The direction of the shape of the workpiece is different from the direction of the cylinder shape)

In the learning device 1 according to the present embodiment, by learning in advance the control that leads to the state where learning cannot be continued, the workpiece shape and the shape of the cylinder are appropriately recognized, and the direction when gripping the workpiece is learned in advance. can do. As a result, in the final stage of learning, it is possible to avoid reaching a state in which learning cannot be continued, so that learning efficiency can be further improved. That is, the time required to achieve the learning purpose can be further shortened.
Other configurations are the same as those of the first embodiment.

  The embodiment of the present invention has been described above. Note that this embodiment is intended to facilitate understanding of the present invention and is not intended to limit the present invention. The present invention can be changed or improved without departing from the spirit of the present invention. For example, each step in the above-described processing flow can be executed in parallel, with some of the steps omitted in a range where no contradiction occurs in the processing content, or the order of the processing steps is arbitrarily changed.

  In the above-described embodiment, the example in which the ability acquired by the machine is managed by the AI technology such as deep learning using the system according to the present invention has been described. However, the present invention is not limited to this and is applied to a wide range of fields. Can be applied. For example, the present invention can be applied to various industrial fields such as identification of good and defective products, food products, machine parts, chemical products, and pharmaceuticals, fishery fields, agricultural fields, forestry fields, service industries, medical care, and health fields. Further, the present invention may be applied to a case where AI technology is applied to products in the embedded field, a system utilizing IT technology such as a social system, analysis of big data, classification processing in a wide range of control devices, and the like.

  In this specification, “unit”, “means”, and “procedure” do not simply mean a physical configuration, but also include a case where processing performed by the “unit” is realized by software. In addition, even if one “unit”, “means”, “procedure”, or processing performed by an apparatus is executed by two or more physical configurations or apparatuses, two or more “parts”, “procedures”, and apparatuses The processing to be performed may be executed by one physical means or apparatus.

Moreover, although a part or all of said embodiment may be described also as the following additional remarks, it is not restricted to the following.
(Appendix 1)
Comprising at least one hardware processor;
The hardware processor is
Accept learning data including learning purpose,
Performing learning based on the learning data;
Output the learning result by the neural network,
Performing the learning includes
The first learning for achieving the initial stage of the learning purpose is executed, and the second learning for learning the control to the state where the operation related to the learning cannot be continued is executed based on the result of the first learning Then, based on the result of the second learning, the third learning for achieving the learning purpose is performed by excluding the control that reaches the state where the continuation becomes impossible.
(Appendix 2)
By at least one or more hardware processors,
A step of learning,
Accepting learning data including learning objectives;
Performing learning based on the learning data;
Outputting a learning result by the neural network;
Run
The step of performing the learning includes:
The first learning for achieving the initial stage of the learning purpose is executed, and the second learning for learning the control to the state where the operation related to the learning cannot be continued is executed based on the result of the first learning Then, based on the result of the second learning, a learning method including a step of performing the third learning for achieving the learning purpose by excluding the control to reach the state where it is impossible to continue.

DESCRIPTION OF SYMBOLS 1 Learning apparatus 10 Control part 20 Machine learning part 21 Learning data input / output part 22 Neural network 23 Learning result output part 30 Action classification part 31 Control data extraction part 32 Action classification result extraction part 40 Storage part 90 Automatic travel control vehicle 91 For control Sensor 92 Actuator 93 Condition detection sensor

Claims (8)

A learning device that learns control of operations related to a predetermined task,
A learning data receiving unit for receiving learning data including a learning purpose;
A neural network for performing learning based on the learning data;
An output unit for outputting a learning result by the neural network;
With
The neural network is
The first learning for achieving the initial stage of the learning purpose is executed, and the second learning for learning the control to the state where the operation related to the learning cannot be continued is executed based on the result of the first learning Then, based on the result of the second learning, the third learning for achieving the learning purpose is performed by excluding the control that reaches the state where the continuation is impossible.
Learning device. The output unit is
Outputting the result of the second learning;
The learning device according to claim 1. The learning device
A learning device for learning control of a series of operations related to a predetermined task,
The task is further divided into a plurality of scenes, and each of the divided scenes further includes a classification unit that identifies a partial action performed in the scene among the series of actions,
The neural network performs the second learning and the third learning for each partial operation.
The learning device according to claim 1. An automatic traveling control learning device that learns control about a series of operations related to automatic traveling of a vehicle that circulates a predetermined course,
A learning data receiving unit for receiving learning data including a learning purpose for the purpose of circulating the course a predetermined number of times within a predetermined time;
A neural network for performing learning based on the learning data;
An output unit for outputting a learning result by the neural network;
With
The neural network is
The first learning is performed to achieve the ability to make one round of the course, and the second learning is performed to learn the control to reach a state where the operation related to the learning cannot be continued based on the result of the first learning. Then, based on the result of the second learning, the third learning for achieving the learning purpose is performed by excluding the control to reach the state where the continuation is impossible.
Automatic travel control learning device. A robot control learning device that learns control about a series of operations related to a task of gripping a predetermined workpiece and stacking it on a placement place corresponding to the shape of the workpiece,
A learning data receiving unit for receiving learning data including a learning purpose for the purpose of accumulating a predetermined number of the workpieces within the predetermined time within a predetermined time;
A neural network for performing learning based on the learning data;
An output unit for outputting a learning result by the neural network;
With
The neural network is
Execute first learning to achieve loading the work in the previous place, and learn control based on the result of the first learning to reach a state where the operation related to the learning cannot be continued. Performing the second learning to perform, and based on the result of the second learning, to perform the third learning for achieving the learning purpose by excluding the control to reach the state where it is impossible to continue,
Robot control learning device. A learning method for learning control of an operation related to a predetermined task, which is executed by a computer including a control unit,
The control unit is
Accepting learning data including learning objectives;
Performing learning based on the learning data;
Outputting a learning result obtained by executing the learning;
Run
The step of performing the learning includes:
The first learning for achieving the initial stage of the learning purpose is executed, and the second learning for learning the control to the state where the operation related to the learning cannot be continued is executed based on the result of the first learning And, based on the result of the second learning, including the step of performing the third learning for achieving the learning purpose by excluding the control that leads to the state where the continuation becomes impossible,
Learning method. A computer that learns the control of operations related to a given task.
Procedures for accepting learning data including learning objectives,
A procedure for performing learning based on the learning data, and a procedure for outputting a learning result by the means for performing learning;
And execute
The procedure for performing the learning is as follows:
The first learning for achieving the initial stage of the learning purpose is executed, and the second learning for learning the control to the state where the operation related to the learning cannot be continued is executed based on the result of the first learning And, based on the result of the second learning, including the procedure of performing the third learning for achieving the learning purpose by excluding the control that reaches the state where the continuation is impossible.
program. An apparatus for performing a predetermined task,
A first sensor that senses information necessary for the device to perform a task;
An actuator,
A second sensor for sensing a change in state of the device by the actuator;
A control unit for controlling the actuator based on sensor values output from the first sensor and the second sensor;
A storage unit for storing a learning result performed by the learning device according to any one of claims 1 to 3;
With
The controller is
Based on the learning result stored in the storage unit, a control amount corresponding to a sensor value output from the first sensor and the second sensor is determined.
apparatus.

Images