JP2020192641A - Control device, control system, machine device system and control method - Google Patents

Abstract

To provide a control device and the like that enable skill to be inherited using a machine learning model.SOLUTION: A control device of a mechanical device comprises: a motion control part that controls motion of the mechanical device according to operation information outputted from an operation device for operating the mechanical device; a calculating part that includes a machine learning model in which first motion information showing motion of the mechanical device is set as input data and a command to activate the mechanical device corresponding to the first motion information is set as output data; and an assisting part that outputs an assist command to assist operation of the operation device on the basis of a difference between motion of the mechanical device controlled by the motion control part and motion of the mechanical device corresponding to the command outputted from the calculating part. The operation device outputs operation information based on second motion information showing the motion of the operation device.SELECTED DRAWING: Figure 12

Description

The present invention relates to control devices, control systems, mechanical device systems and control methods.

Conventionally, there is known a technique of automating a human work by causing a machine learning model including a neural network or the like to perform machine learning of a human work and using a machine device controlled by using the machine learning model. For example, Patent Document 1 discloses a robot camera control device that controls a robot camera using a neural network. The robot camera control device includes a robot camera, a subject detection device that detects the position of the subject, an operator of the robot camera, and a learning control device that has a neural network and controls the imaging operation of the robot camera. The robot camera captures an image of the subject according to the operation on the actuator, and outputs state data indicating the state of the imaging operation to the learning control device. The learning control device causes the neural network to learn the state data using the position data of the subject detected by the subject detection device. At the time of automatic control, the learning control device uses the output of the neural network obtained by inputting the position data of the subject to control the robot camera.

Japanese Unexamined Patent Publication No. 2009-21124

In recent years, there have been problems with skill transfer in various industries. Due to the aging of skilled workers and the shortage of successors, there is a problem that the timing of skill transfer does not match between the two, that is, it is difficult to inherit functions. For example, the robot camera described in Patent Document 1 does not enable inheritance of imaging skills.

An object of the present invention is to provide a control device, a control system, a machine device system, and a control method that enable skill inheritance using a machine learning model.

In order to achieve the above object, the control device according to one aspect of the present invention is a control device of a mechanical device, and the control device of the mechanical device is in accordance with operation information output from the operation device for operating the mechanical device. It includes an operation control unit that controls the operation, and a machine learning model that uses the first operation information indicating the operation of the machine as input data and the command of the operation of the machine corresponding to the first operation information as output data. Based on the difference between the operation of the arithmetic unit, the mechanical device controlled by the motion control unit, and the operation of the mechanical device corresponding to the command output by the arithmetic unit, the operation in the operation device is performed. The operation device includes an auxiliary unit that outputs an auxiliary command to assist, and the operation device outputs the operation information based on the second operation information indicating the operation of the operation device.

Further, the control system according to one aspect of the present invention includes a control device according to one aspect of the present invention and the operation device for operating the mechanical device.

Further, the mechanical device system according to one aspect of the present invention includes a control device according to one aspect of the present invention, the mechanical device, and the operating device for operating the mechanical device.

Further, in the control method according to one aspect of the present invention, the machine device is operated according to the operation information output from the operation device for operating the machine device, and the first operation information indicating the operation of the machine device is input. The data is input to the machine learning model to output a command for the operation of the machine device corresponding to the first operation information, and the operation of the machine device and the command corresponding to the command output by the machine learning model. Based on the difference from the operation of the mechanical device, the auxiliary command for assisting the operation in the operation device is output, and the operation device outputs the operation information based on the second operation information indicating the operation of the operation device. ..

According to the present invention, skill inheritance using a machine learning model becomes possible.

Functional block diagram showing an example of the configuration of the mechanical device system according to the embodiment Diagram showing an example of a machine learning model Diagram showing another example of a machine learning model Functional block diagram showing an example of the configuration of the arithmetic unit according to the embodiment A flowchart showing an example of operation of the mechanical device system according to the embodiment in the automatic operation mode. A flowchart showing an example of operation in the training mode of the manual operation mode of the mechanical device system according to the embodiment. Functional block diagram showing an example of the configuration of the mechanical device system according to the first modification Side view showing an example of the configuration of the robot according to the modified example 1. The figure which shows an example of the appearance of the operation apparatus which concerns on modification 1. Functional block diagram showing an example of the configuration of the operation device according to the first modification Functional block diagram showing an example of the configuration of the calculation unit according to the first modification Functional block diagram showing an example of the configuration of the mechanical device system according to the second modification Functional block diagram showing an example of the configuration of the calculation unit according to the second modification The figure which shows the auxiliary example of the operation based on the auxiliary command

Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that all of the embodiments described below show comprehensive or specific examples. Further, among the components in the following embodiments, the components not described in the independent claims indicating the highest level concept are described as arbitrary components. In addition, each figure in the attached drawings is a schematic view and is not necessarily exactly illustrated. Further, in each figure, substantially the same components are designated by the same reference numerals, and duplicate description may be omitted or simplified. Further, in the present specification and claims, the "device" may mean not only one device but also a system including a plurality of devices.

(Embodiment)
The mechanical device system 1 according to the embodiment will be described. FIG. 1 is a functional block diagram showing an example of the configuration of the mechanical device system 1 according to the embodiment. In FIG. 1, the solid arrow indicates the flow of commands, data, information, and the like for operating the mechanical device 10 in the automatic operation mode. The broken line arrow indicates the flow of commands, data, information, etc. for operating the mechanical device 10 in the training mode of the manual operation mode. The arrow of the alternate long and short dash line indicates the flow of commands, data, information, etc. for the arithmetic unit 36 to learn. These are the same in the following functional block diagram.

[1-1. Constitution]
[1-1-1. Mechanical system]
As shown in FIG. 1, the mechanical device system 1 according to the embodiment includes a mechanical device 10, an operating device 20, a control device 30, an operation information detection device 50, and an output device 60. The mechanical device 10 includes an action unit 11 that applies an action to an object to be processed, and an operation unit 12 that moves the action unit 11 so as to perform the action. The operation device 20 and the control device 30 constitute a control system 100 for controlling the mechanical device 10.

The operation device 20 is a device for operating the mechanical device 10, and outputs operation information, which is information input to the operation device 20, to the control device 30. The control device 30 controls the overall operation of the mechanical device 10. The operation information detection device 50 detects the operation information indicating the operation of the operation unit 11 and the operation unit 12 of the mechanical device 10 and outputs the operation information to the control device 30. For example, in the operation information detection device 50, as operation information, the position of the action unit 11, the force applied by the action unit 11 to the object, the image of the object, the vibration, the impact, the light, the sound, the temperature, and the humidity in the action unit 11 And a sensor that detects atmospheric pressure and the like may be provided. The control device 30 outputs operation information to the operation device 20 and the output device 60 for feedback and presentation of the operation state. The output device 60 converts the operation information into information such as visual and auditory information and presents it to the operator of the operation device 20. For example, an image pickup device such as a camera may be arranged at a position away from the mechanical device 10, and the image captured by the image pickup device may be output to the output device 60. Such an output device 60 can present the state of the mechanical device 10 to the operator. Examples of the output device 60 are, but are not limited to, a liquid crystal display and an organic or inorganic EL display (Electro-Luminescence Display). The output device 60 may include a speaker that emits sound.

Although not limited to this, in the present embodiment, the mechanical device system 1 can cause the mechanical device 10 to execute the operation in the manual operation mode and the operation in the automatic operation mode. The manual operation mode and the automatic operation mode in the present embodiment do not include a teaching (also referred to as "teaching") operation for teaching the mechanical device 10 an operation such as work. In the manual operation mode, the mechanical device 10 executes an operation according to an operation input to the operation device 20 by the operator, that is, an operation that traces the operation. The mechanical device 10 is manually operated by an operator. Further, the manual operation mode includes a training mode and a non-training mode. In the training mode, the mechanical device system 1 assists the operator in operating the operating device 20. At this time, the operator receives assistance from the operating device 20 or the like to operate the operating device 20 for causing the mechanical device 10 to perform a predetermined predetermined operation. In the non-training mode, the mechanical device system 1 does not assist the operator in operating the operating device 20. At this time, the operator can cause the mechanical device 10 to perform an arbitrary operation, and operates the operating device 20 without receiving assistance from the operation.

In the automatic operation mode, the mechanical device 10 executes an operation according to a predetermined operation set in advance. The mechanical device 10 automatically performs an automatic operation to automatically execute a predetermined operation according to the control program. The predetermined operation may be an individual operation such as horizontal movement, vertical movement, and rotation, or may be a composite operation in which a series of a plurality of individual operations are combined according to an execution order. In addition, the individual operation may include one operation or may include two or more operations. Examples of complex operations include holding and moving an object at the action unit 11, cutting the object at the action unit 11, joining two or more objects at the action unit 11, and excavating at the action unit 11. It is work such as doing. In the present embodiment, the mechanical device system 1 can accept the modification of the operation of the action unit 11 and the operation unit 12 using the operation device 20 during the automatic operation. The mechanical device system 1 modifies the operations of the action unit 11 and the operation unit 12 by adding a correction operation corresponding to the operation input to the operation device 20. Further, the automatic operation mode may include a combination of automatic operation and manual operation so that a part of the complex operation is manually operated.

[1-1-2. Mechanical equipment]
As shown in FIG. 1, the mechanical device 10 may be a device operated by power. Examples of the mechanical device 10 include construction machinery, tunnel excavators, cranes, cargo handling vehicles, robots for various purposes such as industrial use, and the like. For example, when the mechanical device 10 is a backhoe of a construction machine, the excavator of the backhoe corresponds to the working portion 11 and the arm corresponds to the moving portion 12. The control device 30 controls a hydraulic device or the like that operates the arm. When the mechanical device 10 is a tunnel excavator, the excavating blade of the tunnel excavator corresponds to the working unit 11, and the operating device for operating the excavating blade corresponds to the operating unit 12. The control device 30 controls the operation of the operating device and the like. When the mechanical device 10 is a cargo handling vehicle, the loading portion or grip portion of the cargo handling device such as a fork of the cargo handling vehicle corresponds to the working unit 11, and the cargo handling device and the driving device of the transport trolley correspond to the operating unit 12. To do. The control device 30 controls the operation of the cargo handling device, the drive device of the transport carriage, and the like. When the mechanical device 10 is an industrial robot, the robot arm of the robot corresponds to the moving unit 12, and the end effector at the tip of the robot arm corresponds to the working unit 11. The control device 30 controls the operation of the robot arm, the drive device for the end effector, and the like. The type of power may be any type. Examples of types of power are electric motors, internal combustion engines, water vapor, hydraulic pressure, pneumatic pressure and the like. The type of control may be any type. Examples of control types are electrical control, hydraulic control, hydraulic control, pneumatic control and the like.

[1-1-3. Operating device]
As shown in FIG. 1, the operation device 20 converts the input by the operator into information corresponding to the input and outputs the operation information to the control device 30. For example, the operating device 20 converts the input by the operator into a signal corresponding to the input and outputs it to the control device 30. In the present embodiment, the operating device 20 is not fixed to another object such as the mechanical device 10, and is configured to be movable in an arbitrary direction in the three-dimensional space. The operating device 20 may be configured to be movable in any direction on a two-dimensional plane or a one-dimensional straight line. The operating device 20 is configured so that it can be gripped by the operator's hand. Therefore, the operator can move the gripping operating device 20 in an arbitrary direction and orient it in an arbitrary posture. The operation device 20 is configured to communicate with the control device 30 via wired communication or wireless communication. Any type of communication may be used regardless of the type of wired communication or wireless communication.

The operating device 20 may be, for example, a device having the same configuration as a general-purpose device such as a game controller, a remote controller, or a smartphone of a home-use game machine, or a dedicated device. .. For example, when the mechanical device 10 is an industrial robot, the dedicated device may be a device corresponding to the function of the end effector. When the end effector is a spray gun for painting, the operating device 20 may be a gun-like device.

In the present embodiment, the operating device 20 includes an inertial measurement unit (IMU) (not shown). The inertial measurement unit includes a 3-axis acceleration sensor and a 3-axis angular velocity sensor, and the operating device 20 outputs operation information based on measurement data of acceleration and angular velocity in the 3-axis direction measured by the inertial measuring unit to the control device 30. .. The operation device 20 may output the measurement data itself to the control device 30. From the measurement data of acceleration and angular velocity in the three axial directions, it is possible to detect various information indicating the operation and acting force of the operating device 20 such as position, posture, movement, moving speed and force. Such an operation device 20 outputs operation information based on the operation operation information which is information indicating the operation of the operation device 20.

Further, in the present embodiment, the operation device 20 includes a haptics device (not shown) that gives feedback of the operating state of the mechanical device 10 that operates according to the operation information to the operator as a tactile sensation. The haptics device receives the operation information of the mechanical device 10 from the operation information detection device 50 via the control device 30, and gives feedback of the operation state of the mechanical device 10 based on the operation information to the operator as a tactile sense. The haptics device is an example of a sensory device.

Here, the operation information includes operation data. The operation data is at least one of force data representing the force applied to the object by the acting unit 11 of the mechanical device 10, that is, the force acting on the working environment, and position data representing the position of the acting unit 11 during operation. including. In the present embodiment, the operation data includes both. The force data may be time series data including the magnitude of the force and the time when the force is generated in association with each other. The position data may be time series data including the position information and the time of the position in association with each other. The motion data including the force data and the position data may be time series data including the magnitude of the force, the time when the force is generated, the position information, and the time at the position. The position of the acting unit 11 may include not only the position of the acting unit 11 in the three-dimensional space but also the posture of the acting unit 11 in the three-dimensional space. As used herein and in the claims, "position" means including a position in three-dimensional space and a position in at least three-dimensional space among postures in three-dimensional space.

The reason why the operation information includes the operation data as essential information is that the control device 30 controls at least one of the "force" that the action unit 11 acts on the work environment and the "position" of the action unit 11 during operation. This is because the operation of the mechanical device 10 is controlled by. The "operation command" in the present embodiment is a force command that indicates a target value or correction value (correction value) of this "force", and a target value or correction value (correction value) of this "position". It includes at least one of a position command which is a command to instruct.

Further, the operation information includes image data of an object to which the action unit 11 acts, vibration data, impact data, optical data, sound data, temperature data, humidity data, etc. generated in the action unit 11, as information other than the operation data. It may include pressure data such as pressure. At least the operation data of the operation information is sent to the operation device 20.

For example, haptics devices include actuators, controllers, drivers and the like. The actuator is exemplified by an eccentric motor, a linear resonance actuator, a piezo, or the like, and gives an operator a sense of tactile force. The controller controls the actuator via a driver, and may have a configuration similar to that of the control device 30 illustrated later. The driver constitutes the interface between the actuator and the controller. The detailed configuration of the haptics device is disclosed in Japanese Patent No. 4111278, Japanese Patent Application Laid-Open No. 2019-60835, etc. For example, a haptics device can give a tactile sensation to an operator while the operator is holding the operating device 20 in the air, and an example of such a tactile sensation is pushed by the operator himself. The sensation, the sensation of pulling by oneself, the sensation of being pulled from the outside, the sensation of being pushed from the outside, the feeling of expansion, the feeling of oppression, the texture indicating the roughness of the surface of the object, and the sensation of pressure indicating the hardness of the object.

[1-1-4. Control device]
The control device 30 shown in FIG. 1 is composed of, for example, an arithmetic unit having a processor, a memory, and the like. The memory is composed of a semiconductor memory such as a volatile memory and a non-volatile memory, and a storage device such as a hard disk (HDD: Hard Disc Drive) and an SSD (Solid State Drive). For example, the function of the arithmetic unit is a computer system consisting of a processor such as a CPU (Central Processing Unit), a volatile memory such as a RAM (Random Access Memory), and a non-volatile memory such as a ROM (Read-Only Memory) (shown in the figure). It may be realized by. Some or all of the functions of the arithmetic unit may be realized by the CPU using the RAM as a work area to execute a program recorded in the ROM. A part or all of the functions of the arithmetic unit may be realized by the above computer system, or may be realized by a dedicated hardware circuit such as an electronic circuit or an integrated circuit, and may be realized by the above computer system and the hardware circuit. It may be realized by a combination.

Specifically, the control device 30 is composed of, for example, a microcontroller, an MPU (Micro Processing Unit), an LSI (Large Scale Integration: large-scale integrated circuit), a system LSI, a PLC (Programmable Logic Controller), a logic circuit, and the like. You may. The plurality of functions of the control device 30 may be realized by being individually integrated into one chip, or may be realized by being integrated into one chip so as to include a part or all of them. Further, each circuit may be a general-purpose circuit or a dedicated circuit. As an LSI, an FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, a reconfigurable processor that can reconfigure the connection and / or setting of circuit cells inside the LSI, or a plurality of functions for a specific application. An ASIC (Application Specific Integrated Circuit) or the like in which the above circuits are integrated may be used.

The control device 30 includes an operation determination unit 31, an operation command unit 32, a correction command unit 33, a drive command unit 34, a correction information detection unit 35, a calculation unit 36, an operation information processing unit 37, and a difference detection. A unit 38, an auxiliary information processing unit 39, a first storage unit 40, a second storage unit 41, and a third storage unit 42 are included as functional components. The operation determination unit 31, the operation command unit 32, the correction command unit 33, the drive command unit 34, the correction information detection unit 35, the calculation unit 36, the operation information processing unit 37, the difference detection unit 38, and the auxiliary information processing unit 39 perform the above calculation. A functional block realized by a computer system, a hardware circuit, or a combination of a computer system and a hardware circuit. The first storage unit 40, the second storage unit 41, and the third storage unit 42 are functional blocks realized by the storage device of the arithmetic unit. The difference detection unit 38 and the auxiliary information processing unit 39 constitute an auxiliary unit.

The operation determination unit 31 determines a predetermined operation to be executed by the mechanical device 10, and acquires operation information (hereinafter, also referred to as “determined operation information”) of the predetermined operation. The operation determination unit 31 receives a command of a predetermined operation to be executed by the mechanical device 10 via the operation device 20 or another input device of the mechanical device system 1. The operation determination unit 31 extracts the operation information corresponding to the received predetermined operation from the third storage unit 42 as the determination operation information. The predetermined operation to be executed by the mechanical device 10 may be an individual operation or a combined operation. In the automatic operation mode, the operation determination unit 31 outputs the determination operation information to the operation command unit 32, but may further output the content of the predetermined operation to the calculation unit 36. In the training mode of the manual operation mode, the operation determination unit 31 outputs the content of a predetermined operation to the calculation unit 36, but it is not necessary to acquire the determination operation information. The motion determination unit 31 does not function in the non-training mode.

The third storage unit 42 stores a predetermined operation that can be executed by the mechanical device 10 in association with the operation information of the predetermined operation. The operation information of the predetermined operation is preset and stored in the third storage unit 42. In the case of complex operations, operation information for each individual operation may be set. For example, the motion information of each individual motion may be set by presetting the target values of the force and the position of the action unit 11. Alternatively, the operation information of each individual operation may be set by using the operation information obtained as a result of operating the mechanical device 10 via the operation device 20 in the manual operation mode. Alternatively, the operation information of each individual operation may be set by using the operation information obtained as a result of actually operating the mechanical device 10 in the automatic operation mode.

The operation command unit 32 generates an operation command (hereinafter, also referred to as an “execution operation command”) for causing the mechanical device 10 to execute the operation. In the training mode and the non-training mode, the operation command unit 32 receives the operation information from the operation device 20 and generates an execution operation command corresponding to the operation information. At this time, the operation command unit 32 generates an operation command (hereinafter, also referred to as “operation operation command”) for causing the action unit 11 to perform an operation corresponding to the operation information, and drives the operation operation command as an execution operation command. Output to the command unit 34 and the difference detection unit 38.

In the automatic operation mode, the operation command unit 32 uses the determined operation information determined by the operation determination unit 31 to generate an execution operation command for causing the mechanical device 10 to execute an operation corresponding to the determined operation information. Output to the correction command unit 33. Further, the operation command unit 32 is configured to receive output data from the calculation unit 36. This output data is a command (hereinafter, also referred to as "execution operation correction command") to be output by the calculation unit 36 by inputting the operation information of the mechanical device 10 as input data. Although not limited to this, in the present embodiment, the execution operation correction command is an operation command. When the operation command unit 32 receives the execution operation correction command from the calculation unit 36, the operation command unit 32 corrects the operation command for executing the determined operation information (hereinafter, “determined operation command”) by using the execution operation correction command. Then, the execution operation command is generated. At this time, the operation command unit 32 adds an execution operation correction command corresponding to the determination operation command, or replaces the determination operation command with the corresponding execution operation correction command. If not received, the operation command unit 32 uses the determined operation command as the execution operation command. The execution operation correction command corresponding to the determination operation command is output data of the calculation unit 36 when the operation information of the mechanical device 10 immediately before executing the operation of the determination operation command is used as input data.

The correction command unit 33 functions in the automatic operation mode. The correction command unit 33 generates a correction operation command, which is a corrected operation command, by modifying the execution operation command received from the operation command unit 32 according to the operation information output from the operation device 20 during the automatic operation mode. And output to the drive command unit 34. For example, when an input is made to the operation device 20, the correction command unit 33 generates an operation operation command according to the operation information corresponding to the input. The correction command unit 33 generates a correction operation command by adding the execution operation command and the operation operation command. The modified operation command is an operation command that reflects the operation information. Further, when there is no input to the operation device 20, the correction command unit 33 determines the execution operation command as the correction operation command.

The drive command unit 34 controls the operation of the mechanical device 10 according to the execution operation command received from the operation command unit 32 or the correction operation command received from the correction command unit 33. The drive command unit 34 controls the operation of each drive device of the mechanical device 10 so that the action unit 11 performs the operation corresponding to the operation command. The drive command unit 34 generates drive data including a command value for driving the drive device in order to execute the above operation, and outputs the drive data to each drive device. Here, the operation command unit 32 and the drive command unit 34 form an operation control unit, and the correction command unit 33 and the drive command unit 34 form a correction control unit.

As described above, in the present embodiment, the "commands" can be added or subtracted, and the "operation command" and the "operation data" can be added or subtracted from each other.

The operation information processing unit 37 receives the operation information of the mechanical device 10 from the operation information detection device 50, and outputs the operation information to the calculation unit 36, the operation device 20, and the output device 60. In the automatic operation mode and the training mode, the operation information processing unit 37 outputs the operation information to the calculation unit 36, the operation device 20 and the output device 60, and in the non-training mode, the operation information is output to the operation device 20 and the output device 60. Output to, but not limited to this. Here, the motion information processing unit 37 is an example of a processing unit.

The correction information detection unit 35 functions in the automatic operation mode. The correction information detection unit 35 detects the correction information indicating the correction made by the correction command unit 33 and stores it in the second storage unit 41. Specifically, when the correction command unit 33 corrects the execution operation command, the correction information detection unit 35 detects the correction operation command generated by the correction command unit 33 as correction information. Further, when the correction command unit 33 does not correct the execution operation command, the correction information detection unit 35 detects the uncorrected execution operation command as the correction information. The correction information detection unit 35 may associate the correction operation command or the execution operation command with the issuance time which is the time when the operation command is issued, and generate time-series data of the operation command. In this case, the correction information detection unit 35 may associate the target value of the "force" and the target value of the "position" included in the operation command with the announcement time to generate time-series data similar to the operation data. ..

The correction information detection unit 35 may detect an operation operation command as correction information. For example, the correction information detection unit 35 detects the operation operation information used for the correction as the correction information when the execution operation command is corrected, and if the execution operation command is not corrected, the correction information You may generate a detection result assuming that there is no.

The first storage unit 40 stores operation information indicating the operation of the mechanical device 10. Specifically, the first storage unit 40 stores the operation information of the mechanical device 10 received from the operation information detection device 50. In the first storage unit 40, the operation information and the time when the operation information is detected by the operation information detection device 50 are stored in association with each other.

The second storage unit 41 stores correction information indicating the correction made by the correction command unit 33. Specifically, the second storage unit 41 stores the correction information received from the correction information detection unit 35. In the second storage unit 41, the correction information and the issuance time of the operation command corresponding to the correction information are stored in association with each other.

The calculation unit 36 includes a machine learning model 36a for machine learning. The machine learning model 36a improves the accuracy of the output data with respect to the input data by learning using the training data. As such machine learning models, there are neural networks such as Deep Learning, Random Forest, Genetic Programming, regression models, tree models, Bayesian models, time series models, clustering models, ensemble learning models, etc. Illustrated. The machine learning model 36a of the present embodiment is a neural network.

In the present embodiment, the calculation unit 36 causes the machine learning model 36a to perform machine learning. The machine learning model 36a performs machine learning using the operation information of the machine device 10 and the correction information corresponding to the operation information. Further, the machine learning model 36a after machine learning uses the operation information of the machine device 10 as input data and the command corresponding to the operation information as output data. In the present embodiment, the output data is an execution operation correction command. For example, in machine learning, the operation information of the machine device 10 may be used as input data, and the correction information executed in the state of the operation information may be used as teacher data. At this time, the weighting of the connection between the nodes in the neural network, which will be described later, is adjusted so that the output data with respect to the input data matches the teacher data. When the operation information of the machine device 10 is input, the machine learning model 36a after such weighting adjustment can output an execution operation correction command to be executed in the state of the operation information.

A neural network is an information processing model modeled on the cranial nerve system. The neural network is composed of a plurality of node layers including an input layer and an output layer. The node layer contains one or more nodes. For example, the machine learning model 36a may be configured by a neural network as shown in FIG. FIG. 2 is a diagram showing an example of the machine learning model 36a. As shown in FIG. 2, when the neural network is composed of an input layer, an intermediate layer, and an output layer, the neural network outputs the information input to the nodes of the input layer from the input layer to the intermediate layer. Output processing from the intermediate layer to the output layer is performed in sequence, and output results that match the input information are output. Each node of one layer is connected to each node of the next layer, and the connection between the nodes is weighted. The information of the nodes of one layer is weighted for the connection between the nodes and output to the nodes of the next layer.

Further, the machine learning model 36a may be composed of a recurrent neural network (also referred to as a “recurrent neural network”) as shown in FIG. FIG. 3 is a diagram showing another example of the machine learning model 36a. As shown in FIG. 3, the recurrent neural network handles time series information. The input data of the recurrent neural network includes the data at the current time t and the output data of the intermediate layer in the recurrent neural network at the time t-1 before the time t. As described above, the recurrent neural network has a network structure in consideration of time series information. Since such a recurrent neural network outputs the motion information in consideration of the behavior over time, the accuracy of the output data can be improved.

An example of the configuration of the calculation unit 36 will be described. FIG. 4 is a functional block diagram showing an example of the configuration of the calculation unit 36 according to the embodiment. As shown in FIG. 4, the calculation unit 36 includes a machine learning model 36a, a data generation unit 36b, a data input unit 36c, and a learning evaluation unit 36d. The configuration of the machine learning model 36a is as described above. Since the machine learning model 36a handles time series data as follows, it is preferably a recurrent neural network.

In this example, each command and each data is acquired at a predetermined sampling interval while the mechanical device 10 executes a predetermined operation once. For example, in the automatic operation mode, the correction information detector 35, time-series data _{Pm 0,} Pm _1, Pm 2 modified operation command Pm as correction information, · · ·, Pm _{u (hereinafter,} the Pm 0 Pm _u (Abbreviated) is acquired at the sampling interval. Operation information detecting unit 50, time-series data _{Pd 0,} Pd _1, Pd 2 operation data Pd of the machine 10, · · ·, Pd _{u (hereinafter,} abbreviated as Pd 0 -PD _u) at the sampling interval get. Hereinafter, the subscript numbers in each time series data represent the order of sampling times (intermittent times). Therefore, the time series data in which the numbers in the subscripts are the same means that the data is acquired at the sampling time which can be regarded as the same or substantially the same. For example, time-series data of the operation data Pd of mechanical apparatus 10 in accordance with time-series data Pm _i corrective action command Pm is executed when a series data Pd _i. Time-series data having the same subscript numbers are time-series data corresponding to each other.

First, the processing of each component of the calculation unit 36 during machine learning will be described. Data generating unit 36b generates the time-series data _pd 0 -PD _u training data pd from the time-series data _Pd 0 -PD _u operation data Pd stored in the first storage unit 40. Also, the data generating unit 36b generates the time-series data _pn 0 to PN _u from time-series data _Pm 0 Pm _u teacher data pn modified operation command Pm stored in the second storage unit 41. The data generation unit 36b outputs the generated time series data to the data input unit 36c.

The data input unit 36c sequentially inputs the time series data pd _{0 to} pd _u of the training data pd to each neuron in the input layer of the neural network of the machine learning model 36a. For example, the data input unit 36c is, by entering the time-series data _pd i training data pd at a sampling time _{t i (i} = _0~u), neural network, the forward calculation, execution at the next sampling time _{t i + 1} The operation correction command Pni _{+ 1} is predicted and output.

Learning evaluation unit 36d, based on the execution operation correction instruction _{Pn i + 1,} by searching time-series data _pn 0 to PN _u teacher data pn, it extracts the time series data _{pn i + 1} at the sampling time _{t i + 1.} Further, the learning evaluation unit 36d adjusts the weights between the neurons of the neural network by a backward calculation so as to match the execution operation correction command Pni _{+ 1} and the time series data pn _{i + 1} or to minimize the error. To do. Further, the data input unit 36c and the learning evaluation unit 36d optimize the weights between neurons by performing the above processing on all of the time series data pd _{0 to} pd _u of the learning data pd.

Next, the processing of each component of the calculation unit 36 at the time of data input / output will be described. The calculation unit 36 functions in the automatic operation mode and the training mode. During operation of the machine 10, the movement information detecting unit 50 detects the operation data Pd _i at the current sampling time t _i, and outputs to the calculation unit 36 through the operation information processing unit 37. Data input unit 36c inputs the operation data Pd _i to the neural network machine learning model 36a. Neural networks will be able to input data to motion data Pd _i, and outputs an execution operation correction command Pn i _{+ 1} at the next sampling time t _{i + 1} as the output data.

In the automatic operation mode, the calculation unit 36 outputs the execution operation correction command Pni _{+ 1} output by the neural network to the operation command unit 32. The operation command unit 32 generates an execution operation command that reflects the execution operation correction command Pni _{+ 1} . At each sampling time _{t i (i = 0~u-1} ), the machine learning model 36a by using a neural network, the sampling time _t sampling time operation data Pd _i as input data in the _{i t i +} execution operation modified in _one command Pn By outputting _{i + 1} , the execution operation command reflecting the execution operation correction command Pni _{+ 1} is output. Further, in the training mode, the calculation unit 36 outputs the execution operation correction command Pni _{+ 1} output by the neural network to the difference detection unit 38.

Note that the neural network machine learning model 36a as the input data, the operation data Pd _i at the sampling time _{t i,} the sampling time _{t i} earlier sampling time _{t i-1 ~t i-n} (n is a predetermined natural number) The operation data Pd _{i-1 to} Pd _{i- n} of the above may be input. In this case, when the machine learning data input unit 36c, with respect to learning data pd at the sampling time _{t i,} the time-series data _{Pd i,} the _{Pd i-1} ~Pd _i-n type to the neural network, the neural network, following Outputs the execution operation correction command Pni _{+ 1} at the sampling time ti _{+ 1} . The learning evaluation unit 36d adjusts the weights between the neurons of the neural network with respect to the execution motion correction command Pni _{+ 1} and the time series data pn _{i + 1 of the} teacher data pn.

Further, when the data input and output, the neural network machine learning models 36a, with respect to the sampling time _{t i,} the sampling time _{t i, t i-1 ~t} i-n operate in data _{Pd i, Pd i-1 ~Pd} i and it outputs an execution operation correction command _{Pn i + 1} at the sampling time _{t i + 1} to _-n as input data. Such a neural network can improve its learning efficiency and learning accuracy. Such a neural network predicts the next movement of the working part 11 of the mechanical device 10 based on not only the movement data at the present moment but also a series of movement data from before that, so that accurate prediction is possible. To.

The neural network as described above is constructed for each type of complex operation that can be executed by the mechanical device 10, and one neural network is configured to correspond to one type of complex operation. In some cases, one neural network may be configured to support a plurality of types of complex operations.

Further, the difference detection unit 38 functions in the training mode. The difference detection unit 38 detects the difference between the operation of the mechanical device 10 controlled by the operation command unit 32 and the operation of the mechanical device 10 corresponding to the command output by the calculation unit 36. Specifically, the difference detection unit 38 detects the difference between the execution operation command received from the operation command unit 32 and the execution operation correction command received from the calculation unit 36. The execution operation command and the execution operation correction command for which the difference is detected are operation commands corresponding to each other. The execution operation correction command corresponding to the execution operation command is output data of the calculation unit 36 when the operation information of the mechanical device 10 immediately before executing the operation of the execution operation command is used as input data. For example, executing the operation command at the sampling time _{t i} corresponds with the operation data _{Pd i-1} the next sampling time _t execution operation correction command in _i Pn _i is output as the input data at the sampling time _{t i-1.}

The auxiliary information processing unit 39 functions in the training mode. The auxiliary information processing unit 39 generates an auxiliary command for assisting the operation in the operation device 20 based on the difference detected by the difference detection unit 38, and outputs the auxiliary command to the operation device 20. The auxiliary information processing unit 39 generates an auxiliary command when the difference is equal to or more than the threshold value, and does not generate an auxiliary command when the difference is less than the threshold value. The auxiliary command is a command for generating a stimulus that makes the operator perceive that the difference is equal to or more than the threshold value. Examples of stimuli are light, sound, vibration, electrical stimuli, temperature, and tactile sensation using haptics devices. The auxiliary command may be a command to generate a stimulus to the operating device 20, or may be a command to generate a stimulus to a device around the operator. In this embodiment, the auxiliary command causes the operating device 20 to generate a stimulus.

If the stimulus is a tactile sensation using a haptics device, the auxiliary command may be a command to generate a tactile sensation in the haptics device so that the degree of tactile sensation corresponds to the magnitude of the difference. .. For example, the auxiliary command may be a command that generates a tactile force sense that guides the operation of the operating device 20 in a direction that reduces the difference. For example, when the amount of movement of the operating device 20 in a certain direction is excessive and the difference exceeds the threshold value, the auxiliary command generates a tactile sensation of pushing the operator's hand in the direction opposite to the above direction. It may be a command to make it. The operator can receive guidance in the operation direction, that is, receive operation assistance by receiving the above-mentioned tactile force sense in the hand holding the operation device 20.

[1-2. motion]
The operation of the mechanical device system 1 according to the embodiment in the automatic operation mode, the operation in the training mode of the manual operation mode, and the operation in the non-training mode of the manual operation mode will be described.

[1-2-1. Operation in automatic operation mode]
FIG. 5 is a flowchart showing an example of the operation of the mechanical device system 1 according to the embodiment in the automatic operation mode. Further, FIG. 5 shows an example in which the mechanical device system 1 causes the mechanical device 10 to perform a predetermined operation for one cycle. In this example, the mechanical device system 1 will be described as having the mechanical device 10 automatically perform all predetermined operations.

As shown in FIG. 5, first, the operator inputs a command to execute a predetermined operation in the automatic operation mode to the mechanical device system 1, and the control device 30 receives the command (step S101). In this case, the operator may input via the operating device 20 or may input via another input device included in the mechanical device system 1. In this example, it is assumed that the predetermined operation is a complex operation.

Next, the operation determination unit 31 of the control device 30 acquires the operation information corresponding to the predetermined operation (step S102). The operation determination unit 31 extracts operation information corresponding to each individual operation included in a predetermined operation from the third storage unit 42, and sequentially outputs the operation information to the operation command unit 32. Further, the operation determination unit 31 outputs the content of the predetermined operation to the calculation unit 36.

Next, the operation command unit 32 determines whether or not there is incomplete operation information among the operation information corresponding to the individual operations included in the predetermined operation, that is, there is an incomplete individual operation. Whether or not it is determined (step S103). The operation command unit 32 proceeds to step S104 when there is incomplete operation information (Yes in step S103), and ends a series of processes when there is no incomplete operation information (No in step S103).

In step S104, the calculation unit 36, operation information such as the action part 11 of the machine 10, specifically, to acquire the operation data Pd _i included in the operation information. Operation data Pd _i is located operation data at time _{t i,} a time _t initial value Pd ₀ in ₀ a at the beginning of the process. At this time, the calculation unit 36 may request the operation information from the operation information processing unit 37 of the control device 30. The motion information processing unit 37 may request the motion information detection device 50 to detect the motion information and acquire the detection result of the motion information detection device 50. Alternatively, the calculation unit 36 may receive operation information from the operation information processing unit 37 in step S112, which will be described later, and acquire operation data from the operation information. In step S112, the calculation unit 36 stores the operation data in the first storage unit 40 of the control device 30. The operation information that has been processed may be acquired.

Then, the calculation unit 36, a neural network machine learning models 36a corresponding to a predetermined operation, to generate an execution operation correction command Pm i _{+ 1} by inputting the operation data Pd _i, operation command execution operation correction command Pm i _{+ 1} Output to unit 32 (step S105).

Next, the operation command unit 32 generates an execution operation command for causing the mechanical device 10 to execute the operation by using the operation information corresponding to the predetermined operation, and outputs the execution operation command to the correction command unit 33 (step S106). Specifically, the operation command unit 32 is for executing the operation information corresponding to the individual operation for the individual operation to be executed at the earliest among the unfinished individual operations included in the predetermined operation. The decision operation command Ps _{i + 1} , which is an operation command, is generated. Further, the operation command unit 32 generates the execution operation command Pe _{i + 1} based on the decision operation command Ps _{i + 1} and the execution operation correction command Pmi _{+ 1} . The decision operation command Ps _{i + 1} and the execution operation correction command Pmi _{+ 1} are commands corresponding to the time ti _{+ 1} .

Next, the correction command unit 33 determines whether or not there is a correction input that is an input for correcting the operation of the mechanical device 10 from the operation device 20 (step S107). The correction command unit 33 proceeds to step S108 when there is a correction input (Yes in step S107), and proceeds to step S109 when there is no correction input (Nо in step S107).

In step S108, the correction command unit 33 corrects the execution operation command Pe _{i + 1} of the operation command unit 32 according to the operation information output from the operation device 20, and outputs the correction command unit 33 to the drive command unit 34. The correction command unit 33 adds the operation operation command Pо _{i + 1} for causing the action unit 11 to perform the operation corresponding to the operation information and the execution operation command Pe _{i + 1 of} the operation command unit 32, thereby issuing the correction operation command Pf _{i + 1} . Generate.

In step S109, the correction command unit 33 outputs the execution operation command Pe _{i + 1} of the operation command unit 32 to the drive command unit 34.

Next, in step S110, the correction information detection unit 35 detects the correction information and stores it in the second storage unit 41. When the execution operation command Pe _{i + 1} is modified, the modification information detection unit 35 detects the modification operation command Pf _{i + 1} as the modification information. If the execution operation command Pe _i is not modified, the correction information detection unit 35 detects the uncorrected execution operation command Pe _{i + 1} as the correction information.

Next, the drive command unit 34 generates drive data which is a command for driving each drive device of the mechanical device 10 so that the action unit 11 performs an operation corresponding to the correction operation command Pfi _{+ 1} or the execution operation command Pe _{i + 1.} , Output to each drive. That is, the drive command unit 34 drives the mechanical device 10 so as to perform an operation corresponding to the command (step S111).

Next, the operation information detection device 50 detects the operation data Pdi _{+ 1} as the operation information of the operating mechanical device 10, and stores it in the first storage unit 40 (step S112). The operation information detection device 50 outputs the detected operation information, which is the detected operation data Pdi _{+ 1} , to the first storage unit 40 and the operation information processing unit 37. The operation information processing unit 37 outputs the detected operation information to the calculation unit 36, the operation device 20, and the output device 60. Further, the motion information processing unit 37 returns to the process of step S103.

The operation device 20 provides the operator with a tactile force sense corresponding to the force data and the position data of the operation data included in the detection operation information. The tactile force sense can indicate the operating state of the acting unit 11. For example, in the operation device 20, the action unit 11 presses the object by giving the operator's hand holding the operation device 20 a tactile force sensation as if the operator pushes the object by himself / herself. Experience the condition. The operating device 20 gives the operator a tactile sensation of pulling by himself / herself, so that he / she can experience a state in which the acting unit 11 is pulling or lifting an object. The operating device 20 gives a tactile sense of the texture of the surface so that the operating portion 11 can experience the roughness state of the surface of the object in contact with the object. The operating device 20 gives a tactile sense of pressure to experience the hardness and softness of the surface of the object with which the acting portion 11 is in contact.

The output device 60 visually and / or aurally indicates to the operator the position and posture of the action unit 11 with respect to the object based on the position data of the operation data included in the detection operation information.

In steps S103 to S112 described above, processing related to the operation to be executed at the sampling time ti _{+ 1} is performed, but in the next steps S103 to S112, processing related to the operation to be executed at the next sampling time ti _{+ 2} is performed.

The controller 30, after completion of a predetermined operation by using the operation information detected at the sampling time t ₀ ~t _u, corresponding to each individual operation included in the predetermined operation is stored in the third storage unit 42 The operation information to be performed may be updated.

In the above, the control device 30, at the sampling time t ₀ ~t _u each timing, to modify the determined operation command with an execution operation correction instruction of the arithmetic unit 36, but generates an execution operation command, to Not limited. The control device 30 may generate the execution operation command as described above at the timing when the individual operations included in the predetermined operation change.

Further, the control device 30 may perform machine learning of the calculation unit 36 at any time. For example, the control device 30 may cause the calculation unit 36 to perform machine learning using the data accumulated in each predetermined work each time the predetermined work by the mechanical device 10 is completed. Alternatively, the control device 30 may cause the calculation unit 36 to perform machine learning using the data accumulated in the predetermined work of the predetermined number of times each time the predetermined work of the mechanical device 10 is completed. Alternatively, the control device 30 causes the calculation unit 36 to use the data accumulated in the predetermined work in the predetermined period for each predetermined period such as a predetermined number of days, a predetermined number of weeks, and a predetermined number of months. Machine learning may be performed.

[1-2-2. Operation in training mode]
FIG. 6 is a flowchart showing an example of the operation of the mechanical device system 1 according to the embodiment in the training mode of the manual operation mode. Further, FIG. 6 shows an example in which the operator causes the mechanical device 10 to perform a predetermined operation for one cycle by using the operating device 20. In this example, the mechanical device system 1 will be described as having the mechanical device 10 perform all predetermined operations according to the operation of the operator.

As shown in FIG. 6, first, the operator inputs a command to execute a predetermined operation in the training mode of the manual operation mode to the mechanical device system 1, and the control device 30 receives the command (step S201).

Next, the operation determination unit 31 of the control device 30 outputs the content of the predetermined operation to the calculation unit 36. Calculation unit 36, operation information such as the action part 11 of the machine 10, specifically, to acquire the operation data Pd _i included in the operation information (step S202). At this time, the calculation unit 36 may request the operation information from the operation information detection device 50 via the operation information processing unit 37 of the control device 30 and acquire the detection result of the operation information detection device 50.

Then, the calculation unit 36, based on the operation data Pd _i, determines whether a predetermined operation has not been completed (step S203). If it is not completed (Yes in step S203), the calculation unit 36 proceeds to step S204, and if it is completed (No in step S203), the calculation unit 36 ends a series of processes.

In step S204, the calculation unit 36, the neural network of the corresponding machine learning models 36a to a predetermined operation, to generate an execution operation correction command _{Pm i + 1} by inputting the operation data Pd _i, the execution operation correction command _{Pm i + 1} Output to the difference detection unit 38.

Next, when the operator operates the operation device 20, the operation command unit 32 receives the operation information input to the operation device 20 by the operator (step S205). Further, the operation command unit 32 generates an operation operation command Pо _{i + 1} from the operation information as an execution operation command based on the operation information (step S206). Operation Operation command Pо _{i + 1} is the operation command generated from the operation information input to the operation unit 20 at time t _{i + 1,} which is to be the operation command to the operation device 20 at time t _{i + 1.} Further, the operation command unit 32 outputs the operation operation command Pо _{i + 1} to the difference detection unit 38 and the drive command unit 34.

Then, the difference detecting unit 38 detects a difference _{D i + 1} of the execution operation correction instruction _{Pm i + 1} and the operating movement command Pо _{i + 1} (step S207). Next, the auxiliary information processing unit 39 detects whether or not the difference _{Di + 1} is equal to or greater than the threshold value Tr, that is, _{Di + 1} ≧ Tr (step S208). When Di _{+ 1} ≧ Tr (Yes in step S208), the auxiliary information processing unit 39 proceeds to step S209, and when Di _{+ 1} <Tr (Nо in step S208), proceeds to step S211.

In step S209, the auxiliary information processing unit 39 generates an auxiliary command for generating a stimulus in the operation device 20 and outputs the auxiliary command to the operation device 20. Such an auxiliary command is a command for indicating that there is a large difference between the output result of the arithmetic unit 36 using machine learning and the operation result of the operator, and the operation result of the operator is an ideal operation. It is a directive to indicate that it deviates from. The auxiliary information processing unit 39 may generate an auxiliary command that generates a stimulus corresponding to the magnitude of the difference _{Di + 1} .

Next, the operating device 20 receives the auxiliary command from the auxiliary information processing unit 39 and gives the operator a stimulus corresponding to the auxiliary command (step S210). The operator who perceives the stimulus can recognize that his / her operation is inappropriate. Further, the operator can easily bring his / her own operation closer to the ideal operation by receiving the stimulus corresponding to the magnitude of the difference _{Di + 1} .

Next, the drive command unit 34 generates drive data for each drive device of the mechanical device 10 so that the action unit 11 performs an operation corresponding to the operation operation command Pо _{i + 1} received from the operation command unit 32 in step S206. Output to each drive. That is, the drive command unit 34 drives the mechanical device 10 so as to perform an operation corresponding to the command (step S211). Further, the drive command unit 34 returns to the process of step S202.

In steps S202 to S211 described above, processing related to the operation executed at the sampling time ti _{+ 1} is performed, but in the next steps S202 to S211, processing related to the operation executed at the next sampling time ti _{+ 2} is performed.

In the above, the controller 30 at the timing of each sampling time t _i, based on the difference in the execution behavior modification command of the arithmetic unit 36 and the operation operation command operation command unit 32, but to generate a stimulus to the operation apparatus 20, Not limited to this. The control device 30 may generate the above-mentioned command at the timing when the individual operations included in the predetermined operations change.

[1-2-3. Operation in non-training mode]
An example of the operation of the mechanical device system 1 in the non-training mode of the manual operation mode will be described with reference to FIG. First, the operator inputs a command to execute the non-training mode of the manual operation mode to the mechanical device system 1, and the control device 30 receives the command. Then, when the operator operates the operation device 20, the operation command unit 32 of the control device 30 receives the operation information input to the operation device 20 by the operator. Further, the operation command unit 32 generates an operation operation command from the operation information as an execution operation command based on the operation information, and outputs the operation operation command to the drive command unit 34.

The drive command unit 34 generates drive data for each drive device of the mechanical device 10 and outputs the drive data to each drive device so that the action unit 11 performs an operation corresponding to the operation operation command received from the operation command unit 32. As a result, the mechanical device 10 operates according to the operation information input to the operation device 20. As described above, when the control device 30 receives an input to the operation device 20, the control device 30 operates the mechanical device 10 according to the input operation information.

[1-3. Effect, etc.]
As described above, in the mechanical device system 1 according to the embodiment, the control device 30 of the mechanical device 10 operates the mechanical device 10 according to the operation information output from the operating device 20 for operating the mechanical device 10. The operation command unit 32 and the drive command unit 34 as the operation control unit to be controlled, and the first operation information indicating the operation of the mechanical device 10 are used as input data, and the operation command of the mechanical device 10 corresponding to the first operation information is output. Between the operation of the arithmetic unit 36 including the machine learning model 36a as data, the operation of the mechanical device 10 controlled by the operation command unit 32, and the operation of the mechanical device 10 corresponding to the command output by the arithmetic unit 36. It includes a difference detection unit 38 and an auxiliary information processing unit 39 as auxiliary units that output an auxiliary command that assists the operation of the operation device 20 based on the difference. Further, the operation device 20 outputs operation information based on the second operation information indicating the operation of the operation device 20.

According to the above configuration, the machine learning model 36a of the calculation unit 36 outputs a command for the operation of the machine device 10 corresponding to the first operation information indicating the operation of the machine device 10. The auxiliary information processing unit 39 outputs an auxiliary command for assisting the operation of the operation device 20 based on the difference between the operation command and the operation of the mechanical device 10. By receiving assistance based on such an auxiliary command, the operator can bring his or her own operation closer to the ideal operation obtained by machine learning, that is, can be trained in the operation. Therefore, the skill inheritance of the operation of the mechanical device 10 using the machine learning model 36a becomes possible.

Further, in the mechanical device system 1 according to the embodiment, the operating device 20 includes a perceptual device that gives a perceptual stimulus to the operator holding the operating device 20, and the auxiliary information processing unit 39 gives the perceptual stimulus to the perceptual device. May be output to the operating device 20 to generate an auxiliary command. The sensory device may be a haptics device that gives a tactile stimulus to the operator. According to the above configuration, the control device 30 gives the operator a perceptual stimulus to assist the operation in the operating device 20. The operator can recognize the assistance for the operation through the experience. Therefore, the operator can easily and surely recognize the assistance. Furthermore, the stimulus using the haptics device can make the operator recognize the contents of various assists.

Further, in the mechanical device system 1 according to the embodiment, the control device 30 includes an operation information processing unit 37 as a processing unit that outputs first operation information indicating the operation of the mechanical device 10 to the operation device 20, and is a perception device. The haptics device as may give the operator a perceptual stimulus corresponding to the operating state of the mechanical device 10 based on the first motion information. According to the above configuration, the operator can operate the operating device 20 while experiencing the operation of the mechanical device 10. Therefore, the operator can appropriately operate the mechanical device 10 using the operating device 20.

Further, in the mechanical device system 1 according to the embodiment, the operation device 20 may include an inertial measurement unit and output operation information based on the measurement data of the inertial measurement unit as the second operation information. According to the above configuration, the operation device 20 outputs operation information based on the second operation information indicating the operation of the operation device 20. Since the operation information is information based on the measurement data of the inertial measurement unit, the operation of the operation device 20 can be accurately shown. Therefore, the accuracy of the operation information is improved, and the operation content of the operation device 20 is reflected in the operation of the mechanical device 10 with high accuracy.

Further, in the mechanical device system 1 according to the embodiment, the operating device 20 may be configured to be movable in an arbitrary direction in a three-dimensional space. According to the above configuration, the operating device 20 can cause the mechanical device 10 to perform various operations.

Further, in the mechanical device system 1 according to the embodiment, the first operation information indicating the operation of the mechanical device 10 may include force data representing the force applied by the mechanical device 10 to the object. According to the above configuration, the calculation unit 36 outputs a command for the operation of the mechanical device 10 in consideration of the force applied by the mechanical device 10 to the object. Therefore, the auxiliary information processing unit 39 can output an auxiliary command in consideration of the state of force of the acting unit 11 of the mechanical device 10. Such an auxiliary command can assist the operation in the operating device 20 with high accuracy.

Further, in the mechanical device system 1 according to the embodiment, the first operation information indicating the operation of the mechanical device 10 may include position data indicating the position of the mechanical device 10. According to the above configuration, the calculation unit 36 outputs a command for the operation of the mechanical device 10 in consideration of the position of the mechanical device 10. Therefore, the auxiliary information processing unit 39 can output an auxiliary command in consideration of the position of the working unit 11 or the like of the mechanical device 10. Such an auxiliary command can assist the operation in the operating device 20 with high accuracy.

Further, in the mechanical device system 1 according to the embodiment, the machine learning model 36a may be configured by a neural network. According to the above configuration, the neural network enables flexible and highly accurate processing. Therefore, the machine learning model 36a can output highly accurate output data for various input data.

Further, in the mechanical device system 1 according to the embodiment, the first operation information indicating the operation of the mechanical device 10 may include the current operation and the past operation of the mechanical device 10. According to the above configuration, the first operation information indicates time series information of the operation of the mechanical device 10. Then, the machine learning model 36a uses such time series information as input data. Therefore, the machine learning model 36a outputs a command for the operation of the machine device 10 in consideration of the behavior of the machine device 10 over time. Therefore, the output accuracy of the machine learning model 36a is improved.

Further, in the mechanical device system 1 according to the embodiment, the control device 30 serves as a correction control unit that corrects the operation of the mechanical device 10 according to the operation information output from the operating device 20 in the automatic operation of the mechanical device 10. The correction command unit 33 and the drive command unit 34, the first storage unit 40 that stores the first operation information of the mechanical device 10, and the second storage unit that stores the correction information indicating the correction made by the correction command unit 33. 41 may be provided. Further, the calculation unit 36 may cause the machine learning model 36a to perform machine learning using the first operation information and the correction information corresponding to the first operation information. Further, the operation command unit 32 controls the operation of the mechanical device 10 based on the command of the calculation unit 36 in the automatic operation, and controls the operation of the mechanical device 10 according to the operation information of the operation device 20 in the manual operation of the mechanical device 10. However, the auxiliary information processing unit 39 may output an auxiliary command in the manual operation.

According to the above configuration, the calculation unit 36 uses as learning data the first operation information indicating the operation of the mechanical device 10 and the correction information indicating the correction of the operation of the mechanical device 10 performed by using the operation device 20. Let the machine learning model 36a perform machine learning. That is, the arithmetic unit 36 executes machine learning by itself. Since the calculation unit 36 causes the machine learning model 36a to perform machine learning using the control result of the machine device 10, the learning accuracy of the machine learning model 36a can be improved as the machine device 10 is operated. Further, since the learning data is generated by the operator modifying the operation of the mechanical device 10 via the operating device 20, the generation is simple. Further, the modification of the operation of the mechanical device 10 is appropriate because it is performed by the operator who has confirmed the operation of the mechanical device 10. Therefore, it is possible to easily generate appropriate learning data. The machine learning model 36a that performs machine learning using such learning data can achieve an output accuracy in a short period of time that outputs a command corresponding to the operation of the machine device 10 that is ideal for an operator or the like. Therefore, the time required for machine learning can be shortened.

Further, the control system 100 according to the embodiment includes the control device 30 according to the embodiment and the operation device 20. According to the above configuration, the same effect as that of the control device 30 according to the embodiment can be obtained.

Further, the mechanical device system 1 according to the embodiment includes a control device 30, a mechanical device 10, and an operating device 20 according to the embodiment. According to the above configuration, the same effect as that of the control device 30 according to the embodiment can be obtained.

(Modification example 1)
The mechanical device system 1A according to the modified example of the embodiment will be described. In this modification, the mechanical device system 1A includes the robot 10A as a mechanical device, and controls the operation of the robot 10A by using not only the motion data but also the image of the processing target of the robot 10A. Hereinafter, the present modification will be described focusing on the points different from the embodiment, and the description of the same points as the embodiment will be omitted as appropriate.

[2-1. Mechanical system]
FIG. 7 is a functional block diagram showing an example of the configuration of the mechanical device system 1A according to the modified example 1. As shown in FIG. 7, the mechanical device system 1A according to the present modification further includes an imaging device 70 as compared with the mechanical device system 1 according to the embodiment. Further, the mechanical device system 1A includes a robot 10A as a mechanical device 10 and a control device 30A as a control device 30. The robot 10A includes an end effector 11A and a robot arm 12A, the end effector 11A corresponds to an action unit 11, and the robot arm 12A corresponds to an operation unit 12. Details of the robot 10A will be described later.

The image pickup device 70 images an object to be processed by the robot 10A. Examples of the image pickup apparatus 70 are a digital camera, a digital video camera, and the like. For example, the image pickup device 70 is arranged on the end effector 11A or the robot arm 12A or the like, but may be arranged at a position away from the end effector 11A or the robot arm 12A. The image pickup device 70 outputs a signal of the captured image to the control device 30A. The image pickup apparatus 70 may output an image signal to the output apparatus 60. As a result, the operator can confirm the processing state of the object by the end effector 11A via the output device 60. Then, the operator can control the operation of the end effector 11A using the operating device 20 while checking the processing state of the object.

[2-2. robot]
FIG. 8 is a side view showing an example of the configuration of the robot 10A according to the first modification. As shown in FIG. 8, the base of the robot arm 12A of the robot 10A is attached and fixed to the base 13, and the end effector 11A is detachably attached to the tip of the robot arm 12A. The end effector 11A is configured so that various actions corresponding to the object such as gripping, suction, lifting, or scooping can be applied to the object. In the example of FIG. 7, the end effector 11A is configured to grip the object W, and the robot 10A performs an operation of assembling the object W gripped by the end effector 11A to the assembly object T. The work of the robot 10A is not limited to assembly, and may be any work. Examples of work of the robot 10A are medical practices such as sorting, assembling, painting, welding, joining, chipping, polishing, sealing, semiconductor manufacturing, drug preparation and surgery.

The robot arm 12A includes joints JT1 to JT6 for sequentially connecting links 12Aa to 12Af and links 12Aa to 12Af arranged in order from the base to the tip, and arm driving devices M1 to rotationally drive each of the joints JT1 to JT6. It is equipped with M6. The operation of the arm drive devices M1 to M6 is controlled by the control device 30A. Although not limited to this, in the present embodiment, each of the arm driving devices M1 to M6 uses electric power as a power source and has a servomotor as an electric motor for driving them. The number of joints of the robot arm 12A is not limited to 6, but may be 7 or more, or 1 or more and 5 or less.

The link 12Aa is attached to the mounting surface 13a of the base 13, and the end effector 11A is attached to the tip of the link 12Af. A mechanical interface is provided at the tip of the link 12Af. The end effector 11A is attached to the mechanical interface via the force sensor 14. An example of the force sensor 14 is a force sensor or the like, and the configuration of the force sensor is not particularly limited, but may be composed of, for example, a three-axis acceleration sensor. The force sensor 14 detects the force exerted by the end effector 11A on the object as a reaction force received from the object. The force detected by the force sensor 14 is converted into force data by an appropriate signal processing means (not shown). This signal processing means is provided in, for example, the force sensor 14 or the control device 30. In the present specification, for convenience, it is expressed that the force sensor 14 detects force data.

The joint JT1 rotatably connects the base 13 and the base end portion of the link 12Aa around an axis in the vertical direction perpendicular to the mounting surface 13a. The joint JT2 rotatably connects the tip end of the link 12Aa and the base end of the link 12Ab around a horizontal axis parallel to the mounting surface 13a. The joint JT3 rotatably connects the tip end of the link 12Ab and the base end of the link 12Ac around an axis in a direction parallel to the mounting surface 13a. The joint JT4 rotatably connects the tip end of the link 12Ac and the proximal end of the link 12Ac about the longitudinal axis of the link 12Ac. The joint JT5 rotatably connects the tip end of the link 12Ad and the base end of the link 12Ae about an axis in a direction orthogonal to the longitudinal direction of the link 12Ad. The joint JT6 connects the tip end of the link 12Ae and the base end of the link 12Af in a twistable and rotatable manner with respect to the link 12Ae.

The arm drive devices M1 to M6 are a servomotor (not shown), a rotation sensor such as an encoder (not shown) that detects the amount of rotation of the rotor of the servomotor, and a current that detects the drive current of the servomotor, respectively. A sensor (not shown) may be provided. Each of the arm drive devices M1 to M6 operates the servomotor according to a command or the like output from the control device 30A, and outputs the detection value of the rotation sensor to the control device 30A. The control device 30A detects the rotation amount and rotation speed of the rotor of the servomotor based on the detection values of the rotation sensors fed back from each servomotor, and uses the detection results to start and rotate the servomotor. Controls stop, rotation speed and rotation torque. As a result, the control device 30A can stop each servomotor at an arbitrary rotation position, rotate it at an arbitrary rotation speed, and operate it at an arbitrary rotation torque. Therefore, the control device 30A can operate the robot arm 12A in various and precise manners.

The operation information calculation unit 43 of the control device 30A, which will be described later, calculates the three-dimensional position of the end effector 11A as position data by integrating the rotation amounts of all the servomotors of the arm drive devices M1 to M6. Further, the data detected by the force sensor 14 is the force data. The position data and the force data are operation data of the robot 10A. The rotation sensor and the force sensor 14 of the arm drive devices M1 to M6 constitute an operation information detection device 50. The detection signals of the current sensors of the arm drive devices M1 to M6 are used by the control device 30A to feedback-control the currents of the servomotors of the arm drive devices M1 to M6 so as to be in accordance with the current command. .. As described above, the robot 10A is configured as a vertical articulated robot, but is not limited thereto.

[2-3. Operating device]
FIG. 9 is a diagram showing an example of the appearance of the operating device 20 according to the modified example 1. FIG. 10 is a functional block diagram showing an example of the configuration of the operation device 20 according to the first modification. As shown in FIG. 9, the operating device 20 includes a housing 20a that can be grasped by a human hand. Further, the operation device 20 includes an input device 21 in the housing 20a. In FIG. 9, the input device 21 is a button switch, but the present invention is not limited to this. Further, the operation device 20 includes an inertial measurement unit 22, a haptics device 23, an operation control device 24, and a communication device 25 (not shown) inside the housing 20a. In the mechanical device system 1A, the control device 30A performs bilateral control on the robot 10A by using the operation device 20.

The components of the operating device 20 will be described with reference to FIG. The haptics device 23 is as described in the embodiment.

The communication device 25 connects the operation device 20 and the control device 30A via wired communication or wireless communication. The communication device 25 may include a communication circuit. The format of wired communication and wireless communication may be any format.

The input device 21 receives input of commands and information by the operator, and transmits the input commands and information to the control device 30A via the operation control device 24 and the communication device 25. Such an input device 21 may accept physical input, voice input, image input, and the like. For example, the input device 21 may include devices such as a slide switch, a button switch, a key, a lever, a touch panel, a microphone, and a camera. For example, the commands and information input to the input device 21 may indicate the operation mode selection and execution command of the robot 10A, the operation selection and execution command of the end effector 11A, and the like.

The inertial measurement unit 22 includes a 3-axis acceleration sensor and a 3-axis angular velocity sensor, and detects acceleration and angular velocity in the 3-axis direction of the operating device 20. The measurement data of the acceleration and the angular velocity in the three axial directions detected by the inertial measurement unit 22 show various movements and acting forces of the operation device 20 such as position, posture, movement, movement speed and force by the operation control device 24. It is converted into information, and the information is transmitted to the control device 30A via the communication device 25 as the operation information of the operation device 20. The measurement data of the acceleration and the angular velocity in the three axial directions may be transmitted to the control device 30A, and the control device 30A may perform an operation to convert the data. The information converted from the measurement data of the inertial measurement unit 22 may indicate the position, posture, movement, moving speed, acting force, etc. of the end effector 11A. The inertial measurement unit 22 may include a geomagnetic sensor, a temperature sensor, and the like. For example, the measurement data of the acceleration and the angular velocity in the three-axis direction may be corrected by using the measurement data of the geomagnetic sensor, the temperature sensor, and the like.

The operation control device 24 controls the overall operation of the operation device 20. The operation control device 24 may have a configuration similar to the configuration exemplified in the embodiment for the control device 30. For example, the operation control device 24 receives a signal from the input device 21, converts the signal into information indicating the corresponding operation, and transmits the signal to the control device 30A. Further, the operation control device 24 converts the measurement data of the inertial measurement unit 22 and transmits the converted data to the control device 30A. Alternatively, the operation control device 24 transmits the measurement data of the inertial measurement unit 22 to the control device 30A. The operation control device 24 receives the operation information of the robot 10A from the control device 30A, converts the operation data and the like included in the operation information into data suitable for input to the haptics device 23, and outputs the data to the haptics device 23. ..

[2-4. Control device]
The configuration of the control device 30A will be described. As shown in FIG. 7, the control device 30A according to the present modification has the operation information calculation unit 43, the image processing unit 44, and the fourth storage unit 45 as compared with the control device 30 according to the embodiment. Further included, the calculation unit 36A is included instead of the calculation unit 36.

The operation information calculation unit 43 converts the data received from the operation information detection device 50 into operation data, and outputs the data to the operation information processing unit 37 and the first storage unit 40. Specifically, the motion information detection device 50 includes data on the amount of rotation of the servomotor detected by using the rotation sensors of the arm drive devices M1 to M6 of the robot 10A, and data on the force of force detected by using the force sensor 14. Is output to the operation information calculation unit 43. The operation information calculation unit 43 calculates the position data indicating the three-dimensional position of the end effector 11A by integrating the rotation amounts of all the servomotors of the arm drive devices M1 to M6. The operation information calculation unit 43 generates and outputs operation data including force data and position data at the same detection time in association with the detection time.

The image processing unit 44 receives image data indicating an image captured by the image pickup apparatus 70, and performs image processing on the image data. The image processing unit 44 extracts the object and the end effector 11A included in the image by image processing, and generates processed image data which is the image data of the image including only the object and the end effector 11A. The image processing unit 44 stores the processed image data in the fourth storage unit 45 in association with the imaging time. The image processing unit 44 may also store the image data before processing in the fourth storage unit 45.

The method for extracting the image of the object and the end effector 11A from the image may be any known method. For example, the image processing unit 44 may extract an image of the object and the end effector 11A by using an image matching method such as feature-based or region-based.

For example, in the case of feature-based, the image processing unit 44 may extract feature points such as edges and corners from the image before processing and calculate the feature amount of the feature points. Further, the image processing unit 44 performs matching between the image before processing and the template of the image of the object and the end effector 11A based on the feature amount of the feature point, so that the image before processing is used as the object and the end. The image of the effector 11A may be extracted. Further, in the case of region-based, the image processing unit 44 may specify each region in the image before processing based on the edge, texture, and the like. Further, the image processing unit 44 performs matching between the image before processing and the template of the image of the object and the end effector 11A based on the specified area, so that the object and the end effector 11A from the image before processing are matched. The image of may be extracted. The template of the image of the object and the end effector 11A may be stored in advance in the fourth storage unit 45.

The fourth storage unit 45 is realized by a storage device in the same manner as the first storage unit 40 and the like. The fourth storage unit 45 stores the image data captured by the image pickup apparatus 70, the processed image data processed by the image processing unit 44, the template of the image of the object and the end effector 11A, and the like.

Further, the calculation unit 36A according to this modification performs machine learning using the operation information of the robot 10A and the correction information corresponding to the operation information. Further, the calculation unit 36A uses the operation information of the robot 10A as input data, and uses the command corresponding to the operation information as output data. In this modification, the motion information of the robot 10A includes the motion data of the end effector 11A of the robot 10A and the image data of the object imaged by the imaging device 70. The image data shows the operation information of the end effector 11A such as the positional relationship between the end effector 11A and the object and the processing status of the object by the end effector 11A. In this modification, the image data is the processed image data processed by the image processing unit 44, but may be the image data before processing. By using the processed image data, it is possible to improve the output accuracy of the neural network.

For example, in machine learning, in the neural network of the arithmetic unit 36A, the operation data of the end effector 11A and the image data of the object are used as input data, and the correction information executed at the time of detecting these data is used as the teacher data. At the time of input / output of data, the neural network accepts the input of the operation data of the end effector 11A and the image data of the object, and outputs an execution operation correction command for causing the robot 10A to execute next.

An example of the configuration of the calculation unit 36A will be described. FIG. 11 is a diagram showing an example of the configuration of the calculation unit 36A according to the first modification. As shown in FIG. 11, the calculation unit 36A includes a machine learning model 36Aa, a data generation unit 36b, a data input unit 36c, and a learning evaluation unit 36d.

In this example as well, each command and each data are acquired at a predetermined sampling interval while the robot 10A executes a predetermined operation once. For example, the correction information detector 35, the time-series data _Pm 0 Pm _u corrective action command Pm is obtained at the sampling interval. The operation information detection device 50 acquires the detection data of the end effector 11A at the sampling interval, and the operation information calculation unit 43 calculates the detection data, so that the time series data Pd ₀ of the operation data Pd of the end effector 11A to get the ~Pd _u. Further, the image pickup apparatus 70 acquires the image data in which the object is captured at the sampling interval, and the image processing unit 44 performs image processing on the image data to time-series data of the processed image data Ip that has been image-processed. to get the Ip ₀ ~Ip _u.

The processing of the calculation unit 36A at the time of machine learning will be described. Data generating unit 36b, by using the time series data _Pd 0 -PD _u operation data Pd of the first storage unit 40, and a time series data _Ip 0 _{~Ip u} processing image data Ip of the fourth memory unit 45, The time series data Ld _{0 to} Ld _u of the training data Ld is generated. Time-series data Ld _i is generated using the time-series data Pd _i and Ip _i. Also, the data generating unit 36b generates the time-series data _pn 0 to PN _u teacher data pn from the time-series data _Pm 0 Pm _u modification operation command Pm of the second storage unit 41.

Data input unit 36c sequentially inputs the time-series data _Ld 0 to Ld _u training data Ld to each neuron of the input layer of the neural network machine learning models 36Aa. Neural network receives the input of the time series data Ld _i training data Ld at sampling time t _i, which predicts the execution behavior correction command Pn i _{+ 1} at the next sampling time t _{i + 1} output.

The learning evaluation unit 36d adjusts the weights between the neurons of the neural network based on the time-series data pn _{i + 1} of the teacher data pn at the sampling time ti _{+ 1} and the execution operation correction command Pn _{i + 1} . Data input unit 36c and the learning evaluation unit 36d for all the time-series data _Ld 0 to Ld _u performing the above process.

The processing of the calculation unit 36A at the time of input / output of data will be described. During operation of the robot 10A, the operation information calculation unit 43 uses the detected data of the operation information detecting unit 50 at the current sampling time t _i, and detects the operation data Pd _i, and outputs to the operation unit 36A. In parallel, the image processing unit 44 uses the image data captured by the imaging device 70 to the sampling time t _i, and generates the processed image data Ip _i, and outputs to the operation unit 36A.

Data input unit 36c inputs the operation data Pd _i and the processed image data Ip _i to the neural network machine learning models 36Aa. Neural networks will be able to input data to motion data Pd _i and the processed image data Ip _i, as output data to perform operation correction command Pn i _{+ 1} at the next sampling time t _{i + 1,} and outputs the operation command unit 32. The operation command unit 32 generates an execution operation command that reflects the execution operation correction command Pni _{+ 1} . At each sampling time t _{0 to} tu _-1 , the machine learning model 36Aa performs the above processing. In such processing by the machine learning model 36Aa, the processing state of the object is taken into consideration.

Note that the neural network machine learning models 36Aa, like the embodiment, as input data, the operation data Pd _i and the processed image data Ip _i at the sampling time _{t i,} a past sampling time _{t i-1} ~t _i may be configured such that the operational data _{Pd i-1 ~Pd i-n} and processed image data _{I i-1 ~I i-n} of _-n is input. Further, since the other configurations and operations of the mechanical device system 1A according to the first modification are the same as those of the embodiment, detailed description thereof will be omitted.

[2-5. Effect, etc.]
According to the mechanical device system 1A according to the modification 1 as described above, the same effect as that of the embodiment can be obtained. Further, in the machine device system 1A, the machine learning model 36Aa performs machine learning using the motion information of the robot 10A and the correction information corresponding to the motion information, and uses the motion information of the robot 10A as input data to obtain the motion information. The corresponding command is used as output data. The operation information of the robot 10A includes the operation data of the end effector 11A of the robot 10A and the image data of the object imaged by the imaging device 70. With the above configuration, the machine learning model 36Aa can output not only the operating state of the end effector 11A but also the state of the object to be processed recognized from the image, that is, the processing state. For example, when the robot 10A performs operations such as painting, welding, chipping, polishing, and sealing, the work performance changes depending on the state of the portion to be processed in the object. The machine learning model 36Aa can output an image suitable for the state of the part by using an image including the part as input data. Therefore, the output accuracy of the machine learning model 36Aa is improved. The machine learning model 36Aa that handles motion information including image data may be used in any mechanical device other than a robot.

(Modification 2)
The mechanical device system 1B according to the second modification of the embodiment will be described. In this modification, the machine device system 1B is different from the embodiment and the modification 1 in that the machine learning model is not trained by itself and processing is performed using the given machine learning model. Hereinafter, the present modification will be described focusing on the points different from the embodiment and the modification 1, and the description of the same points as the embodiment and the modification 1 will be omitted as appropriate.

[3-1. Mechanical system]
FIG. 12 is a functional block diagram showing an example of the configuration of the mechanical device system 1B according to the modified example 2. As shown in FIG. 12, the mechanical device system 1B according to the present modification includes the control device 30B as the control device 30 as compared with the mechanical device system 1 according to the embodiment. Further, the control device 30B does not include the correction command unit 33, the correction information detection unit 35, the first storage unit 40, and the second storage unit 41 as compared with the control device 30 according to the embodiment, and the calculation unit 36 Instead, a calculation unit 36B is provided. The configuration of the operation determination unit 31, the operation command unit 32, the drive command unit 34, the operation information processing unit 37, the difference detection unit 38, the auxiliary information processing unit 39, and the third storage unit 42 of the control device 30B is the same as that of the embodiment. Is. The operation command unit 32 outputs an execution operation command to the drive command unit 34 in all modes, and outputs an execution operation command to the difference detection unit 38 in the training mode.

FIG. 13 is a diagram showing an example of the configuration of the calculation unit 36B according to the modification 2. As shown in FIG. 13, the calculation unit 36B includes a machine learning model 36Ba and a data input unit 36c. The calculation unit 36B executes data input / output functions in the automatic operation mode and the training mode. The data input unit 36c inputs the operation data of the machine device 10 detected by the operation information detection device 50 and the operation information processing unit 37 into the neural network of the machine learning model 36Ba. The machine learning model 36Ba has the same configuration as the machine learning model 36a of the embodiment, but is a machine learning model generated through machine learning outside the machine device system 1B and given to the arithmetic unit 36B. For example, the machine learning model 36Ba may be generated through machine learning using not only the learning data collected in one machine learning system 1B but also the learning data collected in another machine learning system. Since such a machine learning model 36Ba uses a lot of learning data for machine learning, it is possible to output output data corresponding to various input data. The machine learning model 36Ba generated in advance is held in a storage unit (not shown) of the calculation unit 36B, and when a new machine learning model is generated, it may be updated by the machine learning model.

[3-2. Effect, etc.]
According to the mechanical device system 1B according to the modified example 2 as described above, the same effect as that of the embodiment can be obtained. Further, in the machine learning system 1B, the machine learning model 36Ba may be a machine learning model given from the outside of the machine learning system 1B. Such a machine learning model 36Ba can be generated by machine learning using learning data collected not only by the mechanical device system 1B but also by other systems and the like. Since the machine learning model 36Ba can output output data corresponding to various input data, its flexibility can be improved. The configuration of the mechanical device system 1B according to the present modification may be applied to the modification 1.

(Other embodiments)
Although the examples of the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments and modifications. That is, various modifications and improvements are possible within the scope of the present invention. For example, a form in which various modifications are applied to the embodiments and modifications, and a form constructed by combining components in different embodiments and modifications are also included in the scope of the present invention.

For example, in the embodiment and the modified example, the auxiliary information processing unit 39 of the control device gives the operator a perceptual stimulus via the operation device 20 in order to assist the operation in the operation device 20. Not limited to. For example, the auxiliary information processing unit 39 may output an auxiliary command to a device other than the operation device 20 to generate a perceptual stimulus to the operator.

For example, the auxiliary information processing unit 39 may be configured to output an auxiliary command for displaying an image assisting the operation in the operation device 20 to an image output device such as an output device 60. For example, as shown in FIG. 14, the auxiliary command sets the operating device 20 and the indicators Ad1 and Ad2 on the image of the robot 10A captured by the camera on the screen S for the operation of the operator in the output device 60. It may be a command to superimpose and display the represented image. For example, the indexes Ad1 and Ad2 may include an arrow indicating the moving direction of the operating device 20 and a numerical value indicating the moving order. Therefore, the operator can receive assistance through vision.

Further, the auxiliary information processing unit 39 may be configured to output an auxiliary command for outputting a voice assisting the operation in the operation device 20 to a voice output device such as the output device 60. Therefore, the operator can receive assistance through hearing.

Further, in the embodiments and modifications, the operating device 20 includes, but is not limited to, the haptics device 23 for giving the operator a perceptual stimulus. The operating device 20 may include any device that stimulates the operator's perception. For example, the operating device 20 may be configured to provide the operator with at least one of tactile, warm, visual and auditory stimuli. The operating device 20 is a device that gives a tactile stimulus by deformation such as expansion / contraction or expansion / contraction of the operating device 20 and vibration, and expands / contracts by air pressure or hydraulic pressure, a device that generates vibration such as a piezoelectric element, and the like. May be provided. The operating device 20 may be provided with, for example, a heater or the like, which stimulates the sense of warmth by generating heat or the like. The operating device 20 may be provided with a light source such as an LED (Light Emitting Diode), for example, by giving a visual stimulus by emitting light or blinking light. The operating device 20 may provide an auditory stimulus by pronunciation or the like, and may include, for example, a speaker or the like.

Further, in the embodiment and the modified example, the control device is configured to execute the training mode in the manual operation mode, but may be configured to execute the training mode also in the automatic operation mode. For example, when a predetermined operation executed in the automatic operation mode includes an operation by automatic operation and an operation by manual operation, the control device selects either a training mode or a non-training mode for the operation by manual operation. May be configured to run.

Further, in the embodiments and modifications, the control device modifies the operations of the mechanical device 10 and the robot 10A according to the operation information output from one operation device 20 in the automatic operation mode, but the present invention is not limited to this. The control device may modify the operations of the mechanical device 10 and the robot 10A according to the operation information output from the two or more operation devices 20. For example, a priority is set for two or more operation devices 20, and the control device determines the operation information to be adopted for correction from the operation information output from the two or more operation devices 20 according to the priority. You may. Alternatively, the control devices 30 and 30A execute processing such as addition, subtraction, averaging, or other statistical processing on the operation information output from the two or more operation devices 20, and after the processing. The operation information of may be adopted for the correction.

Further, in the embodiments and modifications, the information used by the machine learning model for machine learning is the information acquired in the automatic operation mode, specifically, the operation data as the operation information of the machine device 10 and the robot 10A. And at least the operation data of the image data of the object and the correction operation command as the correction information, but the present invention is not limited to this. For example, the machine learning model may use the information acquired in the manual operation mode for machine learning. Such information includes, for example, an execution operation command based on the operation information of the operation device 20, operation data as operation information of the mechanical device 10 and the robot 10A operated according to the execution operation command, and image data of an object. At least it may be operation data. As a result, the machine learning model also machine-learns the operation results of the mechanical device 10 and the robot 10A by the operator, so that the output can be close to that of a human operation.

Further, in the modified example, the robot 10A was a vertical articulated robot, but the robot 10A is not limited to this, for example, a horizontal articulated robot, a polar coordinate robot, a cylindrical coordinate robot, a right angle coordinate robot, and a vertical articulated robot. It may be configured as a type robot or other robot.

Further, the present invention may be a control method. For example, in the control method according to the present invention, the machine is operated according to the operation information output from the operation device for operating the machine, and the machine uses the first operation information indicating the operation of the machine as input data. Input to the learning model to output a command for the operation of the machine device corresponding to the first operation information, and the operation of the machine device and the machine device corresponding to the command output by the machine learning model. Based on the difference between the operation and the operation, the auxiliary command for assisting the operation in the operation device is output, and the operation device outputs the operation information based on the second operation information indicating the operation of the operation device. According to this control method, the same effect as that of the above-mentioned mechanical device system or the like can be obtained. Such a control method may be realized by a circuit such as a CPU and an LSI, an IC card, a single module, or the like.

Further, the present invention may be a program for executing the control method, or may be a non-temporary computer-readable recording medium on which the program is recorded. Needless to say, the above program can be distributed via a transmission medium such as the Internet.

In addition, the numbers such as the ordinal number and the quantity used above are all exemplified for the purpose of concretely explaining the technique of the present invention, and the present invention is not limited to the illustrated numbers. Further, the connection relationship between the constituent elements is exemplified for concretely explaining the technique of the present invention, and the connection relationship for realizing the function of the present invention is not limited to this.

Further, the division of blocks in the functional block diagram is an example, and even if a plurality of blocks are realized as one block, one block is divided into a plurality of blocks, and / or some functions are transferred to another block. Good. In addition, a single piece of hardware or software may process the functions of a plurality of blocks having similar functions in parallel or in a time division manner.

1,1A, 1B Mechanical device system 10 Mechanical device 10A Robot 20 Operation device 22 Inertial measurement unit 23 Haptics device (perceptual device)
30, 30A, 30B Control device 32 Operation command unit (operation control unit)
33 Correction command unit (correction control unit)
34 Drive command unit (motion control unit, correction control unit)
36, 36A, 36B Calculation unit 36a, 36Aa, 36Ba Machine learning model 37 Motion information processing unit (processing unit)
38 Difference detection unit (auxiliary unit)
39 Auxiliary Information Processing Department (Auxiliary Department)
40 First storage unit (memory unit)
41 Second storage unit (memory unit)
50 Operation information detection device 60 Output device (perception device)
70 Imaging device 100 Control system

Claims (17)

It is a control device for mechanical devices.
An operation control unit that controls the operation of the mechanical device according to the operation information output from the operation device for operating the mechanical device.
An arithmetic unit including a machine learning model in which the first operation information indicating the operation of the machine device is input data and the operation command of the machine device corresponding to the first operation information is output data.
An auxiliary command that assists the operation of the operation device based on the difference between the operation of the machine device controlled by the operation control unit and the operation of the machine device corresponding to the command output by the calculation unit. Including the auxiliary part that outputs
The operation device is a control device that outputs the operation information based on the second operation information indicating the operation of the operation device. The operating device includes a perceptual device that gives a perceptual stimulus to the operator holding the operating device.
The control device according to claim 1, wherein the auxiliary unit outputs the auxiliary command for generating a perceptual stimulus to the sensory device to the operation device. The control device according to claim 2, wherein the sensory device is a haptics device that gives a tactile stimulus to the operator. Further including a processing unit that outputs the first operation information to the operation device,
The control device according to claim 2 or 3, wherein the perceptual device gives the operator a perceptual stimulus corresponding to an operating state of the mechanical device based on the first motion information. The control device according to any one of claims 1 to 4, wherein the auxiliary unit outputs the auxiliary command for displaying an image assisting the operation in the operation device to the image output device. The control device according to any one of claims 1 to 5, wherein the auxiliary unit outputs the auxiliary command for outputting a voice assisting the operation in the operation device to the voice output device. The control device according to any one of claims 1 to 6, wherein the operation device includes an inertial measurement unit and outputs the operation information based on the measurement data of the inertial measurement unit as the second operation information. The control device according to any one of claims 1 to 7, wherein the operating device is configured to be movable in an arbitrary direction in a three-dimensional space. The control device according to any one of claims 1 to 8, wherein the first operation information includes force data representing a force applied by the mechanical device to an object. The control device according to any one of claims 1 to 9, wherein the first operation information includes position data representing the position of the mechanical device. The control device according to any one of claims 1 to 10, wherein the first operation information includes image data obtained by capturing an image of an object processed by the mechanical device. The control device according to any one of claims 1 to 11, wherein the machine learning model is composed of a neural network. The control device according to claim 12, wherein the first operation information includes a current operation and a past operation of the mechanical device. In the automatic operation of the mechanical device, a correction control unit that corrects the operation of the mechanical device according to the operation information output from the operating device, and
Further, a storage unit for storing the first operation information and the correction information indicating the correction made by the correction control unit is provided.
The calculation unit causes the machine learning model to perform machine learning using the first operation information and the correction information corresponding to the first operation information.
The motion control unit
In the automatic operation, the operation of the mechanical device is controlled based on the command of the calculation unit.
In the manual operation of the mechanical device, the operation of the mechanical device is controlled according to the operation information.
The control device according to any one of claims 1 to 13, wherein the auxiliary unit outputs the auxiliary command in the manual operation. The control device according to any one of claims 1 to 14,
A control system including the operating device for operating the mechanical device. The control device according to any one of claims 1 to 14,
With the mechanical device
A mechanical device system including the operating device for operating the mechanical device. The mechanical device is operated according to the operation information output from the operating device for operating the mechanical device.
The first operation information indicating the operation of the mechanical device is input to the machine learning model as input data, and the operation command of the mechanical device corresponding to the first operation information is output.
Based on the difference between the operation of the mechanical device and the operation of the mechanical device corresponding to the command output by the machine learning model, an auxiliary command for assisting the operation in the operating device is output.
The operation device is a control method that outputs the operation information based on the second operation information indicating the operation of the operation device.

Images