WO2020202428A1

WO2020202428A1 - Control unit

Info

Publication number: WO2020202428A1
Application number: PCT/JP2019/014443
Authority: WO
Inventors: 神谷　剛志; 村松　啓且; 悠木下
Original assignee: ヤマハ発動機株式会社
Priority date: 2019-04-01
Filing date: 2019-04-01
Publication date: 2020-10-08

Abstract

The present invention addresses the problem of achieving both high reliability and suitability for complex external environments in a control unit that is used in an autonomously operating apparatus. This control unit comprises: an autonomous control circuit that constitutes a control model constructed by machine learning, the control model indicating the relationship between control of a to-be-controlled object and an external environment detected by an external-environment-sensing unit, whereby the autonomous control circuit outputs an operation control signal for controlling the to-be-controlled object on the basis of an external environment signal inputted from the external-environment-sensing unit, and also outputs a condition index signal indicating the condition of operation of the control model; and a monitoring circuit that constitutes a control unit integrally with the autonomous control circuit, the monitoring circuit having a field programmable gate array that has a re-programmable logic circuit, and inspecting at least one signal selected from the group consisting of the external environment signal, the operation control signal, and the condition index signal, thereby monitoring the operation of the autonomous control circuit that constitutes the control model constructed by machine learning.

Description

Controller unit

The present invention relates to a control unit used in an automatic operation device.

There is a known automatic operation device that operates under the control of a control unit. For example, Patent Document 1 discloses an object handling device including a manipulator and a control unit.

The control unit of the object handling device controls the manipulator so as to grip the object based on the gripping information. The gripping information for gripping the object is generated based on the object information. Object information is generated by, for example, a method using deep learning.

JP-A-2018-158391

By applying the results of machine learning represented by deep learning to the control of automatic operation equipment, it is possible to operate based on information in a complicated external environment.

Such automatic operation equipment is required to have high reliability in addition to applicability to complicated external environments.

An object of the present invention is to achieve both application to a complicated external environment and high reliability in a control unit used for an automatic operation device.

The present inventor conducted a study in view of the above-mentioned problems and obtained the following findings.

In control by machine learning, control is performed based on the model of the algorithm constructed as a result of learning. The algorithm can be developed, for example, by inputting a large amount of sample data from a database into a model and extracting useful control criteria from the model. As a result, the model can handle a complex external environment such as image information.

However, in the control by machine learning, unlike the control based on the rule base, the relationship between the input and the output is not registered as a rule. The input / output relationships in a machine learning model are constructed by optimizing the model while inputting a large amount of sample data at the learning stage. Therefore, the quality of the model as a result of the optimization depends on the sample data and the training conditions. Moreover, the input / output relationship cannot be explicitly confirmed. Therefore, it is not easy to verify the quality of the model in advance.
The evaluation of the quality of the model built by machine learning is realized by the test that actually inputs the data. Therefore, high reliability may not be expected for the machine learning model.

Therefore, it is conceivable to monitor the operation of the model by machine learning with a monitoring circuit equipped with a field programmable gate array having a logic circuit. The logic circuit of the field programmable gate array can be constructed on a rule basis. Therefore, if the above monitoring can be realized, it is possible to suppress a situation in which control using the model becomes inappropriate for the controlled object. Furthermore, the circuit itself of the field programmable gate array has higher reliability than the operation using the processor and the program. Therefore, the operation of the automatic control circuit having the control model constructed by machine learning can be monitored with high reliability.

In order to achieve the above object, according to one viewpoint of the present invention, the control unit has the following configuration.

(1) A control unit used in an automatically operating device equipped with an external environment sensing unit that detects an external environment and a control target that is controlled based on the external environment.
The control unit is
An external environment signal input from the external environment sensing unit by constructing a control model constructed by machine learning that shows the relationship between the external environment detected by the external environment sensing unit and the control of the controlled object. An automatic control circuit that outputs an operation control signal for controlling the control target and outputs a status index signal indicating the operation status of the control model based on the above.
The control unit is integrally formed with the automatic control circuit, has a field programmable gate array having a reprogrammable logic circuit, and is selected from the group consisting of the external environment signal, the operation control signal, and the status index signal. A monitoring circuit for monitoring the operation of the automatic control circuit constituting the control model, which is constructed by machine learning by inspecting at least one kind of signal.

The automatic control circuit of the control unit constitutes a control model that shows the relationship between the external environment and the control of the controlled object. The automatic control circuit constitutes a model constructed by machine learning as the above-mentioned control model. Therefore, for example, even if the external environment is complicated information that is difficult to determine on a rule basis, it is possible to output an operation control signal adapted to the external environment. The monitoring circuit monitors the operation of the automatic control circuit having a control model constructed by machine learning. Therefore, it is possible to prevent a situation in which the control by the control model constructed by machine learning becomes inappropriate as the control of the controlled object. Further, since the monitoring circuit includes a field programmable gate array having a logic circuit, the reliability of the monitoring circuit itself and the reliability of the monitoring operation are high. Therefore, the operation of the automatic control circuit having the control model constructed by machine learning can be monitored with high reliability.
Therefore, in the control unit used for the automatic operation device, it is possible to achieve both application to a complicated external environment and high reliability.

According to one viewpoint of the present invention, the control unit can adopt the following configuration.

(2) The control unit of (1)
The control unit is commercial.

According to the above configuration, in a commercial control unit, it is possible to achieve both application to a complicated external environment and high reliability.

(3) The control unit of (1)
The control unit is a mass production type.

According to the above configuration, in a mass-produced control unit, both application to a complicated external environment and high reliability can be achieved.

(4) The control unit of (1)
The logic circuit is programmed to monitor the operation of the automatic control circuit constituting the control model constructed by machine learning with the logic constructed by the rule base.

According to the above configuration, monitoring is executed by the logic constructed by the rule base. Actions based on control models built by machine learning can produce unexpected results, but monitoring the results is less likely to be unexpected. Therefore, the reliability is further improved.

(5) The control unit of (1)
The monitoring circuit operates the automatic control circuit by inspecting whether or not at least one parameter related to the at least one type of signal satisfies a condition defined so as to be associated with the parameter. It is configured to monitor.

According to the above configuration, the inspection conditions can be flexibly dealt with for each signal to be inspected. Therefore, it is possible to achieve both application to a more complicated external environment and high reliability.
The parameters are not particularly limited. Examples of the parameters include the interval at which the signal is input, the value of the data indicated by the signal, the amount of change in the data indicated by the signal, the signal reception intensity, and the like.

(6) The control unit of (5).
The condition defined to be associated with the parameter is that the parameter is included in the range defined to be associated with the parameter.

According to the above configuration, the inspection can be performed by determining whether or not the parameter is included in the defined range. Therefore, the monitoring circuit can be simply configured. In the configuration of (6), when the parameter is included in the range, the control unit is configured to control the control target based on the operation control signal output by the automatic control circuit. Is preferable. In this case, it is more preferable that the control unit is configured to control the control target based on the operation control signal output by the automatic control circuit without being affected by the monitoring by the monitoring circuit. It is preferable that the monitoring circuit functions only when the parameter is out of the range. The monitoring circuit monitors the parameters as a result of processing by the automatic control circuit so that they do not deviate from the above range. This aspect is an example of monitoring performed by the monitoring circuit for the reliability of the processing result by the automatic control circuit. This aspect is an example in which the results of machine learning are monitored by a rule-based field programmable gate array.

(7) The control unit of (5)
The monitoring circuit is configured so that the conditions can be changed by changing software or hardware.

According to the above configuration, since the conditions are changed by changing the software or hardware, it is easy to respond when the monitoring target or the monitoring level is changed. It is possible to handle a wide variety of applications for automatically operating devices.

(8) The control unit of (1)
When the monitoring circuit detects an abnormality of at least one parameter related to the at least one type of signal, the monitoring circuit prohibits the output of the operation control signal by the automatic control circuit.

According to the above configuration, the output of the operation control signal is prohibited according to the abnormality of one parameter. Therefore, the situation where an abnormal operation control signal is continuously output is suppressed.

(9) The control unit of (1)
The monitoring circuit further detects anomalies by monitoring the voltage of the power supply supplied to the field programmable gate array.

According to the above configuration, the influence of an abnormality in the power supply unit that supplies power can be suppressed.

(10) The control unit of (1).
The automatic control circuit includes a GPU having a multi-core capable of parallel processing.

According to the above configuration, it is possible to carry out the processing of the control model constructed by machine learning so as to correspond to the external environment by the multi-core capable of parallel processing. Therefore, it can be applied to a more complicated external environment. Therefore, it is possible to achieve both application to a more complicated external environment and high reliability.

(11) The control unit of (1).
The monitoring circuit is configured to cut off the power supply to the controlled object when an abnormality of at least one parameter related to the at least one type of signal is detected.

According to the above configuration, the operation of the controlled object can be stopped more reliably.

(12) The control unit of (11).
The monitoring circuit includes a relay for cutting off the power supply to the controlled object.

According to the above configuration, it is possible to make the circuit that cuts off the power supply independent from the circuit for detecting an abnormality. In addition, the relay can quickly cut off the power supply to the controlled object due to the power supply cutoff to the monitoring circuit. Therefore, the operation of the controlled object can be stopped more reliably and quickly. Therefore, the operation can be reliably stopped in response to an abnormality.

(13) The control unit of (1).
When the monitoring circuit detects an abnormality of at least one parameter related to the at least one type of signal, the portion of the controlled object that stops operation is set to at least one parameter related to the at least one type of signal. Change according to the type of abnormality.

According to the above configuration, the part that stops operation differs depending on the type of abnormality. Therefore, it is possible to perform a stop suitable for the type of abnormality.

(14) The control unit of (1).
The automatic control circuit is configured to output the operation control signal and the status index signal based on the external environment signal by executing a software process.
The monitoring circuit is configured to monitor the period and / or time during which the software process is executed.

According to the above configuration, the execution of the software process itself is monitored. Therefore, the decrease in reliability caused by the execution of the software process can be suppressed.

(15) The control unit of (1).
The monitoring circuit monitors at least one parameter selected from the group consisting of the interval at which the external environment signal is input, the value of the data indicated by the external environment signal, and the amount of change in the data indicated by the external environment signal. It is configured as follows.

According to the above configuration, when the automatic control circuit performs control using the control model constructed by machine learning, the situation where an unscheduled external environmental signal is applied to the control model is suppressed. It can be applied to a complicated external environment while improving reliability.

(16) The control unit of (1).
The monitoring circuit comprises an interval at which the signal is input, a value of data indicated by the signal, and a change amount of data indicated by the signal with respect to at least one of the operation control signal and the situation index signal. It is configured to monitor at least one parameter selected from the group.

According to the above configuration, at least one of the operation control signal and the status index signal is monitored. Therefore, the reliability of the automatic control circuit can be further improved.

(17) The control unit of (1).
The monitoring circuit includes a plurality of circuits for monitoring parameters different from each other, and is configured to determine the monitoring result by the logical product, OR, or a combination thereof of the signals output from each circuit. ing.

According to the above configuration, it can be applied more flexibly to a complicated external environment while improving high reliability.

(18) The control unit of (1).
The monitoring circuit includes a program memory in which a program is stored and a non-volatile memory different from the program memory.
In the non-volatile memory, a condition is recorded so that at least one parameter related to the at least one type of signal is associated with the parameter.

According to the above configuration, the inspection conditions can be changed by changing the contents of the non-volatile memory or the non-volatile memory. Therefore, it is possible to flexibly respond to changes in the monitoring target.

(19) The control unit of (18).
In the non-volatile memory, the logical product, the logical sum, or a combination thereof of the signals output when the processing function is executed is recorded.

According to the above configuration, appropriate stoppage is carried out according to the degree of parameter abnormality.

(20) The control unit of (1).
The control target is a traveling device for traveling the automatically operating device.
The automatic control circuit generates the operation control signal for instructing the traveling course of the automatic operation device by processing the external environment.

According to the control unit having the above configuration, the automatic operation device functions as an automatic traveling vehicle. It is possible to improve the high reliability of the autonomous driving vehicle while applying it to the complicated external environment of the autonomous driving vehicle.

(21) The control unit of (20).
When the monitoring circuit detects an abnormality of at least one parameter related to the at least one kind of signal, the monitoring circuit generates a traveling stop signal for operating a brake included in the traveling device.

According to the control unit having the above configuration, the control target can be stopped in the shortest time, so that the influence of the abnormal state can be suppressed to the minimum. Therefore, reliability can be improved.

(22) The control unit of (20).
When the monitoring circuit detects an abnormality of at least one parameter related to the at least one type of signal, the monitoring circuit generates a traveling stop signal for stopping the traveling device at a predetermined deceleration.

According to the control unit having the above configuration, it is possible to suppress the influence of stopping on the automatically operating device and the mounted device.

(23) The control unit of (20).
When the monitoring circuit detects an abnormality of at least one parameter related to the at least one type of signal, the traveling stop signal for stopping the traveling device is used as the operation control signal output by the automatic control circuit. To generate.

The terminology used herein is for the purpose of defining only specific embodiments and is not intended to limit the invention.
As used herein, the term "and / or" includes any or all combinations of one or more related enumerated components.
As used herein, the use of the terms "including, including,""comprising," or "having," and variations thereof, is a feature, process, operation, described. It identifies the presence of elements, components and / or their equivalents, but can include one or more of steps, actions, elements, components, and / or groups thereof.
As used herein, the terms "attached", "connected", "combined" and / or their equivalents are widely used, direct and indirect attachment, connection and Includes both bonds. Further, "connected" and "coupled" are not limited to physical or mechanical connections or connections, but can include direct or indirect electrical connections or connections.
Unless otherwise defined, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.
Terms such as those defined in commonly used dictionaries should be construed to have meaning consistent with the relevant technology and in the context of the present disclosure and are expressly defined herein. Unless it is, it will not be interpreted in an ideal or overly formal sense.
It is understood that a plurality of techniques and processes are disclosed in the description of the present invention.
Each of these has its own benefit and each can be used in conjunction with one or more of the other disclosed techniques, or in some cases all.
Therefore, for clarity, this description refrains from unnecessarily repeating all possible combinations of individual steps.
Nevertheless, the specification and claims should be read with the understanding that all such combinations are within the scope of the present invention and claims.
This specification describes a new control unit.
In the following description, for purposes of illustration, a number of specific details are given to provide a complete understanding of the present invention.
However, it will be apparent to those skilled in the art that the present invention can be practiced without these particular details.
The present disclosure should be considered as an example of the invention and is not intended to limit the invention to the particular embodiments set forth in the drawings or description below.

The autonomous driving device is, for example, an autonomous driving vehicle. However, the automatic operation device is not particularly limited, and may be, for example, an automatic work robot.
Further, the cooperative operation device that cooperates with the automatic operation device is an automatic work robot mounted on the automatic driving vehicle. However, the cooperative operation device is not limited to this, and for example, an autonomous driving vehicle equipped with an automatic work robot as an automatic driving vehicle may be used.

The control target is, for example, an actuator. The actuator is, for example, a motor. The actuator causes the automatic operation device to perform physical output. The actuator provided in the cooperative operation device causes the cooperative operation device to perform physical output. The actuator may be an electromagnetic solenoid. The actuator is controlled by an automatic control circuit. The actuator may be directly controlled by an automatic control circuit. The actuator and the motion control device may be indirectly controlled via a control means different from the automatic control circuit. In this case, the automatic control circuit controls the control means, and the control means controls the actuator.

The control unit is, for example, a navigation device for controlling the automatic driving of an autonomous driving vehicle. However, the control unit is not particularly limited, and may be configured to control a robot that does not travel.

The control unit is preferably commercial. Commercial means that it is used in an automatic operation device so that the actual work is performed by the automatic operation device. The actual work performed by the automatic operation device is a practical work in the application planned for the automatic operation device. The actual work with the auto-acting equipment does not include work for testing or research. That is, the test or research control unit is not included in the commercial unit. The following control units correspond to test or research control units.
-Control unit manufactured for confirmation or investigation of the function or performance of the control unit itself or the automatic operation device-Manufactured for testing or research for the purpose of improving or developing the control unit itself or the automatic operation device Control unit According to the control unit according to one aspect of the present invention described above, it is possible to achieve both application to a complicated external environment and high reliability in a commercial control unit. Therefore, the control unit according to one aspect of the present invention is suitable as a commercial control unit.

The control unit is preferably a mass production type. A mass-produced control unit is a control unit manufactured in large numbers in the same format. A control unit manufactured as a single product such as a prototype does not fall under the mass production type. According to the control unit according to one aspect of the present invention described above, in the mass production type control unit, both application to a complicated external environment and high reliability can be achieved. Therefore, the control unit according to one aspect of the present invention is suitable as a mass production type control unit.

The control unit used for the automatic operation device is mounted on the automatic operation device, for example. However, the control unit is not particularly limited, and may be installed at a position away from the automatic operation device and may be communicably connected to the peripheral device via each connector.

The fact that the monitoring circuit and the automatic control circuit integrally form a control unit means that, for example, the monitoring circuit and the automatic control circuit are formed in one semiconductor device or mounted on one substrate. However, the above configuration is not particularly limited, and for example, a semiconductor device constituting a monitoring circuit and a semiconductor device constituting an automatic control circuit may be provided in the housing of the control unit.

The automatic control circuit is composed of, for example, a GPU and a program executed by the GPU. However, the configuration of the automatic control circuit is not particularly limited, and may be configured by, for example, an FPGA or a multi-core CPU.

The program memory is the memory in which the program is stored. However, data other than the program may be stored in the program memory.

The monitoring circuit comprises a field programmable gate array. Other configurations in the monitoring circuit are not particularly limited, and for example, an additional processor may be provided. Examples of the signal monitored by the monitoring circuit include the following examples.
-External environment signal, operation control signal, and status index signal-External environment signal and operation control signal-External environment signal and status index signal-Operation control signal and status index signal-External environment signal-Operation control signal-Status The signal monitored by the index signal monitoring circuit is preferably at least a status index signal.
The monitoring result is obtained, for example, as the result of the logical sum or AND of any of a plurality of signals included in the external environment signal, the operation control signal, and the situation index signal. Further, the monitoring result is obtained as a result of the combination of the logical sum and the logical product of the plurality of signals described above. Further, the monitoring result is not limited to one result, and may be composed of a plurality of results. Each of the plurality of results is associated with, for example, a plurality of different anomaly handling processes. The plurality of results are, for example, the results of applying different logics to the above-mentioned plurality of signals. Further, the plurality of results are, for example, the results due to the difference in the signals selected as the above-mentioned plurality of signals.

According to the present invention, in a control unit used for an automatic operation device, both application to a complicated external environment and high reliability are achieved.

It is a block diagram which shows the structure of the automatic operation system including the control unit which concerns on 1st Embodiment of this invention. It is a block diagram which shows the structure of the control unit shown in FIG. It is a flowchart explaining the cooperation operation of the control unit shown in FIG. It is a block diagram which shows the 1st application example of the control unit shown in FIG. It is a block diagram which shows the 2nd application example of the control unit shown in FIG. It is a block diagram which shows the flow of an image in the automatic operation system shown in FIG. It is a flowchart explaining the control of the image in the operation control device of the control unit shown in FIG. It is a block diagram which shows the 3rd application example of the control unit shown in FIG. It is a flowchart explaining the cooperation operation of the control unit which concerns on 2nd Embodiment of this invention. It is a block diagram which shows the structure of the automatic operation system which includes the control unit which concerns on 3rd Embodiment of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 is a block diagram showing a configuration of an automatic operation system including a control unit according to the first embodiment of the present invention.

[Automatic operation system]
The automatic operation system S is an operator H, that is, a system capable of automatically operating regardless of the operation of a person. The automatic operation system S detects the external environment of the automatic operation system S by itself. Then, the automatic operation system S recognizes the content of the detection result, and controls the operation of the automatic operation system S based on the recognition result.

The automatic operation system S also has a function of operating according to the operation of the operator H. For example, the automatic operation system S operates in response to the operation of the remote control device 3. The remote control device 3 is a device for remotely controlling the automatic operation system S. The remote control device 3 can communicate with the automatic operation system S by wireless communication. The remote control device 3 transmits operation information to the automatic operation system S. Further, the remote control device 3 receives an image from the automatic operation system S and displays the image.
For example, the automatic operation system S can start or stop the automatic operation in response to the operation of starting or stopping the automatic operation with respect to the remote control device 3. Further, for example, the automatic operation system S can select one pattern from a plurality of automatic operation patterns according to the operation of the selection operation for the remote control device 3. Further, for example, the automatic operation system S can perform sequential operations in response to sequential operations on the remote control device 3. Sequential operations are operations typified by, for example, forward, backward, and stop.

The automatic operation system S includes an automatic operation device 1 and a cooperative operation device 2. The automatic operation device 1 shown in FIG. 1 is a system capable of performing automatic operation.
The automatic operation device 1 detects the external environment. Then, the automatic operation device 1 recognizes the detection result and controls the operation based on the recognition result. However, the automatic operation device 1 also has a function of operating according to the operation of the operator H. The operation according to the operation is the same as the operation for the automatic operation system S described above.
In the example shown in FIG. 1, the cooperative operation device 2 is also a system capable of performing automatic operation. The cooperative operation device 2 detects the external environment. Then, the cooperative operation device 2 recognizes the detection result and controls the operation based on the recognition result. Further, the cooperative operation device 2 also has a function of operating according to the operation of the operator H.
As the cooperative operation device 2, it is also possible to adopt a configuration in which the automatic operation device 1 does not detect the external environment by itself, but operates by receiving the detection result of the external environment in the automatic operation device 1 or the instruction from the automatic operation device 1. Here, first, a configuration for detecting the external environment as the cooperative operation device 2 will be described.

The cooperative operation device 2 shown in FIG. 1 operates in cooperation with the automatic operation device 1. That is, the automatic operation device 1 and the cooperative operation device 2 cooperate with each other to complete the work expected by the operator H. The cooperative operation device 2 can cooperate by performing the same type of operation as the automatic operation device 1, for example. As an example of this, there is a case where the automatic operation device 1 and the cooperative operation device 2 are a pair of robot arms.
Further, for example, the cooperative operation device 2 and the automatic operation device 1 can complete the desired operation by performing different types of operations from each other. That is, the cooperative operation device 2 and the automatic operation device 1 can cooperate with each other. As an example of this, there is a case where the automatic operation device 1 is an automatic traveling vehicle and the cooperative operation device 2 is a robot arm mounted on the automatic traveling vehicle.
The automatic operation device 1 and the cooperative operation device 2 shown in FIG. 1 are mechanically connected.
However, the automatic operation device 1 and the cooperative operation device 2 may be separated from each other. As an example of this, there is a case where the automatic operation device 1 and the cooperative operation device 2 are two vehicles that travel in a common area or share a plurality of adjacent areas.

The linked operation device 2 shown in FIG. 1 is communicably connected to the automatic operation device 1. The automatic operation device 1 and the cooperative operation device 2 shown in FIG. 1 are electrically connected to each other by electric wires. However, as the automatic operation device 1 and the cooperative operation device 2, for example, a configuration for wireless communication can be adopted.

[Automatic operation device]
The automatic operation device 1 includes a control unit 10, an external environment sensing unit 11, an operation unit 12, a remote communication device 13, and a power supply unit 14.

The external environment sensing unit 11 detects the external environment of the automatic operation device 1. The external environment sensing unit 11 outputs external environment data indicating the detection result.
The external environment sensing unit 11 is, for example, a camera that photographs the external environment of the automatically operating device 1. The camera as the external environment sensing unit 11 outputs external image data indicating the shooting result. By using the camera, the control unit 10 can operate the actuator 121 while recognizing the complicated external environment.

The operation unit 12 is controlled based on the external environment. The operation unit 12 includes an actuator 121. The actuator 121 is mechanically operated by electric control to drive a device mounted on the automatic operation device 1 or the automatic operation device 1 itself.

The control unit 10 is connected to the external environment sensing unit 11, the operation unit 12, and the remote communication device 13.
The control unit 10 controls the actuator 121 of the operation unit 12 based on the external environment detected by the external environment sensing unit 11. More specifically, the control unit 10 recognizes the contents of the external environment by processing the external environment data output from the external environment sensing unit 11. The control unit 10 determines the control content based on the recognized content. Then, the control unit 10 controls the actuator 121 based on the determined control content.

The control unit 10 may receive a control command corresponding to the operation of the remote control device 3 from the remote communication device 13 and control the actuator 121 based on the control command. The internal configuration of the control unit 10 will be described later.

The power supply unit 14 supplies electric power to the control unit 10, the external environment sensing unit 11, the operating unit 12, and the remote communication device 13.
The power supply unit 14 has a battery (not shown). The power supply unit 14 supplies the electric power stored in the battery to each unit. The power supply unit 14 supplies electric power based on the control of the control unit 10. The power source used by the power supply unit 14 is not limited to the battery, and various power sources can be used. For example, as the power supply unit 14, a configuration having an engine generator or a fuel cell instead of the battery can be adopted. An engine generator includes, for example, an engine that operates on liquid fuel and a generator that is driven by the engine to generate electricity.
The power supply unit 14 shown in FIG. 1 also supplies electric power to the cooperative operation device 2. However, the linked operation device 2 may be provided with a power supply independent of, for example, the automatic operation device 1.

The remote communication device 13 is communicably connected to the remote control device 3. The remote communication device 13 is communicably connected to the remote control device 3 by wireless communication.
The remote communication device 13 relays communication data between the remote control device 3 and the control unit 10. For example, the remote communication device 13 outputs a control command output from the remote control device 3 to the control unit 10 in response to the operation of the remote control device 3. As a result, the control command from the remote control device 3 is supplied to the control unit 10. Further, for example, the remote communication device 13 supplies data based on the data output from the external environment sensing unit 11 to the remote control device 3. For example, when the external environment sensing unit 11 is a camera, the remote communication device 13 transmits data based on the external image data output from the camera to the remote control device 3. As a result, the image of the camera is displayed on the remote control device 3. In this case, the remote control device 3 transmits an image command to the remote communication device 13. The image command is a command for designating the content of the image and the amount of image data transmitted to the remote control device 3. The remote communication device 13 sends an image command to the control unit 10.

The data exchange between the remote communication device 13 and the control unit 10 described above is the same as the data exchange between the remote communication device 13 and the cooperative control unit 20.

[Collaboration operation device]
The cooperative operation device 2 includes a cooperative control unit 20, a cooperative sensing unit 21, and a cooperative operation unit 22. The roles of the cooperation control unit 20, the cooperation sensing unit 21, and the cooperation operation unit 22 in the cooperation operation device 2 are the same as the roles of the control unit 10, the external environment sensing unit 11, and the operation unit 12 in the above-mentioned automatic operation device 1. are doing. However, the type of environment detected by the cooperative sensing unit 21, the detailed content of the determination of the cooperative control unit 20, and the output of the cooperative operation unit 22 differ depending on the function of the cooperative operation device 2.
The cooperation control unit 20 controls the cooperation operation unit 22 that performs physical output. The physical output of the cooperative operation unit 22 is, for example, the operation of the actuator 221.

The cooperative control unit 20 corresponds to an example of the control unit referred to in the present invention, like the control unit 10.

As an example of the function of the automatic operation device 1 shown in FIG. 1, an automatic traveling vehicle can be mentioned, for example. In this case, the control unit 10 is, for example, an automatic navigation unit. The actuator 121 of the operation unit 12 is a traveling device for traveling the automatic operation device 1. The traveling device is, for example, a motor. The control unit 10 recognizes the content of the image of the traveling region taken by the camera as the external environment sensing unit 11, determines the traveling route based on the recognition result, and instructs the operation unit 12 of the determined traveling route. That is, the control unit 10 controls the operation unit 12 based on the determined travel path. As a result, the automatically operating device 1 automatically travels.
An example of the function of the cooperative operation device 2 shown in FIG. 1 is a work device mounted on an autonomous vehicle. An example of working equipment is a fruit-picking device that picks fruits on a farm. In this case, the cooperative control unit 20 recognizes the content of the fruit tree photographed by the camera as the cooperative sensing unit 21, determines the state and position of the fruit based on the recognition result, and based on the determined position of the fruit, A running command (running position command) is transmitted to the automatic operation device 1. The travel command is an example of an operation command. Further, the actuator 221 of the fruit-picking device as the cooperative operation unit 22 is controlled based on the determined position of the fruit. By linking the automatic operation device 1 and the cooperative operation device 2, the function of picking the fruits on the farm is realized.
Further, the cooperative operation device 2 mounted on the automatic operation device 1 can be replaced with a cooperative operation device having a function different from that of the cooperative operation device 2 shown in FIG.

[Controller unit]
FIG. 2 is a block diagram showing the configuration of the control unit 10 shown in FIG.
The control unit 10 is a unit covered with one housing, and is incorporated in the automatic operation device 1 (see FIG. 1). The control unit 10 is electrically connected to each part of the automatic operation device 1.
The control unit 10 is a commercial unit. The control unit 10 is a mass production type.

The control unit 10 includes an external environment information connector 110, an operation control connector 130, an external communication connection unit 140, and an operation control device 160. Further, the control unit 10 includes a remote data connector 150.

The external environment information connector 110 is electrically connected to the external environment sensing unit 11 shown in FIG. External environment data indicating the detection result is input to the control unit 10 from the external environment sensing unit 11 via the external environment information connector 110.
When the external environment sensing unit 11 is, for example, a camera, the external environment information connector 110 functions as an external image connector. Hereinafter, the external environment information connector 110 will also be referred to as an external image connector 110.

The operation control connector 130 is electrically connected to the operation unit 12 shown in FIG. An operation control signal for controlling the operation of the actuator 121 is output from the control unit 10 to the operation unit 12 via the operation control connector 130.

The external communication connection unit 140 is connected to the cooperation control unit 20 shown in FIG. The external communication connection unit 140 in the example shown in FIG. 1 is an external communication connector that is electrically connected to the cooperation control unit 20. Hereinafter, the external communication connection unit 140 is also referred to as an external communication connector 140. The external communication connector 140 physically includes a plurality of connectors corresponding to a plurality of types of transmission formats. The types of transmission formats are, for example, Controller Area Network (CAN) (registered trademark) and Ethernet (registered trademark). By having a plurality of connectors, the cooperation control unit 20 can be selected from the candidates of a plurality of units having various functions. The versatility of the control unit 10 is improved. The external communication connector 140 is an example of an external communication connection unit that is communicably connected to the cooperation control unit 20. As the external communication connection unit, for example, a configuration in which a wireless communication device is used instead of the external communication connector 140 can be adopted.

The remote data connector 150 is electrically connected to the remote communication device 13 shown in FIG. A control command signal is input to the control unit 10 from the remote control device 3 (see FIG. 1) and the remote communication device 13 via the remote data connector 150. That is, in response to the request for image transmission being input from the remote control device 3 to the remote communication device 13, the control command signal is output from the remote control device 13 and input via the remote data connector 150 for control. It is input to the unit 10. Further, a signal indicating the external environment and the state of the control unit 10 is output from the control unit 10 to the remote communication device 13 via the remote data connector 150. This signal is supplied from the remote communication device 13 to the remote control device 3.

The motion control device 160 controls the actuator 121 of the motion unit 12 based on the external environment detected by the external environment sensing unit 11 shown in FIG. More specifically, the motion control device 160 processes the external environment data output from the external environment sensing unit 11. The motion control device 160 controls the actuator 121 based on the processing result of the external environment data. Further, the motion control device 160 receives a control command corresponding to the operation of the remote control device 3 from the remote communication device 13, and controls the motion unit 12 based on the control command.
Further, the operation control device 160 communicates with the cooperation control unit 20 connected via the external communication connector 140. As described above, various devices can be selected as the collaborative operation device 2 that can be combined with the automatic operation device 1.
There are two types of the cooperative control unit 20 included in the cooperative operation device 2, that is, a master unit and a slave unit. The master unit is a cooperative control unit 20 configured to transmit an operation command to the operation control device 160. The slave unit is a cooperative control unit 20 configured to receive an operation command from the operation control device 160. Either the master unit or the slave unit is connected to the external communication connector 140. The operation control device 160 switches the operation mode according to the type of the cooperation control unit 20.

When the master unit is connected to the external communication connector 140, the operation control device 160 generates an operation control signal based on the operation command received from the master unit. The motion control device 160 outputs the generated motion control signal to the actuator 121 of the motion unit 12 via the motion control connector 130.
On the other hand, when the slave unit is connected to the external communication connector 140, the operation control device 160 controls the cooperative operation device 2 based on the processing result of the external environment data input from the external environment sensing unit 11. Generate an operation command for. The operation control device 160 transmits the generated operation command to the slave unit via the external communication connector 140.
As a result, regardless of whether the master unit or the slave unit is connected to the external communication connector 140 of the control unit 10 as the cooperation control unit 20, precise work can be performed in cooperation with the cooperation control unit 20. Therefore, the versatility of the control unit 10 can be improved.

[Configuration of operation control device]
As shown in FIG. 2, the motion control device 160 includes an automatic control circuit 170 and a monitoring circuit 180.
The automatic control circuit 170 and the monitoring circuit 180 are provided in the housing of the control unit 10.

The automatic control circuit 170 carries out basic control processing in the motion control device 160. More specifically, the automatic control circuit 170 controls the actuator 121 based on the external environment signal from the external environment sensing unit 11. More specifically, the automatic control circuit 170 outputs an operation control signal based on an external environment signal by executing a software process. The automatic control circuit 170 also outputs a status index signal by executing a software process.

The automatic control circuit 170 includes a Graphics Processing Unit (GPU) 171.
GPU 171 is a processor having a multi-core capable of parallel processing. The GPU 171 includes 100 or more arithmetic cores that can operate in parallel. The GPU 171 executes a SIMD (single-instruction multiple-data stream) operation by 100 or more arithmetic cores.
Further, the automatic control circuit 170 includes a non-volatile memory 172, a RAM 173, a control input / output (control IO) 174, and a CPU 175. The non-volatile memory 172 is, for example, a mask ROM flash memory or an EEPROM.
The CPU 175 is a Central Processing Unit. The CPU 175 controls the entire automatic control circuit 170. The GPU 171 and the CPU 175 share and execute the control of the automatic control circuit 170. More specifically, the CPU 175 causes the GPU 171 to perform some of the functions of the automatic control circuit 170. The functions executed by the GPU 171 will be described later.
The non-volatile memory 172 stores a program executed by the CPU 175 and the GPU 171. The CPU 175 sequentially reads and executes the programs stored in the non-volatile memory 172. As a result, control by the automatic control circuit 170 is executed.
Further, the program of the GPU 171 stored in the non-volatile memory 172 is read by the CPU 175 and supplied to the GPU 171.
The RAM 173 holds the result of the processing by the CPU 175 and the result of the processing by the GPU 171. The CPU 175 and the GPU 171 read / write data to / from the RAM 173. The RAM 173 stores data input to the CPU 175 and the GPU 171, data indicating the processing status, and data indicating the operation control signal output from the automatic control circuit 170 as a result of the processing. The data input to the GPU 171 is, for example, data representing an external environment signal. The data indicating the processing status is, for example, one of the parameters indicating the operating status of the automatic control circuit 170.
The control IO 174 relays signals input / output to the CPU 175 and the GPU 171. The CPU 175 and the GPU 171 output a status index signal indicating the operating status of the control model via the control IO 174. The status indicator signal is, for example, a pulse indicating the period and time during which the process of processing the control model is executed. The status index signal is one of the parameters indicating the operating status of the automatic control circuit 170. The stored contents of the RAM 173 can be read out to the FPGA 181 of the monitoring circuit 180 via the control IO 174.

The CPU 175 supplies the program stored in the non-volatile memory 172 to the GPU 171. Further, the CPU 175 outputs a command to execute the program to the GPU 171.
When the GPU 171 executes the program stored in the non-volatile memory 172, the automatic control circuit 170 is configured with the control model 171a constructed by machine learning. The control model 171a is a model showing the relationship between the external environment detected by the external environment sensing unit 11 and the control of the operation unit 12 to be controlled. The control model 171a is a machine learning model using a neural network.
A machine learning model can be obtained, for example, by constructing a model that shows the relationship between external image data and an object that can exist on a traveling path, and optimizing the model. More specifically, the control model 171a optimizes the weighting parameters of the model by referring to, for example, the actual external image data and the data in which the object on the traveling path is associated with the data. This is the result obtained by converting. The function of optimizing the control model 171a with reference to the data is performed not inside the automatic control circuit 170 or the control unit 10, but in the external environment of the automatic operation system S. A program that builds the control model as a result of the optimization is stored in the non-volatile memory 172.
However, the method of optimizing the control model 171a is not limited to this. For example, the automatic control circuit 170 itself can optimize the control model 171a that it configures based on the external image data actually obtained.
Since the GPU 171 can execute SIMD operations on 100 or more arithmetic cores, it is possible to execute the processing of the control model 171a accompanied by the iterative arithmetic of a large-scale matrix at high speed.
The CPU 175 determines the operation of the control unit 10 based on the information of the object obtained as a result of applying the data of the external environment to the machine learning model. The CPU 175 controls the operation unit 12 based on the result of the determination. More specifically, the CPU 175 outputs a command to the operation unit 130 via, for example, the control IO 174 and the communication IF 183 of the monitoring circuit 180.
Further, the CPU 175 transmits a command to the cooperative operation unit 20 based on the result of the determination. Further, the CPU 175 transmits data to the remote communication device 13.
The division of control between the CPU 175 and the GPU 171 and the input / output of the model executed by the GPU 171 are not limited to those described above. For example, the machine learning model may be a model that directly shows, for example, the relationship between the external image data and the optimum traveling path or the motion trajectory of the arm or the like. In this case, the CPU 175 controls the operation unit 12 based on the travel path or operation locus output as the processing result of the GPU 171.

[Monitoring circuit]
The monitoring circuit 180 constitutes the control unit 10 integrally with the automatic control circuit 170. The monitoring circuit 180 monitors the operation of the automatic control circuit 170 constituting the control model 171a.
The monitoring circuit 180 includes a field programmable gate array (FPGA) 181 and a non-volatile memory 182. Further, the monitoring circuit 180 includes a communication interface (communication IF) 183, a relay 184, and a memory 185 for a program.

The FPGA 181 has a reprogrammable logic circuit. The non-volatile memory 182 stores the connection information of the logic circuit for monitoring constructed by the FPGA 181. The FPGA 181 reads the connection information from the non-volatile memory 182 in the initialization process after the power is turned on or after the reset. The FPGA 181 constructs a logic circuit based on connection information. After constructing the logic circuit, the FPGA 181 starts the processing by the logic circuit.
By changing the data stored in the non-volatile memory 182, it is possible to change the function constructed by the logic circuit. Therefore, in the monitoring circuit 180, the monitoring conditions can be changed by changing the connection information, which is software stored in the non-volatile memory 182, and the hardware based on the connection information.
The GPU 171 described above, the CPU 175, or the processor 181p described later will sequentially read the stored programs by accessing the non-volatile memory 172 or the memory 185 for the program during execution after initialization. On the other hand, the FPGA 181 reads out the non-volatile memory 182 only once at the time of initialization to form the logic circuit of the monitoring circuit 180. The FPGA 181 completes reading the non-volatile memory 182 before starting execution of the process. Therefore, the logic circuit for monitoring can operate with high reliability.
The non-volatile memory 182 containing the information of the logic circuit for monitoring may be physically divided into a plurality of parts. For example, the first device of the non-volatile memory 182 composed of a plurality of memory devices stores information on a higher-level logic circuit that monitors the monitoring target and a reference information loader. In the second device, the numerical range of the monitoring target and the logic of the combination of signals determined to be abnormal are stored as reference information. In this case, by resetting the FPGA 181, the information of the upper logic circuit and the reference information loader are read from the first device, and the logic circuit is constructed. Next, the reference information loader starts execution, reads (loads) the reference information from the second device, and complements the numerical range in the logic circuit and the logic of the combination. This completes the construction of the logic circuit in FPGA181. That is, the reading of information is executed in a plurality of steps.
A configuration in which the non-volatile memory 182 is used as three or more devices can also be adopted.

The FPGA 181 may have a fixed logic circuit other than the reprogrammable logic circuit. For example, the FPGA 181 has a processor 181p and a memory as logic circuits. The processor 181p executes processing while sequentially reading the programs stored in the memory 185, for example. This allows for more advanced processing. The memory 185 read by the processor 181p is non-volatile. However, unlike the non-volatile memory 182 for the FPGA 181, the memory 185 stores not the connection information but the programs that are sequentially read by the processor. By dividing the memory according to the application, the reliability of the logic circuit composed of the FPGA 181 is improved.

The communication IF183 is an interface for the FPGA 181 and the automatic control circuit 170 to communicate with the operation unit 12. The communication IF 183 provides, for example, a physical interface for communicating with the operating unit 12. The physical interface is, for example, CAN. The automatic control circuit 170 outputs an operation control signal via the communication IF 183.

The relay 184 cuts off the power supply of the power supply unit 14 (see FIG. 1) to the operating unit 12. More specifically, the relay 184 transmits a supply signal for supplying electric power to the power supply unit 14 by energizing under the control of the FPGA 181. When the energization of the relay 184 is stopped by the control of the FPGA 181, the transmission of the supply signal is stopped. As a result, the power supply from the power supply unit 14 is cut off. By shutting off the power supply, the operation can be reliably stopped.
The supply signal from the relay 184 can pass through a relay (not shown) provided in each part of the automatic operation device 1 and the cooperative operation device 2 outside the monitoring circuit 180. As a result, the power supply is immediately cut off by some cutoff control. Therefore, the operation can be surely performed.

The logic circuit composed of FPGA181 of the monitoring circuit 180 detects an abnormality in the operation of the automatic control circuit 170 by rule-based logic.
The monitoring circuit 180 inspects at least one signal selected from the group consisting of the external environment signal, the operation control signal, and the status index signal of the automatic control circuit 170. The monitoring circuit 180 monitors all of the external environment signal, the operation control signal, and the status index signal, for example.
The monitoring circuit 180 is configured to monitor whether or not the external environment signal, the operation control signal, and the status index signal satisfy the conditions set to be associated with each of them.

For example, the automatic control circuit 170 stores the external environment data representing the external environment signal output from the external environment sensing unit 11 in the RAM 173. The monitoring circuit 180 reads a part of the external environment data of the RAM 173 and monitors whether or not the external environment signal is within the corresponding normal condition range.
In addition, the monitoring circuit 180 also monitors the interval at which the external environment signal is input, the value of the data indicated by the external environment signal, and the amount of change in the data indicated by the external environment signal in the automatic control circuit 170.

Further, the monitoring circuit 180 reads data indicating the processing status and data indicating the operation control signal as a result of the processing from the RAM 173. The monitoring circuit 180 monitors whether or not the read result is within the corresponding normal condition range.
Further, the monitoring circuit 180 monitors whether or not the status index signal output from the control IO 174 of the automatic control circuit 170 is within the corresponding normal condition range. The monitoring circuit 180 monitors whether or not the operation control signal output from the automatic control circuit 170 via the communication IF 183 is within the corresponding normal condition range. Further, the monitoring circuit 180 monitors whether or not the cycle and time for executing the process for processing the control model are within the corresponding normal condition range.
The monitoring circuit 180 also monitors the voltage of the power supply supplied to the FPGA 181. As a result, the monitoring circuit 180 can also monitor the abnormality of the power supply unit 14.

In this way, the monitoring circuit 180 monitors whether or not the parameter to be monitored is included in the range defined so as to be associated with the parameter. When the parameter to be monitored is within the specified range, the monitoring circuit 180 does not limit the output of the automatic control circuit 170. That is, when the parameter to be monitored is within the range defined by the rule, the automatic control circuit 170 is not restricted by the rule. Therefore, it is possible to control with a high degree of freedom by the model of the automatic control circuit 170 while maintaining reliability.

When the monitoring circuit 180 detects an abnormality of the parameter, the portion of the controlled object that stops the operation is changed according to the type of the abnormality of the parameter.
The non-volatile memory 182 records conditions defined so as to be associated with parameters. As a result, the FPGA 181 of the monitoring circuit 180 implements a processing function for inspecting whether or not the parameters satisfy the conditions. The FPGA 181 of the monitoring circuit 180 is configured to determine the result of monitoring by the logical product, OR, or a combination thereof of signals representing the detected abnormalities. A logical product, a logical sum, or a combination thereof is recorded in the non-volatile memory 182.

For example, the monitoring circuit 180 is configured with a logic circuit to select the type of stop described above according to the degree of abnormality of the parameter.
Further, the monitoring circuit 180 generates a stop signal for stopping the operation of the actuator 121 at a predetermined deceleration depending on the type of abnormal parameter. For example, if the anomalous parameter relates to a range of command speeds, the monitoring circuit 180 will generate a stop signal to stop at a predetermined deceleration. In this case, the impact caused by the stop of the actuator 121 is suppressed. When the actuator 121 is a traveling device, the influence of the sudden stop on the mounted equipment is suppressed.
Further, the monitoring circuit 180 generates a stop signal for stopping the operation of the actuator 121 instead of the operation control signal output by the automatic control circuit 170, depending on the type of the abnormal parameter. For example, if the combination of commands is different from the combination of the assumed range, a stop signal is generated. In this case, the actuator 121 is stopped in a short time, and the shock to the mounted device is suppressed to some extent.
Further, the monitoring circuit 180 generates a signal for operating the brake included in the operation unit 12 instead of the operation control signal output by the automatic control circuit 170, depending on the type of abnormal parameter. For example, when an abnormality is detected in the process itself in the automatic control circuit 170, a signal for operating the brake is generated. In this case, since the actuator 121 can be stopped in the shortest time, the influence of the abnormal state can be minimized.

Further, when the monitoring circuit 180 detects an abnormality, the monitoring circuit 180 cuts off the power supply to the controlled object according to the type and number of the abnormal parameters.
The monitoring circuit 180 shuts off the power supply by operating the relay 184, for example, when an abnormality of a plurality of parameters is detected.

Further, for example, the monitoring circuit 180 controls the automatic control circuit 170 so that the image of the camera is forcibly displayed on the remote control device 3 when an abnormality is detected. As a result, the operator can immediately perform the corresponding maneuver.

Further, the output by the monitoring circuit 180 is not limited to the above combination. For example, when the monitoring circuit 180 detects an abnormality of at least one parameter related to at least one type of signal, the monitoring circuit 180 also adopts a configuration in which the output of the operation control signal by the automatic control circuit 170 is prohibited instead of outputting the operation command. It is possible. In this case, the monitoring circuit 180 stops the operation of the communication IF 183 that outputs the signal from the automatic control circuit 170. As a result, the situation in which an abnormal operation control signal is continuously output is suppressed.

In this way, the logical combination constructed by the monitoring circuit 180 implements an appropriate stop according to the degree of parameter abnormality.

The basic hardware structure of the control unit 10 described above is also applied to the cooperative control unit 20. However, when the output content based on the abnormality detection result of the cooperative sensing unit 21 is different from that of the control unit 10, a part of the hardware and the software are different from the control unit 10 according to the difference.

[Linked operation with linked control unit]
The control unit 10 having the above-described configuration operates in cooperation with the cooperative control unit 20.
The control unit 10 operates in cooperation with the cooperative control unit 20 regardless of whether the master unit or the slave unit is connected as the cooperative control unit 20.
Subsequently, the details of the cooperation operation with the cooperation control unit 20 will be described.

FIG. 3 is a flowchart illustrating a cooperative operation among the operations of the control unit shown in FIG.
The operation control device 160 executes the determination of the type of the cooperation control unit 20 and the cooperation operation according to the type.
The type determination and the cooperative operation according to the type are mainly carried out by the automatic control circuit 170 shown in FIG. However, for example, a configuration in which the type of the cooperative control unit 20 is determined by the processor 181p provided in the monitoring circuit 180 can also be adopted.
Here, the cooperative operation will be described as the operation of the operation control device 160.

In the linked operation, the motion control device 160 first determines whether or not it is connected to the linked control unit 20 (S11). For example, the operation control device 160 determines whether or not it is possible to communicate with the cooperation control unit 20 via the external communication connection unit 140.
When the cooperative control unit 20 is connected, the cooperative control unit 20 can communicate with the operation control device 160 of the control unit 10. In this case, the operation control device 160 determines that it is connected to the cooperation control unit 20.

When communication with the cooperation control unit 20 is possible (Yes in S11), the operation control device 160 identifies the type of the cooperation control unit 20 (S12). For example, the operation control device 160 reads the identification information that identifies the cooperation control unit 20 from the cooperation control unit 20 via the external communication connection unit 140.

Next, the cooperation control unit 20 determines the type of the cooperation control unit 20 (S13). The cooperation control unit 20 determines whether the cooperation control unit 20 is either a master unit or a slave unit by referring to a database in which identification information and the type of the cooperation control unit are associated with each other, for example.

When the cooperation control unit 20 is not a master unit (No in S13), the cooperation control unit 20 is a slave unit. In this case, the control unit 10 operates as a master unit.
The operation control device 160 recognizes the content of the external environment data by processing the external environment data (S14).
More specifically, for example, when the automatic operation device 1 is an automatic traveling vehicle, the operation control device 160 performs a process for recognizing the content of image data representing an external image based on the constructed control model.

The motion control device 160 determines the motion based on the recognition result of the content of the external environment data (S15).
More specifically, for example, the motion control device 160 grasps the current position of the automatic motion device 1 and determines the optimum travel route based on the recognition of the content of the image data.

The operation control device 160 generates an operation control signal based on the processing result of the external environment data (S16). The operation control device 160 outputs the generated operation control signal to the actuator 121 of the operation unit 12 via the operation control connector 130.
More specifically, for example, the motion control device 160 generates an motion control signal including a travel and steering command based on the determined travel path, and outputs the motion control signal to the motion unit 12. The operation unit 12 operates the actuator 121. As a result, the automatic operation device 1 operates based on the external environment.

Further, the operation control device 160 transmits an operation command for controlling the linked operation device 2 based on the processing result of the external environment data (S17).
More specifically, for example, the operation control device 160 generates an operation command for the cooperative control unit 20 to perform an operation according to the position of the automatic operation device 1 on the traveling path. The operation control device 160 transmits an operation command to the cooperation control unit 20 via the external communication connection unit 140. As a result, the control unit 10 operates in cooperation with the cooperative control unit 20 that operates as a slave unit.

When the cooperative control unit 20 is a master unit (Yes in S13), the operation control device 160 operates as a slave unit.
In this case, the motion control device 160 recognizes the content of the external environment data by processing the external environment data (S21). Further, the motion control device 160 determines the motion based on the recognition result of the content of the external environment data (S22). These operations are the same as in steps S14 and S15 described above.

Next, the operation control device 160 receives an operation command from the cooperative control unit 20 which is a master unit (S23). The cooperative control unit 20 transmits an operation command to the control unit 10 via the external communication connection unit 140.
More specifically, for example, the cooperative control unit 20 transmits an operation command indicating whether the automatic operation device 1 moves forward or backward according to the position of the work target to the operation control device 160 of the control unit 10.

Next, the operation control device 160 transmits an operation control signal for controlling the cooperative operation device 2 based on the operation command received via the external communication connection unit 140 (S24). The motion control device 160 generates an motion control signal based on the external environment data recognized in step S22 and the motion command.
More specifically, for example, the motion control device 160 generates an motion control signal so as to move forward or backward along the determined travel path, and outputs the motion control signal to the motion unit 12. The operation unit 12 operates the actuator 121. As a result, the automatic operation device 1 operates based on the operation command of the cooperation control unit 20. The control unit 10 operates in cooperation with the cooperative control unit 20 that operates as a master unit.

[First application example of cooperation]
FIG. 4 is a block diagram showing a first application example of the control unit shown in FIG.
An application example shown in FIG. 4 is a case where the automatic operation device 1 is an automatic traveling vehicle and the cooperation control unit 20 is a master unit. That is, the master unit is connected to the external communication connector 140. The cooperative operation device 2 is an autonomous operation robot mounted on an autonomous vehicle.
For example, the cooperation control unit 20 recognizes the position of the work target of the robot based on the image of the cooperation operation device camera (robot camera) as the cooperation sensing unit 21. The cooperative control unit 20 transmits an operation command including forward / backward movement to the operation control device 160 of the control unit 10 according to the position of the work target. The operation control device 160 generates an operation control signal from the cooperation control unit 20 based on the operation command received. The motion control device 160 outputs the generated motion control signal to the actuator 121 of the motion unit 12 via the motion control connector 130. The automatic operation device 1 and the cooperative operation device 2 can cooperate with each other to perform precise work.

[Second application example of cooperation]
FIG. 5 is a block diagram showing a second application example of the control unit shown in FIG.
The application example shown in FIG. 5 is a case where the automatic operation device 1 is an automatic traveling vehicle and the cooperation control unit 20 is a slave unit. That is, the slave unit is connected to the external communication connector 140. The cooperative operation device 2'is a simple work device mounted on the automatic operation device 1. The cooperative operation device 2'is, for example, a sprayer that sprays a chemical or the like toward a work target.
The operation control device 160 generates an operation command for controlling the cooperative operation device 2'based on the processing result of the image data of the external photographing camera as the external environment sensing unit 11. The motion control device 160 generates, for example, an motion command for causing the cooperative motion device 2 to start or stop a work motion based on the position of the autonomous driving vehicle acquired as a result of processing the image data of the external camera. , Transmit to the cooperation control unit 20.
As a result, the cooperative operation device 2 can operate appropriately according to the traveling of the automatic traveling vehicle as the automatic operation device 1.
As described with reference to the example of FIG. 4 and the example of FIG. 5, the cooperation control unit 20 is connected to the external communication connector 140 of the control unit 10 regardless of whether the master unit or the slave unit is connected as the cooperation control unit 20. Can perform precise work in cooperation with. Therefore, the versatility of the control unit 10 can be improved.

[Image transmission during remote control]
For example, when there is an obstacle on the traveling path and it is difficult to work only by automatic operation, or in the case of operation test or confirmation, the automatic operation system S operates in response to the operation of the remote control device 3 by the operator H. ..

In this case, the automatic operation system S operates in response to the operation of the remote control device 3 by the operator H. The remote control device 3 shows both or one of an image based on the image data of the external photographing camera 11 and an image based on the image data of the cooperative operation device camera 21.

FIG. 6 is a block diagram showing an image flow in the automatic operation system shown in FIG.
The solid arrow in FIG. 6 shows the flow of the image when the automatic operation system S is remotely controlled. The broken line arrow in FIG. 6 indicates the flow of the image request signal from the remote control device 3. In FIG. 6, an external photographing camera is shown as an example of the external environment sensing unit 11. Further, a linked operation device camera is shown as the linked sensing unit 21. Hereinafter, the external environment sensing unit 11 will also be referred to as an external photographing camera 11. Further, the linked sensing unit 21 is also referred to as a linked operating device camera 21.

The automatic operation device 1 is equipped with an external photographing camera 11 and an actuator 221 for photographing an external environment. Further, the cooperative operation device 2 is provided with a cooperative operation device camera 21 for photographing the external environment.
The control unit 10 of the automatic operation device 1 is connected to one remote communication device 13.
The control unit 10 of the automatic operation device 1 and the cooperation control unit 20 of the cooperation operation device 2 are connected to one remote communication device 13. That is, the control unit 10 and the cooperative control unit 20 jointly use one remote communication device 13.

As described above, the control unit 10 includes an external image connector 110, an operation control connector 130, a remote data connector 150, and an operation control device 160 (see FIG. 2).
The external image connector 110 (external environment information connector 110) is a connector for inputting external image data indicating a shooting result from the external shooting camera 11 to the control unit 10.
The operation control connector 130 is a connector for the control unit 10 to output an operation control signal for controlling the operation of the actuator 121.
The remote data connector 150 is a connector for inputting / outputting data to / from one remote communication device 13 communicably connected to the remote control device 3.

The operation control device 160 (see FIG. 2) of the control unit 10 remotely transfers monitor image data based on the external image data input via the external image connector 110 from the remote data connector 150 via one remote communication device 13. Output to the control device 3. The operation control device 160 outputs monitor image data based on an image command signal from the outside of the control unit 10.
The cooperative control unit 20 also performs the same processing as the control unit 10 on the image data of the cooperative operation device camera 21. Here, the operation of the control unit 10 in the operation control device 160 will be described as a representative.

[Control of image flow]
FIG. 7 is a flowchart illustrating control of an image in the operation control device 160 of the control unit 10 shown in FIG.

The operation control device 160 determines whether or not an image command signal has been received from the outside (S31). The image command signal is transmitted from the remote communication device 13 via the remote data connector 150. However, the image command signal may be transmitted from the cooperation control unit 20 via the external communication connector 140. The operation control device 160 discriminates about the image command signal transmitted through both connectors.

When the image command signal is received (Yes in S31), the operation control device 160 determines whether the content of the image command signal indicates the start of image transmission (S32).
When the image transmission is started (Yes in S32), the motion control device 160 outputs the monitor image data to the remote control device 3 (S33). The monitor image data is data obtained by processing the external image data input via the external image connector 110. The operation control device 160 generates monitor image data having a smaller amount of data than the external image data by performing image compression processing on the external image data. However, external image data that has not been substantially processed may be used as the monitor image data. The operation control device 160 outputs monitor image data from the remote data connector 150 to one remote communication device 13. The monitor image data is transmitted to the remote control device 3 via the remote communication device 13.

The operation control device 160 determines whether the content of the image command signal indicates that the image transmission is stopped (S34).
When the image transmission is stopped (Yes in S34), the operation control device 160 stops the output of the monitor image data (S35). As a result, the transmission of the monitor image data to the remote control device 3 is stopped.

The operation control device 160 determines whether the content of the image command signal indicates that the image transmission is stopped (S34).
When the image transmission is stopped (Yes in S34), the operation control device 160 stops the output of the monitor image data (S35). As a result, the transmission of the monitor image data to the remote control device 3 is stopped. The image command signal indicating that the image transmission is stopped may be output from the cooperation control unit 20. For example, the cooperative control unit 20 that has received a command to transmit only the image of the cooperative operation device camera 21 from the remote control device 3 sends an image command signal indicating that the image transmission is stopped to the control unit 10. By stopping the transmission of data based on the image data of the external photographing camera 11, the remote control device 3 can display the image that the operator H wants to pay attention to during maneuvering while reducing the amount of data to be transmitted.

The operation control device 160 determines whether the content of the image command signal indicates frame thinning (S36).
In the case of frame thinning (Yes in S36), the operation control device 160 performs frame thinning processing on the monitor image data (S37). As a result, the amount of monitor image data transmitted to the remote control device 3 is reduced.

The operation control device 160 determines whether the content of the image command signal indicates the image compression rate (S38).
In the case of the image compression rate (Yes in S38), the operation control device 160 changes the compression rate of the image compression process (S39). As a result, the amount of monitor image data transmitted to the remote control device 3 is reduced.

The operation control device 160 determines whether the content of the image command signal indicates a cutout of a part of the image (S41).
In the case of cutting out the area (Yes in S41), the operation control device 160 performs the area cutting process on the monitor image data (S42). That is, the motion control device 160 extracts an image of a part of the area designated by the image command from the images captured by the external photographing camera 11 to generate monitor image data. More specifically, for example, when the range captured by the external photographing camera 11 is wide, only an image of a part of the traveling direction necessary for maneuvering is transmitted / displayed. As a result, the amount of monitor image data transmitted to the remote control device 3 is reduced.
To cut out a part of the image, for example, a plurality of external photographing cameras 11 are connected to the control unit 10, and the motion control device 160 processes the images of the plurality of areas photographed by the plurality of external photographing cameras 11. It also applies if you have one. In the case of cropping a region (Yes in S41), the motion control device 160 uses only the image of the designated region as monitor image data. Also in this case, the amount of monitor image data transmitted to the remote control device 3 is reduced.

When it is determined in step S31 that the image command signal has not been received (No in S31), the operation control device 160 stops the output of the monitor image data (S45).
Next, the operation control device 160 determines whether or not an abnormal state has occurred in the control unit 10 (S46). The abnormal state of the control unit 10 is detected by, for example, a logic circuit composed of FPGA 181 of the monitoring circuit 180.
The operation control device 160 determines that an abnormal state has occurred, for example, when any of the parameters to be monitored is not within the range defined by the rule.
When an abnormal state has occurred (Yes in S46), the motion control device 160 outputs monitor image data to the remote control device 3 (S47). The monitor image data is transmitted to the remote control device 3 via the remote communication device 13.
As a result, the operator H can recognize the occurrence of the abnormal state at an early stage by the display screen of the remote control device 3. In addition, the operator H can recognize the cause of the abnormal state at an early stage from the display screen.

[Example of separation operation of linked operation device]
So far, an example in which the cooperative operation device 2 is connected to the automatic operation device 1 has been described. However, the control unit 10 of the present embodiment can be configured to correspond to the cooperative operation device 4 which is not connected to the automatic operation device 1 and operates separately.
FIG. 8 is a block diagram showing a third application example of the control unit shown in FIG.
In the application example shown in FIG. 8, the cooperative operation device 4 of the automatic operation system S is not connected to the automatic operation device 1. The cooperative operation device 4 operates away from the automatic operation device 1. The cooperative operation device 4 is, for example, an automatic traveling vehicle having substantially the same configuration as the automatic operation device 1. However, the cooperative control unit 40 of the cooperative operation device 4 operates as either a master unit or a slave unit.
The external communication connection unit 140 included in the control unit 10 of the automatic operation device 1 is a wireless communication device. Further, the cooperative operation device 4 includes an external communication connection unit 440 that performs wireless communication with the external communication connection unit 140. The control unit 10 of the automatic operation device 1 communicates with the cooperation control unit 40 of the cooperation operation device 4 via wireless communication. The cooperation control unit 40 controls the cooperation operation unit 42.

When the cooperative control unit 40 is the master unit, the control unit 10 of the automatic operation device 1 outputs an operation control signal based on the operation command from the cooperative control unit 40. For example, the control unit 10 travels in response to an operation command from the cooperation control unit 40. For example, the control unit 10 operates following the cooperative operation device 4. In this case, the control unit 10 can, for example, photograph the cooperative operation device 4 with the external photographing camera 11 and travel following the cooperative operation device 4 based on the image data of the external photographing camera 11.

When the cooperation control unit 40 is a slave unit, the control unit 10 generates an operation command for controlling the cooperation operation device 4 based on the image data of the external photographing camera 11. The control unit 10 transmits an operation command to the cooperative control unit 40. The cooperative operation device 4 operates following the automatic operation device 1.

[Second Embodiment]
In the above-described embodiment, an example of determining the type of the cooperative control unit 20 when the control unit 10 is connected to the cooperative control unit 20 has been described.
Subsequently, a mode in which the control unit 10 determines the type of the cooperation control unit 20 after being connected to the cooperation control unit 20 will be described.

FIG. 9 is a flowchart illustrating the cooperative operation of the control unit 10 according to the second embodiment of the present invention.
The operation of the control unit shown in FIG. 9 is different from the operation described with reference to FIG. 3 in steps S51 and S53.

In step S51, the operation control device 160 (see FIG. 2) of the control unit 10 determines whether or not an operation command has been received from the cooperative control unit 20.

When the operation control device 160 receives an operation command from the cooperation control unit 20 (Yes in S51), the operation control device 160 determines that the cooperation control unit 20 is the master unit. That is, the control unit 10 operates as a slave unit. In this case, the cooperation control unit 20 executes the processes of steps S21, S22, S53, and S24. In step S53, the operation control device 160 executes the process based on the operation command received in S51.

When the operation command is not received from the cooperation control unit 20 (Yes in S51), the operation control device 160 determines that the cooperation control unit 20 is a slave unit. That is, the control unit 10 operates as a master unit. In this case, the cooperation control unit 20 carries out the processes of steps S14 to S17.

As described above, the control unit 10 of the present embodiment determines that the cooperation control unit 20 is the master unit when the operation command is input from the cooperation control unit. The control unit 10 determines that the cooperative control unit is the slave unit until an operation command is input.

[Third Embodiment]
In the above-described embodiment, for example, with reference to FIG. 1, an example in which the control unit 10 of the automatic operation device 1 and the cooperation control unit 20 of the cooperation operation device 2 are directly connected to one remote communication device 13 will be described. did.
FIG. 10 is a block diagram showing a configuration of an automatic operation system including a control unit according to a third embodiment of the present invention.
The automatic operation device 1 in the present embodiment is different from the first embodiment in that it has a hub 13a. Further, the automatic operation device 1 in the present embodiment is connected to the cooperation control unit 20 via the hub 13a instead of directly.
Since other points in this embodiment are the same as those in the first embodiment, each part is designated by the same reference numerals as those in the first embodiment.
The control unit 10 and the cooperation control unit 20 in FIG. 10 are connected to one remote communication device 13 via the hub 13a. The hub 13a relays data between the control unit 10, the cooperative control unit 20, and the remote communication device 13. The control unit 10 and the cooperative control unit 20 connected to the hub 13a transmit data in a common transmission format.
Further, the automatic operation device 1 in FIG. 10 communicates with the cooperation control unit 20 via the hub 13a.
The hub 13a in the present embodiment has an independent data mixing function in the remote communication device 13 of the first embodiment. That is, the hub 13a has a part of the functions of the remote communication device 13. Therefore, even in this embodiment, it can be said that the cooperative control unit 20 of the automatic operation device 1 and the cooperative control unit 20 of the cooperative operation device 2 are connected to one remote communication device 13.

1

Automatic operation device

2, 2', 4 Cooperative operation device 3 Remote control device 10 Control unit 11 External environment sensing unit (external camera)
12 Operation unit 13 Remote communication device 14

Power supply unit

20, 40

Coordination control unit

21, 41 Coordination sensing unit (cooperation operation device camera)
22, 42 Cooperative operation unit 40 Cooperative control unit 110 External environment information connector (external image connector)
121 Actuator 130 Operation control connector 140,440 External communication connection (external communication connector)
150 Remote data connector 160 Operation control device 170 Automatic control circuit 171 GPU
171a Control model 172 Non-volatile memory 180 Monitoring circuit 181 FPGA
182 Non-volatile memory 184 Relay 185 Memory 221 Actuator S Automatic operation system

Claims

A control unit used in an automatically operating device equipped with an external environment sensing unit that detects an external environment and a controlled object that is controlled based on the external environment.
The control unit is
An external environment signal input from the external environment sensing unit by constructing a control model constructed by machine learning that shows the relationship between the external environment detected by the external environment sensing unit and the control of the controlled object. An automatic control circuit that outputs an operation control signal for controlling the control target and outputs a status index signal indicating the operation status of the control model based on the above.
The control unit is integrally formed with the automatic control circuit, has a field programmable gate array having a reprogrammable logic circuit, and is selected from the group consisting of the external environment signal, the operation control signal, and the status index signal. A monitoring circuit for monitoring the operation of the automatic control circuit constituting the control model, which is constructed by machine learning by inspecting at least one kind of signal.
The control unit according to claim 1.
The control unit is commercial.
The control unit according to claim 1.
The control unit is a mass production type.
The control unit according to claim 1.
The logic circuit is programmed to monitor the operation of the automatic control circuit constituting the control model constructed by machine learning with the logic constructed by the rule base.
The control unit according to claim 1.
The monitoring circuit operates the automatic control circuit by inspecting whether or not at least one parameter related to the at least one type of signal satisfies a condition defined so as to be associated with the parameter. It is configured to monitor.
The control unit according to claim 5.
The condition defined to be associated with the parameter is that the parameter is included in the range defined to be associated with the parameter.
The control unit according to claim 5.
The monitoring circuit is configured so that the conditions can be changed by changing software or hardware.
The control unit according to claim 1.
When the monitoring circuit detects an abnormality of at least one parameter related to the at least one type of signal, the monitoring circuit prohibits the output of the operation control signal by the automatic control circuit.
The control unit according to claim 1.
The monitoring circuit further detects anomalies by monitoring the voltage of the power supply supplied to the field programmable gate array.
The control unit according to claim 1.
The automatic control circuit includes a GPU having a multi-core capable of parallel processing.
The control unit according to claim 1.
The monitoring circuit is configured to cut off the power supply to the controlled object when an abnormality of at least one parameter related to the at least one type of signal is detected.
The control unit according to claim 11.
The monitoring circuit includes a relay for cutting off the power supply to the controlled object.
The control unit according to claim 1.
When the monitoring circuit detects an abnormality of at least one parameter related to the at least one type of signal, the portion of the controlled object that stops operation is set to at least one parameter related to the at least one type of signal. Change according to the type of abnormality.
The control unit according to claim 1.
The automatic control circuit is configured to output the operation control signal and the status index signal based on the external environment signal by executing a software process.
The monitoring circuit is configured to monitor the period and / or time during which the software process is executed.
The control unit according to claim 1.
The monitoring circuit monitors at least one parameter selected from the group consisting of the interval at which the external environment signal is input, the value of the data indicated by the external environment signal, and the amount of change in the data indicated by the external environment signal. It is configured as follows.
The control unit according to claim 1.
The monitoring circuit comprises an interval at which the signal is input, a value of data indicated by the signal, and a change amount of data indicated by the signal with respect to at least one of the operation control signal and the situation index signal. It is configured to monitor at least one parameter selected from the group.
The control unit according to claim 1.
The monitoring circuit includes a plurality of circuits for monitoring parameters different from each other, and is configured to determine the monitoring result by the logical product, OR, or a combination thereof of the signals output from each circuit. ing.
The control unit according to claim 1.
The monitoring circuit includes a program memory in which a program is stored and a non-volatile memory different from the program memory.
In the non-volatile memory, a condition is recorded so that at least one parameter related to the at least one type of signal is associated with the parameter.
The control unit according to claim 18.
In the non-volatile memory, a logical product, a logical sum, or a combination thereof of signals output when the processing function is executed is recorded.
The control unit according to claim 1.
The control target is a traveling device for traveling the automatically operating device.
The automatic control circuit generates the operation control signal for instructing the traveling course of the automatic operation device by processing the external environment.
The control unit according to claim 20.
When the monitoring circuit detects an abnormality of at least one parameter related to the at least one kind of signal, the monitoring circuit generates a traveling stop signal for operating a brake included in the traveling device.
The control unit according to claim 20.
When the monitoring circuit detects an abnormality of at least one parameter related to the at least one type of signal, the monitoring circuit generates a traveling stop signal for stopping the traveling device at a predetermined deceleration.
The control unit according to claim 20.
When the monitoring circuit detects an abnormality of at least one parameter related to the at least one type of signal, the traveling stop signal for stopping the traveling device is used as the operation control signal output by the automatic control circuit. To generate.