WO2020203968A1

WO2020203968A1 - Control unit

Info

Publication number: WO2020203968A1
Application number: PCT/JP2020/014546
Authority: WO
Inventors: 啓二西村; 村松　啓且; 裕二平松
Original assignee: ヤマハ発動機株式会社
Priority date: 2019-04-01
Filing date: 2020-03-30
Publication date: 2020-10-08

Abstract

The present invention addresses the problem of providing a control unit with which it is possible to enhance the degree of freedom for being combined with other equipment, to improve the versatility of the control unit. This control unit is provided with an external environment information connector, an operation control connector, an external communication connecting portion, and an operation control device, wherein, if the operation control device receives an operation command from a collaborative operation machine, the operation control device generates an operation control signal on the basis of the operation command received from the collaborative operation machine, and outputs the operation control signal to an actuator via the operation control connector, and if the collaborative operation machine is caused to operate on the basis of a result obtained by processing external environment data, the operation control device generates the operation control signal on the basis of the result obtained by processing the external environment data, generates an operation command for controlling the collaborative operation machine, and sends the operation command to the collaborative operation machine via the external communication connecting portion.

Description

Controller unit

The present invention relates to a control unit used in an automatic operation device (Autonomous Operation Machine).

There are known automatic operation devices that operate under the control of a control unit.
For example, Patent Document 1 discloses an automatic guided vehicle with a robot arm. The transport vehicle is a self-propelled vehicle. The automatic guided vehicle includes an AGV (Automatic Guided Vehicle), an AGV control unit, a robot arm, and a robot controller. The AGV control unit controls the movement of the AGV. The robot controller controls the robot arm.
The robot controller notifies the AGV control unit of a signal indicating that the robot arm can be connected to the power source and can work. This regulates the movement of the AGV. Further, the detachment control unit of the robot controller notifies the AGV control unit of a signal indicating that the work is completed. As a result, the movement of the AGV starts. In this way, the robot controller controls the movement of the AGV.

Japanese Unexamined Patent Publication No. 2017-132002

There are a wide variety of types of work required for work machines. The control unit used in the automatic operation device is required to increase the degree of freedom of combination with the control device of other working machines and improve the versatility.

An object of the present invention is to provide a control unit capable of increasing the degree of freedom of combination with other devices and improving versatility.

The present inventor conducted a study in view of the above-mentioned problems and obtained the following findings.

The above-mentioned automatic operation device may want to perform work other than the work by the robot arm. In response to such a request, it is conceivable to replace the working machine attached to the transport vehicle with another type of working machine.
The types of work required for the work machine include, for example, relatively simple work that operates simply according to the position of the autonomous vehicle. Among the work machines that perform such work, for example, there is a work machine configured so as not to output a signal for traveling to an autonomous driving vehicle. In this case, it is not easy to link the work machine and the autonomous driving vehicle. As a result, the types of work machines that can be combined with autonomous vehicles are limited.

Therefore, the present inventor has a configuration in which the automatic operation device operates based on the command when the work machine outputs a command, and outputs the command when the automatic operation device needs to output the command to the work machine. I thought.
In this case, the automatic operation device can be applied to a form in which a working machine that does not output a command is combined in addition to a form in which a working machine that outputs a command is combined. Further, the working machine to be attached can be arbitrarily selected from the above two types of working machines while suppressing the modification of the hardware configuration and the software configuration in the automatic operation device. In addition, the work machine attached to the automatic operation device can be easily replaced with another type of work machine. Therefore, the degree of freedom of combination can be increased and the versatility can be improved.

In order to achieve the above object, according to one viewpoint of the present invention, the control unit has the following configuration.

(1) A control unit used in an automatically operating device equipped with an external environment sensing unit that detects an external environment and an actuator that is controlled based on the external environment.
The control unit is
An external environment information connector for inputting external environment data indicating the detection result from the external environment sensing unit, and
An operation control connector for outputting an operation control signal for controlling the operation of the actuator, and an operation control connector.
It is configured to be communicably connected to the cooperative control unit that controls the cooperative operation device that outputs physical or non-physical output by wire or wirelessly, and controls the automatic operation device transmitted from the cooperative control unit. An external communication connection unit that outputs an operation command for controlling the control unit and transmits an operation command for controlling the linked operation device output from the control unit to the linked control unit.
When an operation command is received from the cooperative operation device, the operation control signal is generated based on the operation command received from the cooperative control unit, and the operation control signal is output to the actuator via the operation control connector. Then, when the cooperative operation device is operated based on the processing result of the external environment data, the operation control signal is generated based on the processing result of the external environment data and the cooperative operation device is controlled. It is provided with an operation control device that generates an operation command and transmits an operation command for controlling the cooperative operation device to the cooperative control unit via the external communication connection unit.

When the operation command is received from the cooperative control unit, the operation control device controls the actuator of the automatic operation device at least based on the operation command from the cooperative operation device. At this time, the motion control device generates an motion control signal at least based on the motion command, and outputs the motion control signal to the actuator. The operation control device may generate an operation control signal based only on the operation command, or may generate an operation control signal based on the operation command and the external environment data. In this way, the actuator of the automatic operation device is controlled based on the operation command of the cooperative control unit. Therefore, the operation of the automatically operating device and the operation of the linked operating device can be precisely linked.
Further, when operating the cooperative operation device based on the processing result of the external environment data, the operation control device transmits an operation command generated based on the processing result of the external environment data to the cooperative control unit. Therefore, the cooperative operation device is controlled based on the external environment of the automatic operation device. Therefore, the operation of the automatically operating device and the operation of the linked operating device can be precisely linked.
In this way, regardless of whether the linked control unit connected to the control unit used for the automatic operating device belongs to the type that outputs a command or the type that does not output a command, the operation of the automatic operating device and the linked operating device The operation of can be precisely linked. Further, while suppressing modification of the hardware configuration and software configuration in the automatically operating device, the linked operating device to be attached can be arbitrarily selected and attached from any of the two types of devices. In addition, the cooperative operation device attached to the automatic operation device can be easily replaced with a different type of device. Therefore, it is possible to increase the degree of freedom of combination with the automatically operating device and improve the versatility.

According to one viewpoint of the present invention, the control unit can adopt the following configuration.

(2) The control unit of (1)
The external environment sensing unit is a camera.
The external environment information connector inputs image data output from the camera as the external environment data.

According to the control unit having the above configuration, by using the image data from the camera, it is possible to execute the operation while recognizing the complicated external environment.
The control unit can cause the cooperative control unit to operate with high accuracy based on the image data from the camera. Further, the control unit can operate the actuator with high accuracy based on the operation command from the cooperative control unit and the image data from the camera. Therefore, while improving the versatility of the control unit, it is possible to make the connected linked control unit operate with high accuracy.

(3) The control unit of (1) or (2).
The actuator mounted on the automatic operation device is a traveling device for traveling the automatic operation device.
The motion control device is
When an operation command is received from the cooperative control unit, an operation control signal for instructing the traveling of the automatic operation device based on the operation command received from the cooperative control unit and the processing result of the external environment data. To generate
When operating the cooperative operation device based on the processing result of the external environment data, an operation control signal for instructing the running of the automatic operation device is generated based on the processing result of the external environment data. , The operation command for controlling the cooperative operation device is generated, and the operation command for controlling the cooperative operation device is transmitted to the cooperation control unit via the external communication connection unit.

According to the control unit having the above configuration, the automatic operation device functions as an automatic traveling vehicle. For example, when the control unit receives an operation command from the cooperative control unit, the autonomous vehicle makes the cooperative control unit operate with high accuracy while traveling based on external environmental data such as image data from a camera. be able to. Further, when the cooperative operation device is operated based on the processing result of the external environment data, the autonomous driving vehicle has high accuracy based on the operation command from the cooperation control unit and the external environment data such as the image data from the camera. You can drive with. Therefore, it is possible to make the cooperative control unit operate with high accuracy while improving the versatility of the autonomous driving vehicle.

(4) The control unit of any one of (1) to (3).
The operation control device generates the operation control signal based on the operation command received from the cooperation control unit and outputs the operation control signal to the actuator, and connects the operation command to the cooperation control unit by the external communication. The process of transmitting via the unit is executed in sequence.

According to the control unit having the above configuration, the operation control device sequentially executes the process based on the operation command received from the cooperative control unit and the process of transmitting the operation command to the cooperative control unit. Therefore, the operation of the actuator can be controlled according to the type of the constant cooperation control unit immediately after being connected to the cooperation control unit. Therefore, the versatility of the control unit can be improved.

(5) The control unit of any one of (1) to (4).
The external communication connection unit includes a plurality of connectors corresponding to a plurality of different types of transmission formats.

According to the control unit having the above configuration, the linked control unit to be connected can be selected from the candidates of a plurality of units having various functions. Therefore, the versatility of the control unit can be further improved.

The terminology used herein is for the purpose of defining only specific embodiments and is not intended to limit the invention.
As used herein, the term "and / or" includes any or all combinations of one or more related enumerated components.
As used herein, the use of the terms "including, including,""comprising," or "having," and variations thereof, is a feature, process, operation, described. It identifies the presence of elements, components and / or their equivalents, but can include one or more of steps, actions, elements, components, and / or groups thereof.
As used herein, the terms "attached", "connected", "combined" and / or their equivalents are widely used, direct and indirect attachment, connection and Includes both bonds. Further, "connected" and "coupled" are not limited to physical or mechanical connections or connections, but can include direct or indirect electrical connections or connections.
Unless otherwise defined, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.
Terms such as those defined in commonly used dictionaries should be construed to have meaning consistent with the relevant technology and in the context of the present disclosure and are expressly defined herein. Unless it is, it will not be interpreted in an ideal or overly formal sense.
It is understood that a plurality of techniques and processes are disclosed in the description of the present invention.
Each of these has its own benefit and each can be used in conjunction with one or more of the other disclosed techniques, or in some cases all.
Therefore, for clarity, this description refrains from unnecessarily repeating all possible combinations of individual steps.
Nevertheless, the specification and claims should be read with the understanding that all such combinations are within the scope of the present invention and claims.
This specification describes a new control unit.
In the following description, for purposes of illustration, a number of specific details are given to provide a complete understanding of the present invention.
However, it will be apparent to those skilled in the art that the present invention can be practiced without these particular details.
The present disclosure should be considered as an example of the invention and is not intended to limit the invention to the particular embodiments set forth in the drawings or description below.

The detection of the external environment is to detect the external environment of the automatically operating device. The external environment is an external condition that determines the operation of an automatically operating device. The external environment is the environment around the automatically operating device.
The external environment data as a result of being detected by the external environment sensing unit is, for example, an image taken of the outside of the automatically operating device. The external environment sensing unit in this case is a camera. However, the external environment data and the external environment sensing unit are not particularly limited. The external environment data is, for example, the distance to an external object of the automatic operation device. The external environment sensing unit in this case is, for example, a distance sensor. The distance sensor is, for example, a sonar (SONAR: sound navigation ranking) that uses ultrasonic waves. The distance sensor is, for example, a distance measuring device using a laser. Further, the external environment data is, for example, a laser scan image of an object outside the automatic operation device. The external environment sensing unit in this case is, for example, LIDAR (Laser Imaging Detection and Ranking). The external environment data is, for example, position information in the work area of the automatic operation device. The external environment sensing unit in this case is, for example, GNSS (Global Navigation Satellite System).

An automatically operating device is a device that can operate automatically regardless of human operation. However, the automatic operation device does not exclude human operation. For example, the automatic operation device may perform some operations by human operation and automatically perform some operations regardless of human operation. For example, the automatic operation device may be instructed about a rough upper level operation by human operation, and may determine and execute a more detailed lower level operation for realizing the upper level operation. Further, for example, the automatic operation device may perform an operation in a certain range according to a human operation, and may perform the above-mentioned operation or an operation similar to the above-mentioned operation in a range different from the above-mentioned certain range. Good. The autonomous driving device is, for example, an autonomous driving vehicle. However, the automatic operation device is not particularly limited, and may be, for example, an automatic work robot.
Further, the cooperative operation device that cooperates with the automatic operation device is, for example, an automatic work robot mounted on an automatic driving vehicle. However, the cooperative operation device is not limited to this, and for example, an autonomous driving vehicle equipped with an automatic work robot may be used.

When the cooperative operating device is equipped with an actuator, the actuator is, for example, a motor. The actuator causes the automatic operation device to perform physical output. The actuator provided in the cooperative operation device causes the cooperative operation device to perform physical output. The actuator may be, for example, an electromagnetic solenoid. The actuator is controlled by the motion control device. The actuator may be directly controlled by the motion control device. The actuator and the motion control device may be indirectly controlled via a control means different from the motion control device. In this case, the motion control device controls the control means, and the control means controls the actuator.

The control unit is, for example, a navigation device for controlling the automatic driving of an autonomous driving vehicle. However, the control unit is not particularly limited, and may be configured to control, for example, a robot that does not travel.
The control unit used in the automatic operation device is mounted on the automatic operation device, for example. However, the mounting position of the control unit is not particularly limited, and for example, the control unit may be installed at a position away from the automatic operation device and may be communicably connected to the peripheral device via each connector.

The external communication connection part is, for example, a connector to which an electric wire is connected. However, the external communication connection unit may be a wireless communication device connected by wireless communication. That is, the communicable connection is, for example, directly electrically connected via a connector and an electric wire. However, the means of connection capable of communicating is not particularly limited, and may be, for example, a state in which communication is possible via a wireless communication device, which is arranged at physically distant positions. Further, the communicable connection may be in a state in which communication is possible via a central communication relay device capable of communicating with a large number of devices.

The cooperative operation device is, for example, a device that operates in cooperation with an automatic operation device. The cooperative operation device operates, for example, in cooperation with the automatic operation device to achieve a common work purpose. The cooperative operation device realizes parallel execution for the operation of the automatic operation device by, for example, executing the same type of operation as the automatic operation device. The cooperative operation device completes one operation in cooperation with the automatic operation device, for example, by complementing the operation of the automatic operation device. The cooperative operation device is, for example, a device connected to an automatic operation device. The cooperative operation device is, for example, an automatic work robot connected to an automatic driving vehicle as an automatic operation device. However, the cooperative operation device is not particularly limited, and may be, for example, a second automatic driving vehicle that travels at a position away from the automatic driving vehicle as the automatic driving device. In this case, the cooperative control unit is, for example, a navigation device for controlling the automatic driving of the autonomous driving vehicle as a cooperative operation device. In this case, a plurality of autonomous driving vehicles can travel in cooperation with each other.

The physical output is an action that involves the physical movement of at least some of the members. The physical output of the linked operating device is, for example, the movement of the entire linked operating device. However, the physical output is not particularly limited, and for example, a part of the linked operating device may be physically moved. The physical output is, for example, the operation of the robot arm when the cooperative operation device includes the robot arm. The physical output is, for example, the operation of a valve when the coordinated operating device includes a valve that opens and closes a flow path. The physical output is, for example, the rotation of the fan when the cooperating device includes a blower fan.
Non-physical output is an action that does not involve physical movement. The non-physical output is, for example, the output of information. The non-physical output is, for example, the output of captured image data when the cooperative operating device is equipped with a camera. The non-physical output is, for example, a display when the cooperative operating device is equipped with an image display device.

The operation command is a command exchanged between the automatic operation device and the cooperative operation device. More specifically, the operation command is a signal exchanged between the control unit of the automatic operation device and the cooperative control unit of the cooperative operation device. The operation command can also be referred to as an operation command signal. The type of operation command sent by the automatic operation device to the cooperative operation device may be different from the type of operation command sent by the cooperative operation device to the automatic operation device. Moreover, these types may be common.
The operation command sent by the automatic operation device to the cooperative operation device and the operation command sent by the cooperative operation device to the automatic operation device may be distinguished from each other. For example, an operation command sent from a control unit of an automatic operation device to a cooperative control unit of a cooperative operation device is called an active operation command, and an operation command sent from a cooperative control unit of a cooperative operation device to a control unit of an automatic operation device is a passive operation. It can also be called a directive.
A device other than the linked operating device linked with the automatic operating device may be further linked to the automatic operating device. That is, a plurality of devices may cooperate with the automatically operating device.
The operation control signal for instructing the traveling is, for example, a signal for instructing the traveling route. However, the operation control signal for instructing the running may be a signal for instructing the start or stop of the running.

A plurality of transmission formats supported by a plurality of connectors are, for example, CAN (Control Area Network) (registered trademark) Bus or Ethernet (registered trademark).

According to the present invention, it is possible to realize a control unit capable of increasing the degree of freedom of combination with other devices and improving versatility.

It is a block diagram which shows the structure of the automatic operation system including the control unit which concerns on 1st Embodiment of this invention. It is a block diagram which shows the structure of the automatic operation system which includes the control unit which concerns on 2nd Embodiment of this invention. It is a block diagram which shows the structure of the control unit shown in FIG. It is a flowchart explaining the cooperation operation of the control unit shown in FIG. It is a block diagram which shows the 1st application example of the control unit shown in FIG. It is a block diagram which shows the 2nd application example of the control unit shown in FIG. It is a block diagram which shows the flow of an image in the automatic operation system shown in FIG. It is a flowchart explaining the control of the image in the operation control device of the control unit shown in FIG. 7. It is a block diagram which shows the 3rd application example of the control unit shown in FIG. It is a block diagram which shows the structure of the automatic operation system which includes the control unit which concerns on 3rd Embodiment of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 is a block diagram showing a configuration of a control unit and its peripheral devices according to the first embodiment of the present invention.

The control unit 10 is used for the automatic operation device 1. The automatic operation device 1 is equipped with an external environment sensing unit 11 and an actuator 121.
The automatic operation device 1 can operate automatically regardless of human operation. However, the automatic operation device 1 does not exclude human operation. For example, the automatic operation device 1 may perform some operations by human operation. The automatic operation device is, for example, an automatic driving vehicle or an automatic work robot.
The external environment sensing unit 11 detects the external environment of the automatic operation device 1. The external environment sensing unit 11 outputs external environment data indicating the detection result. The external environment sensing unit 11 is, for example, a camera that photographs the external environment of the automatically operating device 1. The external environment sensing unit 11 may be, for example, a distance sensor that measures the distance to an external object of the automatic operation device 1.
The actuator 121 is a device that operates mechanically by electrical control. The actuator 121 drives a device mounted on the automatic operation device 1 or the automatic operation device 1 itself. The actuator 121 is, for example, a motor or an electromagnetic solenoid.

The control unit 10 includes an external environment information connector 110, an operation control connector 130, an external communication connection unit 140, and an operation control device 160.

External environment data indicating the detection result is input from the external environment sensing unit 11 to the external environment information connector 110.
The motion control connector 130 outputs an motion control signal for controlling the motion of the actuator 121.

The external communication connection unit 140 is communicably connected to the cooperation control unit 20. The cooperation control unit 20 controls the cooperation operation device 2 that outputs physical or non-physical output. The cooperative operation device 2 is a device provided separately from the automatic operation device 1. The cooperative operation device 2 is physically connected to, for example, the automatic operation device 1. However, the cooperative operation device 2 may be separated from the automatic operation device 1, for example. The cooperative operation device 2 may include its own sensing unit and its own actuator. The cooperative control unit 20 can transmit an operation command for controlling the automatic operation device 1 to the control unit 10. Further, the cooperation control unit 20 can receive an operation command for controlling the cooperation operation device 2 from the control unit 10.
The external communication connection unit 140 is, for example, a communication device for communicating with the cooperation control unit 20 according to a specific agreement. The external communication connection unit 140 is, for example, a communication device for wired communication. The external communication connection unit 140 is, for example, an Ethernet (registered trademark, the same applies hereinafter) communication device. The external communication connection unit 140 includes, for example, an Ethernet connector. The external communication connection unit 140 may be physically connected to a device other than the cooperation control unit 20 via a connector and a cable. The external communication connection unit 140 may be physically connected to the cooperation control unit 20 via a hub or router, for example. The external communication connection unit 140 may be a device that performs communication based on an agreement different from Ethernet. The external communication connection unit 140 may perform communication based on Wi-Fi (registered trademark, the same shall apply hereinafter) or Bluetooth (registered trademark, the same shall apply hereinafter), for example. Further, the external communication connection unit 140 may be a connector for communicating simple serial communication data. For example, the external communication connection unit 140 may perform communication based on the Controller Area Network (CAN) (registered trademark, the same shall apply hereinafter) Bus. The external communication connection unit 140 may communicate in accordance with an agreement other than an international standard.

The motion control device 160 controls the motion of the actuator 121. The motion control device 160 generates an motion control signal for controlling the motion of the actuator 121. The operation control device 160 outputs an operation control signal via the operation control connector 130.
The operation control device 160 generates, for example, an operation control signal based on the processing result of the external environment data. The operation control device 160 can also generate an operation control signal based on an operation command received from the cooperative operation device 2.

More specifically, when the operation control device 160 receives an operation command from the cooperative operation device 2, the operation control device 160 generates an operation control signal based on the operation command received from the cooperative operation device 2. The motion control device 160 outputs a motion control signal to the actuator 121 via the motion control connector 130.
Further, when the operation control device 160 operates the cooperative operation device 2 based on the processing result of the external environment data, the operation control device 160 generates an operation control signal based on the processing result of the external environment data and controls the cooperative operation device 2. Generate an operation command for. The operation control device 160 transmits an operation command to the cooperative operation device via the external communication connection unit 140.

When the operation command is received from the cooperative operation device 2, the operation control device 160 controls the actuator 121 of the automatic operation device 1 at least based on the operation command from the cooperative operation device 2. At this time, the operation control device 160 generates an operation control signal at least based on the operation command, and outputs the operation control signal to the actuator 121. The operation control device 160 may generate an operation control signal based only on the operation command, or may generate an operation control signal based on the operation command and the external environment data. In this way, the actuator 121 of the automatic operation device 1 is controlled based on the operation command of the cooperative operation device 2. Therefore, the operation of the automatic operation device 1 and the operation of the cooperative operation device 2 can be precisely linked.
Further, when the cooperative operation device 2 is operated based on the processing result of the external environment data, the operation control device 160 transmits an operation command generated based on the processing result of the external environment data to the cooperative control unit 20. Therefore, the cooperative operation device 2 is controlled based on the external environment of the automatic operation device 1. Therefore, the operation of the automatic operation device 1 and the operation of the cooperative operation device 2 can be precisely linked.
In this way, the operation of the automatic operation device 1 is performed regardless of whether the cooperative operation device 2 connected to the control unit 10 used for the automatic operation device 1 is a type that outputs a command or a type that does not output a command. And cooperation operation The operation of the device 2 can be precisely linked. Further, the linked operation device 2 to be attached can be arbitrarily selected and attached from any of the two types of devices while suppressing the modification of the hardware configuration and the software configuration in the automatic operation device 1. Further, the cooperative operation device 2 attached to the automatic operation device 1 can be easily replaced with a different type of device. Therefore, the degree of freedom of combination with the automatic operation device 1 can be increased and the versatility can be improved.

[Second Embodiment]
FIG. 2 is a block diagram showing a configuration of an automatic operation system including a control unit according to a second embodiment of the present invention.
In the present embodiment, as an application example of the control unit 10, the entire automatic operation system S including the automatic operation device 1 will be described. Further, in the present embodiment, an example in which a functional unit is further added to the automatic operation device 1 of the first embodiment will be described. The parts common to the parts described in the first embodiment will be described with the same reference numerals.

[Automatic operation system]
The automatic operation system S is an operator H, that is, a system capable of automatically operating regardless of the operation of a person. The automatic operation system S detects the external environment of the automatic operation system S by itself. Then, the automatic operation system S recognizes the content of the detection result, and controls the operation of the automatic operation system S based on the recognition result.

The automatic operation system S also has a function of operating according to the operation of the operator H. For example, the automatic operation system S operates in response to the operation of the remote control device 3. The remote control device 3 is a device for remotely controlling the automatic operation system S. The remote control device 3 can communicate with the automatic operation system S by wireless communication. The remote control device 3 transmits operation information to the automatic operation system S. Further, the remote control device 3 receives an image from the automatic operation system S and displays the image.
For example, the automatic operation system S can start or stop the automatic operation in response to the operation of starting or stopping the automatic operation with respect to the remote control device 3. Further, for example, the automatic operation system S can select one pattern from a plurality of automatic operation patterns according to the operation of the selection operation for the remote control device 3. Further, for example, the automatic operation system S can perform sequential operations in response to sequential operations on the remote control device 3. Sequential operations are operations typified by, for example, forward, backward, and stop.

The automatic operation system S includes an automatic operation device 1 and a cooperative operation device 2. The automatic operation device 1 shown in FIG. 2 is a system capable of performing automatic operation.
The automatic operation device 1 detects the external environment. Then, the automatic operation device 1 recognizes the detection result and controls the operation based on the recognition result. However, the automatic operation device 1 also has a function of operating according to the operation of the operator H. The operation according to the operation is the same as the operation for the automatic operation system S described above.
In the example shown in FIG. 2, the cooperative operation device 2 is also a system capable of performing automatic operation. The cooperative operation device 2 detects the external environment. Then, the cooperative operation device 2 recognizes the detection result and controls the operation based on the recognition result. Further, the cooperative operation device 2 also has a function of operating according to the operation of the operator H.
As the cooperative operation device 2, it is also possible to adopt a configuration in which the automatic operation device 1 does not detect the external environment by itself, but operates by receiving the detection result of the external environment in the automatic operation device 1 or the instruction from the automatic operation device 1. Here, first, a configuration for detecting the external environment as the cooperative operation device 2 will be described.

The cooperative operation device 2 shown in FIG. 2 operates in cooperation with the automatic operation device 1. That is, the automatic operation device 1 and the cooperative operation device 2 cooperate with each other to complete the work expected by the operator H. The cooperative operation device 2 can cooperate by performing the same type of operation as the automatic operation device 1, for example. As an example of this, there is a case where the automatic operation device 1 and the cooperative operation device 2 are a pair of robot arms.
Further, for example, the cooperative operation device 2 and the automatic operation device 1 can complete the desired operation by performing different types of operations from each other. That is, the cooperative operation device 2 and the automatic operation device 1 can cooperate with each other. As an example of this, there is a case where the automatic operation device 1 is an automatic traveling vehicle and the cooperative operation device 2 is a robot arm mounted on the automatic traveling vehicle.
The automatic operation device 1 and the cooperative operation device 2 shown in FIG. 2 are mechanically connected.
However, the automatic operation device 1 and the cooperative operation device 2 may be separated from each other. As an example of this, there is a case where the automatic operation device 1 and the cooperative operation device 2 are two vehicles that travel in a common area or share a plurality of adjacent areas.

The linked operation device 2 shown in FIG. 2 is communicably connected to the automatic operation device 1. The automatic operation device 1 and the cooperative operation device 2 shown in FIG. 2 are electrically connected to each other by electric wires. However, as the automatic operation device 1 and the cooperative operation device 2, for example, a configuration for wireless communication can be adopted.

[Automatic operation device]
The automatic operation device 1 includes a control unit 10, an external environment sensing unit 11, an operation unit 12, a remote communication device 13, and a power supply unit 14.

The external environment sensing unit 11 detects the external environment of the automatic operation device 1. The external environment sensing unit 11 outputs external environment data indicating the detection result.
The external environment sensing unit 11 is, for example, a camera that photographs the external environment of the automatically operating device 1. The camera as the external environment sensing unit 11 outputs external image data indicating the shooting result. By using the camera, the control unit 10 can operate the actuator 121 while recognizing the complicated external environment.

The operation unit 12 is controlled based on the external environment. The operation unit 12 includes an actuator 121. The actuator 121 is mechanically operated by electric control to drive a device mounted on the automatic operation device 1 or the automatic operation device 1 itself.

The control unit 10 is connected to the external environment sensing unit 11, the operation unit 12, and the remote communication device 13.
The control unit 10 controls the actuator 121 of the operation unit 12 based on the external environment detected by the external environment sensing unit 11. More specifically, the control unit 10 recognizes the contents of the external environment by processing the external environment data output from the external environment sensing unit 11. The control unit 10 determines the control content based on the recognized content. Then, the control unit 10 controls the actuator 121 based on the determined control content.

The control unit 10 may receive a control command corresponding to the operation of the remote control device 3 from the remote communication device 13 and control the actuator 121 based on the control command. The internal configuration of the control unit 10 will be described later.

The power supply unit 14 supplies electric power to the control unit 10, the external environment sensing unit 11, the operating unit 12, and the remote communication device 13.
The power supply unit 14 has a battery (not shown). The power supply unit 14 supplies the electric power stored in the battery to each unit. The power supply unit 14 supplies electric power to meet the demand of each unit. The power source used by the power supply unit 14 is not limited to the battery, and various power sources can be used. For example, as the power supply unit 14, a configuration having an engine generator or a fuel cell instead of the battery can be adopted. An engine generator includes, for example, an engine that operates on liquid fuel and a generator that is driven by the engine to generate electricity.
The power supply unit 14 shown in FIG. 2 also supplies electric power to the cooperative operation device 2. However, the linked operation device 2 may be provided with a power supply independent of, for example, the automatic operation device 1.

The remote communication device 13 is communicably connected to the remote control device 3. The remote communication device 13 is communicably connected to the remote control device 3 by wireless communication.
The remote communication device 13 relays communication data between the remote control device 3 and the control unit 10. For example, the remote communication device 13 outputs a control command output from the remote control device 3 to the control unit 10 in response to the operation of the remote control device 3. As a result, the control command from the remote control device 3 is supplied to the control unit 10. Further, for example, the remote communication device 13 supplies data based on the data output from the external environment sensing unit 11 to the remote control device 3. For example, when the external environment sensing unit 11 is a camera, the remote communication device 13 transmits data based on the external image data output from the camera to the remote control device 3. As a result, the image of the camera is displayed on the remote control device 3. In this case, the remote control device 3 transmits an image command to the remote communication device 13. The image command is a command for designating the content of the image and the amount of image data transmitted to the remote control device 3. The remote communication device 13 sends an image command to the control unit 10.

The data exchange between the remote communication device 13 and the control unit 10 described above is the same as the data exchange between the remote communication device 13 and the cooperative control unit 20.

[Collaboration operation device]
The cooperative operation device 2 includes a cooperative control unit 20, a cooperative sensing unit 21, and a cooperative operation unit 22. The roles of the cooperation control unit 20, the cooperation sensing unit 21, and the cooperation operation unit 22 in the cooperation operation device 2 are the same as the roles of the control unit 10, the external environment sensing unit 11, and the operation unit 12 in the above-mentioned automatic operation device 1. are doing. However, the type of environment detected by the cooperative sensing unit 21, the detailed content of the determination of the cooperative control unit 20, and the output of the cooperative operation unit 22 differ depending on the function of the cooperative operation device 2.
The cooperation control unit 20 controls the cooperation operation unit 22 that performs physical output. The physical output of the cooperative operation unit 22 is, for example, the operation of the actuator 221.

The cooperative control unit 20 corresponds to an example of the control unit referred to in the present invention, like the control unit 10.

As an example of the function of the automatic operation device 1 shown in FIG. 2, for example, an automatic traveling vehicle can be mentioned. In this case, the control unit 10 is, for example, an automatic navigation unit. The actuator 121 of the operation unit 12 is a traveling device for traveling the automatic operation device 1. The traveling device is, for example, a motor. The control unit 10 recognizes the content of the image of the traveling region taken by the camera as the external environment sensing unit 11, determines the traveling route based on the recognition result, and instructs the operation unit 12 of the determined traveling route. That is, the control unit 10 controls the operation unit 12 based on the determined travel path. As a result, the automatically operating device 1 automatically travels. It should be noted that the control unit 10 can also adopt a configuration in which, for example, the control unit 10 does not determine the route itself, but determines the start or stop of traveling on a preset traveling route.
An example of the function of the cooperative operation device 2 shown in FIG. 2 is a work device mounted on an autonomous vehicle. An example of working equipment is a fruit-picking device that picks fruits on a farm. In this case, the cooperative control unit 20 recognizes the content of the fruit tree photographed by the camera as the cooperative sensing unit 21, determines the state and position of the fruit based on the recognition result, and based on the determined position of the fruit, A running command (running position command) is transmitted to the automatic operation device 1. The travel command is an example of an operation command. Further, the actuator 221 of the fruit-picking device as the cooperative operation unit 22 is controlled based on the determined position of the fruit. By linking the automatic operation device 1 and the cooperative operation device 2, the function of picking the fruits on the farm is realized.
Further, the cooperative operation device 2 mounted on the automatic operation device 1 can be replaced with a cooperative operation device having a function different from that of the cooperative operation device 2 shown in FIG.

[Controller unit]
FIG. 3 is a block diagram showing the configuration of the control unit 10 shown in FIG.
The control unit 10 is a unit covered with one housing, and is incorporated in the automatic operation device 1 (see FIG. 2). The control unit 10 is electrically connected to each part of the automatic operation device 1.
The control unit 10 is a commercial unit. The control unit 10 is a mass production type.

The external environment information connector 110 is electrically connected to the external environment sensing unit 11 shown in FIG. External environment data indicating the detection result is input to the control unit 10 from the external environment sensing unit 11 via the external environment information connector 110.
When the external environment sensing unit 11 is, for example, a camera, the external environment information connector 110 functions as an external image connector. Hereinafter, the external environment information connector 110 will also be referred to as an external image connector 110.

The operation control connector 130 is electrically connected to the operation unit 12 shown in FIG. An operation control signal for controlling the operation of the actuator 121 is output from the control unit 10 to the operation unit 12 via the operation control connector 130.

The external communication connection unit 140 is electrically connected to the remote communication device 13 shown in FIG. The remote communication device 13 is connected to the cooperation control unit 20. The remote communication device 13 in this embodiment functions as a hub. That is, the remote communication device 13 outputs the signal output from the external communication connection unit 140 to the cooperation control unit 20, and outputs the signal output from the cooperation control unit 20 to the external communication connection unit 140. Therefore, the external communication connection unit 140 is communicably connected to the cooperation control unit 20 shown in FIG. The external communication connection unit 140 in the example shown in FIG. 2 is an external communication connector that is electrically connected to the remote communication device 13. As described above, the remote communication device 13 functions as a hub. That is, when viewed from the external communication connection unit 140, the cooperation control unit 20 can be identified with the remote communication device 13. Therefore, the external communication connection unit 140 can be said to be an external communication connector that is electrically connected to the cooperation control unit 20. Hereinafter, the external communication connection unit 140 is also referred to as an external communication connector 140. The external communication connector 140 physically includes a plurality of connectors corresponding to a plurality of types of transmission formats. The types of transmission formats are, for example, Controller Area Network (CAN) (registered trademark) and Ethernet (registered trademark). By having a plurality of connectors, the cooperation control unit 20 can be selected from the candidates of a plurality of units having various functions. The versatility of the control unit 10 is improved. The external communication connector 140 is an example of an external communication connection unit that is communicably connected to the cooperation control unit 20. As the external communication connection unit, for example, a configuration in which a wireless communication device is used instead of the external communication connector 140 can be adopted.

The external communication connection unit 140 is electrically connected to the remote communication device 13 shown in FIG. A control command signal is input to the control unit 10 from the remote control device 3 (see FIG. 2) and the remote control device 13 via the external communication connection unit 140. That is, in response to the request for transmission of the image being input from the remote control device 3 to the remote communication device 13, the control command signal is output from the remote control device 13 and input via the external communication connection unit 140. It is input to the control unit 10. Further, a signal indicating the external environment and the state of the control unit 10 is output from the control unit 10 to the remote communication device 13 via the external communication connection unit 140. This signal is supplied from the remote communication device 13 to the remote control device 3.

The motion control device 160 controls the actuator 121 of the motion unit 12 based on the external environment detected by the external environment sensing unit 11 shown in FIG. More specifically, the motion control device 160 processes the external environment data output from the external environment sensing unit 11. The motion control device 160 controls the actuator 121 based on the processing result of the external environment data. Further, the motion control device 160 receives a control command corresponding to the operation of the remote control device 3 from the remote communication device 13, and controls the motion unit 12 based on the control command.
Further, the operation control device 160 communicates with the cooperation control unit 20 connected via the external communication connector 140. As described above, various devices can be selected as the collaborative operation device 2 that can be combined with the automatic operation device 1.
In a certain situation, the cooperative operation device 2 transmits an operation command to the operation control device 160. Further, the cooperative operation device 2 is a cooperative control unit 20 configured to receive an operation command from the operation control device 160 in a situation different from the above.
The operation control device 160 switches the type of operation according to the situation of the cooperative control unit 20.

When the operation control device 160 receives an operation command from the cooperation control unit 20, the operation control device 160 generates an operation control signal based on the operation command. The motion control device 160 outputs the generated motion control signal to the actuator 121 of the motion unit 12 via the motion control connector 130.
On the other hand, when the operation command is not received from the cooperation control unit 20, the operation control device 160 controls the cooperation operation device 2 based on the processing result of the external environment data input from the external environment sensing unit 11. Generate an operation command. The operation control device 160 transmits the generated operation command to the cooperation control unit 20 via the external communication connector 140.
As a result, the external communication connector 140 of the control unit 10 is linked as the linked control unit 20, regardless of whether the device whose input / output of the operation command changes according to the situation or the device whose input / output is fixed is connected. Precise work can be performed in cooperation with the control unit 20. Therefore, the versatility of the control unit 10 can be improved.

[Configuration of operation control device]
As shown in FIG. 3, the motion control device 160 includes an automatic control circuit 170 and a monitoring circuit 180.
The automatic control circuit 170 and the monitoring circuit 180 are provided in the housing of the control unit 10.

The automatic control circuit 170 carries out basic control processing in the motion control device 160. More specifically, the automatic control circuit 170 controls the actuator 121 based on the external environment signal from the external environment sensing unit 11. More specifically, the automatic control circuit 170 outputs an operation control signal based on an external environment signal by executing a software process. The automatic control circuit 170 also outputs a status index signal by executing a software process.

The automatic control circuit 170 includes a Graphics Processing Unit (GPU) 171.
GPU 171 is a processor having a multi-core capable of parallel processing. The GPU 171 includes 100 or more arithmetic cores that can operate in parallel. The GPU 171 executes a SIMD (single-instruction multiple-data stream) operation by 100 or more arithmetic cores.
Further, the automatic control circuit 170 includes a non-volatile memory 172, a RAM 173, a control input / output (control IO) 174, and a CPU 175. The non-volatile memory 172 is, for example, a mask ROM flash memory or an EEPROM.
The CPU 175 is a Central Processing Unit. The CPU 175 controls the entire automatic control circuit 170. The GPU 171 and the CPU 175 share and execute the control of the automatic control circuit 170. More specifically, the CPU 175 causes the GPU 171 to perform some of the functions of the automatic control circuit 170. The functions executed by the GPU 171 will be described later.
The non-volatile memory 172 stores a program executed by the CPU 175 and the GPU 171. The CPU 175 sequentially reads and executes the programs stored in the non-volatile memory 172. As a result, control by the automatic control circuit 170 is executed.
Further, the program of the GPU 171 stored in the non-volatile memory 172 is read by the CPU 175 and supplied to the GPU 171.
The RAM 173 holds the result of the processing by the CPU 175 and the result of the processing by the GPU 171. The CPU 175 and the GPU 171 read / write data to / from the RAM 173. The RAM 173 stores data input to the CPU 175 and the GPU 171, data indicating the processing status, and data indicating the operation control signal output from the automatic control circuit 170 as a result of the processing. The data input to the GPU 171 is, for example, data representing an external environment signal. The data indicating the processing status is, for example, one of the parameters indicating the operating status of the automatic control circuit 170.
The control IO 174 relays signals input / output to the CPU 175 and the GPU 171. The CPU 175 and the GPU 171 output a status index signal indicating the operating status of the control model via the control IO 174. The status indicator signal is, for example, a pulse indicating the period and time during which the process of processing the control model is executed. The status index signal is one of the parameters indicating the operating status of the automatic control circuit 170. The stored contents of the RAM 173 can be read out to the FPGA 181 of the monitoring circuit 180 via the control IO 174.

The CPU 175 supplies the program stored in the non-volatile memory 172 to the GPU 171. Further, the CPU 175 outputs a command to execute the program to the GPU 171.
When the GPU 171 executes the program stored in the non-volatile memory 172, the automatic control circuit 170 is configured with the control model 171a constructed by machine learning. The control model 171a is a model showing the relationship between the external environment detected by the external environment sensing unit 11 and the control of the operation unit 12 to be controlled. The control model 171a is a machine learning model using a neural network.
Since the GPU 171 can execute SIMD operations on 100 or more arithmetic cores, it is possible to execute the processing of the control model 171a accompanied by the iterative arithmetic of a large-scale matrix at high speed.
The CPU 175 determines the operation of the control unit 10 based on the information of the object obtained as a result of applying the data of the external environment to the machine learning model. The CPU 175 controls the operation unit 12 based on the result of the determination. More specifically, the CPU 175 outputs a command to the operation control connector 130 via, for example, the control IO 174 and the communication IF 183 of the monitoring circuit 180.
Further, the CPU 175 transmits a command to the cooperation control unit 20 based on the result of the determination. Further, the CPU 175 transmits data to the remote communication device 13.
The division of control between the CPU 175 and the GPU 171 and the input / output of the model executed by the GPU 171 are not limited to those described above. For example, the machine learning model may be a model that directly shows, for example, the relationship between the external image data and the optimum traveling path or the motion trajectory of the arm or the like. In this case, the CPU 175 controls the operation unit 12 based on the travel path or operation locus output as the processing result of the GPU 171.

The monitoring circuit 180 constitutes the control unit 10 integrally with the automatic control circuit 170.
The monitoring circuit 180 includes a field programmable gate array (FPGA) 181 and a non-volatile memory 182. Further, the monitoring circuit 180 includes a communication interface (communication IF) 183, a relay 184, and a memory 185 for a program.

The FPGA 181 has a reprogrammable logic circuit. The non-volatile memory 182 stores the connection information of the logic circuit for monitoring constructed by the FPGA 181. The FPGA 181 reads the connection information from the non-volatile memory 182 in the initialization process after the power is turned on or after the reset. The FPGA 181 constructs a logic circuit based on connection information. After constructing the logic circuit, the FPGA 181 starts the processing by the logic circuit.
By changing the data stored in the non-volatile memory 182, it is possible to change the function constructed by the logic circuit. Therefore, in the monitoring circuit 180, the monitoring conditions can be changed by changing the connection information, which is software stored in the non-volatile memory 182, and the hardware based on the connection information.

The FPGA 181 may have a fixed logic circuit other than the reprogrammable logic circuit. For example, the FPGA 181 has a processor 181p and a memory as logic circuits. The processor 181p executes processing while sequentially reading the programs stored in the memory 185, for example. This allows for more advanced processing. The memory 185 read by the processor 181p is non-volatile. However, unlike the non-volatile memory 182 for the FPGA 181, the memory 185 stores not the connection information but the programs that are sequentially read by the processor. By dividing the memory according to the application, the reliability of the logic circuit composed of the FPGA 181 is improved.

The communication IF183 is an interface for the FPGA 181 and the automatic control circuit 170 to communicate with the operation unit 12. The communication IF 183 provides, for example, a physical interface for communicating with the operating unit 12. The physical interface is, for example, CAN. The automatic control circuit 170 outputs an operation control signal via the communication IF 183.

The relay 184 cuts off the power supply of the power supply unit 14 (see FIG. 2) to the operating unit 12. More specifically, the relay 184 transmits a supply signal for supplying electric power to the power supply unit 14 by energizing under the control of the FPGA 181. When the energization of the relay 184 is stopped by the control of the FPGA 181, the transmission of the supply signal is stopped. As a result, the power supply from the power supply unit 14 is cut off. By shutting off the power supply, the operation can be reliably stopped.
The supply signal from the relay 184 can pass through a relay (not shown) provided in each part of the automatic operation device 1 and the cooperative operation device 2 outside the monitoring circuit 180. As a result, the power supply is immediately cut off by some cutoff control. Therefore, the operation can be reliably stopped.

The logic circuit composed of FPGA181 of the monitoring circuit 180 detects an abnormality in the operation of the automatic control circuit 170 by rule-based logic.

For example, the monitoring circuit 180 controls the automatic control circuit 170 so that the image of the camera is forcibly displayed on the remote control device 3 when an abnormality is detected. As a result, the operator can immediately perform the corresponding maneuver.

Further, the output by the monitoring circuit 180 is not limited to the above combination. For example, when the monitoring circuit 180 detects an abnormality of at least one parameter related to at least one type of signal, the monitoring circuit 180 also adopts a configuration in which the output of the operation control signal by the automatic control circuit 170 is prohibited instead of outputting the operation command. It is possible. In this case, the monitoring circuit 180 stops the operation of the communication IF 183 that outputs the signal from the automatic control circuit 170. As a result, the situation in which an abnormal operation control signal is continuously output is suppressed.

The basic hardware structure of the control unit 10 described above is also applied to the cooperative control unit 20. However, when the output content based on the abnormality detection result of the cooperative sensing unit 21 is different from that of the control unit 10, a part of the hardware and the software are different from the control unit 10 according to the difference.

[Linked operation with linked control unit]
The control unit 10 having the above-described configuration operates in cooperation with the cooperative control unit 20.
As the cooperation control unit 20, the control unit 10 cooperates with the cooperation control unit 20 regardless of whether the device whose input / output of the operation command changes according to the situation or the device whose input / output is fixed is connected. Works.
Subsequently, the details of the cooperation operation with the cooperation control unit 20 will be described.

FIG. 4 is a flowchart illustrating a cooperative operation among the operations of the control unit shown in FIG.
The operation control device 160 executes the determination of the type of the cooperation control unit 20 and the cooperation operation according to the type.
The type determination and the cooperative operation according to the type are mainly executed by the automatic control circuit 170 shown in FIG. However, for example, a configuration in which the type of the cooperative control unit 20 is determined by the processor 181p provided in the monitoring circuit 180 can also be adopted. Further, the configuration instructed by the remote control device 3 can also be adopted for the type determination and the cooperative operation according to the type.
Here, the cooperative operation will be described as the operation of the operation control device 160.

In the linked operation, the motion control device 160 first determines whether or not it is connected to the linked control unit 20 (S11). For example, the operation control device 160 determines whether or not it is possible to communicate with the cooperation control unit 20 via the external communication connection unit 140.
When the cooperative control unit 20 is connected, the cooperative control unit 20 can communicate with the operation control device 160 of the control unit 10. In this case, the operation control device 160 determines that it is connected to the cooperation control unit 20.

Next, the operation control device 160 determines whether or not an operation command has been received from the cooperative control unit 20 (S13).

When the operation command is not received from the linked operation device 2 (No in S13), the operation control device 160 recognizes the content of the external environment data by processing the external environment data (S14).
More specifically, for example, when the automatic operation device 1 is an automatic traveling vehicle, the operation control device 160 performs a process for recognizing the content of image data representing an external image based on the constructed control model.

The motion control device 160 determines the motion based on the recognition result of the content of the external environment data (S15).
More specifically, for example, the motion control device 160 grasps the current position of the automatic motion device 1 and determines the optimum travel route based on the recognition of the content of the image data.

The operation control device 160 generates an operation control signal based on the processing result of the external environment data (S16). The operation control device 160 outputs the generated operation control signal to the actuator 121 of the operation unit 12 via the operation control connector 130.
More specifically, for example, the motion control device 160 generates an motion control signal including a travel and steering command based on the determined travel path, and outputs the motion control signal to the motion unit 12. The operation unit 12 operates the actuator 121. As a result, the automatic operation device 1 operates based on the external environment.

Further, the operation control device 160 transmits an operation command for controlling the linked operation device 2 based on the processing result of the external environment data (S17).
More specifically, for example, the operation control device 160 generates an operation command for the cooperative control unit 20 to perform an operation according to the position of the automatic operation device 1 on the traveling path. The operation control device 160 transmits an operation command to the cooperation control unit 20 via the external communication connection unit 140.

When an operation command is received from the linked operation device 2 (Yes in S13), the operation control device 160 recognizes the content of the external environment data by processing the external environment data (S21). Further, the motion control device 160 determines the motion based on the recognition result of the content of the external environment data (S22). These operations are the same as in steps S14 and S15 described above.

Next, the operation control device 160 performs processing based on the operation command from the cooperation control unit 20 (S23). The cooperative control unit 20 transmits an operation command to the control unit 10 via the external communication connection unit 140. The operation control device 160 performs processing based on the operation command.
More specifically, for example, the cooperative control unit 20 transmits an operation command indicating whether the automatic operation device 1 moves forward or backward according to the position of the work target to the operation control device 160 of the control unit 10.

Next, the operation control device 160 transmits an operation control signal for controlling the cooperative operation device 2 based on the operation command received via the external communication connection unit 140 (S24). The motion control device 160 generates an motion control signal based on the external environment data recognized in step S21 and the motion command.
More specifically, for example, the motion control device 160 generates an motion control signal so as to move forward or backward along the determined travel path, and outputs the motion control signal to the motion unit 12. The operation unit 12 operates the actuator 121. As a result, the automatic operation device 1 operates based on the operation command of the cooperation control unit 20.
A process of outputting an operation control signal to the actuator 121 based on an operation command received from the cooperative control unit 20 and a process of transmitting an operation command to the cooperative control unit 20 via the external communication connection unit 140 in sequence. It is possible to execute.

[First application example of cooperation]
FIG. 5 is a block diagram showing a first application example of the control unit shown in FIG.
An application example shown in FIG. 5 is a case where the automatic operation device 1 is an automatic traveling vehicle and the cooperation control unit 20 outputs an operation command with high frequency. The cooperative operation device 2 is an autonomous operation robot mounted on an autonomous vehicle.
For example, the cooperation control unit 20 recognizes the position of the work target of the robot based on the image of the cooperation operation device camera (robot camera) as the cooperation sensing unit 21. The cooperative control unit 20 transmits an operation command including forward / backward movement to the operation control device 160 of the control unit 10 according to the position of the work target. The operation control device 160 generates an operation control signal from the cooperation control unit 20 based on the operation command received. The motion control device 160 outputs the generated motion control signal to the actuator 121 of the motion unit 12 via the motion control connector 130. The automatic operation device 1 and the cooperative operation device 2 can cooperate with each other to perform precise work.

[Second application example of cooperation]
FIG. 6 is a block diagram showing a second application example of the control unit shown in FIG.
An application example shown in FIG. 6 is a case where the automatic operation device 1 is an automatic traveling vehicle and the cooperation control unit 20 outputs an operation command at a low frequency or does not output. The cooperative operation device 2'is a simple work device mounted on the automatic operation device 1. The cooperative operation device 2'is, for example, a sprayer that sprays a chemical or the like toward a work target.
The operation control device 160 generates an operation command for controlling the cooperative operation device 2'based on the processing result of the image data of the external photographing camera as the external environment sensing unit 11. The motion control device 160 generates, for example, an motion command for causing the cooperative motion device 2 to start or stop a work motion based on the position of the autonomous driving vehicle acquired as a result of processing the image data of the external camera. , Transmit to the cooperation control unit 20.
As a result, the cooperative operation device 2 can operate appropriately according to the traveling of the automatic traveling vehicle as the automatic operation device 1.
As described with reference to the example of FIG. 5 and the example of FIG. 6, the external communication connector 140 of the control unit 10 is used as the linked control unit 20, and the input / output of the operation command changes depending on the situation. Regardless of which device has a fixed input / output, it is possible to perform precise work in cooperation with the cooperation control unit 20. Therefore, the versatility of the control unit 10 can be improved.

[Image transmission during remote control]
For example, when there is an obstacle on the traveling path and it is difficult to work only by automatic operation, or in the case of operation test or confirmation, the automatic operation system S operates in response to the operation of the remote control device 3 by the operator H. ..

In this case, the automatic operation system S operates in response to the operation of the remote control device 3 by the operator H. The remote control device 3 shows both or one of an image based on the image data of the external photographing camera 11 and an image based on the image data of the cooperative operation device camera 21.

FIG. 7 is a block diagram showing an image flow in the automatic operation system shown in FIG.
The solid arrow in FIG. 7 shows the flow of the image when the automatic operation system S is remotely controlled. The broken line arrow in FIG. 7 indicates the flow of the image request signal from the remote control device 3. In FIG. 7, an external photographing camera is shown as an example of the external environment sensing unit 11. Further, a linked operation device camera is shown as the linked sensing unit 21. Hereinafter, the external environment sensing unit 11 will also be referred to as an external photographing camera 11. Further, the linked sensing unit 21 is also referred to as a linked operating device camera 21.
It is also possible to adopt a configuration in which the flow of the image request signal from the remote control device 3 is directed to the cooperative control unit 20 as shown by the alternate long and short dash line in FIG. 7. In such a configuration, for example, at a certain timing, one of the flow of the image request signal toward the cooperative control unit 20 (dashed line) and the flow of the image request signal toward the control unit 10 (broken line) is enabled. It can be adopted. As a result, for example, in the remote control device 3, the image of the external photographing camera 11 and the image of the cooperative operation device camera 21 are switched.
Further, in addition to the configuration in which one of the above is enabled, at a certain timing, the flow of the image request signal toward the cooperative control unit 20 (dashed line) and the flow of the image request signal toward the control unit 10 (broken line). A configuration that enables both of the above can also be adopted. For example, in the remote control device 3, the image of the external photographing camera 11 and the image of the cooperative operation device camera 21 are displayed at the same time.
Furthermore, in a configuration in which both of the above are valid, for example, the content of the image command for each of the cooperative control unit 20 and the control unit 10 is determined according to the state of wireless communication between the remote communication device 13 and the remote control device 3. A configuration that can be adjusted can also be adopted.

The automatic operation device 1 is equipped with an external photographing camera 11 and an actuator 121 for photographing the external environment. Further, the cooperative operation device 2 is provided with a cooperative operation device camera 21 for photographing the external environment.
The control unit 10 of the automatic operation device 1 is connected to one remote communication device 13.
The control unit 10 of the automatic operation device 1 and the cooperation control unit 20 of the cooperation operation device 2 are connected to one remote communication device 13. That is, the control unit 10 and the cooperative control unit 20 jointly use one remote communication device 13.

As described above, the control unit 10 includes an external image connector 110, an operation control connector 130, and an operation control device 160 (see FIG. 3).
The external image connector 110 (external environment information connector 110) is a connector for inputting external image data indicating a shooting result from the external shooting camera 11 to the control unit 10.
The operation control connector 130 is a connector for the control unit 10 to output an operation control signal for controlling the operation of the actuator 121.
The external communication connection unit 140 functions as a remote data connector for inputting / outputting data to / from one remote communication device 13 communicably connected to the remote control device 3. The external communication connection unit 140 also functions as a connector for inputting / outputting data to / from the cooperation control unit 20.

The operation control device 160 (see FIG. 3) of the control unit 10 is also responsible for image control including image command signal acquisition.
The operation control device 160 (see FIG. 3) of the control unit 10 transfers monitor image data based on the external image data input via the external image connector 110 to one remote from the external communication connection unit 140 that functions as a remote data connector. It is output to the remote control device 3 via the communication device 13. The operation control device 160 outputs monitor image data based on an image command signal from the outside of the control unit 10.
The cooperative control unit 20 also executes the same processing as the control unit 10 for the image data of the linked operation device camera 21. Here, the operation of the control unit 10 in the operation control device 160 will be described as a representative.

[Control of image flow]
FIG. 8 is a flowchart illustrating control of an image in the operation control device 160 of the control unit 10 shown in FIG. 7.

The operation control device 160 determines whether or not an image command signal has been received from the outside (S31). The image command signal is transmitted from the remote communication device 13 via the external communication connection unit 140. However, the image command signal may be transmitted from the cooperation control unit 20 via the external communication connector 140. The operation control device 160 discriminates about the image command signal transmitted through both connectors.

When the image command signal is received (Yes in S31), the operation control device 160 determines whether the content of the image command signal indicates the start of image transmission (S32).
When the image transmission is started (Yes in S32), the motion control device 160 transmits the monitor image data to the remote control device 3 (S33). That is, the motion control device 160 outputs the monitor image data to the remote control device 3. The monitor image data is data obtained by processing the external image data input via the external image connector 110. The operation control device 160 generates monitor image data having a smaller amount of data than the external image data by performing image compression processing on the external image data. However, external image data that has not been substantially processed may be used as the monitor image data. The operation control device 160 outputs monitor image data from the external communication connection unit 140 to one remote communication device 13. The monitor image data is transmitted to the remote control device 3 via the remote communication device 13.

The operation control device 160 determines whether the content of the image command signal indicates that the image transmission is stopped (S34).
When the image transmission is stopped (Yes in S34), the operation control device 160 stops the transmission of the monitor image data (S35). That is, the operation control device 160 stops the output of the monitor image data. As a result, the transmission of the monitor image data to the remote control device 3 is stopped.

The image command signal indicating that the image transmission is stopped may be output from the cooperation control unit 20. For example, the cooperative control unit 20 that has received a command to transmit only the image of the cooperative operation device camera 21 from the remote control device 3 sends an image command signal indicating that the image transmission is stopped to the control unit 10. By stopping the transmission of data based on the image data of the external photographing camera 11, the remote control device 3 can display the image that the operator H wants to pay attention to during maneuvering while reducing the amount of data to be transmitted.

The operation control device 160 determines whether the content of the image command signal indicates frame thinning (S36).
In the case of frame thinning (Yes in S36), the operation control device 160 executes the frame thinning process on the monitor image data (S37). As a result, the amount of monitor image data transmitted to the remote control device 3 is reduced.

The operation control device 160 determines whether the content of the image command signal indicates the image compression rate (S38).
In the case of the image compression rate (Yes in S38), the operation control device 160 specifies the compression rate of the image compression process (S39). As a result, the amount of monitor image data transmitted to the remote control device 3 is reduced.

The operation control device 160 determines whether the content of the image command signal indicates a cutout of a part of the image (S41).
In the case of area clipping (Yes in S41), the operation control device 160 executes the area clipping process on the monitor image data (S42). That is, the motion control device 160 extracts an image of a part of the area designated by the image command from the images captured by the external photographing camera 11 to generate monitor image data. More specifically, for example, when the range captured by the external photographing camera 11 is wide, only an image of a part of the traveling direction necessary for maneuvering is transmitted / displayed. As a result, the amount of monitor image data transmitted to the remote control device 3 is reduced.
To cut out a part of the image, for example, a plurality of external photographing cameras 11 are connected to the control unit 10, and the motion control device 160 processes the images of the plurality of areas photographed by the plurality of external photographing cameras 11. It also applies if you have one. In the case of cropping a region (Yes in S41), the motion control device 160 uses only the image of the designated region as monitor image data. Also in this case, the amount of monitor image data transmitted to the remote control device 3 is reduced.

If it is determined in step S31 that the image command signal has not been received (No in S31), the operation control device 160 stops transmitting the monitor image data (S45). That is, the operation control device 160 stops the output of the monitor image data.
Next, the operation control device 160 determines whether or not an abnormal state has occurred in the control unit 10 (S46). The abnormal state of the control unit 10 is detected by, for example, a logic circuit composed of FPGA 181 of the monitoring circuit 180.
The operation control device 160 determines that an abnormal state has occurred, for example, when any of the parameters to be monitored is not within the range defined by the rule.
When an abnormal state has occurred (Yes in S46), the motion control device 160 transmits monitor image data to the remote control device 3 (S47). That is, the motion control device 160 outputs the monitor image data to the remote control device 3. The monitor image data is transmitted to the remote control device 3 via the remote communication device 13.
As a result, the operator H can recognize the occurrence of the abnormal state at an early stage by the display screen of the remote control device 3. In addition, the operator H can recognize the cause of the abnormal state at an early stage from the display screen.

[Example of separation operation of linked operation device]
So far, an example in which the cooperative operation device 2 is connected to the automatic operation device 1 has been described. However, the control unit 10 of the present embodiment can be configured to correspond to the cooperative operation device 4 which is not connected to the automatic operation device 1 and operates separately.
FIG. 9 is a block diagram showing a third application example of the control unit shown in FIG.
In the application example shown in FIG. 9, the cooperative operation device 4 of the automatic operation system S is not connected to the automatic operation device 1. The cooperative operation device 4 operates away from the automatic operation device 1. The cooperative operation device 4 is, for example, an automatic traveling vehicle having substantially the same configuration as the automatic operation device 1.
The external communication connection unit 140 included in the control unit 10 of the automatic operation device 1 includes an external communication connector 141 and a wireless communication unit. Further, the cooperative operation device 4 includes an external communication connection unit 440 that performs wireless communication with the external communication connection unit 140. The control unit 10 of the automatic operation device 1 communicates with the cooperation control unit 40 of the cooperation operation device 4 via wireless communication. The cooperation control unit 40 controls the cooperation operation unit 42.

The control unit 10 of the automatic operation device 1 outputs an operation control signal based on an operation command from the cooperative control unit 40. For example, the control unit 10 travels in response to an operation command from the cooperation control unit 40. In this case, for example, the control unit 10 photographs the outside of the automatic operation device 1 with the external photography camera 11, determines the traveling route by itself based on the image data of the external photography camera 11, and controls the cooperation of the cooperation operation device 4. The start or stop of running of the automatic operation device 1 is determined according to the operation command received from the unit 40. In this way, the automatic operation device 1 can also travel in cooperation with the cooperative operation device 4.

Further, the control unit 10 generates an operation command for controlling the linked operation device 4 based on the image data of the external photographing camera 11. The control unit 10 transmits an operation command to the cooperative control unit 40. For example, the cooperative operation device 4 photographs the outside of the cooperative operation device 4 with the cooperative operation device camera 41, determines the traveling route by itself based on the image data of the cooperative operation device camera 41, and controls the automatic operation device 1. The start or stop of running of the cooperative operation device 4 is determined according to the operation command received from 10. In this way, the cooperative operation device 4 operates in cooperation with the automatic operation device 1.

[Third Embodiment]
In the above-described embodiment, for example, with reference to FIG. 2, an example in which the control unit 10 of the automatic operation device 1 and the cooperation control unit 20 of the cooperation operation device 2 are directly connected to one remote communication device 13 will be described. did.
FIG. 10 is a block diagram showing a configuration of an automatic operation system including a control unit according to a third embodiment of the present invention.
The automatic operation device 1 in the present embodiment is different from the second embodiment in that it has a hub 13a. Further, the automatic operation device 1 in the present embodiment is connected to the cooperation control unit 20 via the hub 13a instead of directly.
Since other points in this embodiment are the same as those in the second embodiment, each part is designated by the same reference numerals as those in the second embodiment.
The control unit 10 and the cooperation control unit 20 in FIG. 10 are connected to one remote communication device 13 via the hub 13a. The hub 13a relays data between the control unit 10, the cooperative control unit 20, and the remote communication device 13. The control unit 10 and the cooperative control unit 20 connected to the hub 13a transmit data in a common transmission format.
Further, the automatic operation device 1 in FIG. 10 communicates with the cooperation control unit 20 via the hub 13a.
The hub 13a in the present embodiment has an independent data mixing function in the remote communication device 13 of the first embodiment. That is, the hub 13a has a part of the functions of the remote communication device 13. Therefore, even in this embodiment, it can be said that the control unit 10 of the automatic operation device 1 and the cooperation control unit 20 of the cooperation operation device 2 are connected to one remote communication device 13.

In the second to third embodiments described above, in order to make the explanation easier to understand, an example of a device traveling as an automatic operation device 1 is given. However, the embodiment is not limited to this, and the automatic operation device 1 may be, for example, a robot that does not travel.

1

Automatic operation device

2, 2', 4 Cooperative operation device 3 Remote control device 10 Control unit 11 External environment sensing unit (external camera)
12 Operation unit 13 Remote communication device 14

Power supply unit

20, 40

Coordination control unit

21, 41 Coordination sensing unit (cooperation operation device camera)
22, 42 Cooperative operation unit 40 Cooperative control unit 110 External environment information connector (external image connector)
121 Actuator 130 Operation control connector 140,440 External communication connection (external communication connector)
160 Operation control device 170 Automatic control circuit 171 GPU
171a Control model 172 Non-volatile memory 180 Monitoring circuit 181 FPGA
182 Non-volatile memory 184 Relay 185 Memory 221 Actuator S Automatic operation system

Claims

A control unit used in an automatically operating device equipped with an external environment sensing unit that detects an external environment and an actuator that is controlled based on the external environment.
The control unit is
An external environment information connector for inputting external environment data indicating the detection result from the external environment sensing unit, and
An operation control connector for outputting an operation control signal for controlling the operation of the actuator, and an operation control connector.
It is configured to be communicably connected to the cooperative control unit that controls the cooperative operation device that outputs physical or non-physical output by wire or wirelessly, and controls the automatic operation device transmitted from the cooperative control unit. An external communication connection unit that outputs an operation command for controlling the control unit and transmits an operation command for controlling the linked operation device output from the control unit to the linked control unit.
When an operation command is received from the cooperative operation device, the operation control signal is generated based on the operation command received from the cooperative control unit, and the operation control signal is output to the actuator via the operation control connector. Then, when the cooperative operation device is operated based on the processing result of the external environment data, the operation control signal is generated based on the processing result of the external environment data and the cooperative operation device is controlled. It is provided with an operation control device that generates an operation command and transmits an operation command for controlling the cooperative operation device to the cooperative control unit via the external communication connection unit.
The control unit according to claim 1.
The external environment sensing unit is a camera.
The external environment information connector inputs image data output from the camera as the external environment data.
The control unit according to claim 1 or 2.
The actuator mounted on the automatic operation device is a traveling device for traveling the automatic operation device.
The motion control device is
When an operation command is received from the cooperation control unit, operation control for instructing the running of the automatic operation device based on the operation command received from the cooperation control unit and the processing result of the external environment data. Generate a signal,
When operating the cooperative operation device based on the processing result of the external environment data, an operation control signal for instructing the running of the automatic operation device is generated based on the processing result of the external environment data. , The operation command for controlling the cooperative operation device is generated, and the operation command for controlling the cooperative operation device is transmitted to the cooperation control unit via the external communication connection unit.
The control unit according to any one of claims 1 to 3.
The operation control device generates the operation control signal based on the operation command received from the cooperation control unit and outputs the operation control signal to the actuator, and connects the operation command to the cooperation control unit by the external communication. The process of transmitting via the unit is executed in sequence.
The control unit according to any one of claims 1 to 4.
The external communication connection unit includes a plurality of connectors corresponding to a plurality of different types of transmission formats.