WO2023190877A1

WO2023190877A1 - Assistance device, work machine, program

Info

Publication number: WO2023190877A1
Application number: PCT/JP2023/013195
Authority: WO
Inventors: 竜次續木; 大稀安達; 孝介原
Original assignee: 住友重機械工業株式会社
Priority date: 2022-03-31
Filing date: 2023-03-30
Publication date: 2023-10-05

Abstract

Provided is a technology that makes it possible to more appropriately operate a work machine. According to one embodiment of the present disclosure, a controller 30 comprises a work target shape acquisition unit 302B that acquires data about the shape of a work target in the surroundings of a shovel 100, an inference unit 302C that infers operations that are suited to the shape of the work target in the surroundings of the shovel 100 from among a plurality of candidate operations for the shovel 100 for prescribed work on the basis of the data about the shape of the work target in the surroundings of the shovel 100 using a trained model LM that has been trained by machine learning using training data about operations of the shovel 100 under operation by a relatively highly proficient operator as associated with the shapes of work targets, and a suggestion unit 302D that suggests one or more operations from among the plurality of candidate operations to a user on the basis of the inference results from the inference unit 302C.

Description

Support equipment, working machines, programs

The present disclosure relates to a support device for a working machine, etc.

Working machines such as shovels are known (for example, see Patent Document 1).

Japanese Patent Application Publication No. 2020-029769

By the way, when performing a certain work using a work machine, it is necessary to use a plurality of motions depending on the state of the work object, such as the topographical shape. For example, in the case of soil leveling work with an excavator, the action of sweeping earth and sand forward with the back of the bucket, the horizontal pulling action, the compaction action, etc. are used. Therefore, for example, in the case of an inexperienced operator, it may be difficult to select an appropriate motion from among a plurality of candidate motions, which may result in a decrease in work efficiency.

Therefore, in view of the above problems, it is an object of the present invention to provide a technology that allows working machines to operate more appropriately.

To achieve the above object, in one embodiment of the present disclosure,
an acquisition unit that acquires data regarding the shape of a work target around the work machine;
a proposal unit that proposes to the user an operation among a plurality of candidate operations of the working machine in a predetermined work based on the data acquired by the acquisition unit;
Assistive equipment is provided.

Additionally, in other embodiments of the present disclosure,
an acquisition unit that acquires data regarding the shape of a work target around the work machine;
a proposal unit that proposes to the user an operation among a plurality of candidate operations of the working machine in a predetermined work based on the data acquired by the acquisition unit;
Working machinery is provided.

In still other embodiments of the present disclosure,
support equipment,
an acquisition step of acquiring data regarding the shape of the work object around the work machine;
a proposing step of proposing to a user an action among a plurality of candidate actions of the working machine in a predetermined work based on the data obtained in the obtaining step;
program will be provided.

According to the embodiment described above, the work machine can be operated more appropriately.

FIG. 1 is a diagram showing an example of a shovel operation support system. It is a top view showing an example of a shovel. FIG. 2 is a diagram showing an example of a configuration related to remote control of an excavator. FIG. 2 is a block diagram showing an example of the hardware configuration of an excavator. FIG. 1 is a diagram illustrating an example of a hardware configuration of an information processing device. FIG. 2 is a functional block diagram showing a first example of a functional configuration regarding a motion proposal function of the excavator operation support system. 2 is a flowchart schematically showing a first example of processing related to a motion suggestion function of the shovel. FIG. 3 is a functional block diagram showing a second example of a functional configuration regarding a motion proposal function of the excavator operation support system. 12 is a flowchart schematically showing a second example of processing related to a motion suggestion function of the shovel. FIG. 3 is a diagram illustrating a first example of display content on a display device regarding a shovel motion suggestion function. FIG. 7 is a diagram illustrating a second example of display content on the display device regarding the shovel motion suggestion function. It is a figure which shows the 3rd example of the display content of a display device regarding the motion suggestion function of an excavator. It is a figure which shows the 3rd example of the display content of a display device regarding the motion suggestion function of an excavator. It is a figure which shows the 4th example of the display content of a display device regarding the motion suggestion function of an excavator. It is a figure which shows the 5th example of the display content of a display device regarding the motion suggestion function of an excavator. FIG. 2 is a functional block diagram illustrating an example of a functional configuration related to generation of a target trajectory of a working part of an excavator. FIG. 3 is a diagram illustrating an example of a screen related to generation of a target trajectory of a working part of an excavator. FIG. 7 is a diagram illustrating another example of a screen related to generation of a target trajectory of a working part of an excavator. FIG. 7 is a diagram illustrating still another example of a screen related to generation of a target trajectory of a working part of an excavator. 2 is a flowchart schematically showing an example of processing related to generation of a target trajectory of a working part of an excavator.

Hereinafter, embodiments will be described with reference to the drawings.

[Overview of operation support system]
First, an overview of the operation support system SYS according to this embodiment will be explained with reference to FIGS. 1 to 3.

FIG. 1 is a diagram showing an example of the operation support system SYS. In FIG. 1, the excavator 100 is shown in a left side view. FIG. 2 is a top view showing an example of the shovel 100. FIG. 3 is a diagram showing an example of a configuration related to remote control of an excavator. Hereinafter, the direction on the shovel 100 or the direction seen from the shovel 100 may be described by defining the direction in which the attachment AT extends (upward direction in FIG. 2) as seen from the top of the shovel 100 as "front".

As shown in FIG. 1, the operation support system SYS includes an excavator 100 and an information processing device 200.

The operation support system SYS uses the information processing device 200 to cooperate with the excavator 100 and provides support regarding the operation of the excavator 100.

The number of excavators 100 included in the operation support system SYS may be one or multiple.

The excavator 100 is a work machine to which operation support is provided in the operation support system SYS.

As shown in FIGS. 1 and 2, the excavator 100 includes a lower traveling body 1, an upper rotating body 3, an attachment AT including a boom 4, an arm 5, and a bucket 6, and a cabin 10.

The lower traveling body 1 causes the excavator 100 to travel using the crawler 1C. The crawler 1C includes a left crawler 1CL and a right crawler 1CR. The crawler 1CL is hydraulically driven by a travel hydraulic motor 1ML. Similarly, the crawler 1CL is hydraulically driven by a travel hydraulic motor 1MR. Thereby, the lower traveling body 1 can self-propel.

The upper rotating body 3 is rotatably mounted on the lower traveling body 1 via the rotating mechanism 2. For example, the upper rotating structure 3 turns with respect to the lower traveling structure 1 by hydraulically driving the turning mechanism 2 by the turning hydraulic motor 2M.

The boom 4 is attached to the center of the front part of the upper revolving body 3 so that it can be raised and raised about a rotation axis along the left-right direction. The arm 5 is attached to the tip of the boom 4 so as to be rotatable about a rotation axis extending in the left-right direction. The bucket 6 is attached to the tip of the arm 5 so as to be rotatable about a rotation axis extending in the left-right direction.

The bucket 6 is an example of an end attachment, and is used, for example, in excavation work.

The bucket 6 is attached to the tip of the arm 5 in such a manner that it can be replaced as appropriate depending on the work content of the shovel 100. That is, instead of the bucket 6, a bucket of a different type than the bucket 6, such as a relatively large bucket, a slope bucket, a dredging bucket, etc., may be attached to the tip of the arm 5. Further, an end attachment of a type other than the bucket, such as an agitator, a breaker, a crusher, etc., may be attached to the tip of the arm 5. Furthermore, a preliminary attachment such as a quick coupling or a tiltrotator may be provided between the arm 5 and the end attachment.

The boom 4, arm 5, and bucket 6 are hydraulically driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, respectively.

The cabin 10 is a control room where an operator boards and operates the shovel 100. The cabin 10 is mounted, for example, on the front left side of the upper revolving body 3.

For example, the excavator 100 moves the lower traveling body 1 (that is, the pair of left and right crawlers 1CL, 1CR), the upper revolving body 3, the boom 4, the arm 5, the bucket 6, etc. to operate the driven element of.

Further, instead of being configured to be operable by an operator riding in the cabin 10, or in addition to being configured to be operable by an operator riding in the cabin 10, the shovel 100 may be configured to be remotely controlled from outside the shovel 100. When the excavator 100 is remotely controlled, the interior of the cabin 10 may be unmanned. The following description will proceed on the premise that the operator's operations include at least one of an operator's operation on the operating device 26 by an operator in the cabin 10 and a remote control by an external operator.

For example, as shown in FIG. 3, the remote control includes a mode in which the shovel 100 is operated by an operation input regarding the actuator of the shovel 100 performed by the remote control support device 300.

The remote operation support device 300 is provided, for example, in a management center or the like that manages the work of the excavator 100 from the outside. Further, the remote operation support device 300 may be a portable operation terminal, in which case the operator can remotely control the excavator 100 while directly checking the working status of the excavator 100 from around the excavator 100. can.

For example, the excavator 100 transmits an image representing the surroundings including the front of the excavator 100 (hereinafter referred to as "surrounding image") based on a captured image output by the imaging device 40 (described later) to the remote operation support device through the communication device 60 (described later). 300. Then, the remote operation support device 300 may display the image (surrounding image) received from the excavator 100 on the display device. Further, various information images (information screens) displayed on the output device 50 (display device 50A) inside the cabin 10 of the excavator 100 may be similarly displayed on the display device of the remote operation support device 300. As a result, an operator using the remote operation support device 300 can, for example, remotely operate the shovel 100 while checking the display contents such as an image or information screen showing the surroundings of the shovel 100 displayed on the display device. I can do it. Then, the excavator 100 operates the actuators to operate the lower traveling structure 1, the upper rotating structure 3, and the boom 4 in response to a remote control signal indicating the content of the remote control received from the remote control support device 300 through the communication device 60. , arm 5, and bucket 6 may be driven.

Further, the remote control may include, for example, a mode in which the shovel 100 is operated by external voice input or gesture input to the shovel 100 by a person (for example, a worker) around the shovel 100. Specifically, the excavator 100 receives sounds uttered by surrounding workers, etc. through an audio input device (for example, a microphone), a gesture input device (for example, an imaging device), etc. mounted on the excavator 100. Recognizes gestures etc. performed by Then, the excavator 100 operates the actuator according to the content of the recognized voice or gesture, and moves the lower traveling body 1 (left and right crawlers 1C), the upper rotating body 3, the boom 4, the arm 5, the bucket 6, etc. The driven element may also be driven.

Additionally, the work of the shovel 100 may be remotely monitored. In this case, a remote monitoring support device having the same functions as remote operation support device 300 may be provided. The remote monitoring support device is, for example, the information processing device 200. Thereby, the supervisor who is the user of the remote monitoring support device can monitor the working status of the excavator 100 while checking the peripheral image displayed on the display device of the remote monitoring support device. For example, if the supervisor determines that it is necessary from a safety perspective, the supervisor may intervene in the operator's operation of the excavator 100 and bring it to an emergency stop by inputting a predetermined input using the input device of the remote monitoring support device. be able to.

The information processing device 200 cooperates with the shovel 100 by communicating with the shovel 100, and provides support regarding the operation of the shovel 100.

The information processing device 200 is, for example, a server installed in a management office within the work site of the excavator 100 or a management center that manages the operating status of the excavator 100, etc. located at a location different from the work site of the excavator 100. It is a terminal device for management purposes. The management terminal device may be a stationary terminal device such as a desktop PC (Personal Computer), or a portable terminal device such as a tablet terminal, smartphone, or laptop PC. terminal). In the latter case, workers at the work site, supervisors who supervise work, managers who manage the work site, and the like can carry the portable information processing device 200 and move around the work site. In the latter case, the operator can, for example, bring the portable information processing device 200 into the cabin of the excavator 100. Further, a plurality of information processing apparatuses 200 may be provided depending on the purpose, for example, for remote monitoring, for processing regarding a function of suggesting the operation of the shovel 100 to an operator, which will be described later.

The information processing device 200 acquires data regarding the operating state from the excavator 100, for example. Thereby, the information processing device 200 can grasp the operating state of the shovel 100 and monitor whether there is any abnormality in the shovel 100 or the like. Further, the information processing device 200 can display data regarding the operating state of the excavator 100 for the user to confirm through a display device 208, which will be described later.

Furthermore, the information processing device 200 transmits to the shovel 100, for example, various data such as programs and reference data used in processing by the controller 30, etc. of the shovel 100. Thereby, the excavator 100 can perform various processes related to the operation of the excavator 100 using various data downloaded from the information processing device 200.

Additionally, the information processing device 200 performs processing to support, for example, a function related to proposing a motion of the shovel 100 to an operator (hereinafter referred to as "motion proposal function"), which will be described later (see FIG. 6). Details will be described later.

[Hardware configuration of operation support system]
Next, the hardware configuration of the operation support system SYS will be described with reference to FIGS. 4 and 5 in addition to FIGS. 1 to 3.

<Excavator hardware configuration>
FIG. 4 is a block diagram showing an example of the hardware configuration of shovel 100.

In Figure 4, the path through which mechanical power is transmitted is a double line, the path through which high-pressure hydraulic oil that drives the hydraulic actuator flows is a solid line, the path through which pilot pressure is transmitted is a broken line, and the path through which electrical signals are transmitted is shown. Each route is indicated by a dotted line.

The excavator 100 includes a hydraulic drive system for hydraulically driving the driven elements, an operation system for operating the driven elements, a user interface system for exchanging information with the user, a communication system for communicating with the outside, a control system for various controls, etc. Contains each component of.

≪Hydraulic drive system≫
As shown in FIG. 4, the hydraulic drive system of the excavator 100 includes hydraulic pressure for hydraulically driving each of the driven elements such as the lower traveling body 1 (left and right crawlers 1C), the upper rotating body 3, and the attachment AT, as described above. Includes actuator HA. Further, the hydraulic drive system of the excavator 100 according to the present embodiment includes an engine 11, a regulator 13, a main pump 14, and a control valve 17.

The hydraulic actuator HA includes travel hydraulic motors 1ML and 1MR, a swing hydraulic motor 2M, a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, and the like.

Note that in the excavator 100, part or all of the hydraulic actuator HA may be replaced with an electric actuator. In other words, the excavator 100 may be a hybrid excavator or an electric excavator.

The engine 11 is the prime mover of the excavator 100 and is the main power source in the hydraulic drive system. The engine 11 is, for example, a diesel engine that uses light oil as fuel. The engine 11 is mounted, for example, at the rear of the upper revolving structure 3. The engine 11 rotates at a predetermined target rotation speed under direct or indirect control by a controller 30, which will be described later, and drives the main pump 14 and the pilot pump 15.

Note that in place of or in addition to the engine 11, another prime mover (for example, an electric motor) or the like may be mounted on the excavator 100.

The regulator 13 controls (adjusts) the discharge amount of the main pump 14 under the control of the controller 30. For example, the regulator 13 adjusts the angle of the swash plate (hereinafter referred to as "tilt angle") of the main pump 14 in accordance with a control command from the controller 30.

The main pump 14 supplies hydraulic oil to the control valve 17 through a high-pressure hydraulic line. The main pump 14 is, for example, mounted at the rear of the upper revolving structure 3, like the engine 11. The main pump 14 is driven by the engine 11 as described above. The main pump 14 is, for example, a variable displacement hydraulic pump, and as described above, the stroke length of the piston is adjusted by adjusting the tilt angle of the swash plate by the regulator 13 under the control of the controller 30, and the stroke length of the piston is adjusted. The flow rate and discharge pressure are controlled.

The control valve 17 drives the hydraulic actuator HA in accordance with the contents of the operator's operation on the operating device 26 or remote control, or the operation command corresponding to the automatic operation function. The control valve 17 is mounted, for example, in the center of the upper revolving body 3. As described above, the control valve 17 is connected to the main pump 14 via a high-pressure hydraulic line, and controls the hydraulic fluid supplied from the main pump 14 according to an operator's operation or an operation command corresponding to an automatic operation function. , selectively supplying each hydraulic actuator. Specifically, the control valve 17 includes a plurality of control valves (also referred to as "direction switching valves") that control the flow rate and flow direction of the hydraulic oil supplied from the main pump 14 to each of the hydraulic actuators HA.

≪Operation system≫
As shown in FIG. 4, the operating system of the excavator 100 includes a pilot pump 15, an operating device 26, a hydraulic control valve 31, a shuttle valve 32, and a hydraulic control valve 33.

The pilot pump 15 supplies pilot pressure to various hydraulic devices via the pilot line 25. The pilot pump 15 is, for example, mounted at the rear of the upper revolving structure 3, like the engine 11. The pilot pump 15 is, for example, a fixed capacity hydraulic pump, and is driven by the engine 11 as described above.

Note that the pilot pump 15 may be omitted. In this case, the relatively high pressure hydraulic oil discharged from the main pump 14 may be reduced in pressure by a predetermined pressure reducing valve, and then the relatively low pressure hydraulic oil may be supplied as pilot pressure to various hydraulic devices.

The operating device 26 is provided near the cockpit of the cabin 10 and is used by the operator to operate various driven elements. Specifically, the operating device 26 is used for an operator to operate the hydraulic actuator HA that drives each driven element, and as a result, the operator operates the driven element to be driven by the hydraulic actuator HA. can be realized. The operating device 26 includes a pedal device and a lever device for operating each driven element (hydraulic actuator HA).

For example, as shown in FIG. 4, the operating device 26 is of a hydraulic pilot type. Specifically, the operating device 26 utilizes hydraulic oil supplied from the pilot pump 15 through the pilot line 25 and a pilot line 25A branching from the pilot line 25, and applies pilot pressure according to the operation content to the pilot line 27A on the secondary side. Output to. Pilot line 27A is connected to one inlet port of shuttle valve 32 and connected to control valve 17 via pilot line 27, which is connected to an outlet port of shuttle valve 32. Thereby, a pilot pressure can be input to the control valve 17 via the shuttle valve 32 in accordance with the operation contents regarding various driven elements (hydraulic actuator HA) in the operating device 26. Therefore, the control valve 17 can drive each hydraulic actuator HA according to the operation performed on the operating device 26 by an operator or the like.

Additionally, the operating device 26 may be electrical. In this case, the pilot line 27A, shuttle valve 32, and hydraulic control valve 33 are omitted. Specifically, the operating device 26 outputs an electrical signal (hereinafter referred to as an "operating signal") according to the content of the operation, and the operating signal is taken into the controller 30. Then, the controller 30 outputs a control command according to the content of the operation signal, that is, a control signal according to the content of the operation on the operating device 26 to the hydraulic control valve 31. As a result, pilot pressure corresponding to the operation details of the operating device 26 is inputted from the hydraulic control valve 31 to the control valve 17, and the control valve 17 drives each hydraulic actuator HA according to the operation details of the operating device 26. be able to.

Furthermore, the control valves (directional switching valves) built into the control valve 17 and driving the respective hydraulic actuators HA may be of an electromagnetic solenoid type. In this case, the operation signal output from the operation device 26 may be directly input to the control valve 17, that is, to an electromagnetic solenoid type control valve.

Furthermore, as described above, part or all of the hydraulic actuator HA may be replaced with an electric actuator. In this case, the controller 30 may output a control command according to the operation content of the operating device 26 or the remote control content specified by the remote control signal to the electric actuator or a driver driving the electric actuator. Moreover, when the shovel 100 is remotely controlled, the operating device 26 may be omitted.

The hydraulic control valve 31 is provided for each driven element (hydraulic actuator HA) to be operated by the operating device 26 and for each drive direction of the driven element (hydraulic actuator HA) (for example, the raising direction and lowering direction of the boom 4). . That is, two hydraulic control valves 31 are provided for each double-acting hydraulic actuator HA. The hydraulic control valve 31 is provided, for example, in the pilot line 25B between the pilot pump 15 and the control valve 17, and is configured to be able to change its flow path area (that is, the cross-sectional area through which hydraulic oil can flow). good. Thereby, the hydraulic control valve 31 can output a predetermined pilot pressure to the secondary side pilot line 27B using the hydraulic oil of the pilot pump 15 supplied through the pilot line 25B. Therefore, as shown in FIG. 4, the hydraulic control valve 31 indirectly applies a predetermined pilot pressure according to a control signal from the controller 30 to the control valve through the shuttle valve 32 between the pilot line 27B and the pilot line 27. 17. Therefore, the controller 30 can cause the hydraulic control valve 31 to supply pilot pressure to the control valve 17 according to the operation details of the operating device 26, thereby realizing the operation of the shovel 100 based on the operator's operation. Furthermore, the controller 30 can cause the hydraulic control valve 31 to supply pilot pressure to the control valve 17 according to an operation command corresponding to the automatic operation function, thereby realizing operation of the excavator 100 according to the automatic operation function.

Furthermore, the controller 30 may control the hydraulic control valve 31 to realize remote control of the excavator 100, for example. Specifically, the controller 30 outputs to the hydraulic control valve 31 a control signal corresponding to the content of the remote operation specified by the remote operation signal received from the remote operation support device 300, using the communication device 60. Thereby, the controller 30 can cause the hydraulic control valve 31 to supply pilot pressure corresponding to the content of the remote control to the control valve 17, and realize the operation of the shovel 100 based on the operator's remote control.

The shuttle valve 32 has two inlet ports and one outlet port, and outputs the hydraulic oil having the higher pilot pressure of the pilot pressures input to the two inlet ports to the outlet port. The shuttle valve 32 is provided for each driven element (hydraulic actuator HA) to be operated by the operating device 26 and for each drive direction of the driven element (hydraulic actuator HA). One of the two inlet ports of the shuttle valve 32 is connected to the pilot line 27A on the secondary side of the operating device 26 (specifically, the above-mentioned lever device or pedal device included in the operating device 26), and the other is It is connected to the pilot line 27B on the secondary side of the hydraulic control valve 31. The outlet port of shuttle valve 32 is connected to the pilot port of the corresponding control valve of control valve 17 through pilot line 27 . The corresponding control valve is a control valve that drives a hydraulic actuator that is operated by the above-mentioned lever device or pedal device connected to one inlet port of the shuttle valve 32. Therefore, these shuttle valves 32 each control the higher of the pilot pressure in the pilot line 27A on the secondary side of the operating device 26 and the pilot pressure on the pilot line 27B on the secondary side of the hydraulic control valve 31, respectively. It can act on the pilot port of the control valve. In other words, the controller 30 controls the corresponding control valve by causing the hydraulic control valve 31 to output a pilot pressure higher than the pilot pressure on the secondary side of the operating device 26, regardless of the operator's operation on the operating device 26. be able to. Therefore, the controller 30 can control the operation of the driven elements (the lower traveling body 1, the upper rotating body 3, the attachment AT) and realize a remote control function, regardless of the operation state of the operating device 26 by the operator. .

The hydraulic control valve 33 is provided in the pilot line 27A that connects the operating device 26 and the shuttle valve 32. The hydraulic control valve 33 is configured to be able to change its flow path area, for example. The hydraulic control valve 33 operates according to a control signal input from the controller 30. Thereby, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26 when the operating device 26 is being operated by the operator. Therefore, even when the operating device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator corresponding to the operation of the operating device 26. Further, the controller 30 can reduce the pilot pressure output from the operating device 26 to be lower than the pilot pressure output from the hydraulic control valve 31, for example, even when the operating device 26 is being operated. I can do it. Therefore, by controlling the hydraulic control valve 31 and the hydraulic control valve 33, the controller 30 applies a desired pilot pressure to the pilot port of the control valve in the control valve 17, for example, regardless of the operation details of the operating device 26. It can be made to work reliably. Therefore, by controlling the hydraulic control valve 33 in addition to the hydraulic control valve 31, for example, the controller 30 can more appropriately realize the remote control function and automatic operation function of the excavator 100.

≪User interface system≫
As shown in FIG. 4, the user interface system of excavator 100 includes an operating device 26, an output device 50, and an input device 52.

The output device 50 outputs various information to the user of the excavator 100 (for example, the operator in the cabin 10 or an external remote control operator) and the people around the excavator 100 (for example, a worker or a driver of a work vehicle). Output.

For example, the output device 50 includes a lighting device that outputs various information in a visual manner, a display device 50A (see FIG. 6), and the like. The lighting equipment is, for example, a warning light (indicator lamp) or the like. The display device 50A is, for example, a liquid crystal display or an organic EL (Electroluminescence) display. For example, as shown in FIG. 2, lighting equipment and a display device 50A may be provided inside the cabin 10 and output various information visually to an operator inside the cabin 10. Further, the lighting equipment and the display device 50A may be provided, for example, on the side surface of the revolving upper structure 3, and may output various information visually to workers and the like around the excavator 100.

Furthermore, for example, the output device 50 includes a sound output device 50B (see FIG. 6) that outputs various information in an auditory manner. The sound output device 50B includes, for example, a buzzer, a speaker, and the like. The sound output device 50B is provided, for example, in at least one of the interior and exterior of the cabin 10, and outputs various information in an auditory manner to the operator inside the cabin 10 and the people (workers, etc.) around the excavator 100. It's fine.

Furthermore, for example, the output device 50 may include a device that outputs various information using a tactile method such as vibration of the cockpit.

The input device 52 accepts various inputs from the user of the excavator 100, and signals corresponding to the accepted inputs are taken into the controller 30. The input device 52 is provided inside the cabin 10 , for example, and receives input from an operator inside the cabin 10 . Further, the input device 52 may be provided, for example, on a side surface of the revolving upper structure 3, and may receive input from a worker or the like around the excavator 100.

For example, the input device 52 includes an operation input device that accepts operation input. The operation input device may include a touch panel mounted on the display device, a touch pad installed around the display device, a button switch, a lever, a toggle, a knob switch provided on the operation device 26 (lever device), etc. .

Furthermore, for example, the input device 52 may include a voice input device that accepts voice input from the user. The audio input device includes, for example, a microphone.

Furthermore, for example, the input device 52 may include a gesture input device that accepts gesture input from the user. The gesture input device includes, for example, an imaging device that captures an image of a gesture performed by a user.

Furthermore, for example, the input device 52 may include a biometric input device that receives biometric input from the user. The biometric input includes, for example, input of biometric information such as a user's fingerprint or iris.

≪Communication system≫
As shown in FIG. 4, the communication system of the excavator 100 according to this embodiment includes a communication device 60.

The communication device 60 is connected to an external communication line and communicates with a device provided separately from the excavator 100. Devices provided separately from the excavator 100 may include devices external to the excavator 100 as well as portable terminal devices (portable terminals) brought into the cabin 10 by the user of the excavator 100. The communication device 60 may include, for example, a mobile communication module that complies with standards such as 4G ( ^4th Generation) and 5G ( ^5th Generation). Further, the communication device 60 may include, for example, a satellite communication module. Further, the communication device 60 may include, for example, a WiFi communication module, a Bluetooth (registered trademark) communication module, or the like. Furthermore, the communication device 60 may include a plurality of communication devices depending on the communication lines to be connected.

For example, the communication device 60 communicates with external devices such as the information processing device 200 and the remote operation support device 300 in the work site through a local communication line built at the work site. The local communication line is, for example, a local 5G (so-called local 5G) mobile communication line built at a work site or a local area network (LAN) using WiFi 6.

Further, for example, the communication device 60 communicates with an information processing device 200, a remote operation support device 300, etc. located outside the work site through a wide area communication line that includes the work site, that is, a wide area network (WAN). I do. The wide area network includes, for example, a wide area mobile communication network, a satellite communication network, an Internet network, and the like.

≪Control system≫
As shown in FIG. 4, the control system of excavator 100 includes a controller 30. Further, the control system of the excavator 100 according to the present embodiment includes an operating pressure sensor 29, an imaging device 40, and sensors S1 to S5.

The controller 30 performs various controls regarding the shovel 100.

The functions of the controller 30 may be realized by arbitrary hardware or a combination of arbitrary hardware and software. For example, as shown in FIG. 4, the controller 30 includes an auxiliary storage device 30A, a memory device 30B, a CPU (Central Processing Unit) 30C, and an interface device 30D, which are connected via a bus B1.

The auxiliary storage device 30A is a non-volatile storage means, and stores installed programs as well as necessary files, data, etc. The auxiliary storage device 30A is, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory) or a flash memory.

For example, when there is an instruction to start a program, the memory device 30B loads the program in the auxiliary storage device 30A so that it can be read by the CPU 30C. The memory device 30B is, for example, an SRAM (Static Random Access Memory).

For example, the CPU 30C executes a program loaded into the memory device 30B, and implements various functions of the controller 30 according to instructions of the program.

The interface device 30D functions as a communication interface for connecting to a communication line inside the excavator 100, for example. The interface device 30D may include a plurality of different types of communication interfaces depending on the type of communication line to be connected.

Additionally, the interface device 30D functions as an external interface for reading data from and writing data to the recording medium. The recording medium is, for example, a dedicated tool that is connected to a connector installed inside the cabin 10 with a detachable cable. Further, the recording medium may be a general-purpose recording medium such as an SD memory card or a USB (Universal Serial Bus) memory. Thereby, programs for realizing various functions of the controller 30 can be provided by, for example, a portable recording medium and installed in the auxiliary storage device 30A of the controller 30. Further, the program may be downloaded from another computer outside the excavator 100 through the communication device 60 and installed in the auxiliary storage device 30A.

Note that some of the functions of the controller 30 may be realized by another controller (control device). That is, the functions of the controller 30 may be realized in a distributed manner by a plurality of controllers.

The operating pressure sensor 29 detects the pilot pressure on the secondary side (pilot line 27A) of the hydraulic pilot type operating device 26, that is, the pilot pressure corresponding to the operating state of each driven element (hydraulic actuator) in the operating device 26. To detect. A detection signal of pilot pressure corresponding to the operating state of each driven element (hydraulic actuator HA) in the operating device 26 by the operating pressure sensor 29 is taken into the controller 30.

Note that if the operating device 26 is an electric type, the operating pressure sensor 29 is omitted. This is because the controller 30 can grasp the operating state of each driven element through the operating device 26 based on the operating signal taken in from the operating device 26.

The imaging device 40 acquires images around the excavator 100. The imaging device 40 also generates three-dimensional data (hereinafter simply referred to as "the object's three-dimensional shape") representing the position and external shape of the object around the shovel 100 within the imaging range (angle of view) based on the acquired image and distance-related data described below. "original data") may be obtained (generated). The three-dimensional data of objects around the shovel 100 is, for example, coordinate information data of a point group representing the surface of the object, distance image data, and the like.

For example, as shown in FIG. 2, the imaging device 40 includes a camera 40F that images the front of the upper revolving structure 3, a camera 40B that images the rear of the upper revolving structure 3, and a camera 40L that images the left side of the upper revolving structure 3. , and a camera 40R that images the right side of the upper rotating body 3. Thereby, the imaging device 40 can image the entire circumference of the excavator 100, that is, the range covering the angular direction of 360 degrees, when the excavator 100 is viewed from above. In addition, the operator visually recognizes peripheral images such as captured images of the

cameras

40B, 40L, and 40R and processed images generated based on the captured images through the output device 50 (display device) and the remote control display device, and rotates the upper part. The left, right, and rear sides of the body 3 can be confirmed. In addition, the operator can check the operation of the attachment AT including the bucket 6 by visually checking peripheral images such as images captured by the camera 40F and processed images generated based on the captured images through the remote control display device. , the excavator 100 can be remotely controlled. Hereinafter, the

cameras

40F, 40B, 40L, and 40R may be collectively or individually referred to as "camera 40X."

The camera 40X is, for example, a monocular camera. In addition, the camera 40X acquires data regarding distance (depth) in addition to two-dimensional images, such as a stereo camera, a TOF (Time Of Flight) camera, etc. (hereinafter collectively referred to as a "3D camera"). It may be possible.

Output data (for example, image data, three-dimensional data of objects around the excavator 100, etc.) of the imaging device 40 (camera 40X) is taken into the controller 30 through a one-to-one communication line or an in-vehicle network. Thereby, for example, the controller 30 can monitor objects around the excavator 100 based on the output data of the camera 40X. Further, for example, the controller 30 can determine the surrounding environment of the excavator 100 based on the output data of the camera 40X. Further, for example, the controller 30 can determine the posture state of the attachment AT shown in the captured image based on the output data of the camera 40X (camera 40F). Further, for example, the controller 30 can determine the attitude state of the body of the excavator 100 (the upper revolving body 3) based on the output data of the camera 40X, with reference to objects around the excavator 100.

Note that some of the

cameras

40F, 40B, 40L, and 40R may be omitted. For example, when the excavator 100 is not remotely controlled, the camera 40F and the camera 40L may be omitted. This is because it is relatively easy for the operator in the cabin 10 to check the front and left side of the excavator 100. Further, instead of or in addition to the imaging device 40 (camera 40X), a distance sensor may be provided in the upper revolving body 3. The distance sensor is attached to the upper part of the upper revolving body 3, for example, and acquires data regarding the distance and direction of surrounding objects with respect to the shovel 100 as a reference. Further, the distance sensor may acquire (generate) three-dimensional data (for example, coordinate information data of a point group) of objects around the shovel 100 within the sensing range based on the acquired data. The distance sensor is, for example, LIDAR (Light Detection and Ranging). Further, for example, the distance sensor may be, for example, a millimeter wave radar, an ultrasonic sensor, an infrared sensor, or the like.

The sensor S1 is attached to the boom 4 and detects the attitude angle (hereinafter referred to as "boom angle") around the rotation axis of the base end corresponding to the connection part of the boom 4 with the upper revolving structure 3. The sensor S1 includes, for example, a rotary potentiometer, a rotary encoder, an acceleration sensor, an angular acceleration sensor, a 6-axis sensor, an IMU (Inertial Measurement Unit), and the like. The same may apply to the sensors S2 to S4 below. Further, the sensor S1 may include a cylinder sensor that detects the extended/contracted position of the boom cylinder 7. The same may apply to the sensors S2 and S3 below. A detection signal of the boom angle by the sensor S1 is taken into the controller 30. Thereby, the controller 30 can grasp the attitude state of the boom 4.

The sensor S2 is attached to the arm 5 and detects the posture angle (hereinafter referred to as "arm angle") around the rotation axis of the base end of the arm 5, which corresponds to the connection part with the boom 4. The arm angle detection signal from the sensor S2 is taken into the controller 30. Thereby, the controller 30 can grasp the posture state of the arm 5.

The sensor S3 is attached to the bucket 6 and detects the attitude angle (hereinafter referred to as "arm angle") around the rotation axis of the base end corresponding to the connection part with the arm 5 of the bucket 6. A detection signal of the arm angle by the sensor S3 is taken into the controller 30. Thereby, the controller 30 can grasp the attitude state of the bucket 6.

The sensor S4 detects the inclination state of the aircraft body (for example, the upper rotating body 3) with respect to a predetermined reference plane (for example, a horizontal plane). For example, the sensor S4 is attached to the revolving upper structure 3, and measures the inclination angle of the excavator 100 (i.e., the revolving upper structure 3) about two axes in the front-rear direction and the left-right direction (hereinafter, "front-rear inclination angle" and "lateral inclination angle"). ”) is detected. A detection signal corresponding to the inclination angle (front/rear inclination angle and left/right inclination angle) detected by the sensor S<b>4 is taken into the controller 30 . Thereby, the controller 30 can grasp the tilting state of the aircraft body (upper rotating body 3).

The sensor S5 is attached to the revolving upper structure 3 and outputs detection information regarding the turning state of the revolving upper structure 3. The sensor S5 detects, for example, the turning angular velocity and turning angle of the upper rotating body 3. The sensor S5 includes, for example, a gyro sensor, a resolver, a rotary encoder, and the like. Detection information regarding the turning state detected by the sensor S5 is taken into the controller 30. Thereby, the controller 30 can grasp the turning state such as the turning angle of the upper rotating body 3.

In addition, when the sensor S4 includes a gyro sensor, a 6-axis sensor, an IMU, etc. that can detect angular velocity around three axes, the turning state (for example, turning angular velocity) of the upper rotating structure 3 is detected based on the detection signal of the sensor S4. may be done. In this case, sensor S5 may be omitted. Further, if it is possible to grasp the attitude state of the upper rotating body 3, attachment AT, etc. based on the output of the imaging device 40 or the distance sensor, at least some of the sensors S1 to S5 may be omitted.

<Hardware configuration of information processing device>
FIG. 5 is a block diagram showing an example of the hardware configuration of the information processing device 200.

The functions of the information processing device 200 are realized by arbitrary hardware or a combination of arbitrary hardware and software. For example, as shown in FIG. 5, the information processing device 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU 204, a high-speed arithmetic device 205, a communication interface 206, an input device 207, and and a display device 208.

The external interface 201 functions as an interface for reading data from and writing data to the recording medium 201A. The recording medium 201A includes, for example, a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disc), a BD (Blu-ray (registered trademark) Disc), an SD memory card, a USB memory, and the like. Thereby, the information processing device 200 can read various data used in processing through the recording medium 201A, store it in the auxiliary storage device 202, and install programs that implement various functions.

Note that the information processing device 200 may obtain various data and programs used in processing from an external device through the communication interface 206.

The auxiliary storage device 202 stores various installed programs, as well as files, data, etc. necessary for various processes. The auxiliary storage device 202 includes, for example, an HDD (Hard Disc Drive), an SSD (Solid State Disc), a flash memory, and the like.

The memory device 203 reads and stores the program from the auxiliary storage device 202 when there is an instruction to start the program. The memory device 203 includes, for example, DRAM (Dynamic Random Access Memory) and SRAM.

The CPU 204 executes various programs loaded from the auxiliary storage device 202 to the memory device 203, and implements various functions related to the information processing device 200 according to the programs.

The high-speed arithmetic unit 205 works in conjunction with the CPU 204 and performs arithmetic processing at a relatively high speed. The high-speed calculation device 205 includes, for example, a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), and the like.

Note that the high-speed calculation device 205 may be omitted depending on the required speed of calculation processing.

The communication interface 206 is used as an interface for communicably connecting to an external device. Thereby, the information processing device 200 can communicate with an external device such as the excavator 100, for example, through the communication interface 206. Furthermore, the communication interface 206 may have a plurality of types of communication interfaces depending on the communication method with the connected device.

The input device 207 receives various inputs from the user.

The input device 207 includes, for example, an operation input device that accepts mechanical operation input from the user. The operation input device includes, for example, a button, a toggle, a lever, and the like. Further, the operation input device includes, for example, a touch panel mounted on the display device 208, a touch pad provided separately from the display device 208, and the like.

Furthermore, the input device 207 includes, for example, a voice input device that can accept voice input from a user. The voice input device includes, for example, a microphone that can collect the user's voice.

Furthermore, the input device 207 includes, for example, a gesture input device that can accept gesture input from the user. The gesture input device includes, for example, a camera that can capture images of the user's gestures.

Furthermore, the input device 207 includes, for example, a biometric input device that can accept biometric input from a user. The biometric input device includes, for example, a camera that can acquire image data that includes information about a user's fingerprint or iris.

The display device 208 displays information screens and operation screens for the user. For example, display device 208 includes the above-mentioned remote control display device. The display device 208 is, for example, a liquid crystal display, an organic EL (Electroluminescence) display, or the like.

Note that, like the information processing device 200, the remote operation support device 300 may also be realized by arbitrary hardware or a combination of arbitrary hardware and software, and a similar hardware configuration may be adopted. For example, like the information processing device 200 (FIG. 5), the remote operation support device 300 is mainly configured with a computer including a CPU, a memory device, an auxiliary storage device, an interface device, an input device, and a display device. The memory device is, for example, SRAM or DRAM. The auxiliary storage device is, for example, an HDD, SSD, EEPROM, flash memory, or the like. The interface device includes an external interface for connecting to an external recording medium and a communication interface for communicating with the outside, such as the shovel 100. The input device includes, for example, a lever-type operation input device. As a result, the operator can use the operation input device to perform operation input regarding the actuator of the shovel 100, and the remote operation support device 300 can use the communication interface to transmit a signal corresponding to the operation input to the shovel 100. can. Therefore, the operator can remotely control the excavator 100 using the remote control support device.

[First example of motion suggestion function]
Next, with reference to FIGS. 6 and 7 in addition to FIGS. 1 to 5, a first example of a function for suggesting the operation of the excavator 100 to the user (operator) (operation proposal function) will be described.

<Functional configuration>
FIG. 6 is a functional block diagram showing a first example of a functional configuration related to the operation proposal function of the operation support system SYS.

The excavator 100 includes a support device 150. Support device 150 supports operation of excavator 100 by an operator.

As shown in FIG. 6, the support device 150 includes a controller 30, an imaging device 40, an output device 50, and a communication device 60.

The controller 30 includes an operation log providing unit 301 and a work support unit 302 as functional units.

Note that when there are multiple excavators 100 included in the operation support system SYS, the controller 30 includes only the former of the operation log providing unit 301 and the work support unit 302, and the excavator 100 that includes only the latter. May exist. In this case, the former shovel 100 only has the function of acquiring the operation log of the shovel 100 and providing it to the information processing device 200, which is used for the operator operation support function (motion suggestion function) of the latter shovel 100. The same may apply to the second example (FIG. 8) of the motion suggestion function described below.

The information processing device 200 includes an operation log acquisition unit 2001, an operation log storage unit 2002, a teacher data generation unit 2003, a machine learning unit 2004, a learned model storage unit 2005, and a distribution unit 2006 as functional units. include.

The operation log providing unit 301 is a functional unit that acquires the operation log of the excavator 100, which is the original data for realizing the operation proposal function, and provides it to the information processing device 200. Specifically, a relatively experienced operator (hereinafter referred to as an "expert" for convenience) who has a long history of operating the excavator 100 obtains an operation log when operating the excavator 100 and stores it in the information processing device 200. provide.

The operation log of the shovel 100 includes data regarding the shape of the work target around the shovel 100 and data regarding the operation of the shovel 100 performed on the shape of the work target. The data regarding the shape of the work target around the shovel 100 is, for example, data regarding the topographical shape of the ground at the work site as the work target of the shovel 100. The data regarding the shape of the work target of the shovel 100 is, for example, image data of the imaging device 40 or three-dimensional data of the work target obtained from the image data. The data regarding the operation of the shovel 100 is, for example, data representing the details of the operator's operation. The data representing the contents of the operator's operation may be, for example, the output data of the operating pressure sensor 29 in the case of the hydraulic pilot type operating device 26 or the output data of the operating device 26 (operation signal data) in the case of the electric operating device 26. It is. Furthermore, the data regarding the operation of the shovel 100 may be data representing the operation state of the shovel 100 actually executed in response to an operation by an operator. The data representing the operating state of the shovel 100 is, for example, the output data of the sensors S1 to S5, or the data related to the posture state of the shovel 100 obtained from the output data of the sensors S1 to S5.

The operation log providing section 301 includes an operation log recording section 301A, an operation log storage section 301B, and an operation log transmission section 301C.

The operation log recording unit 301A acquires the operation log of the shovel 100 and records it in the operation log storage unit 301B. For example, every time an operation of the shovel 100 is executed, the operation log recording unit 301A records data regarding the shape of the work target around the shovel 100 at the start of execution of the operation or immediately before execution, and data regarding the operation of the shovel 100. is recorded in the operation log storage unit 301B.

The operation log storage unit 301B stores operation logs of the shovel 100 in an accumulated manner. For example, the operation log storage unit 301B stores data regarding the shape of a work target around the shovel 100 for each operation of the shovel 100 and data regarding the operation of the shovel 100 in a linked form. Specifically, the operation log storage unit 301B stores record data representing the correspondence relationship between data regarding the shape of the work target around the shovel 100 and data regarding the operation of the shovel 100 for each operation of the shovel 100. A database of logs may be constructed.

Note that the operation log in the operation log storage unit 301B that has been transmitted to the information processing device 200 by the operation log transmission unit 301C, which will be described later, may be deleted after the fact.

The operation log transmission unit 301C transmits the operation log of the shovel 100, which is stored in the operation log storage unit 301B, to the information processing device 200 via the communication device 60. In addition, the operation log transmitting unit 301C also transmits to the information processing device 200 data regarding the shape of the work target around the shovel 100 for each operation of the shovel 100 and record data representing the correspondence between the data regarding the operations of the shovel 100. You may.

For example, the operation log transmitting unit 301C may store the operation log in the operation log storage unit 301B in response to a signal requesting transmission of the operation log of the excavator 100 (hereinafter referred to as a “transmission request signal”) received from the information processing device 200. The operation log of the shovel 100 that has not been sent yet is sent to the information processing device 200. Further, the operation log transmitting unit 301C may automatically transmit the unsent operation log of the shovel 100, which is stored in the operation log storage unit 301B, to the information processing device 200 at a predetermined timing. The predetermined timing is, for example, when the excavator 100 stops operating (the key switch is turned off) or when the excavator 100 starts operating (the key switch is turned on).

The operation log acquisition unit 2001 acquires the operation log of the shovel 100, which is received from the shovel 100.

The operation log acquisition unit 2001 acquires the operation log of the excavator 100 by transmitting a transmission request signal to the excavator 100 in response to an operation by the user of the information processing device 200 or automatically at a predetermined timing. Further, the operation log acquisition unit 2001 may acquire an operation log of the excavator 100 that is transmitted from the excavator 100 at a predetermined timing.

The operation log storage unit 2002 stores operation logs of the shovel 100 acquired by the operation log acquisition unit 2001 in an accumulated manner. For example, in the operation log storage unit 2002, as in the case of the operation log storage unit 301B, data regarding the shape of the work object around the shovel 100 for each operation of the shovel 100 and data regarding the operation of the shovel 100 are linked. remembered in form.

The teacher data generation unit 2003 generates teacher data for machine learning based on the operation log of the excavator 100 in the operation log storage unit 2002. The teacher data generation unit 2003 may automatically generate the teacher data by batch processing, or may generate the teacher data in response to input from the user of the information processing apparatus 200. The training data includes data regarding the shape of the work target around the shovel 100 as input data, and data representing the operation of the shovel 100 corresponding to the shape of the work target corresponding to the input data as correct output data (hereinafter referred to as " This is the data in combination with "correct answer data").

The correct answer data includes, for example, data representing the type of motion selected from a plurality of candidate motions that can be performed in a predetermined task. For example, in the case of leveling the ground at a work site, the plurality of candidate operations include a sweeping operation, a leveling operation, a compaction operation, a broom operation, and the like. The sweeping operation is, for example, an operation in which the attachment AT is operated to push the bucket 6 forward along the ground, thereby sweeping out earth and sand forward on the back surface of the bucket 6. In the sweeping operation, for example, the attachment AT performs a lowering operation of the boom 4 and an opening operation of the arm 5. The horizontal pulling operation is, for example, an operation to smooth out irregularities on the ground (the surface of the terrain) by operating the attachment AT and moving the toe of the bucket 6 along the ground almost horizontally toward you. be. In the horizontal pulling operation, for example, the attachment AT performs a raising operation of the boom 4 and a closing operation of the arm 5. The rolling operation is, for example, an operation of operating the attachment AT and pressing the back surface of the bucket 6 against the ground. In addition, the compaction operation is performed by pushing the bucket 6 forward along the ground, sweeping the earth and sand to a predetermined position in front with the back of the bucket 6, and then rolling the ground at a predetermined position with the back of the bucket 6. It may also be a pressing motion. In the rolling operation, for example, the attachment AT lowers the boom 4 when pressing against the ground. The broom operation is, for example, an operation in which the upper rotating body 3 is operated and the bucket 6 is rotated left and right while keeping it along the ground. Further, the broom operation may be, for example, an operation of pushing the bucket 6 forward while operating the attachment AT and the upper rotating body 3 and rotating the bucket 6 alternately left and right while keeping the bucket 6 along the ground. In the broom movement, for example, the upper revolving body 3 alternately repeats left and right turning movements. In addition, in the broom operation, for example, in addition to the alternating right and left turning operations of the upper revolving structure 3, the attachment AT may perform a lowering operation of the boom 4 and an opening operation of the arm 5, as in the case of the sweeping operation. Further, the correct data may include, for example, data representing the trajectory of the bucket 6 during operation of the shovel 100.

The machine learning unit 2004 performs machine learning on the base learning model based on the set of teacher data generated by the teacher data generation unit 2003 to generate a learned model LM. The learned model LM (base learning model) includes, for example, a neural network such as a DNN (Deep Neural Network).

The trained model LM outputs predicted probabilities for each of a plurality of candidate movements to be executed in a predetermined work, using, for example, data regarding the shape of a work target around the shovel 100 as an input condition. This predicted probability represents the reliability of the candidate's motion. This is because, as described above, the learned model LM reflects the operation log when the shovel 100 is operated by an expert, and it is considered that the higher the prediction probability, the higher the reliability of selecting the candidate operation. Further, this predicted probability represents the degree of conformity to the shape of the work target around the shovel 100 as an input condition. This is because it is considered that the higher the prediction probability, the higher the possibility that the expert will judge that the candidate motion is suitable for the shape of the work target. In addition, the learned model LM outputs data representing the trajectory of the bucket 6 for each of the plurality of candidate movements (hereinafter referred to as "target trajectory"), using data regarding the shape of the work target around the excavator 100 as an input condition. Good too. In addition, the trained model outputs a plurality of pieces of data representing the target trajectory of the bucket 6 for each of the plurality of candidate movements, using data regarding the shape of the work target around the excavator 100 as an input condition, and The predicted probability for each target trajectory may be output. As in the case of the predicted probability of the candidate motion, this predicted probability represents the degree of reliability of the target trajectory of the object and the degree of conformity to the shape of the work object around the shovel 100 as an input condition. Furthermore, the learned model LM may be generated for each of a plurality of different tasks. For example, the learned model LM is generated for each task such as land leveling work, slope construction work, and embankment work.

The trained model storage unit 2005 stores the trained model LM output by the machine learning unit 2004.

The distribution unit 2006 distributes the learned model LM to the excavator 100.

For example, when the learned model LM is generated by the machine learning unit 2004, the distribution unit 2006 distributes the most recently generated learned model LM to the excavator 100. Further, the distribution unit 2006 may distribute the latest learned model LM in the learned model storage unit 2005 to the shovel 100 in response to a signal received from the excavator 100 requesting distribution of the learned model LM. .

The work support unit 302 is a functional unit that supports the work of the shovel 100 operated by the operator.

The work support unit 302 includes a learned model storage unit 302A, a work target shape acquisition unit 302B, an estimation unit 302C, and a proposal unit 302D.

The trained model storage unit 302A stores the trained model LM distributed from the information processing device 200 and received through the communication device 60.

The work target shape acquisition unit 302B acquires data regarding the shape of the work target (terrain shape) around the shovel 100 based on the output of the imaging device 40 and the distance sensor.

Based on the data regarding the shape of the work object around the shovel 100, the estimation unit 302C determines the reliability of the shape of the work work around the shovel 100 among a plurality of candidate movements that can be performed in a predetermined work. and the motion with relatively high fitness. Furthermore, the estimating unit 302C determines the target trajectory of one or more buckets 6 with relatively high reliability and suitability for each of the plurality of candidate movements based on the data regarding the shape of the work target around the shovel 100. It may be estimated.

Specifically, the estimation unit 302C uses the trained model LM to calculate the reliability and the shape of the work target around the shovel 100 using data regarding the shape of the work target around the shovel 100 as an input condition. A motion with a relatively high degree of fitness may be estimated. In addition, the estimation unit 302C uses the learned model LM to set one or more goals for the bucket 6 with relatively high reliability and suitability, using data regarding the shape of the work target around the shovel 100 as an input condition. Orbits may be estimated.

Based on the estimation result of the estimating section 302C, the proposing section 302D suggests an operation of the excavator 100 that has a relatively high degree of reliability and conformity to the shape of the work target around the excavator 100 through the output device 50 such as the display device 50A. A proposal is made to the operator of cabin 10. Thereby, even an inexperienced operator can select a more appropriate operation according to the shape of the current work target around the shovel 100. Therefore, the operator's convenience can be improved, and the working efficiency of the excavator 100 can be improved. The number of actions suggested to the operator may be one or multiple. For example, the proposal unit 302D notifies the numerical value of the degree of conformity (reliability) to the shape of the work target around the shovel 100 for all or part of the motions of the plurality of candidates, so that An action is proposed (see FIG. 10 below).

Further, based on the estimation result of the estimating unit 302C, the proposing unit 302D sends, via the output device 50, a list of buckets 6 in the motion of the proposed target that has a relatively high degree of reliability and conformity to the shape of the work target around the shovel 100. One target trajectory may be proposed. As a result, even an inexperienced operator can understand a more appropriate target trajectory of the bucket 6 according to the shape of the current work target around the excavator 100, and can move the excavator 100 to realize the target trajectory. can be operated. Therefore, the operator's convenience can be further improved, and the working efficiency of the excavator 100 can be further improved.

In addition, the proposal unit 302D proposes a plurality of target trajectories of the bucket 6 in the motion to be proposed, which have a relatively high degree of reliability and conformity to the shape of the work target around the shovel 100, through the output device 50 of the motion target. You may. This allows the operator to understand multiple target trajectories for the bucket 6 that are more appropriate according to the shape of the work target around the current excavator 100, and to move the excavator so that the operator can realize the one target trajectory that he/she selects. 100 can be operated. Therefore, the working efficiency of the excavator 100 can be improved in a manner that reflects the operator's intention. For example, the proposal unit 302D notifies the numerical value of the degree of conformity (reliability) to the shape of the work target around the shovel 100 for each of the plurality of target trajectories of the motion to be proposed. A target trajectory for bucket 6 is proposed (see FIGS. 11 and 13 described later).

In addition, when the excavator 100 is remotely controlled, the proposal unit 302D provides the operator using the remote operation support device 300 with information on operations with relatively high reliability and suitability and the target trajectory of the bucket 6 via the communication device 60. You may make suggestions. In this case, the proposal unit 302D transmits data representing the content of the proposal to the remote operation support device 300 via the communication device 60. As a result, the remote operation support device 300 uses a display device, a sound output device, etc. to inform the operator using the remote operation support device 300 of operations with relatively high degrees of reliability and suitability and the target trajectory of the bucket 6. I can make suggestions.

<Processing>
FIG. 7 is a flowchart schematically showing a first example of processing related to the motion suggestion function of the shovel 100.

The flowchart in FIG. 7 is started when a predetermined input for starting the motion suggestion function is received, for example, through the input device 52 or the input device of the remote operation support device 300. The same may apply to the flowchart of FIG. 9, which will be described later.

As shown in FIG. 7, in step S102 (an example of an acquisition step), the work target shape acquisition unit 302B acquires data regarding the shape of the work target around the shovel 100 based on the output of the imaging device 40.

Upon completion of the process in step S102, the controller 30 proceeds to step S104.

In step S104, the estimating unit 302C estimates a motion that has a relatively high degree of suitability (reliability) for the shape of the work target around the current shovel 100, based on the data acquired in step S102.

Upon completion of the process in step S104, the controller 30 proceeds to step S106.

In step S106 (an example of a proposing step), the proposing unit 302D causes the display device 50A to display the proposed motion among the plurality of candidate motions and the target trajectory of the motion, based on the estimation result in step S104.

Upon completion of the process in step S106, the controller 30 proceeds to step S108.

In step S108, the controller 30 determines whether the driven element (actuator) has been operated. If the driven element is not being operated, the controller 30 proceeds to step S110, and if the driven element is being operated, the process proceeds to step S112.

In step S110, the controller 30 determines whether the termination condition is satisfied. The termination condition is, for example, that a predetermined input indicating termination of the motion suggestion function is received from the operator through the input device 52 or the input device of the remote operation support device 300. Further, the termination condition may be that a predetermined input indicating the end of the work is received from the operator through the input device 52 or the input device of the remote operation support device 300. Further, the end condition may be that the controller 30 determines the end of the work based on the captured image of the imaging device 40. If the termination condition is satisfied, the controller 30 terminates the process of the current flowchart, and if the termination condition is not satisfied, the process returns to step S108.

On the other hand, in step S112, the controller 30 determines whether or not the operation of the driven element corresponding to one operation of the shovel 100 has been completed, based on the operating state of the operating device 26, the operating state of the shovel 100, etc. . The controller 30 can grasp the operating state of the operating device 26, the operating state of the shovel 100, etc. based on the output of the operating pressure sensor 29, the operating signal output from the operating device 26, the outputs of the sensors S1 to S5, etc. . If the operation of the driven element corresponding to one operation of the shovel 100 has been completed, the controller 30 proceeds to step S114, and if the operation has not been completed, the controller 30 waits until the operation is completed (repeat the process of step S112).

In step S114, the controller 30 determines whether the termination condition is satisfied. If the termination condition is satisfied, the controller 30 terminates the process of the current flowchart, and if the termination condition is not satisfied, the process returns to step S102.

As described above, in this example, the support device 150 displays the operation of the shovel 100 and the target trajectory of the bucket 6 that have a high degree of conformity (reliability) to the shape of the work target around the shovel 100 on the display device 50A. A proposal can be made to the operator through the remote operation support device 300.

[Second example of motion suggestion function]
Next, with reference to FIGS. 8 and 9 in addition to FIGS. 1 to 5, a second example of the function of suggesting the operation of the excavator 100 to the user (operator) will be described.

Hereinafter, the same reference numerals will be given to the same or corresponding configurations as in the above-mentioned first example, and the explanation will focus on the parts that are different from the above-mentioned first example, and the same or corresponding contents as in the above-mentioned first example will be explained. It may be simplified or omitted.

<Functional configuration>
FIG. 8 is a functional block diagram showing a first example of the functional configuration regarding the operation proposal function of the operation support system SYS.

As shown in FIG. 8, the support device 150 of the excavator 100 includes a controller 30, a hydraulic control valve 31, an imaging device 40, an output device 50 (display device 50A), an input device 52, and a communication device 60. include.

The controller 30 includes, as functional units, an operation log providing unit 301 and a work support unit 302, as in the first example described above.

The work support unit 302 includes a learned model storage unit 302A, a work target shape acquisition unit 302B, an estimation unit 302C, a proposal unit 302D, and an operation control unit 302E.

The operation control unit 302E controls the hydraulic control valve 31 in response to instructions input from the operator received through the input device 52 and the communication device 60, and automatically performs the operation of the excavator 100 suggested to the operator by the suggestion unit 302D. Run it with As a result, the support device 150 can cause the shovel 100 to automatically perform the proposed operation in accordance with the shape of the current work target around the shovel 100, on the premise that instructions are input from the operator. Therefore, even an inexperienced operator can more appropriately operate the shovel 100 in accordance with the shape of the current work target around the shovel 100 by simply inputting instructions. Therefore, the operator's convenience can be further improved, and the working efficiency of the excavator 100 can be further improved.

For example, when there is one motion to be proposed, the motion control unit 302E controls the hydraulic control valve 31 in response to an instruction input from the operator, and automatically controls the motion of the shovel 100 to be proposed by the suggestion unit 302D. Run it with Further, for example, when there are a plurality of motions to be proposed, the motion control unit 302E automatically executes one of the motions to be suggested, which is selected by inputting an instruction from the operator. Further, for example, when the proposal unit 302D proposes one target trajectory for the bucket 6, the motion control unit 302E controls the operation of the proposed shovel 100 so that the bucket 6 moves along the proposed target trajectory. Let it run automatically. Further, for example, when the proposal unit 302D proposes a plurality of target trajectories for the bucket 6, the operation control unit 302E selects one target trajectory for the bucket 6 from among the plurality of target trajectories, which is selected by inputting an instruction from the operator. Automatically execute the proposed action so that the object moves.

The action log recording unit 301A stores an action log including data regarding the shape of the work object acquired by the work object shape acquisition unit 302B, and data representing the actions and target trajectory performed by the action control unit 302E, into the action log storage unit 301B. to be recorded. As a result, the operation log transmitter 301C transmits an operation log that includes data regarding the shape of the work target and data representing the target trajectory and the motion of the excavator 100 actually executed by the operator based on the shape of the work target. It can be stored in the log storage unit 301B. The operation log transmitter 301C can then upload the accumulated operation logs to the information processing device 200. Therefore, the machine learning unit 2004 can update the learned model LM by relearning or additionally learning the learned model LM using the operation log.

In addition, the machine learning unit 2004 compares the trained model LM that has been retrained or additionally trained and the current trained model LM using predetermined evaluation data, and if the evaluation result of the former is high, the learning The trained model LM in the trained model storage unit 2005 may be updated.

<Processing>
FIG. 9 is a flowchart schematically showing a second example of processing related to the motion suggestion function of the shovel 100.

As shown in FIG. 9, the processing in steps S202 to S206 is the same as steps S102 to S106 in FIG. 7, so the explanation will be omitted.

Upon completion of the process in step S206, the controller 30 proceeds to step S208.

In step S208, the controller 30 determines whether an input from the operator instructing execution of the proposed action (hereinafter referred to as "input of execution instruction") is received through the input device 52 or the communication device 60. If the input of the execution instruction is not accepted, the controller 30 proceeds to step S210, and if the input is accepted, the process proceeds to step S212.

In step S210, the controller 30 determines whether the termination condition is satisfied. If the termination condition is satisfied, the controller 30 terminates the process of the current flowchart, and if the termination condition is not satisfied, the process returns to step S208.

On the other hand, in step S212, the operation control unit 302E controls the hydraulic control valve 31 to automatically execute the operation specified by the input of the execution instruction. In addition, when the target trajectory is specified by inputting the execution instruction, the operation control unit 302E controls the shovel 100 specified by the input of the execution instruction so that the bucket 6 moves on the target trajectory specified by the input of the execution instruction. Execute the action.

Upon completion of the process in step S212, the controller 30 proceeds to step S214.

In step S214, the action log recording unit 301A records an action log including data regarding the shape of the work object acquired by the work object shape acquisition unit 302B, and data representing the actions and target trajectory performed by the action control unit 302E. It is recorded in the operation log storage unit 301B.

Upon completion of the process in step S214, the controller 30 proceeds to step S216.

Step S216: The controller 30 determines whether the termination condition is satisfied. If the termination condition is satisfied, the controller 30 terminates the process of the current flowchart, and if the termination condition is not satisfied, the process returns to step S202.

In this example, the work target shape acquisition unit 302B may acquire data regarding the shape of the work target by predicting a change in the shape of the work target due to the operation of the shovel 100 in the previous process of step S212. .

As described above, in this example, the support device 150 determines the operation of the shovel 100 and the target of the bucket 6 that have a high degree of conformity (reliability) to the shape of the work target around the shovel 100, in accordance with instructions from the operator. Trajectories can be executed automatically.

Furthermore, in this example, the support device 150 can accumulate an operation log that includes data regarding the shape of the work target around the shovel 100, and data regarding the automatically executed actions of the shovel 100 and the target trajectory. Therefore, the support device 150 can update the learned model LM using the accumulated operation logs.

[Specific example of display content related to excavator motion suggestion function]
Next, with reference to FIGS. 10 to 15, a specific example of the display content of the display device 50A regarding the motion suggestion function of the shovel 100 will be described.

Note that the display contents of FIGS. 10 to 15 may be displayed on the display device of the remote operation support device 300.

<First example>
FIG. 10 is a diagram showing a first example (screen 1000) of display content on the display device 50A regarding the motion suggestion function of the excavator 100.

Screen 1000 includes images 1001 to 1006.

Image 1001 is an image representing a work target around shovel 100. In this example, the image 1001 is generated based on the output (image data) of the imaging device 40 using a known image processing technique, and is an image of the surroundings of the excavator 100 when viewed from a predetermined viewpoint. This is an image representing the work target (the ground at the work site).

Image 1002 is an image schematically representing shovel 100. In this example, image 1002 is an image schematically representing shovel 100 when viewed from the same viewpoint as image 1001, and is displayed superimposed on image 1001.

The image 1003 is an image that represents, in a list format, the motions to be proposed by the proposal unit 302D among the plurality of candidate motions. In this example, the image 1003 includes images 1003A to 1003D representing each row of a sweeping motion, a horizontal pulling motion, a rolling motion, and a broom motion from among a plurality of candidate motions as motions to be proposed. . Furthermore, in this example, the images 1003A to 1003D express the reliability (degree of suitability) for each of the broom operation, sweeping operation, horizontal pulling operation, and rolling operation. Thereby, the operator can select the operation to be performed by the excavator 100 by his own operation or automatically, taking into consideration the reliability (degree of suitability) from among the proposed operations.

Note that the image 1003 may represent only the motion with the highest degree of reliability (degree of suitability) (in this example, the sweeping motion). In other words, the suggestion unit 302D may suggest to the operator, through the image 1003, only the motion with the highest degree of reliability (degree of suitability) among the plurality of candidate motions for a predetermined task. Furthermore, among the plurality of candidate motions, only motions (for example, sweeping motion and horizontal pulling motion) whose reliability (degree of suitability) is equal to or higher than a predetermined standard (for example, 30%) are expressed in the image 1003. Good too. In other words, the proposal unit 302D may suggest to the operator only those operations whose reliability (degree of suitability) is equal to or higher than a predetermined standard among the plurality of candidate operations.

Image 1004 is an image representing the target trajectory for each proposed motion, which is represented in image 1003. Image 1004 is displayed around image 1002, superimposed on image 1001. Thereby, the operator can easily grasp the target trajectory for each motion to be proposed while comparing the image 1001 representing the state of the ground at the work site around the shovel 100 and the image 1002 representing the shovel 100. Image 1004 includes images 1004A to 1004D.

Image 1004A is an image representing the target trajectory of the sweeping operation.

Image 1004B is an image representing the target trajectory of the horizontal pulling operation.

Image 1004C is an image representing the target trajectory of the rolling operation.

Image 1004D is an image representing the target trajectory of the broom motion.

Note that the images 1004A to 1004D may be expressed in such a manner that the portion of the target trajectory that contacts the work object (ground) and the other portions can be distinguished. For example, the images 1004A to 1004D may have different colors between the portion of the target trajectory that contacts the work object and the other portions. Thereby, it is possible to assist the sense of perspective on the image 1001 of the images 1004A to 1004D corresponding to the target trajectory.

In this example, a matte cursor is expressed in the image 1003A corresponding to the sweeping operation. Furthermore, in this example, the image 1004A corresponding to the sweeping motion of the image 1004 is expressed by a thicker line than the images 1004B to 1004D corresponding to other motions. Thus, in this example, a state in which the sweeping operation is selected is expressed. For example, the operator uses the input device 52 to specify any one of the images 1003A to 1003D, thereby selecting one of the sweeping operation, horizontal pulling operation, compaction operation, and broom operation. be able to. Similarly, for example, by specifying any one of the images 1004A to 1004D using the input device 52, the operator can perform any one of the sweeping operation, horizontal pulling operation, rolling operation, and broom operation. can be selected.

The image 1005 is an icon for confirming the execution of the action selected by the user (operator) from among the proposed actions.

For example, by operating the image 1005 using the input device 52, the operator can cause the excavator 100 to automatically perform the selected operation.

Image 1006 is an icon for terminating the motion suggestion function of excavator 100. The same applies to

images

1106, 1206, 1306, 1406, and 1506, which will be described later.

In this way, in this example, the controller 30 selects a plurality of motions from among a plurality of candidate motions in the ground leveling work based on the reliability (adaptability) of each of the current shape of the work target (terrain shape) around the excavator 100. degree) on the display device 50A. Thereby, the controller 30 can suggest to the operator an operation that has a relatively high degree of reliability (degree of suitability) for the current shape of the work target (terrain shape) around the excavator 100.

<Second example>
FIG. 11 is a diagram showing a second example (screen 1100) of display content on the display device 50A regarding the motion suggestion function of the shovel 100.

Hereinafter, the explanation will focus on the parts that are different from the above-mentioned first example, and the explanation of the same or corresponding contents as the above-mentioned first example may be simplified or omitted.

Screen 1100 includes images 1101 to 1106.

Image 1101, like image 1001 in FIG. 10, is an image representing a work target around shovel 100.

Image 1102 is an image schematically representing shovel 100, similar to image 1002 in FIG.

The image 1103 is an image that represents, in a list format, a plurality of target trajectories of the bucket 6 for one motion to be proposed by the proposal unit 302D among the plurality of candidate motions. In this example, the image 1103 includes images 1103A to 1103D representing each row of four target trajectories ("sweeping I" to "sweeping IV") of the bucket 6 regarding the sweeping motion as one of the motions to be proposed. included. Further, in this example, the reliability (degree of suitability) of each of the four target trajectories of the bucket 6 is expressed in the images 1103A to 1103D. As a result, the operator can select the bucket 6 to be executed by the excavator 100 either by the operator's own operation or automatically, considering the reliability among the four target trajectories of the bucket 6 for one proposed operation (sweeping operation). A target trajectory can be selected.

Incidentally, the image 1103 may represent only the target trajectory of the bucket 6 (in this example, "sweep I"), which has the highest degree of reliability (degree of suitability). In other words, the proposal unit 302D may suggest to the operator, through the image 1103, only the motion with the highest degree of reliability (degree of suitability) among the plurality of target trajectories for one motion to be proposed in a predetermined task. The image 1103 also includes target trajectories (for example, "Sweeping I" and "Sweeping II") whose reliability (fitness) is higher than a predetermined standard (for example, 30%) among the plurality of target trajectories of the bucket 6. may be expressed only. That is, the proposal unit 302D may propose to the operator only those target trajectories whose reliability (degree of suitability) is equal to or higher than a predetermined standard among the plurality of target trajectories in the bucket 6 for one motion to be proposed. The same may apply to image 1203A, which will be described later.

Image 1104 is an image representing four target trajectories for one motion of the proposal target, which is represented in image 1103. Similar to the image 1004, the image 1104 is displayed in the vicinity of the image 1002, superimposed on the image 1001. As a result, the operator can easily determine multiple target trajectories for one motion (sweeping motion) to be proposed while comparing the image 1101 representing the ground condition of the work site around the shovel 100 and the image 1102 representing the shovel 100. can be grasped. Image 1104 includes images 1104A to 1104D.

Image 1104A is an image representing the target trajectory of the sweeping operation, which corresponds to image 1103A ("Sweeping I").

Image 1104B is an image representing the target trajectory of the sweeping operation, which corresponds to image 1103B ("Sweeping II").

Image 1104C is an image representing the target trajectory of the sweeping operation, which corresponds to image 1103C ("Sweeping III").

Image 1104D is an image representing the target trajectory of the sweeping operation, which corresponds to image 1103C ("Sweeping IV").

In this example, a matte cursor is expressed in the image 1103A corresponding to "sweeping operation I". Further, in this example, an image 1104A corresponding to the target trajectory of "sweeping operation I" in the image 1104 is expressed by a thicker line than images 1104B to 1104D corresponding to other target trajectories. As a result, in this example, a state is expressed in which the "sweeping I" target trajectory is selected among the four target trajectories. For example, the operator can select any one of the four target trajectories by specifying any one of the images 1103A to 1103D using the input device 52. Similarly, for example, the operator can select any one of the four target trajectories for the bucket 6 by specifying any one of the images 1104A to 1104D using the input device 52. .

The image 1105 is an icon for executing one of the proposed operations (sweeping operation) so that the bucket 6 moves on a target trajectory selected by the user (operator) from among a plurality of target trajectories.

For example, by manipulating the image 1105 using the input device 52, the operator can cause the excavator 100 to automatically perform one of the proposed operations so that the bucket 6 moves along the selected target trajectory. .

As described above, in this example, the controller 30 determines the plurality of target trajectories of the bucket 6 for one operation related to the ground leveling work based on the reliability ( degree of suitability) on the display device 50A. Thereby, the controller 30 can suggest to the operator a plurality of target trajectories for the bucket 6 that have a relatively high degree of reliability (degree of suitability) for the shape of the work target (terrain shape) around the current excavator 100. .

<3rd example>
12 and 13 are diagrams showing a third example (screens 1200, 1300) of display contents of the display device 50A regarding the motion suggestion function of the excavator 100.

Hereinafter, the explanation will focus on the parts that are different from the above-mentioned first and second examples, and the explanation of the same or corresponding contents as the above-mentioned first and second examples may be simplified or omitted.

Screen 1200 includes images 1201 to 1206.

Image 1201, like image 1001 in FIG. 10, is an image representing a work target around shovel 100.

Image 1202 is an image schematically representing shovel 100, similar to image 1002 in FIG.

Similar to the image 1003 in FIG. 10, the image 1203 is an image that represents, in a list format, the motions to be proposed by the proposal unit 302D among the plurality of candidate motions. In this example, the image 1203 includes images 1203A to 1203D representing each row of a sweeping motion, a horizontal pulling motion, a rolling motion, and a broom motion from among a plurality of candidate motions as motions to be proposed. . Further, in this example, the images 1203A to 1203D express the reliability (degree of suitability) for each of the broom operation, sweeping operation, horizontal pulling operation, and rolling operation.

Similarly to the image 1103 in FIG. 11, the image 1203A is an image that represents a plurality of target trajectories of the bucket 6 in a list format for one motion (sweeping motion) to be proposed. In this example, image 1203A shows each of the four target trajectories ("sweep I" to "sweep IV") of bucket 6 for the sweep motion with the highest degree of reliability (fitness) among the multiple candidate motions. Images 1203A1 to 1203A4 representing rows of are included. Further, in this example, the reliability (degree of suitability) of each of the four target trajectories of the bucket 6 is expressed in the images 1203A1 to 1203A4.

Similar to the image 1004 in FIG. 10, the image 1204 is an image representing the target trajectory for each proposed motion, which is expressed in the image 1203. Image 1204 includes images 1204A to 1204D.

Image 1204A is an image representing the target trajectory of the sweeping operation. Specifically, the image represents the target trajectory ("sweeping I") with the highest reliability among the target trajectories ("sweeping I" to "sweeping IV") of the bucket 6 corresponding to the sweeping operation.

The images 1204B to 1204D are the same as the images 1004B to 1004D in FIG. 10, so a description thereof will be omitted.

In this example, a matte cursor is expressed in the image 1003A corresponding to the sweeping operation. Furthermore, in this example, the image 1004A corresponding to the sweeping motion of the image 1004 is expressed by a thicker line than the images 1004B to 1004D corresponding to other motions. Thus, in this example, a state in which the sweeping operation is selected is expressed.

The image 1205 is an icon for confirming execution of the action selected by the user (operator) from among the proposed actions.

In addition, the image 1205 is for transitioning to a screen 1300 for selecting four target trajectories of the bucket 6 corresponding to the sweeping motion in a state where the sweeping motion with the highest reliability among the motions to be proposed is selected. This is the icon. That is, when the image 1205 is operated through the input device 52 while the screen 1200 is displayed, the screen shifts to the screen 1300.

Screen 1300 includes images 1301 to 1306.

Similar to the image 1203 in FIG. 12, the image 1303 is an image that represents, in a list format, the motions to be proposed by the proposal unit 302D among the plurality of candidate motions. Specifically, the image 1303 includes images 1203A to 1203D representing each row of a sweeping motion, a horizontal pulling motion, a rolling motion, and a broom motion from among a plurality of candidate motions as motions to be proposed. It will be done.

Also, like the image 1203A in FIG. 12, the image 1303A is an image that represents a plurality of target trajectories of the bucket 6 in a list format for one motion (sweeping motion) to be proposed. Specifically, the image 1203A shows four target trajectories ("sweep I" to "sweep IV") of bucket 6 for the sweep motion with the highest degree of reliability (fitness) among the multiple candidate motions. Images 1303A1 to 1303A4 representing each row are included.

Image 1304 is similar to image 1104 in FIG. 11, and image 1104 is an image representing four target trajectories for one motion of the proposed object, which is expressed in image 1103. Specifically, image 1304 includes images 1304A to 1304D.

Images 1304A to 1304D are the same as images 1104A to 1104D in FIG. 11, respectively, so their description will be omitted.

In this example, a matte cursor is expressed in the image 1303A1 corresponding to "sweeping operation I". Further, in this example, an image 1304A corresponding to the target trajectory of "sweeping operation I" in the image 1304 is expressed by a thicker line than images 1304B to 1304D corresponding to other target trajectories. As a result, in this example, a state is expressed in which the "sweeping I" target trajectory is selected among the four target trajectories.

The image 1305 is an icon for executing one of the proposed operations (sweeping operation) so that the bucket 6 moves on a target trajectory selected by the user (operator) from among a plurality of target trajectories.

In this way, in this example, the controller 30 causes the display device 50A to display a plurality of motions among a plurality of candidate motions related to the ground leveling work together with their reliability, and also displays a plurality of targets for the motion with the highest reliability. The trajectory is displayed on the display device 50A. Thereby, the controller 30 selects a plurality of operations that have a relatively high degree of reliability with respect to the current shape of the work target (terrain shape) around the excavator 100, and a plurality of target trajectories of the bucket 6 for the most reliable operation. can be proposed to the operator.

<4th example>
FIG. 14 is a diagram showing a fourth example (screen 1400) of display content on the display device 50A regarding the motion suggestion function of the shovel 100.

Hereinafter, the explanation will focus on the parts that are different from the above-mentioned first to third examples, and the explanation of the same or corresponding content as the above-mentioned first to third examples may be simplified or omitted.

Screen 1400 includes images 1401 to 1406.

Image 1401, like image 1001 in FIG. 10, is an image representing a work target around shovel 100.

Image 1402 is an image schematically representing shovel 100, similar to image 1002 in FIG.

Similar to the image 1003 in FIG. 10, the image 1403 is an image that represents, in a list format, the motions to be proposed by the proposal unit 302D among the plurality of candidate motions. Specifically, the image 1403 includes images 1403A to 1403D representing each row of a sweeping motion, a horizontal pulling motion, a rolling motion, and a broom motion from among a plurality of candidate motions as motions to be proposed. It will be done.

Similar to the image 1004 in FIG. 10, the image 1404 is an image representing the target trajectory for each proposed motion expressed in the image 1403. Specifically, image 1404 includes images 1404A to 1404D.

Images 1404A to 1404D are the same as images 1004A to 1004D in FIG. 10, respectively, so their description will be omitted.

In this example, the worker W is shown in the image area where the image 1403A of the image 1401 is displayed in a superimposed manner. Therefore, if the most reliable operation (sweeping operation) is selected, there is a possibility that the attachment AT will come too close to the worker W or that the attachment AT will come into contact with the worker W.

In contrast, in this example, a satin-textured cursor is expressed in image 1403B corresponding to the horizontal pulling motion, and image 1404B corresponding to the horizontal pulling motion is thicker than

images

1404A, 1404C, and 1404D corresponding to other motions. It is represented by a line. That is, in this example, the operator selects an operation other than the sweeping operation (horizontal pulling operation) through the input device 52, and is about to have the excavator 100 execute it. Thereby, it is possible to prevent the attachment AT from coming too close to the worker W or coming into contact with the worker W.

The image 1405 is the same as the image 1005 in FIG. 10, so its description will be omitted.

In this way, in this example, by displaying the target trajectory of the proposed motion superimposed on the image 1401 representing the surroundings of the excavator 100, the operator can compare the target trajectory with the worker W at the work site. It is possible to understand the relationship with obstacles. Therefore, the safety of the shovel 100 can be improved while improving the operator's convenience and the working efficiency of the shovel 100.

<Fifth example>
FIG. 15 is a diagram showing a fifth example (screen 1500) of display content on the display device 50A regarding the motion suggestion function of the shovel 100.

Hereinafter, the explanation will focus on the parts that are different from the above-mentioned first to fourth examples, and the explanation of contents that are the same as or corresponding to the above-mentioned first to fourth examples may be simplified or omitted.

Screen 1500 includes images 1501 to 1506.

Image 1501, like image 1001 in FIG. 10, is an image representing a work target around shovel 100.

Image 1502 is an image schematically representing shovel 100, similar to image 1002 in FIG.

Similar to the image 1003 in FIG. 10, the image 1503 is an image that represents, in a list format, the motions to be proposed by the proposal unit 302D among the plurality of candidate motions. Specifically, the image 1503 includes images 1503A to 1503D representing each row of a horizontal pulling action, a sweeping action, a rolling action, and a broom action from among a plurality of candidate actions as suggested actions. It will be done.

Similar to the image 1004 in FIG. 10, the image 1504 is an image representing the target trajectory for each proposed motion expressed in the image 1503. Specifically, image 1504 includes images 1504A to 1404D.

Image 1004D is an image representing the target trajectory of the broom motion.

In this example, the image area in which the

images

1504A, 1504C, and 1504D of the image 1501 are displayed in a superimposed manner is included in the area 1501A in which the leveling work has already been completed. Therefore, if the operation with the highest reliability (the leveling operation corresponding to image 1504A) is selected, not only will wasteful work be performed on the area where the leveling work has already been completed, but also the impact of the wasteful work will be You will need to do some work to restore it. As a result, there is a possibility that the working efficiency of the shovel 100 will be reduced and the progress of the work at the work site will be delayed.

On the other hand, in this example, a matte cursor is expressed in the image 1503B corresponding to the sweeping motion, and the image 1504B corresponding to the sweeping motion is drawn with thicker lines than the

images

1504A, 1504C, and 1504D corresponding to other motions. It is expressed. That is, in this example, the operator selects a different operation (horizontal pulling operation) from the sweeping operation and is trying to have the excavator 100 execute it. This can prevent the shovel 100 from performing work on an area where work has already been completed. Therefore, it is possible to suppress a decrease in the work efficiency of the shovel 100 and a delay in the progress of work at the work site.

Furthermore, in this example, an image (such as an image covering the area 1501A with diagonal lines) indicating that the area 1501A is an area where land leveling work has already been completed is displayed in a superimposed manner. This allows the operator to more reliably understand that the area 1501A is an area where the leveling work has been completed. In this case, information regarding the area where the work has been completed at the work site is distributed from the information processing device 200 to the shovel 100, for example.

The image 1505 is the same as the image 1005 in FIG. 10, so its description will be omitted.

In this way, in this example, by displaying the target trajectory of the proposed motion superimposed on the image 1401 representing the surroundings of the excavator 100, the operator can check the target trajectory and the completion status of the work at the work site. It is possible to understand the relationship between Therefore, the operator can select a more appropriate motion or target trajectory according to the completion status of the work at the work site.

[Other examples of motion suggestion function]
Next, another example of the motion suggestion function will be explained.

The contents of the first and second examples of the above-described motion suggestion function may be combined, modified or changed as appropriate.

For example, in the first and second examples of the motion proposal function described above, the proposal unit 302D proposes only the motion to be proposed out of a plurality of candidate motions that can be performed in a predetermined task, and Proposal of a target trajectory corresponding to the motion may be omitted.

Furthermore, in the first example, the second example, and their variations of the motion proposal function described above, the proposal unit 302D only selects target trajectories that have a relatively high degree of reliability (degree of conformity) to the shape of the work target around the shovel 100. Instead, motions or target trajectories with relatively low reliability (fitness) may be intentionally proposed. For example, the motion with relatively low reliability (fitness) is the motion of the excavator 100 or the target trajectory of the bucket 6, which is estimated based on the trained model LM and whose reliability (fitness) is lower than a predetermined standard. . In addition, motions with relatively low reliability (fitness) are handled by a trained model other than the trained model LM, that is, a trained model that has been machine learned using a teacher dataset that includes inappropriate motions and target trajectories. The motion of the shovel 100 or the target trajectory of the bucket 6 estimated based on the model may be used. Thereby, it is possible to suppress the operator's dependence on the motion suggestion function and encourage the operator to use the motion suggestion function based on his or her own appropriate judgment. In this case, when an operation with a relatively low degree of reliability suitability is selected, a log to that effect may be recorded in the auxiliary storage device 30A of the controller 30 or the like. Further, the log may include identification information of the operator. As a result, for example, the support device 150 (controller 30) can label an operator who has a high probability of selecting an operation of the shovel 100 or a target trajectory of the bucket 6 that has a relatively low degree of reliability (degree of suitability). I can do it. Therefore, a manager or the like at a work site can manage the usage status of the motion suggestion function for each operator by checking logs, labels, etc. after the fact.

Further, in the first example, the second example, and the modified examples of the above-described motion suggestion function, part or all of the functions of the support device 150 may be transferred to the remote operation support device 300. For example, the functions of the proposal unit 302D are transferred to the remote operation support device 300. Further, in addition to the proposal section 302D, the functions of the estimation section 302C may be transferred to the remote operation support device 300. Further, in addition to the proposal section 302D and the estimation section 302C, the functions of the work object shape acquisition section 302B may be transferred to the remote operation support device 300. In this case, image data of the imaging device 40 is transmitted from the excavator 100 to the remote operation support device 300.

Further, in the first example, the second example, and the modified examples of the above-described motion suggestion function, part or all of the functions of the support device 150 may be transferred to the information processing device 200. For example, the work target shape acquisition unit 302B is transferred to the information processing device 200. In this case, image data of the imaging device 40 is transmitted from the excavator 100 to the information processing device 200. Further, in addition to the work target shape acquisition unit 302B, the function of the estimation unit 302C may be transferred to the information processing device 200. Furthermore, in addition to the work target shape acquisition section 302B and the estimation section 302C, the functions of the proposal section 302D may be transferred to the information processing apparatus 200. Thereby, for example, the portable information processing device 200 brought into the cabin 10 suggests to the operator one or more operations from among a plurality of candidate operations of the shovel 100 that can be performed in a predetermined work. You can also suggest the trajectory of the movement.

Further, the support device 150 according to the first example, the second example, or a modification thereof of the above-mentioned motion suggestion function may perform one or more of the plurality of candidate motions in a predetermined work of a working machine different from the excavator 100. A plurality of actions may be suggested to the operator. For example, another working machine is a forestry machine with a harvester device. In this case, the support device 150 may propose an operation for one or more trees to be operated by the harvester device from among operations for a plurality of candidate trees existing in the field.

[Function of support device (1)]
Next, the operation of the support device according to this embodiment will be explained.

In this embodiment, the support device includes an acquisition unit and a proposal unit. The support device is, for example, the support device 150. The working machine is, for example, a shovel 100. The acquisition unit is, for example, a work target shape acquisition unit 302B. The proposal unit is, for example, the proposal unit 302D. Specifically, the acquisition unit acquires data regarding the shape of the work target (for example, topographic shape) around the working machine. Then, the proposal unit proposes to the user a motion among the plurality of candidate motions of the work machine in a predetermined work based on the data regarding the shape of the work target around the work machine acquired by the acquisition unit.

Furthermore, in this embodiment, the work machine may include the above-mentioned support device.

Thereby, the support device can, for example, suggest to the operator of the work machine an action that is more suitable for the shape of the work target around the work machine, from among a plurality of candidate actions that can be performed in a predetermined work. Therefore, the work machine can be operated more appropriately. Therefore, the working efficiency of the working machine can be improved.

Additionally, in this embodiment, the support device includes an estimator. The estimator is, for example, the estimator 302C. Specifically, the estimation unit uses a trained model that has been machine-learned using training data related to the operation of the work machine by operations of a relatively highly skilled operator, which is associated with the shape of the work target. Based on the data regarding the shape of the work target around the work machine acquired by the acquisition unit, a motion matching the shape of the work target around the work machine is estimated from among the plurality of candidate motions. The learned model is, for example, the learned model LM. Then, the proposal unit may propose a motion among the plurality of candidate motions based on the estimation result of the estimation unit.

Thereby, the support device can use the learned model to suggest a motion that is more suitable for the shape of the work target around the work machine, from among a plurality of candidate motions that can be performed in a predetermined work.

Further, in the present embodiment, the proposal unit may propose a plurality of motions among the plurality of candidate motions to the user based on the data regarding the shape of the work target around the work machine acquired by the acquisition unit. good.

Thereby, the support device can provide options to the operator and encourage the operator to make a decision based on his or her own will. Therefore, by reflecting the operator's judgment, the work machine can be operated more appropriately.

Furthermore, in this embodiment, the plurality of motions to be proposed may include motions that have a relatively low degree of adaptation to the shape of the work target around the work machine among the plurality of candidate motions. good.

Thereby, the support device can encourage the operator to make a decision based on his or her own will. Therefore, by reflecting the operator's judgment, the work machine can be operated more appropriately.

Further, in the present embodiment, the proposal unit selects a motion among the plurality of candidate motions based on the data regarding the shape of the work target around the work machine acquired by the acquisition unit. It may be proposed together with the degree of conformity to the shape of the work target.

Thereby, the support device can provide the operator with information to decide whether or not to execute the operation. Therefore, the support device can prompt the operator to make more appropriate decisions and cause the work machine to operate more appropriately.

Further, in the present embodiment, the proposal unit selects a plurality of motions among the plurality of candidate motions based on the data regarding the shape of the work target around the work machine acquired by the acquisition unit. It may be proposed together with the degree of conformity to the shape of the work object around the work machine.

Thereby, the support device can provide the operator with a plurality of options regarding the operation of the work machine, and can also provide material for making decisions regarding the selection. Therefore, the support device can prompt the operator to make more appropriate decisions and cause the work machine to operate more appropriately.

Furthermore, in the present embodiment, the proposal unit selects a motion among the plurality of candidate motions based on the data regarding the shape of the work target around the work machine acquired by the acquisition unit. It may be proposed together with the trajectory of the part. The work part is, for example, the bucket 6.

Further, in the present embodiment, the proposing unit selects a motion among the plurality of candidate motions based on the data regarding the shape of the work target around the work machine acquired by the acquisition unit. It may be proposed along with the orbit of .

Thereby, the support device can provide the operator with multiple options regarding the trajectory of the work part that corresponds to the proposed motion. Therefore, the support device can prompt the operator to make more appropriate decisions and cause the work machine to operate more appropriately.

Further, in the present embodiment, the proposing unit selects a motion among the plurality of candidate motions based on the data regarding the shape of the work target around the work machine acquired by the acquisition unit. It may be proposed together with the trajectory of the object and the degree of conformity to the shape of the work object around the working machine for each of the plurality of trajectories.

Thereby, the support device can provide the operator with a plurality of options for the work part, and can also provide information for making a decision on the selection. Therefore, the support device can prompt the operator to make more appropriate decisions and cause the work machine to operate more appropriately.

Furthermore, in this embodiment, the support device may include a display section. The display section is, for example, a display device 50A. Then, the proposal unit may display the trajectory of the work part according to the motion to be proposed among the plurality of candidate motions on the display unit, superimposed on the image representing the surroundings of the work machine.

This allows the operator to more easily understand the relationship between the trajectory of the work site and the shape of the work target around the work machine. Therefore, the support device can prompt the operator to make more appropriate decisions and cause the work machine to operate more appropriately.

Further, in the present embodiment, the proposal unit displays on the display unit the trajectory portion that contacts the work target in the trajectory due to the proposed motion among the plurality of candidate motions and the other trajectory portions in different manners. You may let them.

Thereby, the support device can assist the sense of perspective on the image representing the surroundings of the work machine, allowing the operator to grasp the trajectory more appropriately.

Furthermore, in this embodiment, the support device includes a control section. The control unit is, for example, the operation control unit 302E. Specifically, the control unit may cause the work machine to automatically execute the operation proposed by the suggestion unit in response to an input instruction from the user.

Thereby, it is possible to further improve the convenience for the user (operator) and to operate the work machine more appropriately.

Furthermore, in the present embodiment, the acquisition unit obtains data regarding the shape of the work target around the work machine by predicting the shape of the work target around the work machine after the operation automatically executed by the control unit. may be obtained.

Thereby, the support device can propose the operation of the work machine more smoothly when supporting the work of the work machine while repeatedly proposing the operation of the work machine. Therefore, the working efficiency of the working machine can be further improved.

[Functions related to generating the trajectory of the excavator's working area]
Next, with reference to FIGS. 16 to 19 in addition to FIGS. 1 to 5, functions related to generation of a trajectory (hereinafter referred to as a "target trajectory") of the work area of the excavator 100 will be described.

Hereinafter, the same reference numerals will be given to the same or corresponding configurations as those related to the above-mentioned action suggestion function, and the description will focus on the parts that are different from the above-mentioned action suggestion function, and the explanation will focus on the parts that are different from the above-mentioned action suggestion function. Description may be simplified or omitted.

<Functional configuration>
FIG. 16 is a block diagram illustrating a first example of a functional configuration related to generation of a target trajectory of a working part of excavator 100. FIG. 17 is a diagram illustrating an example of a screen (screen 700) related to generation of a target trajectory of a working part of excavator 100, which is displayed on display device 50A. FIG. 18 is a diagram illustrating another example of a screen (screen 800) related to generation of a target trajectory of a working part of excavator 100, which is displayed on display device 50A. FIG. 19 is a diagram showing still another example (screen 900) of a screen related to generation of a target trajectory of a working part of excavator 100, which is displayed on display device 50A.

Note that when the excavator 100 is remotely controlled, a screen similar to that shown in FIGS. 17 and 18 is displayed on the remote operation support device 300 (display device).

The work area of the shovel 100 is, for example, the toe or back of the bucket 6.

The excavator 100 includes a support device 150. The support device 150 provides support regarding the work of the excavator 100.

As shown in FIG. 16, the support device 150 includes an operating device 26, a controller 30, an imaging device 40, and an output device 50. Further, when the excavator 100 is remotely controlled, the support device 150 may include the communication device 60.

Note that when there are multiple excavators 100 included in the operation support system SYS, the controller 30 includes only the former of the operation log providing unit 301 and the work support unit 302, and the excavator 100 that includes only the latter. May exist. In this case, the former excavator 100 acquires an operation log of the excavator 100 used for the operator operation support function (a function related to generating the trajectory of the work part) of the latter excavator 100, and provides it to the information processing device 200. It has only a function.

The operation log providing unit 301 is a functional unit that acquires the operation log of the shovel 100, which is the original data for realizing the function of generating the target trajectory of the working part of the excavator 100, and provides it to the information processing device 200. be. Specifically, a relatively experienced operator (hereinafter referred to as an "expert" for convenience) who has a long history of operating the excavator 100 obtains an operation log when operating the excavator 100 and stores it in the information processing device 200. provide.

The teacher data generation unit 2003 generates teacher data for machine learning based on the operation log of the excavator 100 in the operation log storage unit 2002. The teacher data generation unit 2003 may automatically generate the teacher data by batch processing, or may generate the teacher data in response to input from the user of the information processing apparatus 200. The training data includes data regarding the shape of the work object around the shovel 100 as input data, and data representing the trajectory (trajectory) of the work part of the shovel 100 corresponding to the input data as correct answer output data (hereinafter referred to as "correct answer"). data").

The data representing the trajectory of the working part of the shovel 100 is generated based on the output data of the sensors S1 to S5, which is included in the data regarding the operation of the shovel 100, for example.

The learned model LM outputs data representing the target trajectory of the work part of the shovel 100 and predicted probabilities using, for example, data regarding the type of operation of the shovel 100 and the shape of the work object around the shovel 100 as input conditions. . The trained model LM also includes data representing the target trajectory of the work area of the shovel 100, and the types of movements of the shovel 100, such as digging movement, sweeping movement, horizontal pulling movement, compaction movement, broom movement, etc. include. The sweeping operation is, for example, an operation in which the attachment AT is operated to push the bucket 6 forward along the ground, thereby sweeping out earth and sand forward on the back surface of the bucket 6. In the sweeping operation, for example, the attachment AT lowers the boom 4 and opens the arm 5. The horizontal pulling operation is, for example, an operation of smoothing out unevenness on the surface of the ground by operating the attachment AT and moving the toe of the bucket 6 along the ground substantially horizontally toward the user. In the horizontal pulling operation, for example, the attachment AT performs a raising operation of the boom 4 and a closing operation of the arm 5. The rolling operation is, for example, an operation of operating the attachment AT and pressing the back surface of the bucket 6 against the ground. In addition, the compaction operation is performed by pushing the bucket 6 forward along the ground, sweeping the earth and sand to a predetermined position in front with the back of the bucket 6, and then rolling the ground at a predetermined position with the back of the bucket 6. It may also be a pressing motion. In the rolling operation, for example, the attachment AT lowers the boom 4 when pressing against the ground. The broom operation is, for example, an operation in which the upper rotating body 3 is operated and the bucket 6 is rotated left and right while keeping it along the ground. Further, the broom operation may be, for example, an operation of pushing the bucket 6 forward while operating the attachment AT and the upper rotating body 3 and rotating the bucket 6 alternately left and right while keeping the bucket 6 along the ground. In the broom movement, for example, the upper revolving body 3 alternately repeats left and right turning movements. Further, in the broom operation, for example, in addition to the alternating left and right turning operations of the upper revolving structure 3, the boom 4 may be lowered and the arm 5 may be opened, as in the case of the sweeping operation. The predicted probability represents the reliability of the target trajectory of the work part. This is because, as described above, the learned model LM reflects the operation log when the shovel 100 is operated by an expert, and it is considered that the higher the prediction probability, the higher the reliability of the target trajectory of the work part. Furthermore, the predicted probability represents the degree of suitability of the target trajectory of the work area to the shape of the work object around the shovel 100 as an input condition. This is because it is considered that the higher the prediction probability, the higher the possibility that the expert will judge that the candidate motion is suitable for the shape of the work target. For example, the learned model LM is generated for each task such as land leveling work, slope construction work, and embankment work.

The work support unit 302 includes a learned model storage unit 302A, a work target shape acquisition unit 302B, a motion selection unit 302F, a condition setting unit 302G, a trajectory generation unit 302H, a display processing unit 302I, and a motion control unit 302E. including.

The motion selection unit 302F selects (the type of motion) of the shovel 100 from among a plurality of motion candidates in response to input from the user (operator) received through the input device 52. Further, when the excavator 100 is remotely controlled, the motion selection unit 302F selects one of a plurality of motion candidates in response to an input from a user (operator) using the remote operation support device 300, which is received through the communication device 60. The operation of the shovel 100 may be selected from the following.

The condition setting unit 302G sets preconditions regarding the generation of the target trajectory of the work area of the excavator 100 in response to input from the user (operator) received through the input device 52. When the excavator 100 is remotely operated, the condition setting unit 302G sets preconditions regarding the target trajectory of the excavator 100 in response to input from the user (operator) using the remote operation support device 300, which is received through the communication device 60. May be set. Further, the condition setting unit 302G may automatically set the preconditions without depending on input from the user. For example, the condition setting unit 302G uses a history of data of combinations of data regarding the shape of the work target and preconditions set for the shape of the work target as a training data set to create a trained model. Based on this, the preconditions may be automatically set. In this case, the condition setting unit 302G may automatically modify the preconditions that have already been set in response to input from the user.

The precondition is, for example, a point in the topographical shape around the shovel 100 that is a target during the operation of the shovel 100 (hereinafter referred to as a "target point"). The target points include, for example, a target point through which a work part passes when the shovel 100 is in operation, a point corresponding to a place where earth and sand from the bucket 6 is to be discharged when the shovel 100 is in operation, and the like. Further, the preconditions may include the attitude state of the bucket 6 at the target point (the attitude angle of the bucket 6).

The trajectory generation unit 302H generates a target shape of the excavator 100 based on the data acquired by the work target shape acquisition unit 302B, the target shape of the work target, the motion selected by the motion selection unit 302F, and the preconditions set by the condition setting unit 302G. The target trajectory of the work part is generated. The target shape of the work target is, for example, a target construction surface representing a flat or curved surface as a construction target, which is formed by work on the work target (the ground at the work site). The target shape of the work object is set, for example, by inputting parameters representing a plane or a curved surface from the user through the input device 52 or the remote operation support device 300 (input device). Further, the target shape of the work object may be distributed to the excavator 100 from an external device such as the information processing device 200, for example. The trajectory generation unit 302H uses as input data the data acquired by the work target shape acquisition unit 302B, the target shape of the work target, the motion selected by the motion selection unit 302F, and the preconditions set by the condition setting unit 302G. Apply the learned model LM. In addition, the trajectory generation unit 302H uses the target shape of the work target, the motion selected by the motion selection unit 302F, and the data acquired by the work target shape acquisition unit 302B as input data, and generates a target shape of the work part from the learned model LM. The trajectory may also be output. Then, the trajectory generation unit 302H may generate the target trajectory of the work area by optimizing the output target trajectory of the work area using the preconditions set by the condition setting unit 302G.

The display processing unit 302I causes the display device 50A to display a screen related to the generation of the target trajectory of the work area of the excavator 100 (see FIGS. 17 and 18). On the screen related to the generation of the target trajectory of the work area of the excavator 100, the user (operator) can input operations related to the operation of the shovel 100 selected by the operation selection section 302F and the preconditions set by the condition setting section 302G, for example. Contains an operation screen for performing the operations. Further, the screen related to the generation of the target trajectory of the working part of the shovel 100 includes a screen that displays the target trajectory of the working part of the shovel 100, which is generated by the trajectory generating section 302H. Further, when the excavator 100 is remotely operated, the display processing unit 302I may transmit data related to a screen related to generation of a target trajectory of the work area of the excavator 100 to the remote operation support device 300 via the communication device 60. Thereby, the display processing unit 302I can cause the remote operation support device 300 (display device) to display a screen related to the generation of the target trajectory of the work part of the excavator 100.

For example, as shown in FIGS. 17 and 18, the display processing unit 302I displays screens 700 and 800 on the display device 50A.

As shown in FIG. 17, the screen 700 includes images TG, CG, SB, and PB1.

The image TG is an image representing the topographical shape around the excavator 100. The image TG is generated based on data acquired by the work target shape acquisition unit 302B. In this example, the image TG is an image representing the topographic shape around the shovel 100 as seen from a predetermined viewpoint outside the shovel 100. The predetermined viewpoint can be changed, for example, according to input from the user (operator) through the input device 52 or the remote operation support device 300 (input device).

The image CG is an image representing the shovel 100.

The positional relationship between images TG and CG is set to be the same as the actual positional relationship between the topographical shape around the shovel 100 and the shovel 100.

Image SB is an image representing candidate motions that can be selected by motion selection section 302F. In this example, the image SB includes images SB1 to SB5 representing operations of the excavator 100 candidate that may be performed during land leveling work.

The image SB1 is an operation icon for the user to select a combination of the excavation operation and the earth removal operation of the shovel 100.

The image SB2 is an operation icon for the user to select the sweeping operation of the shovel 100.

The image SB3 is an operation icon for the user to select the horizontal pulling operation of the shovel 100.

The image SB4 is an operation icon for the user to select the broom operation of the shovel 100.

The image SB5 is an operation icon for the user to select the rolling operation of the shovel 100.

The user can specify any one of the images SB1 to SB5 through the input device 52 or the remote operation support device 300 (input device), and select the operation of the shovel 100 through the operation selection section 302F. In this example, there is a cursor (matte texture in the figure) on image SB1, and a state is represented in which a combination of excavation and earth removal operations of the shovel 100 is selected.

Note that in addition to images SB1 to SB5, image SB may display operation icons for the user to select other actions different from the actions corresponding to images SB1 to SB5. Further, in place of at least one of the images SB1 to SB5, an operation icon for the user to select another operation different from the images SB1 to SB5 may be displayed on the image SB.

In this example, the image TG includes image regions TG1 and TG2.

The image region TG1 represents a convex portion on the ground around (in front of) the shovel 100.

The image region TG2 represents a recess on the ground around (in front of) the shovel 100.

Furthermore, in this example, the screen 700 includes images P1 and P2 corresponding to the target point.

Image P1 is displayed superimposed on image region TG1.

Image P2 is displayed superimposed on image region TG2.

For example, the user can set the target points corresponding to the images P1 and P2 through the condition setting section 302G by specifying the image regions TG1 and TG2 through the input device 52 or the remote operation support device 300 (input device). I can do it. The user may be able to set the target point in the entire range of the image TG through the input device 52 or the remote operation support device 300 (input device), or may be able to set the target point in the entire range of the image TG. It may be possible to set the target point within a range that can be reached. In the former case, when the target point is set in a range within the entire range of the image TG that can reach the work area of the bucket 6, display content indicating an error (warning) is displayed on the screen 700. It's okay. In the latter case, an image representing a range within the entire range of the image TG that can be reached by the work site of the bucket 6 may be displayed superimposed on the image TG. Further, the user may be able to delete a set target point through the input device 52 or the remote operation support device 300 (input device).

Furthermore, in this example, images RC1 and RC2 are displayed on the screen 700 so as to accompany images P1 and P2, respectively.

The image RC1 is an image representing the preconditions for the attitude angle of the bucket 6 corresponding to the target point corresponding to the image P1.

The image RC2 is an image representing the preconditions for the attitude angle of the bucket 6 corresponding to the target point corresponding to the image P2.

For example, by specifying the images P1 and P2 through the input device 52 or the remote operation support device 300 (input device), the user can set the premise of the posture angle of the bucket 6 corresponding to the images RC1 and RC2 through the condition setting unit 302G. Conditions can be set.

The image PB1 is an icon for operation to cause the trajectory generation unit 302H to generate the trajectory of the work part of the bucket 6 in accordance with the operation selected on the screen 800 and the preconditions set on the screen 800.

For example, the user can generate the target trajectory of the bucket 6 through the trajectory generation unit 302H by operating the image PB1 through the input device 52 or the remote operation support device 300 (input device).

When image PB1 is operated, the display content of display device 50A transitions from screen 700 to screen 800.

The screen 800, like the screen 700, includes images TG, CG, and SB. Further, like the screen 700, the screen 800 includes images P1 and P2. Further, the screen 800 includes images OG, CG1, and PB2.

Image OG is an image representing the target trajectory.

The image CG1 is an image representing the bucket 6 that is displayed along with the image OG corresponding to the target trajectory.

In this example, image OG represents a target trajectory for realizing an operation of scooping up earth and sand at a target point corresponding to image P1 by an excavation operation and discharging it to a target point corresponding to image P2. The image OG may be expressed in such a way that the trajectory portion where the working part of the bucket 6 comes into contact with the earth and sand can be distinguished from the other trajectory portions. For example, in the image OG, the track portion where the working part of the bucket 6 comes into contact with earth and sand and the other track portions are displayed in different colors.

The image PB2 is an icon for operation to reproduce on the screen 800 with a moving image (animation) the operation of moving the work part of the bucket 6 along the target trajectory corresponding to the image OG.

For example, by operating the image PB2 through the input device 52 or the remote operation support device 300 (input device), the user can create a video in which the image CG1 corresponding to the bucket 6 moves along the image OG corresponding to the target trajectory. The image can be displayed on screen 800. Therefore, the user can determine whether or not the target trajectory is appropriate by checking the moving image.

Further, in the moving image, the shape of the work target (topographic shape) after the operation of the shovel 100 to move the bucket 6 along the target trajectory may be displayed. In other words, the screen 800 shows the predicted shape of the work target around the shovel 100 (terrain shape) after the shovel 100 is operated so as to move the bucket 6 along the target trajectory corresponding to the image OG. may be displayed. Thereby, the user can more appropriately determine whether the target trajectory is appropriate by checking the moving image and the predicted change in topographical shape.

When image PB1 is operated, the display content of display device 50A transitions from screen 800 to screen 900.

The screen 900, like the screen 800, includes images TG, CG, and SB. Further, like the screen 800, the screen 900 includes images P1 and P2. Further, the screen 900 includes images OG and CG1. Further, the screen 900 includes an image PB3.

The image PB3 is an operation icon for automatically operating the shovel 100 so as to move the working part of the bucket 6 along the target trajectory corresponding to the image OG.

For example, by operating the image PB3 through the input device 52 or the remote operation support device 300 (input device), the user can control the shovel so that the bucket 6 moves on a target trajectory corresponding to the image OG through the operation control unit 302E. 100 can be operated automatically.

Note that it may be possible to automatically operate the shovel 100 so that the bucket 6 moves along the target trajectory corresponding to the image OG without the user checking the above-mentioned moving image. In this case, in addition to image PB2, an operation icon corresponding to image PB3 is displayed on screen 800.

Returning to FIG. 17, the operation control unit 302E causes the work area of the bucket 6 to move along the target trajectory generated by the trajectory generation unit 302H in response to input from the user (operator) received through the input device 52. The excavator 100 is operated as follows. Specifically, the operation control unit 302E controls the hydraulic control valve 31 while grasping the position of the work area of the bucket 6 from the outputs of the sensors S1 to S5, etc., so as to control the work of the bucket 6 along the target trajectory. The shovel 100 can be operated so that the part moves.

For example, the operation control unit 302E operates the shovel 100 so that the working part of the bucket 6 moves along the target trajectory generated by the trajectory generation unit 302H in response to an instruction to execute an operation from the user. .

In addition, the operation control unit 302E moves the work area of the bucket 6 along the target trajectory generated by the trajectory generation unit 302H in accordance with the operation of the operating device 26 or a remote control signal in a manner that assists the operator's operation. The excavator 100 may be operated as follows.

<Processing>
Next, with reference to FIG. 20, a process related to generation of a target trajectory of a working part of excavator 100 will be described.

FIG. 20 is a flowchart schematically illustrating an example of processing related to generation of a target trajectory of a working part of the excavator 100.

The flowchart in FIG. 20 is repeatedly executed, for example, during operation of a function related to generation of a target trajectory for the work area of the excavator 100. The function related to the generation of the target trajectory of the working part of the excavator 100 is activated (activated) by inputting an instruction from the user, which is received through the input device 52 or the remote operation support device 300 (input device).

As shown in FIG. 20, in step S302 (an example of an acquisition step), the work object shape acquisition unit 302B acquires data regarding the shape of the work object around the shovel 100 from the imaging device 40.

Upon completion of the process in step S302, the controller 30 proceeds to step S304.

In step S304 (an example of a display step), the display processing unit 302I displays a setting screen (for example, screen 700) including an image representing the topographical shape on the display device 50A or in a remote control support system based on the data acquired in step S302. The information is displayed on the device 300 (display device).

Upon completion of the process in step S304, the controller 30 proceeds to step S306.

In step S306, the motion selection unit 302F selects one motion from among the plurality of candidate motions of the shovel 100 in response to the input from the user.

Upon completion of the process in step S306, the controller 30 proceeds to step S308.

In step S308, the condition setting unit 302G (an example of a setting step) sets preconditions regarding the generation of the target trajectory of the work area of the shovel 100 in response to input from the user.

Upon completion of the process in step S308, the controller 30 proceeds to step S310.

Note that the order of steps S306 and S308 may be changed depending on the input from the user.

In step S310 (an example of a generation step), the trajectory generation unit 302H generates a target trajectory for the work area of the shovel 100 for the operation selected in step S306 under the preconditions set in step S308.

Upon completion of the process in step S310, the controller 30 proceeds to step S312.

In step S312, the display processing unit 302I displays the image representing the target trajectory generated in step S310 on the display device 50A or the remote operation support device 300 (display device).

Upon completion of the process in step S312, the controller 30 proceeds to step S314.

In step S314, the controller 30 determines whether an operation input instructing the shovel 100 to perform an operation to move the working part of the bucket 6 along the target trajectory generated in step S312 has been received. . When the controller 30 receives an operation input instructing the execution of the operation of the excavator 100, the process proceeds to step S316, and other operations, specifically, an operation for generating a target trajectory again, are accepted. If so, the process returns to step S306.

In step S316, the operation control unit 302E controls the hydraulic control valve 31 to automatically operate the shovel 100 so that the working part of the bucket 6 moves on the target trajectory generated in the process of the most recent step S310. .

When the process of step S316 is completed, the controller 30 ends the process of the current flowchart.

Incidentally, upon completion of the process in step S316, the shovel 100 (attachment AT) may be in a state where the working part of the bucket 6 is at the end point of the target trajectory. The posture may be returned to the state before the start of the process in step S314.

In this way, in this example, the support device 150 (controller 30) can generate a target trajectory that matches the shape of the work target. Therefore, the working efficiency of the shovel 100 can be improved.

Furthermore, in this example, the support device 150 can generate a target trajectory that meets preconditions such as the target point and the attitude angle of the bucket 6. Therefore, a more appropriate target trajectory can be generated by reflecting the user's judgment and intention based on the shape of the work target.

Furthermore, in this example, the support device 150 can automatically operate the shovel 100 so that the working part of the bucket 6 moves along the generated target trajectory. Therefore, even an inexperienced operator can make the shovel 100 perform appropriate operations, and as a result, the working efficiency of the shovel 100 can be further improved.

[Other examples of functions related to generation of work part trajectory]
Next, another example of a function related to generation of a trajectory of a work part will be described.

The contents of the embodiments of the functions related to the generation of the trajectory of the work part described above may be combined as appropriate, or may be modified or changed.

For example, in the above embodiment, the trajectory generation unit 302H may generate the target trajectory without using the learned model LM. For example, a trajectory that serves as a reference for a work part is defined in advance for each of a plurality of candidate motions, and the trajectory generation section 302H generates a trajectory that serves as a reference for the motion selected by the motion selection section 302F for a work target around the excavator 100. The target trajectory of the work area may be generated by optimizing data on the shape (topographical shape) of the target area and preconditions.

Furthermore, in the above-described embodiments and modifications thereof, data regarding the shape of the work target around the shovel 100 may be acquired based on data from an imaging device, a distance sensor, etc. installed outside the shovel 100. For example, data from an imaging device or a distance sensor installed at a work site is received by the excavator 100 through the communication device 60, so that the work object shape acquisition unit 302B acquires data regarding the shape of the work object around the shovel 100. can be obtained. Further, for example, data from an imaging device or a distance sensor mounted on a drone flying above the work site is received by the excavator 100 through the communication device 60, so that the work object shape acquisition unit 302B acquires information about the surroundings of the excavator 100. It is possible to obtain data regarding the shape of the work target.

Further, the support device 150 according to the above-described embodiment or its modification may generate a trajectory of a working part of a working machine other than the excavator 100.

In addition, in the above-described embodiments and modifications thereof, the support device 150 may have the above-described function of proposing the operation of the working machine in addition to the function of generating the trajectory of the working part of the working machine.

Furthermore, in the above-described embodiments and modifications thereof, part or all of the functions of the support device 150 may be transferred to the remote operation support device 300.

Furthermore, in the above-described embodiments and modifications thereof, part or all of the functions of the support device 150 may be transferred to the information processing device 200.

[Function of support device (2)]
Next, the operation of the support device according to this embodiment will be explained.

For example, a technique is known in which a teaching point is set and a trajectory of a working part of a working machine is generated based on the teaching point (see, for example, Japanese Patent Application Publication No. 2021-50576).

However, for example, in Patent Document 1, it is necessary to set teaching points by operating a shovel and operating a working machine, and as a result, it takes a lot of time and effort to generate the target trajectory of the work part. There is a possibility that it will happen.

In contrast, in this embodiment, the support device includes an acquisition section, a display section, a setting section, and a generation section. The support device is, for example, the support device 150. The acquisition unit is, for example, a work target shape acquisition unit 302B. The display section is, for example, a display device 50A. The setting section is, for example, a condition setting section 302G. The generation unit is, for example, a trajectory generation unit 302H. Specifically, the acquisition unit acquires data regarding the topographic shape of the construction target around the working machine. The working machine is, for example, a shovel 100. Further, the display unit displays an image representing the topographical shape of the construction target based on the data acquired by the acquisition unit. Further, the setting unit sets a point (target point) that is a target during operation of the working machine in the topographical shape of the construction target. Then, the generation unit generates a trajectory of the working part of the working machine based on the data acquired by the acquisition unit, the target shape of the construction target, and the points set by the setting unit.

Thereby, the trajectory of the working part of the working machine can be generated more easily. Further, since the target point is set, it is possible to generate a more appropriate trajectory of the work part that reflects the judgment and intention of the user who visually recognized the shape of the work target around the work machine, for example. Therefore, the working efficiency of the working machine can be improved.

Furthermore, in this embodiment, the support device may include a selection section. Specifically, the selection unit may select one motion from among a plurality of candidate motions of the working machine in response to an input from the user. The generation unit may generate a trajectory of the work part due to one operation of the work machine based on the data acquired by the acquisition unit and the points set by the setting unit.

Thereby, the support device can define the operation of the work machine and generate the trajectory of the work part. Therefore, the working efficiency of the working machine can be further improved.

Further, in the present embodiment, the setting unit determines, in response to input from the user, a point (target point) that is a target during operation of the working machine in the topographical shape around the working machine, and a work part corresponding to the point. You may also set the posture of

Thereby, the support device can generate a more appropriate trajectory of the work part that reflects the user's judgment and intention regarding the posture of the work part. The work efficiency of the working machine can be further improved.

Furthermore, in the present embodiment, the display unit may display an image representing the trajectory generated by the generation unit, superimposed on an image representing the topographical shape around the working machine.

This allows the user to visually check the generated image. Moreover, by simultaneously visually recognizing the terrain shape around the work machine and the generated trajectory, the user can more appropriately judge the validity of the generated trajectory.

Furthermore, in the present embodiment, the display unit may display a moving image of the work part moving along the trajectory generated by the generation unit, superimposed on an image representing the topographical shape around the work machine.

With this, the user can more appropriately judge the validity of the generated trajectory by checking the moving image.

Furthermore, in the present embodiment, the display unit may display an image representing the predicted shape of the work target around the work machine after the work part moves on the trajectory generated by the generation unit.

As a result, the validity of the generated trajectory can be more appropriately determined by checking the shape of the work target after the work part moves along the generated trajectory.

Furthermore, in this embodiment, the support device may include a control unit that automatically operates the work machine based on the trajectory generated by the generation unit in response to user input. The control unit is, for example, the operation control unit 302E.

Thereby, even a relatively inexperienced operator can operate the work part along the target trajectory. Therefore, work efficiency can be improved. Furthermore, user convenience can be improved.

Furthermore, in this embodiment, the working machine may include the above-mentioned operation support device.

Thereby, the work machine can more easily generate the target trajectory and improve work efficiency.

Although the embodiments have been described in detail above, the present disclosure is not limited to such specific embodiments, and various modifications and changes can be made within the scope of the gist described in the claims.

Finally, this application claims priority based on Japanese Patent Application No. 2022-058984 filed on March 31, 2022 and Japanese Patent Application No. 2022-060273 filed on March 31, 2022. , and the entire content of the Japanese patent application is incorporated by reference into this application.

1 Lower traveling body 1C, 1CL, 1CR Crawler 1ML, 1MR Travel hydraulic motor 2 Swing mechanism 2M Swing hydraulic motor 3 Upper rotating body 4 Boom 5 Arm 6 Bucket 7 Boom cylinder 8 Arm cylinder 9 Bucket cylinder 10 Cabin 11 Engine 13 Regulator 14 Main Pump 15 Pilot pump 17 Control valve 26 Operating device 29 Operating pressure sensor 30 Controller 31 Hydraulic control valve 32 Shuttle valve 33 Hydraulic control valve 40 Imaging device 50 Output device 50A Display device 50B Sound output device 52 Input device 60 Communication device 100 Excavator 150 Support Device 200 Information processing device 300 Remote operation support device 301 Operation log providing unit 301A Operation log recording unit 301B Operation log storage unit 301C Operation log transmission unit 302 Work support unit 302A Learned model storage unit 302B Work object shape acquisition unit

302C Estimation unit

302D Proposal section 302E Motion control section 302F Motion selection section 302G Condition setting section 302H Trajectory generation section 302I Display processing section 2001 Motion log acquisition section 2002 Motion log storage section 2003 Teacher data generation section 2004 Machine learning section 2005 Learned model storage section 2006 Distribution section AT Attachment HA Hydraulic actuator LM Learned models S1 to S5 Sensor SYS Operation support system

Claims

an acquisition unit that acquires data regarding the shape of a work target around the work machine;
a proposal unit that proposes to the user an operation among a plurality of candidate operations of the working machine in a predetermined work based on the data acquired by the acquisition unit;
Support equipment.
The data acquired by the acquisition unit is applied to the data acquired by the acquisition unit using a trained model that has undergone machine learning based on training data related to the operation of the work machine by the operation of a relatively highly skilled operator, which is associated with the shape of the work target. an estimator for estimating a motion that matches the shape of a work object around the working machine from among the plurality of candidate motions based on the above,
The proposal unit proposes a motion among the plurality of candidate motions based on the estimation result of the estimation unit.
The support device according to claim 1.
The suggestion unit proposes a plurality of actions among the plurality of candidate actions to the user based on the data acquired by the acquisition unit.
The support device according to claim 1 or 2.
The plurality of motions to be proposed include, among the plurality of candidate motions, motions that have a relatively low degree of adaptation to the shape of the work target around the work machine;
The support device according to claim 3.
The proposal unit proposes a motion among the plurality of candidate motions, along with a degree of compatibility of the motion with a shape of a work object around the working machine, based on the data acquired by the acquisition unit.
The support device according to any one of claims 1 to 4.
The proposal unit proposes a plurality of motions among the plurality of candidate motions, along with a degree of suitability of each of the plurality of motions to the shape of a work object around the working machine, based on the data acquired by the acquisition unit. do,
The support device according to claim 5.
The proposal unit proposes a motion among the plurality of candidate motions, along with a trajectory of a working part of the working machine due to the motion, based on the data acquired by the acquisition unit.
The support device according to any one of claims 1 to 6.
The proposal unit proposes a motion among the plurality of candidate motions, along with a plurality of trajectories of the work part due to the motion, based on the data acquired by the acquisition unit.
The support device according to claim 7.
The proposal unit is configured to determine, based on the data acquired by the acquisition unit, a motion among the plurality of candidate motions, a plurality of trajectories of the work area due to the motion, and a surrounding area of the work machine for each of the plurality of trajectories. We propose the following along with the degree of compatibility with the shape of the work target.
The support device according to claim 8.
Equipped with a display section,
The proposal unit causes the display unit to display a trajectory of the work part according to a motion to be proposed among the plurality of candidate motions, superimposed on an image representing the surroundings of the work machine.
The support device according to any one of claims 7 to 9.
The proposal unit causes the display unit to display in different manners a trajectory portion that contacts the work object and other trajectory portions in the trajectory due to the motion of the proposal target among the plurality of candidate motions.
The support device according to claim 10.
a control unit that causes the work machine to automatically execute the operation proposed by the suggestion unit in response to an instruction input from a user;
The support device according to any one of claims 1 to 11.
The acquisition unit acquires data regarding the shape of the work target around the work machine by predicting the shape of the work target around the work machine after the operation automatically executed by the control unit.
The support device according to claim 12.
a display unit that displays an image representing the shape of the work target based on the data acquired by the acquisition unit;
a setting unit that sets a target point during operation of the working machine in the shape of the work object;
a generation unit that generates a trajectory of a working part of the working machine based on the data acquired by the acquisition unit, the target shape of the work object, and the points set by the setting unit;
The support device according to any one of claims 1 to 13.
an acquisition unit that acquires data regarding the shape of a work target around the work machine;
a proposal unit that proposes to the user an operation among a plurality of candidate operations of the working machine in a predetermined work based on the data acquired by the acquisition unit;
working machine.
support equipment,
an acquisition step of acquiring data regarding the shape of the work object around the work machine;
a proposing step of proposing to the user an action among a plurality of candidate actions of the work machine in a predetermined work based on the data obtained in the obtaining step;
program.