WO2023190877A1 - Dispositif d'assistance, machine de travail, programme - Google Patents

Dispositif d'assistance, machine de travail, programme Download PDF

Info

Publication number
WO2023190877A1
WO2023190877A1 PCT/JP2023/013195 JP2023013195W WO2023190877A1 WO 2023190877 A1 WO2023190877 A1 WO 2023190877A1 JP 2023013195 W JP2023013195 W JP 2023013195W WO 2023190877 A1 WO2023190877 A1 WO 2023190877A1
Authority
WO
WIPO (PCT)
Prior art keywords
work
unit
shovel
image
target
Prior art date
Application number
PCT/JP2023/013195
Other languages
English (en)
Japanese (ja)
Inventor
竜次 續木
大稀 安達
孝介 原
Original Assignee
住友重機械工業株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from JP2022058984A external-priority patent/JP2023150082A/ja
Priority claimed from JP2022060273A external-priority patent/JP2023150920A/ja
Application filed by 住友重機械工業株式会社 filed Critical 住友重機械工業株式会社
Publication of WO2023190877A1 publication Critical patent/WO2023190877A1/fr

Links

Images

Classifications

    • E—FIXED CONSTRUCTIONS
    • E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02F—DREDGING; SOIL-SHIFTING
    • E02F3/00—Dredgers; Soil-shifting machines
    • E02F3/04—Dredgers; Soil-shifting machines mechanically-driven
    • E02F3/28—Dredgers; Soil-shifting machines mechanically-driven with digging tools mounted on a dipper- or bucket-arm, i.e. there is either one arm or a pair of arms, e.g. dippers, buckets
    • E02F3/36—Component parts
    • E02F3/42—Drives for dippers, buckets, dipper-arms or bucket-arms
    • E02F3/43—Control of dipper or bucket position; Control of sequence of drive operations
    • E—FIXED CONSTRUCTIONS
    • E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02F—DREDGING; SOIL-SHIFTING
    • E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
    • E02F9/20—Drives; Control devices
    • E—FIXED CONSTRUCTIONS
    • E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02F—DREDGING; SOIL-SHIFTING
    • E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
    • E02F9/26—Indicating devices

Definitions

  • the present disclosure relates to a support device for a working machine, etc.
  • an acquisition unit that acquires data regarding the shape of a work target around the work machine; a proposal unit that proposes to the user an operation among a plurality of candidate operations of the working machine in a predetermined work based on the data acquired by the acquisition unit; Assistive equipment is provided.
  • an acquisition unit that acquires data regarding the shape of a work target around the work machine; a proposal unit that proposes to the user an operation among a plurality of candidate operations of the working machine in a predetermined work based on the data acquired by the acquisition unit; Working machinery is provided.
  • support equipment an acquisition step of acquiring data regarding the shape of the work object around the work machine; a proposing step of proposing to a user an action among a plurality of candidate actions of the working machine in a predetermined work based on the data obtained in the obtaining step; program will be provided.
  • the work machine can be operated more appropriately.
  • FIG. 1 is a diagram showing an example of a shovel operation support system. It is a top view showing an example of a shovel.
  • FIG. 2 is a diagram showing an example of a configuration related to remote control of an excavator.
  • FIG. 2 is a block diagram showing an example of the hardware configuration of an excavator.
  • FIG. 1 is a diagram illustrating an example of a hardware configuration of an information processing device.
  • FIG. 2 is a functional block diagram showing a first example of a functional configuration regarding a motion proposal function of the excavator operation support system. 2 is a flowchart schematically showing a first example of processing related to a motion suggestion function of the shovel.
  • FIG. 1 is a diagram showing an example of a shovel operation support system. It is a top view showing an example of a shovel.
  • FIG. 2 is a diagram showing an example of a configuration related to remote control of an excavator.
  • FIG. 2 is a block diagram showing an example of the hardware configuration
  • FIG. 3 is a functional block diagram showing a second example of a functional configuration regarding a motion proposal function of the excavator operation support system.
  • 12 is a flowchart schematically showing a second example of processing related to a motion suggestion function of the shovel.
  • FIG. 3 is a diagram illustrating a first example of display content on a display device regarding a shovel motion suggestion function.
  • FIG. 7 is a diagram illustrating a second example of display content on the display device regarding the shovel motion suggestion function. It is a figure which shows the 3rd example of the display content of a display device regarding the motion suggestion function of an excavator. It is a figure which shows the 3rd example of the display content of a display device regarding the motion suggestion function of an excavator.
  • FIG. 2 is a functional block diagram illustrating an example of a functional configuration related to generation of a target trajectory of a working part of an excavator.
  • FIG. 3 is a diagram illustrating an example of a screen related to generation of a target trajectory of a working part of an excavator.
  • FIG. 7 is a diagram illustrating another example of a screen related to generation of a target trajectory of a working part of an excavator.
  • FIG. 7 is a diagram illustrating still another example of a screen related to generation of a target trajectory of a working part of an excavator.
  • 2 is a flowchart schematically showing an example of processing related to generation of a target trajectory of a working part of an excavator.
  • FIG. 1 is a diagram showing an example of the operation support system SYS.
  • the excavator 100 is shown in a left side view.
  • FIG. 2 is a top view showing an example of the shovel 100.
  • FIG. 3 is a diagram showing an example of a configuration related to remote control of an excavator.
  • the direction on the shovel 100 or the direction seen from the shovel 100 may be described by defining the direction in which the attachment AT extends (upward direction in FIG. 2) as seen from the top of the shovel 100 as "front".
  • the operation support system SYS includes an excavator 100 and an information processing device 200.
  • the operation support system SYS uses the information processing device 200 to cooperate with the excavator 100 and provides support regarding the operation of the excavator 100.
  • the number of excavators 100 included in the operation support system SYS may be one or multiple.
  • the excavator 100 is a work machine to which operation support is provided in the operation support system SYS.
  • the excavator 100 includes a lower traveling body 1, an upper rotating body 3, an attachment AT including a boom 4, an arm 5, and a bucket 6, and a cabin 10.
  • the lower traveling body 1 causes the excavator 100 to travel using the crawler 1C.
  • the crawler 1C includes a left crawler 1CL and a right crawler 1CR.
  • the crawler 1CL is hydraulically driven by a travel hydraulic motor 1ML.
  • the crawler 1CL is hydraulically driven by a travel hydraulic motor 1MR.
  • the lower traveling body 1 can self-propel.
  • the upper rotating body 3 is rotatably mounted on the lower traveling body 1 via the rotating mechanism 2.
  • the upper rotating structure 3 turns with respect to the lower traveling structure 1 by hydraulically driving the turning mechanism 2 by the turning hydraulic motor 2M.
  • the boom 4 is attached to the center of the front part of the upper revolving body 3 so that it can be raised and raised about a rotation axis along the left-right direction.
  • the arm 5 is attached to the tip of the boom 4 so as to be rotatable about a rotation axis extending in the left-right direction.
  • the bucket 6 is attached to the tip of the arm 5 so as to be rotatable about a rotation axis extending in the left-right direction.
  • the bucket 6 is an example of an end attachment, and is used, for example, in excavation work.
  • the bucket 6 is attached to the tip of the arm 5 in such a manner that it can be replaced as appropriate depending on the work content of the shovel 100. That is, instead of the bucket 6, a bucket of a different type than the bucket 6, such as a relatively large bucket, a slope bucket, a dredging bucket, etc., may be attached to the tip of the arm 5. Further, an end attachment of a type other than the bucket, such as an agitator, a breaker, a crusher, etc., may be attached to the tip of the arm 5. Furthermore, a preliminary attachment such as a quick coupling or a tiltrotator may be provided between the arm 5 and the end attachment.
  • the boom 4, arm 5, and bucket 6 are hydraulically driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, respectively.
  • the cabin 10 is a control room where an operator boards and operates the shovel 100.
  • the cabin 10 is mounted, for example, on the front left side of the upper revolving body 3.
  • the excavator 100 moves the lower traveling body 1 (that is, the pair of left and right crawlers 1CL, 1CR), the upper revolving body 3, the boom 4, the arm 5, the bucket 6, etc. to operate the driven element of.
  • the lower traveling body 1 that is, the pair of left and right crawlers 1CL, 1CR
  • the upper revolving body 3 that is, the pair of left and right crawlers 1CL, 1CR
  • the boom 4 the arm 5, the bucket 6, etc. to operate the driven element of.
  • the shovel 100 may be configured to be remotely controlled from outside the shovel 100.
  • the interior of the cabin 10 may be unmanned. The following description will proceed on the premise that the operator's operations include at least one of an operator's operation on the operating device 26 by an operator in the cabin 10 and a remote control by an external operator.
  • the remote control includes a mode in which the shovel 100 is operated by an operation input regarding the actuator of the shovel 100 performed by the remote control support device 300.
  • the remote operation support device 300 is provided, for example, in a management center or the like that manages the work of the excavator 100 from the outside. Further, the remote operation support device 300 may be a portable operation terminal, in which case the operator can remotely control the excavator 100 while directly checking the working status of the excavator 100 from around the excavator 100. can.
  • the excavator 100 transmits an image representing the surroundings including the front of the excavator 100 (hereinafter referred to as "surrounding image") based on a captured image output by the imaging device 40 (described later) to the remote operation support device through the communication device 60 (described later). 300. Then, the remote operation support device 300 may display the image (surrounding image) received from the excavator 100 on the display device. Further, various information images (information screens) displayed on the output device 50 (display device 50A) inside the cabin 10 of the excavator 100 may be similarly displayed on the display device of the remote operation support device 300.
  • an operator using the remote operation support device 300 can, for example, remotely operate the shovel 100 while checking the display contents such as an image or information screen showing the surroundings of the shovel 100 displayed on the display device. I can do it. Then, the excavator 100 operates the actuators to operate the lower traveling structure 1, the upper rotating structure 3, and the boom 4 in response to a remote control signal indicating the content of the remote control received from the remote control support device 300 through the communication device 60. , arm 5, and bucket 6 may be driven.
  • the remote control may include, for example, a mode in which the shovel 100 is operated by external voice input or gesture input to the shovel 100 by a person (for example, a worker) around the shovel 100.
  • the excavator 100 receives sounds uttered by surrounding workers, etc. through an audio input device (for example, a microphone), a gesture input device (for example, an imaging device), etc. mounted on the excavator 100. Recognizes gestures etc. performed by Then, the excavator 100 operates the actuator according to the content of the recognized voice or gesture, and moves the lower traveling body 1 (left and right crawlers 1C), the upper rotating body 3, the boom 4, the arm 5, the bucket 6, etc.
  • the driven element may also be driven.
  • a remote monitoring support device having the same functions as remote operation support device 300 may be provided.
  • the remote monitoring support device is, for example, the information processing device 200.
  • the supervisor who is the user of the remote monitoring support device can monitor the working status of the excavator 100 while checking the peripheral image displayed on the display device of the remote monitoring support device. For example, if the supervisor determines that it is necessary from a safety perspective, the supervisor may intervene in the operator's operation of the excavator 100 and bring it to an emergency stop by inputting a predetermined input using the input device of the remote monitoring support device. be able to.
  • the information processing device 200 cooperates with the shovel 100 by communicating with the shovel 100, and provides support regarding the operation of the shovel 100.
  • the information processing device 200 is, for example, a server installed in a management office within the work site of the excavator 100 or a management center that manages the operating status of the excavator 100, etc. located at a location different from the work site of the excavator 100. It is a terminal device for management purposes.
  • the management terminal device may be a stationary terminal device such as a desktop PC (Personal Computer), or a portable terminal device such as a tablet terminal, smartphone, or laptop PC. terminal). In the latter case, workers at the work site, supervisors who supervise work, managers who manage the work site, and the like can carry the portable information processing device 200 and move around the work site.
  • the operator can, for example, bring the portable information processing device 200 into the cabin of the excavator 100.
  • a plurality of information processing apparatuses 200 may be provided depending on the purpose, for example, for remote monitoring, for processing regarding a function of suggesting the operation of the shovel 100 to an operator, which will be described later.
  • the information processing device 200 acquires data regarding the operating state from the excavator 100, for example. Thereby, the information processing device 200 can grasp the operating state of the shovel 100 and monitor whether there is any abnormality in the shovel 100 or the like. Further, the information processing device 200 can display data regarding the operating state of the excavator 100 for the user to confirm through a display device 208, which will be described later.
  • the information processing device 200 transmits to the shovel 100, for example, various data such as programs and reference data used in processing by the controller 30, etc. of the shovel 100.
  • the excavator 100 can perform various processes related to the operation of the excavator 100 using various data downloaded from the information processing device 200.
  • the information processing device 200 performs processing to support, for example, a function related to proposing a motion of the shovel 100 to an operator (hereinafter referred to as "motion proposal function”), which will be described later (see FIG. 6). Details will be described later.
  • motion proposal function a function related to proposing a motion of the shovel 100 to an operator
  • FIG. 4 is a block diagram showing an example of the hardware configuration of shovel 100.
  • the path through which mechanical power is transmitted is a double line
  • the path through which high-pressure hydraulic oil that drives the hydraulic actuator flows is a solid line
  • the path through which pilot pressure is transmitted is a broken line
  • the path through which electrical signals are transmitted is shown. Each route is indicated by a dotted line.
  • the excavator 100 includes a hydraulic drive system for hydraulically driving the driven elements, an operation system for operating the driven elements, a user interface system for exchanging information with the user, a communication system for communicating with the outside, a control system for various controls, etc. Contains each component of.
  • the hydraulic drive system of the excavator 100 includes hydraulic pressure for hydraulically driving each of the driven elements such as the lower traveling body 1 (left and right crawlers 1C), the upper rotating body 3, and the attachment AT, as described above. Includes actuator HA. Further, the hydraulic drive system of the excavator 100 according to the present embodiment includes an engine 11, a regulator 13, a main pump 14, and a control valve 17.
  • the hydraulic actuator HA includes travel hydraulic motors 1ML and 1MR, a swing hydraulic motor 2M, a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, and the like.
  • the excavator 100 part or all of the hydraulic actuator HA may be replaced with an electric actuator.
  • the excavator 100 may be a hybrid excavator or an electric excavator.
  • the engine 11 is the prime mover of the excavator 100 and is the main power source in the hydraulic drive system.
  • the engine 11 is, for example, a diesel engine that uses light oil as fuel.
  • the engine 11 is mounted, for example, at the rear of the upper revolving structure 3.
  • the engine 11 rotates at a predetermined target rotation speed under direct or indirect control by a controller 30, which will be described later, and drives the main pump 14 and the pilot pump 15.
  • the regulator 13 controls (adjusts) the discharge amount of the main pump 14 under the control of the controller 30.
  • the regulator 13 adjusts the angle of the swash plate (hereinafter referred to as "tilt angle") of the main pump 14 in accordance with a control command from the controller 30.
  • the main pump 14 supplies hydraulic oil to the control valve 17 through a high-pressure hydraulic line.
  • the main pump 14 is, for example, mounted at the rear of the upper revolving structure 3, like the engine 11.
  • the main pump 14 is driven by the engine 11 as described above.
  • the main pump 14 is, for example, a variable displacement hydraulic pump, and as described above, the stroke length of the piston is adjusted by adjusting the tilt angle of the swash plate by the regulator 13 under the control of the controller 30, and the stroke length of the piston is adjusted.
  • the flow rate and discharge pressure are controlled.
  • the control valve 17 drives the hydraulic actuator HA in accordance with the contents of the operator's operation on the operating device 26 or remote control, or the operation command corresponding to the automatic operation function.
  • the control valve 17 is mounted, for example, in the center of the upper revolving body 3.
  • the control valve 17 is connected to the main pump 14 via a high-pressure hydraulic line, and controls the hydraulic fluid supplied from the main pump 14 according to an operator's operation or an operation command corresponding to an automatic operation function. , selectively supplying each hydraulic actuator.
  • the control valve 17 includes a plurality of control valves (also referred to as "direction switching valves") that control the flow rate and flow direction of the hydraulic oil supplied from the main pump 14 to each of the hydraulic actuators HA.
  • the operating system of the excavator 100 includes a pilot pump 15, an operating device 26, a hydraulic control valve 31, a shuttle valve 32, and a hydraulic control valve 33.
  • the pilot pump 15 supplies pilot pressure to various hydraulic devices via the pilot line 25.
  • the pilot pump 15 is, for example, mounted at the rear of the upper revolving structure 3, like the engine 11.
  • the pilot pump 15 is, for example, a fixed capacity hydraulic pump, and is driven by the engine 11 as described above.
  • pilot pump 15 may be omitted.
  • the relatively high pressure hydraulic oil discharged from the main pump 14 may be reduced in pressure by a predetermined pressure reducing valve, and then the relatively low pressure hydraulic oil may be supplied as pilot pressure to various hydraulic devices.
  • the operating device 26 is provided near the cockpit of the cabin 10 and is used by the operator to operate various driven elements. Specifically, the operating device 26 is used for an operator to operate the hydraulic actuator HA that drives each driven element, and as a result, the operator operates the driven element to be driven by the hydraulic actuator HA. can be realized.
  • the operating device 26 includes a pedal device and a lever device for operating each driven element (hydraulic actuator HA).
  • the operating device 26 is of a hydraulic pilot type. Specifically, the operating device 26 utilizes hydraulic oil supplied from the pilot pump 15 through the pilot line 25 and a pilot line 25A branching from the pilot line 25, and applies pilot pressure according to the operation content to the pilot line 27A on the secondary side. Output to. Pilot line 27A is connected to one inlet port of shuttle valve 32 and connected to control valve 17 via pilot line 27, which is connected to an outlet port of shuttle valve 32. Thereby, a pilot pressure can be input to the control valve 17 via the shuttle valve 32 in accordance with the operation contents regarding various driven elements (hydraulic actuator HA) in the operating device 26. Therefore, the control valve 17 can drive each hydraulic actuator HA according to the operation performed on the operating device 26 by an operator or the like.
  • the operating device 26 may be electrical.
  • the pilot line 27A, shuttle valve 32, and hydraulic control valve 33 are omitted.
  • the operating device 26 outputs an electrical signal (hereinafter referred to as an "operating signal") according to the content of the operation, and the operating signal is taken into the controller 30.
  • the controller 30 outputs a control command according to the content of the operation signal, that is, a control signal according to the content of the operation on the operating device 26 to the hydraulic control valve 31.
  • pilot pressure corresponding to the operation details of the operating device 26 is inputted from the hydraulic control valve 31 to the control valve 17, and the control valve 17 drives each hydraulic actuator HA according to the operation details of the operating device 26. be able to.
  • control valves built into the control valve 17 and driving the respective hydraulic actuators HA may be of an electromagnetic solenoid type.
  • the operation signal output from the operation device 26 may be directly input to the control valve 17, that is, to an electromagnetic solenoid type control valve.
  • part or all of the hydraulic actuator HA may be replaced with an electric actuator.
  • the controller 30 may output a control command according to the operation content of the operating device 26 or the remote control content specified by the remote control signal to the electric actuator or a driver driving the electric actuator.
  • the operating device 26 may be omitted.
  • the hydraulic control valve 31 is provided for each driven element (hydraulic actuator HA) to be operated by the operating device 26 and for each drive direction of the driven element (hydraulic actuator HA) (for example, the raising direction and lowering direction of the boom 4). . That is, two hydraulic control valves 31 are provided for each double-acting hydraulic actuator HA.
  • the hydraulic control valve 31 is provided, for example, in the pilot line 25B between the pilot pump 15 and the control valve 17, and is configured to be able to change its flow path area (that is, the cross-sectional area through which hydraulic oil can flow). good. Thereby, the hydraulic control valve 31 can output a predetermined pilot pressure to the secondary side pilot line 27B using the hydraulic oil of the pilot pump 15 supplied through the pilot line 25B. Therefore, as shown in FIG.
  • the hydraulic control valve 31 indirectly applies a predetermined pilot pressure according to a control signal from the controller 30 to the control valve through the shuttle valve 32 between the pilot line 27B and the pilot line 27. 17. Therefore, the controller 30 can cause the hydraulic control valve 31 to supply pilot pressure to the control valve 17 according to the operation details of the operating device 26, thereby realizing the operation of the shovel 100 based on the operator's operation. Furthermore, the controller 30 can cause the hydraulic control valve 31 to supply pilot pressure to the control valve 17 according to an operation command corresponding to the automatic operation function, thereby realizing operation of the excavator 100 according to the automatic operation function.
  • the controller 30 may control the hydraulic control valve 31 to realize remote control of the excavator 100, for example. Specifically, the controller 30 outputs to the hydraulic control valve 31 a control signal corresponding to the content of the remote operation specified by the remote operation signal received from the remote operation support device 300, using the communication device 60. Thereby, the controller 30 can cause the hydraulic control valve 31 to supply pilot pressure corresponding to the content of the remote control to the control valve 17, and realize the operation of the shovel 100 based on the operator's remote control.
  • the shuttle valve 32 has two inlet ports and one outlet port, and outputs the hydraulic oil having the higher pilot pressure of the pilot pressures input to the two inlet ports to the outlet port.
  • the shuttle valve 32 is provided for each driven element (hydraulic actuator HA) to be operated by the operating device 26 and for each drive direction of the driven element (hydraulic actuator HA).
  • One of the two inlet ports of the shuttle valve 32 is connected to the pilot line 27A on the secondary side of the operating device 26 (specifically, the above-mentioned lever device or pedal device included in the operating device 26), and the other is It is connected to the pilot line 27B on the secondary side of the hydraulic control valve 31.
  • the outlet port of shuttle valve 32 is connected to the pilot port of the corresponding control valve of control valve 17 through pilot line 27 .
  • the corresponding control valve is a control valve that drives a hydraulic actuator that is operated by the above-mentioned lever device or pedal device connected to one inlet port of the shuttle valve 32. Therefore, these shuttle valves 32 each control the higher of the pilot pressure in the pilot line 27A on the secondary side of the operating device 26 and the pilot pressure on the pilot line 27B on the secondary side of the hydraulic control valve 31, respectively. It can act on the pilot port of the control valve.
  • the controller 30 controls the corresponding control valve by causing the hydraulic control valve 31 to output a pilot pressure higher than the pilot pressure on the secondary side of the operating device 26, regardless of the operator's operation on the operating device 26. be able to. Therefore, the controller 30 can control the operation of the driven elements (the lower traveling body 1, the upper rotating body 3, the attachment AT) and realize a remote control function, regardless of the operation state of the operating device 26 by the operator. .
  • the hydraulic control valve 33 is provided in the pilot line 27A that connects the operating device 26 and the shuttle valve 32.
  • the hydraulic control valve 33 is configured to be able to change its flow path area, for example.
  • the hydraulic control valve 33 operates according to a control signal input from the controller 30.
  • the controller 30 can forcibly reduce the pilot pressure output from the operating device 26 when the operating device 26 is being operated by the operator. Therefore, even when the operating device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator corresponding to the operation of the operating device 26.
  • the controller 30 can reduce the pilot pressure output from the operating device 26 to be lower than the pilot pressure output from the hydraulic control valve 31, for example, even when the operating device 26 is being operated. I can do it.
  • the controller 30 applies a desired pilot pressure to the pilot port of the control valve in the control valve 17, for example, regardless of the operation details of the operating device 26. It can be made to work reliably. Therefore, by controlling the hydraulic control valve 33 in addition to the hydraulic control valve 31, for example, the controller 30 can more appropriately realize the remote control function and automatic operation function of the excavator 100.
  • the user interface system of excavator 100 includes an operating device 26, an output device 50, and an input device 52.
  • the output device 50 outputs various information to the user of the excavator 100 (for example, the operator in the cabin 10 or an external remote control operator) and the people around the excavator 100 (for example, a worker or a driver of a work vehicle). Output.
  • the output device 50 includes a lighting device that outputs various information in a visual manner, a display device 50A (see FIG. 6), and the like.
  • the lighting equipment is, for example, a warning light (indicator lamp) or the like.
  • the display device 50A is, for example, a liquid crystal display or an organic EL (Electroluminescence) display.
  • lighting equipment and a display device 50A may be provided inside the cabin 10 and output various information visually to an operator inside the cabin 10.
  • the lighting equipment and the display device 50A may be provided, for example, on the side surface of the revolving upper structure 3, and may output various information visually to workers and the like around the excavator 100.
  • the output device 50 includes a sound output device 50B (see FIG. 6) that outputs various information in an auditory manner.
  • the sound output device 50B includes, for example, a buzzer, a speaker, and the like.
  • the sound output device 50B is provided, for example, in at least one of the interior and exterior of the cabin 10, and outputs various information in an auditory manner to the operator inside the cabin 10 and the people (workers, etc.) around the excavator 100. It's fine.
  • the output device 50 may include a device that outputs various information using a tactile method such as vibration of the cockpit.
  • the input device 52 accepts various inputs from the user of the excavator 100, and signals corresponding to the accepted inputs are taken into the controller 30.
  • the input device 52 is provided inside the cabin 10 , for example, and receives input from an operator inside the cabin 10 . Further, the input device 52 may be provided, for example, on a side surface of the revolving upper structure 3, and may receive input from a worker or the like around the excavator 100.
  • the input device 52 includes an operation input device that accepts operation input.
  • the operation input device may include a touch panel mounted on the display device, a touch pad installed around the display device, a button switch, a lever, a toggle, a knob switch provided on the operation device 26 (lever device), etc. .
  • the input device 52 may include a voice input device that accepts voice input from the user.
  • the audio input device includes, for example, a microphone.
  • the input device 52 may include a gesture input device that accepts gesture input from the user.
  • the gesture input device includes, for example, an imaging device that captures an image of a gesture performed by a user.
  • the input device 52 may include a biometric input device that receives biometric input from the user.
  • the biometric input includes, for example, input of biometric information such as a user's fingerprint or iris.
  • the communication system of the excavator 100 includes a communication device 60.
  • the communication device 60 is connected to an external communication line and communicates with a device provided separately from the excavator 100.
  • Devices provided separately from the excavator 100 may include devices external to the excavator 100 as well as portable terminal devices (portable terminals) brought into the cabin 10 by the user of the excavator 100.
  • the communication device 60 may include, for example, a mobile communication module that complies with standards such as 4G ( 4th Generation) and 5G ( 5th Generation). Further, the communication device 60 may include, for example, a satellite communication module. Further, the communication device 60 may include, for example, a WiFi communication module, a Bluetooth (registered trademark) communication module, or the like. Furthermore, the communication device 60 may include a plurality of communication devices depending on the communication lines to be connected.
  • the communication device 60 communicates with external devices such as the information processing device 200 and the remote operation support device 300 in the work site through a local communication line built at the work site.
  • the local communication line is, for example, a local 5G (so-called local 5G) mobile communication line built at a work site or a local area network (LAN) using WiFi 6.
  • the communication device 60 communicates with an information processing device 200, a remote operation support device 300, etc. located outside the work site through a wide area communication line that includes the work site, that is, a wide area network (WAN).
  • a wide area network includes, for example, a wide area mobile communication network, a satellite communication network, an Internet network, and the like.
  • control system of excavator 100 includes a controller 30. Further, the control system of the excavator 100 according to the present embodiment includes an operating pressure sensor 29, an imaging device 40, and sensors S1 to S5.
  • the controller 30 performs various controls regarding the shovel 100.
  • the controller 30 may be realized by arbitrary hardware or a combination of arbitrary hardware and software.
  • the controller 30 includes an auxiliary storage device 30A, a memory device 30B, a CPU (Central Processing Unit) 30C, and an interface device 30D, which are connected via a bus B1.
  • auxiliary storage device 30A a memory device 30B
  • CPU Central Processing Unit
  • interface device 30D an interface device 30D
  • the auxiliary storage device 30A is a non-volatile storage means, and stores installed programs as well as necessary files, data, etc.
  • the auxiliary storage device 30A is, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory) or a flash memory.
  • the memory device 30B loads the program in the auxiliary storage device 30A so that it can be read by the CPU 30C.
  • the memory device 30B is, for example, an SRAM (Static Random Access Memory).
  • the CPU 30C executes a program loaded into the memory device 30B, and implements various functions of the controller 30 according to instructions of the program.
  • the interface device 30D functions as a communication interface for connecting to a communication line inside the excavator 100, for example.
  • the interface device 30D may include a plurality of different types of communication interfaces depending on the type of communication line to be connected.
  • the interface device 30D functions as an external interface for reading data from and writing data to the recording medium.
  • the recording medium is, for example, a dedicated tool that is connected to a connector installed inside the cabin 10 with a detachable cable.
  • the recording medium may be a general-purpose recording medium such as an SD memory card or a USB (Universal Serial Bus) memory.
  • programs for realizing various functions of the controller 30 can be provided by, for example, a portable recording medium and installed in the auxiliary storage device 30A of the controller 30. Further, the program may be downloaded from another computer outside the excavator 100 through the communication device 60 and installed in the auxiliary storage device 30A.
  • controller 30 may be realized by another controller (control device). That is, the functions of the controller 30 may be realized in a distributed manner by a plurality of controllers.
  • the operating pressure sensor 29 detects the pilot pressure on the secondary side (pilot line 27A) of the hydraulic pilot type operating device 26, that is, the pilot pressure corresponding to the operating state of each driven element (hydraulic actuator) in the operating device 26. To detect. A detection signal of pilot pressure corresponding to the operating state of each driven element (hydraulic actuator HA) in the operating device 26 by the operating pressure sensor 29 is taken into the controller 30.
  • the operating device 26 is an electric type, the operating pressure sensor 29 is omitted. This is because the controller 30 can grasp the operating state of each driven element through the operating device 26 based on the operating signal taken in from the operating device 26.
  • the imaging device 40 acquires images around the excavator 100.
  • the imaging device 40 also generates three-dimensional data (hereinafter simply referred to as "the object's three-dimensional shape") representing the position and external shape of the object around the shovel 100 within the imaging range (angle of view) based on the acquired image and distance-related data described below. "original data”) may be obtained (generated).
  • the three-dimensional data of objects around the shovel 100 is, for example, coordinate information data of a point group representing the surface of the object, distance image data, and the like.
  • the imaging device 40 includes a camera 40F that images the front of the upper revolving structure 3, a camera 40B that images the rear of the upper revolving structure 3, and a camera 40L that images the left side of the upper revolving structure 3. , and a camera 40R that images the right side of the upper rotating body 3.
  • the imaging device 40 can image the entire circumference of the excavator 100, that is, the range covering the angular direction of 360 degrees, when the excavator 100 is viewed from above.
  • the operator visually recognizes peripheral images such as captured images of the cameras 40B, 40L, and 40R and processed images generated based on the captured images through the output device 50 (display device) and the remote control display device, and rotates the upper part. The left, right, and rear sides of the body 3 can be confirmed.
  • the operator can check the operation of the attachment AT including the bucket 6 by visually checking peripheral images such as images captured by the camera 40F and processed images generated based on the captured images through the remote control display device.
  • the excavator 100 can be remotely controlled.
  • the cameras 40F, 40B, 40L, and 40R may be collectively or individually referred to as "camera 40X.”
  • the camera 40X is, for example, a monocular camera.
  • the camera 40X acquires data regarding distance (depth) in addition to two-dimensional images, such as a stereo camera, a TOF (Time Of Flight) camera, etc. (hereinafter collectively referred to as a "3D camera"). It may be possible.
  • Output data (for example, image data, three-dimensional data of objects around the excavator 100, etc.) of the imaging device 40 (camera 40X) is taken into the controller 30 through a one-to-one communication line or an in-vehicle network.
  • the controller 30 can monitor objects around the excavator 100 based on the output data of the camera 40X.
  • the controller 30 can determine the surrounding environment of the excavator 100 based on the output data of the camera 40X.
  • the controller 30 can determine the posture state of the attachment AT shown in the captured image based on the output data of the camera 40X (camera 40F).
  • the controller 30 can determine the attitude state of the body of the excavator 100 (the upper revolving body 3) based on the output data of the camera 40X, with reference to objects around the excavator 100.
  • the cameras 40F, 40B, 40L, and 40R may be omitted.
  • the camera 40F and the camera 40L may be omitted. This is because it is relatively easy for the operator in the cabin 10 to check the front and left side of the excavator 100.
  • a distance sensor may be provided in the upper revolving body 3. The distance sensor is attached to the upper part of the upper revolving body 3, for example, and acquires data regarding the distance and direction of surrounding objects with respect to the shovel 100 as a reference.
  • the distance sensor may acquire (generate) three-dimensional data (for example, coordinate information data of a point group) of objects around the shovel 100 within the sensing range based on the acquired data.
  • the distance sensor is, for example, LIDAR (Light Detection and Ranging).
  • the distance sensor may be, for example, a millimeter wave radar, an ultrasonic sensor, an infrared sensor, or the like.
  • the sensor S1 is attached to the boom 4 and detects the attitude angle (hereinafter referred to as "boom angle") around the rotation axis of the base end corresponding to the connection part of the boom 4 with the upper revolving structure 3.
  • the sensor S1 includes, for example, a rotary potentiometer, a rotary encoder, an acceleration sensor, an angular acceleration sensor, a 6-axis sensor, an IMU (Inertial Measurement Unit), and the like. The same may apply to the sensors S2 to S4 below. Further, the sensor S1 may include a cylinder sensor that detects the extended/contracted position of the boom cylinder 7. The same may apply to the sensors S2 and S3 below.
  • a detection signal of the boom angle by the sensor S1 is taken into the controller 30. Thereby, the controller 30 can grasp the attitude state of the boom 4.
  • the sensor S2 is attached to the arm 5 and detects the posture angle (hereinafter referred to as "arm angle") around the rotation axis of the base end of the arm 5, which corresponds to the connection part with the boom 4.
  • the arm angle detection signal from the sensor S2 is taken into the controller 30. Thereby, the controller 30 can grasp the posture state of the arm 5.
  • the sensor S3 is attached to the bucket 6 and detects the attitude angle (hereinafter referred to as "arm angle") around the rotation axis of the base end corresponding to the connection part with the arm 5 of the bucket 6.
  • a detection signal of the arm angle by the sensor S3 is taken into the controller 30. Thereby, the controller 30 can grasp the attitude state of the bucket 6.
  • the sensor S4 detects the inclination state of the aircraft body (for example, the upper rotating body 3) with respect to a predetermined reference plane (for example, a horizontal plane).
  • the sensor S4 is attached to the revolving upper structure 3, and measures the inclination angle of the excavator 100 (i.e., the revolving upper structure 3) about two axes in the front-rear direction and the left-right direction (hereinafter, "front-rear inclination angle” and “lateral inclination angle”). ”) is detected.
  • a detection signal corresponding to the inclination angle (front/rear inclination angle and left/right inclination angle) detected by the sensor S ⁇ b>4 is taken into the controller 30 . Thereby, the controller 30 can grasp the tilting state of the aircraft body (upper rotating body 3).
  • the sensor S5 is attached to the revolving upper structure 3 and outputs detection information regarding the turning state of the revolving upper structure 3.
  • the sensor S5 detects, for example, the turning angular velocity and turning angle of the upper rotating body 3.
  • the sensor S5 includes, for example, a gyro sensor, a resolver, a rotary encoder, and the like. Detection information regarding the turning state detected by the sensor S5 is taken into the controller 30. Thereby, the controller 30 can grasp the turning state such as the turning angle of the upper rotating body 3.
  • the senor S4 includes a gyro sensor, a 6-axis sensor, an IMU, etc. that can detect angular velocity around three axes
  • the turning state for example, turning angular velocity
  • sensor S5 may be omitted.
  • at least some of the sensors S1 to S5 may be omitted.
  • FIG. 5 is a block diagram showing an example of the hardware configuration of the information processing device 200.
  • the functions of the information processing device 200 are realized by arbitrary hardware or a combination of arbitrary hardware and software.
  • the information processing device 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU 204, a high-speed arithmetic device 205, a communication interface 206, an input device 207, and and a display device 208.
  • the external interface 201 functions as an interface for reading data from and writing data to the recording medium 201A.
  • the recording medium 201A includes, for example, a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disc), a BD (Blu-ray (registered trademark) Disc), an SD memory card, a USB memory, and the like.
  • the information processing device 200 can read various data used in processing through the recording medium 201A, store it in the auxiliary storage device 202, and install programs that implement various functions.
  • the information processing device 200 may obtain various data and programs used in processing from an external device through the communication interface 206.
  • the auxiliary storage device 202 stores various installed programs, as well as files, data, etc. necessary for various processes.
  • the auxiliary storage device 202 includes, for example, an HDD (Hard Disc Drive), an SSD (Solid State Disc), a flash memory, and the like.
  • the memory device 203 reads and stores the program from the auxiliary storage device 202 when there is an instruction to start the program.
  • the memory device 203 includes, for example, DRAM (Dynamic Random Access Memory) and SRAM.
  • the CPU 204 executes various programs loaded from the auxiliary storage device 202 to the memory device 203, and implements various functions related to the information processing device 200 according to the programs.
  • the high-speed arithmetic unit 205 works in conjunction with the CPU 204 and performs arithmetic processing at a relatively high speed.
  • the high-speed calculation device 205 includes, for example, a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), and the like.
  • the high-speed calculation device 205 may be omitted depending on the required speed of calculation processing.
  • the communication interface 206 is used as an interface for communicably connecting to an external device. Thereby, the information processing device 200 can communicate with an external device such as the excavator 100, for example, through the communication interface 206. Furthermore, the communication interface 206 may have a plurality of types of communication interfaces depending on the communication method with the connected device.
  • the input device 207 receives various inputs from the user.
  • the input device 207 includes, for example, an operation input device that accepts mechanical operation input from the user.
  • the operation input device includes, for example, a button, a toggle, a lever, and the like.
  • the operation input device includes, for example, a touch panel mounted on the display device 208, a touch pad provided separately from the display device 208, and the like.
  • the input device 207 includes, for example, a voice input device that can accept voice input from a user.
  • the voice input device includes, for example, a microphone that can collect the user's voice.
  • the input device 207 includes, for example, a gesture input device that can accept gesture input from the user.
  • the gesture input device includes, for example, a camera that can capture images of the user's gestures.
  • the input device 207 includes, for example, a biometric input device that can accept biometric input from a user.
  • the biometric input device includes, for example, a camera that can acquire image data that includes information about a user's fingerprint or iris.
  • the display device 208 displays information screens and operation screens for the user.
  • display device 208 includes the above-mentioned remote control display device.
  • the display device 208 is, for example, a liquid crystal display, an organic EL (Electroluminescence) display, or the like.
  • the remote operation support device 300 may also be realized by arbitrary hardware or a combination of arbitrary hardware and software, and a similar hardware configuration may be adopted.
  • the remote operation support device 300 is mainly configured with a computer including a CPU, a memory device, an auxiliary storage device, an interface device, an input device, and a display device.
  • the memory device is, for example, SRAM or DRAM.
  • the auxiliary storage device is, for example, an HDD, SSD, EEPROM, flash memory, or the like.
  • the interface device includes an external interface for connecting to an external recording medium and a communication interface for communicating with the outside, such as the shovel 100.
  • the input device includes, for example, a lever-type operation input device.
  • the operator can use the operation input device to perform operation input regarding the actuator of the shovel 100, and the remote operation support device 300 can use the communication interface to transmit a signal corresponding to the operation input to the shovel 100. can. Therefore, the operator can remotely control the excavator 100 using the remote control support device.
  • FIG. 6 is a functional block diagram showing a first example of a functional configuration related to the operation proposal function of the operation support system SYS.
  • the excavator 100 includes a support device 150.
  • Support device 150 supports operation of excavator 100 by an operator.
  • the support device 150 includes a controller 30, an imaging device 40, an output device 50, and a communication device 60.
  • the controller 30 includes an operation log providing unit 301 and a work support unit 302 as functional units.
  • the controller 30 includes only the former of the operation log providing unit 301 and the work support unit 302, and the excavator 100 that includes only the latter. May exist.
  • the former shovel 100 only has the function of acquiring the operation log of the shovel 100 and providing it to the information processing device 200, which is used for the operator operation support function (motion suggestion function) of the latter shovel 100.
  • the same may apply to the second example (FIG. 8) of the motion suggestion function described below.
  • the information processing device 200 includes an operation log acquisition unit 2001, an operation log storage unit 2002, a teacher data generation unit 2003, a machine learning unit 2004, a learned model storage unit 2005, and a distribution unit 2006 as functional units. include.
  • the operation log providing unit 301 is a functional unit that acquires the operation log of the excavator 100, which is the original data for realizing the operation proposal function, and provides it to the information processing device 200.
  • a relatively experienced operator hereinafter referred to as an "expert" for convenience
  • who has a long history of operating the excavator 100 obtains an operation log when operating the excavator 100 and stores it in the information processing device 200. provide.
  • the operation log of the shovel 100 includes data regarding the shape of the work target around the shovel 100 and data regarding the operation of the shovel 100 performed on the shape of the work target.
  • the data regarding the shape of the work target around the shovel 100 is, for example, data regarding the topographical shape of the ground at the work site as the work target of the shovel 100.
  • the data regarding the shape of the work target of the shovel 100 is, for example, image data of the imaging device 40 or three-dimensional data of the work target obtained from the image data.
  • the data regarding the operation of the shovel 100 is, for example, data representing the details of the operator's operation.
  • the data representing the contents of the operator's operation may be, for example, the output data of the operating pressure sensor 29 in the case of the hydraulic pilot type operating device 26 or the output data of the operating device 26 (operation signal data) in the case of the electric operating device 26. It is. Furthermore, the data regarding the operation of the shovel 100 may be data representing the operation state of the shovel 100 actually executed in response to an operation by an operator.
  • the data representing the operating state of the shovel 100 is, for example, the output data of the sensors S1 to S5, or the data related to the posture state of the shovel 100 obtained from the output data of the sensors S1 to S5.
  • the operation log providing section 301 includes an operation log recording section 301A, an operation log storage section 301B, and an operation log transmission section 301C.
  • the operation log recording unit 301A acquires the operation log of the shovel 100 and records it in the operation log storage unit 301B. For example, every time an operation of the shovel 100 is executed, the operation log recording unit 301A records data regarding the shape of the work target around the shovel 100 at the start of execution of the operation or immediately before execution, and data regarding the operation of the shovel 100. is recorded in the operation log storage unit 301B.
  • the operation log storage unit 301B stores operation logs of the shovel 100 in an accumulated manner.
  • the operation log storage unit 301B stores data regarding the shape of a work target around the shovel 100 for each operation of the shovel 100 and data regarding the operation of the shovel 100 in a linked form.
  • the operation log storage unit 301B stores record data representing the correspondence relationship between data regarding the shape of the work target around the shovel 100 and data regarding the operation of the shovel 100 for each operation of the shovel 100.
  • a database of logs may be constructed.
  • operation log in the operation log storage unit 301B that has been transmitted to the information processing device 200 by the operation log transmission unit 301C may be deleted after the fact.
  • the operation log transmission unit 301C transmits the operation log of the shovel 100, which is stored in the operation log storage unit 301B, to the information processing device 200 via the communication device 60.
  • the operation log transmitting unit 301C also transmits to the information processing device 200 data regarding the shape of the work target around the shovel 100 for each operation of the shovel 100 and record data representing the correspondence between the data regarding the operations of the shovel 100. You may.
  • the operation log transmitting unit 301C may store the operation log in the operation log storage unit 301B in response to a signal requesting transmission of the operation log of the excavator 100 (hereinafter referred to as a “transmission request signal”) received from the information processing device 200.
  • the operation log of the shovel 100 that has not been sent yet is sent to the information processing device 200.
  • the operation log transmitting unit 301C may automatically transmit the unsent operation log of the shovel 100, which is stored in the operation log storage unit 301B, to the information processing device 200 at a predetermined timing.
  • the predetermined timing is, for example, when the excavator 100 stops operating (the key switch is turned off) or when the excavator 100 starts operating (the key switch is turned on).
  • the operation log acquisition unit 2001 acquires the operation log of the shovel 100, which is received from the shovel 100.
  • the operation log acquisition unit 2001 acquires the operation log of the excavator 100 by transmitting a transmission request signal to the excavator 100 in response to an operation by the user of the information processing device 200 or automatically at a predetermined timing. Further, the operation log acquisition unit 2001 may acquire an operation log of the excavator 100 that is transmitted from the excavator 100 at a predetermined timing.
  • the operation log storage unit 2002 stores operation logs of the shovel 100 acquired by the operation log acquisition unit 2001 in an accumulated manner.
  • the operation log storage unit 2002 as in the case of the operation log storage unit 301B, data regarding the shape of the work object around the shovel 100 for each operation of the shovel 100 and data regarding the operation of the shovel 100 are linked. remembered in form.
  • the teacher data generation unit 2003 generates teacher data for machine learning based on the operation log of the excavator 100 in the operation log storage unit 2002.
  • the teacher data generation unit 2003 may automatically generate the teacher data by batch processing, or may generate the teacher data in response to input from the user of the information processing apparatus 200.
  • the training data includes data regarding the shape of the work target around the shovel 100 as input data, and data representing the operation of the shovel 100 corresponding to the shape of the work target corresponding to the input data as correct output data (hereinafter referred to as " This is the data in combination with "correct answer data").
  • the correct answer data includes, for example, data representing the type of motion selected from a plurality of candidate motions that can be performed in a predetermined task.
  • the plurality of candidate operations include a sweeping operation, a leveling operation, a compaction operation, a broom operation, and the like.
  • the sweeping operation is, for example, an operation in which the attachment AT is operated to push the bucket 6 forward along the ground, thereby sweeping out earth and sand forward on the back surface of the bucket 6.
  • the attachment AT performs a lowering operation of the boom 4 and an opening operation of the arm 5.
  • the horizontal pulling operation is, for example, an operation to smooth out irregularities on the ground (the surface of the terrain) by operating the attachment AT and moving the toe of the bucket 6 along the ground almost horizontally toward you. be.
  • the attachment AT performs a raising operation of the boom 4 and a closing operation of the arm 5.
  • the rolling operation is, for example, an operation of operating the attachment AT and pressing the back surface of the bucket 6 against the ground.
  • the compaction operation is performed by pushing the bucket 6 forward along the ground, sweeping the earth and sand to a predetermined position in front with the back of the bucket 6, and then rolling the ground at a predetermined position with the back of the bucket 6. It may also be a pressing motion.
  • the attachment AT lowers the boom 4 when pressing against the ground.
  • the broom operation is, for example, an operation in which the upper rotating body 3 is operated and the bucket 6 is rotated left and right while keeping it along the ground. Further, the broom operation may be, for example, an operation of pushing the bucket 6 forward while operating the attachment AT and the upper rotating body 3 and rotating the bucket 6 alternately left and right while keeping the bucket 6 along the ground.
  • the upper revolving body 3 alternately repeats left and right turning movements.
  • the attachment AT may perform a lowering operation of the boom 4 and an opening operation of the arm 5, as in the case of the sweeping operation.
  • the correct data may include, for example, data representing the trajectory of the bucket 6 during operation of the shovel 100.
  • the machine learning unit 2004 performs machine learning on the base learning model based on the set of teacher data generated by the teacher data generation unit 2003 to generate a learned model LM.
  • the learned model LM includes, for example, a neural network such as a DNN (Deep Neural Network).
  • the trained model LM outputs predicted probabilities for each of a plurality of candidate movements to be executed in a predetermined work, using, for example, data regarding the shape of a work target around the shovel 100 as an input condition.
  • This predicted probability represents the reliability of the candidate's motion. This is because, as described above, the learned model LM reflects the operation log when the shovel 100 is operated by an expert, and it is considered that the higher the prediction probability, the higher the reliability of selecting the candidate operation. Further, this predicted probability represents the degree of conformity to the shape of the work target around the shovel 100 as an input condition. This is because it is considered that the higher the prediction probability, the higher the possibility that the expert will judge that the candidate motion is suitable for the shape of the work target.
  • the learned model LM outputs data representing the trajectory of the bucket 6 for each of the plurality of candidate movements (hereinafter referred to as "target trajectory"), using data regarding the shape of the work target around the excavator 100 as an input condition.
  • target trajectory data representing the trajectory of the bucket 6 for each of the plurality of candidate movements
  • the trained model outputs a plurality of pieces of data representing the target trajectory of the bucket 6 for each of the plurality of candidate movements, using data regarding the shape of the work target around the excavator 100 as an input condition, and
  • the predicted probability for each target trajectory may be output.
  • this predicted probability represents the degree of reliability of the target trajectory of the object and the degree of conformity to the shape of the work object around the shovel 100 as an input condition.
  • the learned model LM may be generated for each of a plurality of different tasks. For example, the learned model LM is generated for each task such as land leveling work, slope construction work, and embankment work.
  • the trained model storage unit 2005 stores the trained model LM output by the machine learning unit 2004.
  • the distribution unit 2006 distributes the learned model LM to the excavator 100.
  • the distribution unit 2006 distributes the most recently generated learned model LM to the excavator 100. Further, the distribution unit 2006 may distribute the latest learned model LM in the learned model storage unit 2005 to the shovel 100 in response to a signal received from the excavator 100 requesting distribution of the learned model LM. .
  • the work support unit 302 is a functional unit that supports the work of the shovel 100 operated by the operator.
  • the work support unit 302 includes a learned model storage unit 302A, a work target shape acquisition unit 302B, an estimation unit 302C, and a proposal unit 302D.
  • the trained model storage unit 302A stores the trained model LM distributed from the information processing device 200 and received through the communication device 60.
  • the work target shape acquisition unit 302B acquires data regarding the shape of the work target (terrain shape) around the shovel 100 based on the output of the imaging device 40 and the distance sensor.
  • the estimation unit 302C determines the reliability of the shape of the work work around the shovel 100 among a plurality of candidate movements that can be performed in a predetermined work. and the motion with relatively high fitness. Furthermore, the estimating unit 302C determines the target trajectory of one or more buckets 6 with relatively high reliability and suitability for each of the plurality of candidate movements based on the data regarding the shape of the work target around the shovel 100. It may be estimated.
  • the estimation unit 302C uses the trained model LM to calculate the reliability and the shape of the work target around the shovel 100 using data regarding the shape of the work target around the shovel 100 as an input condition. A motion with a relatively high degree of fitness may be estimated.
  • the estimation unit 302C uses the learned model LM to set one or more goals for the bucket 6 with relatively high reliability and suitability, using data regarding the shape of the work target around the shovel 100 as an input condition. Orbits may be estimated.
  • the proposing section 302D suggests an operation of the excavator 100 that has a relatively high degree of reliability and conformity to the shape of the work target around the excavator 100 through the output device 50 such as the display device 50A.
  • a proposal is made to the operator of cabin 10. Thereby, even an inexperienced operator can select a more appropriate operation according to the shape of the current work target around the shovel 100. Therefore, the operator's convenience can be improved, and the working efficiency of the excavator 100 can be improved.
  • the number of actions suggested to the operator may be one or multiple.
  • the proposal unit 302D notifies the numerical value of the degree of conformity (reliability) to the shape of the work target around the shovel 100 for all or part of the motions of the plurality of candidates, so that An action is proposed (see FIG. 10 below).
  • the proposing unit 302D sends, via the output device 50, a list of buckets 6 in the motion of the proposed target that has a relatively high degree of reliability and conformity to the shape of the work target around the shovel 100.
  • One target trajectory may be proposed.
  • the proposal unit 302D proposes a plurality of target trajectories of the bucket 6 in the motion to be proposed, which have a relatively high degree of reliability and conformity to the shape of the work target around the shovel 100, through the output device 50 of the motion target. You may. This allows the operator to understand multiple target trajectories for the bucket 6 that are more appropriate according to the shape of the work target around the current excavator 100, and to move the excavator so that the operator can realize the one target trajectory that he/she selects. 100 can be operated. Therefore, the working efficiency of the excavator 100 can be improved in a manner that reflects the operator's intention.
  • the proposal unit 302D notifies the numerical value of the degree of conformity (reliability) to the shape of the work target around the shovel 100 for each of the plurality of target trajectories of the motion to be proposed.
  • a target trajectory for bucket 6 is proposed (see FIGS. 11 and 13 described later).
  • the proposal unit 302D provides the operator using the remote operation support device 300 with information on operations with relatively high reliability and suitability and the target trajectory of the bucket 6 via the communication device 60. You may make suggestions.
  • the proposal unit 302D transmits data representing the content of the proposal to the remote operation support device 300 via the communication device 60.
  • the remote operation support device 300 uses a display device, a sound output device, etc. to inform the operator using the remote operation support device 300 of operations with relatively high degrees of reliability and suitability and the target trajectory of the bucket 6. I can make suggestions.
  • FIG. 7 is a flowchart schematically showing a first example of processing related to the motion suggestion function of the shovel 100.
  • the flowchart in FIG. 7 is started when a predetermined input for starting the motion suggestion function is received, for example, through the input device 52 or the input device of the remote operation support device 300. The same may apply to the flowchart of FIG. 9, which will be described later.
  • step S102 an example of an acquisition step
  • the work target shape acquisition unit 302B acquires data regarding the shape of the work target around the shovel 100 based on the output of the imaging device 40.
  • step S102 Upon completion of the process in step S102, the controller 30 proceeds to step S104.
  • step S104 the estimating unit 302C estimates a motion that has a relatively high degree of suitability (reliability) for the shape of the work target around the current shovel 100, based on the data acquired in step S102.
  • step S104 Upon completion of the process in step S104, the controller 30 proceeds to step S106.
  • step S106 an example of a proposing step
  • the proposing unit 302D causes the display device 50A to display the proposed motion among the plurality of candidate motions and the target trajectory of the motion, based on the estimation result in step S104.
  • step S106 Upon completion of the process in step S106, the controller 30 proceeds to step S108.
  • step S108 the controller 30 determines whether the driven element (actuator) has been operated. If the driven element is not being operated, the controller 30 proceeds to step S110, and if the driven element is being operated, the process proceeds to step S112.
  • step S110 the controller 30 determines whether the termination condition is satisfied.
  • the termination condition is, for example, that a predetermined input indicating termination of the motion suggestion function is received from the operator through the input device 52 or the input device of the remote operation support device 300. Further, the termination condition may be that a predetermined input indicating the end of the work is received from the operator through the input device 52 or the input device of the remote operation support device 300. Further, the end condition may be that the controller 30 determines the end of the work based on the captured image of the imaging device 40. If the termination condition is satisfied, the controller 30 terminates the process of the current flowchart, and if the termination condition is not satisfied, the process returns to step S108.
  • step S112 the controller 30 determines whether or not the operation of the driven element corresponding to one operation of the shovel 100 has been completed, based on the operating state of the operating device 26, the operating state of the shovel 100, etc. .
  • the controller 30 can grasp the operating state of the operating device 26, the operating state of the shovel 100, etc. based on the output of the operating pressure sensor 29, the operating signal output from the operating device 26, the outputs of the sensors S1 to S5, etc. . If the operation of the driven element corresponding to one operation of the shovel 100 has been completed, the controller 30 proceeds to step S114, and if the operation has not been completed, the controller 30 waits until the operation is completed (repeat the process of step S112).
  • step S114 the controller 30 determines whether the termination condition is satisfied. If the termination condition is satisfied, the controller 30 terminates the process of the current flowchart, and if the termination condition is not satisfied, the process returns to step S102.
  • the support device 150 displays the operation of the shovel 100 and the target trajectory of the bucket 6 that have a high degree of conformity (reliability) to the shape of the work target around the shovel 100 on the display device 50A.
  • a proposal can be made to the operator through the remote operation support device 300.
  • FIG. 8 is a functional block diagram showing a first example of the functional configuration regarding the operation proposal function of the operation support system SYS.
  • the support device 150 of the excavator 100 includes a controller 30, a hydraulic control valve 31, an imaging device 40, an output device 50 (display device 50A), an input device 52, and a communication device 60. include.
  • the controller 30 includes, as functional units, an operation log providing unit 301 and a work support unit 302, as in the first example described above.
  • the work support unit 302 includes a learned model storage unit 302A, a work target shape acquisition unit 302B, an estimation unit 302C, a proposal unit 302D, and an operation control unit 302E.
  • the operation control unit 302E controls the hydraulic control valve 31 in response to instructions input from the operator received through the input device 52 and the communication device 60, and automatically performs the operation of the excavator 100 suggested to the operator by the suggestion unit 302D. Run it with As a result, the support device 150 can cause the shovel 100 to automatically perform the proposed operation in accordance with the shape of the current work target around the shovel 100, on the premise that instructions are input from the operator. Therefore, even an inexperienced operator can more appropriately operate the shovel 100 in accordance with the shape of the current work target around the shovel 100 by simply inputting instructions. Therefore, the operator's convenience can be further improved, and the working efficiency of the excavator 100 can be further improved.
  • the motion control unit 302E controls the hydraulic control valve 31 in response to an instruction input from the operator, and automatically controls the motion of the shovel 100 to be proposed by the suggestion unit 302D. Run it with Further, for example, when there are a plurality of motions to be proposed, the motion control unit 302E automatically executes one of the motions to be suggested, which is selected by inputting an instruction from the operator. Further, for example, when the proposal unit 302D proposes one target trajectory for the bucket 6, the motion control unit 302E controls the operation of the proposed shovel 100 so that the bucket 6 moves along the proposed target trajectory. Let it run automatically.
  • the operation control unit 302E selects one target trajectory for the bucket 6 from among the plurality of target trajectories, which is selected by inputting an instruction from the operator. Automatically execute the proposed action so that the object moves.
  • the action log recording unit 301A stores an action log including data regarding the shape of the work object acquired by the work object shape acquisition unit 302B, and data representing the actions and target trajectory performed by the action control unit 302E, into the action log storage unit 301B. to be recorded.
  • the operation log transmitter 301C transmits an operation log that includes data regarding the shape of the work target and data representing the target trajectory and the motion of the excavator 100 actually executed by the operator based on the shape of the work target. It can be stored in the log storage unit 301B.
  • the operation log transmitter 301C can then upload the accumulated operation logs to the information processing device 200. Therefore, the machine learning unit 2004 can update the learned model LM by relearning or additionally learning the learned model LM using the operation log.
  • the machine learning unit 2004 compares the trained model LM that has been retrained or additionally trained and the current trained model LM using predetermined evaluation data, and if the evaluation result of the former is high, the learning The trained model LM in the trained model storage unit 2005 may be updated.
  • FIG. 9 is a flowchart schematically showing a second example of processing related to the motion suggestion function of the shovel 100.
  • steps S202 to S206 are the same as steps S102 to S106 in FIG. 7, so the explanation will be omitted.
  • step S206 Upon completion of the process in step S206, the controller 30 proceeds to step S208.
  • step S208 the controller 30 determines whether an input from the operator instructing execution of the proposed action (hereinafter referred to as "input of execution instruction") is received through the input device 52 or the communication device 60. If the input of the execution instruction is not accepted, the controller 30 proceeds to step S210, and if the input is accepted, the process proceeds to step S212.
  • input of execution instruction an input from the operator instructing execution of the proposed action
  • step S210 the controller 30 determines whether the termination condition is satisfied. If the termination condition is satisfied, the controller 30 terminates the process of the current flowchart, and if the termination condition is not satisfied, the process returns to step S208.
  • step S212 the operation control unit 302E controls the hydraulic control valve 31 to automatically execute the operation specified by the input of the execution instruction.
  • the operation control unit 302E controls the shovel 100 specified by the input of the execution instruction so that the bucket 6 moves on the target trajectory specified by the input of the execution instruction. Execute the action.
  • step S212 Upon completion of the process in step S212, the controller 30 proceeds to step S214.
  • step S214 the action log recording unit 301A records an action log including data regarding the shape of the work object acquired by the work object shape acquisition unit 302B, and data representing the actions and target trajectory performed by the action control unit 302E. It is recorded in the operation log storage unit 301B.
  • step S214 Upon completion of the process in step S214, the controller 30 proceeds to step S216.
  • Step S216 The controller 30 determines whether the termination condition is satisfied. If the termination condition is satisfied, the controller 30 terminates the process of the current flowchart, and if the termination condition is not satisfied, the process returns to step S202.
  • the work target shape acquisition unit 302B may acquire data regarding the shape of the work target by predicting a change in the shape of the work target due to the operation of the shovel 100 in the previous process of step S212. .
  • the support device 150 determines the operation of the shovel 100 and the target of the bucket 6 that have a high degree of conformity (reliability) to the shape of the work target around the shovel 100, in accordance with instructions from the operator. Trajectories can be executed automatically.
  • the support device 150 can accumulate an operation log that includes data regarding the shape of the work target around the shovel 100, and data regarding the automatically executed actions of the shovel 100 and the target trajectory. Therefore, the support device 150 can update the learned model LM using the accumulated operation logs.
  • FIGS. 10 to 15 may be displayed on the display device of the remote operation support device 300.
  • FIG. 10 is a diagram showing a first example (screen 1000) of display content on the display device 50A regarding the motion suggestion function of the excavator 100.
  • Screen 1000 includes images 1001 to 1006.
  • Image 1001 is an image representing a work target around shovel 100.
  • the image 1001 is generated based on the output (image data) of the imaging device 40 using a known image processing technique, and is an image of the surroundings of the excavator 100 when viewed from a predetermined viewpoint. This is an image representing the work target (the ground at the work site).
  • Image 1002 is an image schematically representing shovel 100.
  • image 1002 is an image schematically representing shovel 100 when viewed from the same viewpoint as image 1001, and is displayed superimposed on image 1001.
  • the image 1003 is an image that represents, in a list format, the motions to be proposed by the proposal unit 302D among the plurality of candidate motions.
  • the image 1003 includes images 1003A to 1003D representing each row of a sweeping motion, a horizontal pulling motion, a rolling motion, and a broom motion from among a plurality of candidate motions as motions to be proposed.
  • the images 1003A to 1003D express the reliability (degree of suitability) for each of the broom operation, sweeping operation, horizontal pulling operation, and rolling operation. Thereby, the operator can select the operation to be performed by the excavator 100 by his own operation or automatically, taking into consideration the reliability (degree of suitability) from among the proposed operations.
  • the image 1003 may represent only the motion with the highest degree of reliability (degree of suitability) (in this example, the sweeping motion).
  • the suggestion unit 302D may suggest to the operator, through the image 1003, only the motion with the highest degree of reliability (degree of suitability) among the plurality of candidate motions for a predetermined task.
  • the plurality of candidate motions only motions (for example, sweeping motion and horizontal pulling motion) whose reliability (degree of suitability) is equal to or higher than a predetermined standard (for example, 30%) are expressed in the image 1003. Good too.
  • the proposal unit 302D may suggest to the operator only those operations whose reliability (degree of suitability) is equal to or higher than a predetermined standard among the plurality of candidate operations.
  • Image 1004 is an image representing the target trajectory for each proposed motion, which is represented in image 1003.
  • Image 1004 is displayed around image 1002, superimposed on image 1001. Thereby, the operator can easily grasp the target trajectory for each motion to be proposed while comparing the image 1001 representing the state of the ground at the work site around the shovel 100 and the image 1002 representing the shovel 100.
  • Image 1004 includes images 1004A to 1004D.
  • Image 1004A is an image representing the target trajectory of the sweeping operation.
  • Image 1004B is an image representing the target trajectory of the horizontal pulling operation.
  • Image 1004C is an image representing the target trajectory of the rolling operation.
  • Image 1004D is an image representing the target trajectory of the broom motion.
  • the images 1004A to 1004D may be expressed in such a manner that the portion of the target trajectory that contacts the work object (ground) and the other portions can be distinguished.
  • the images 1004A to 1004D may have different colors between the portion of the target trajectory that contacts the work object and the other portions. Thereby, it is possible to assist the sense of perspective on the image 1001 of the images 1004A to 1004D corresponding to the target trajectory.
  • a matte cursor is expressed in the image 1003A corresponding to the sweeping operation.
  • the image 1004A corresponding to the sweeping motion of the image 1004 is expressed by a thicker line than the images 1004B to 1004D corresponding to other motions.
  • a state in which the sweeping operation is selected is expressed.
  • the operator uses the input device 52 to specify any one of the images 1003A to 1003D, thereby selecting one of the sweeping operation, horizontal pulling operation, compaction operation, and broom operation. be able to.
  • the operator can perform any one of the sweeping operation, horizontal pulling operation, rolling operation, and broom operation. can be selected.
  • the image 1005 is an icon for confirming the execution of the action selected by the user (operator) from among the proposed actions.
  • the operator can cause the excavator 100 to automatically perform the selected operation.
  • Image 1006 is an icon for terminating the motion suggestion function of excavator 100. The same applies to images 1106, 1206, 1306, 1406, and 1506, which will be described later.
  • the controller 30 selects a plurality of motions from among a plurality of candidate motions in the ground leveling work based on the reliability (adaptability) of each of the current shape of the work target (terrain shape) around the excavator 100. degree) on the display device 50A. Thereby, the controller 30 can suggest to the operator an operation that has a relatively high degree of reliability (degree of suitability) for the current shape of the work target (terrain shape) around the excavator 100.
  • FIG. 11 is a diagram showing a second example (screen 1100) of display content on the display device 50A regarding the motion suggestion function of the shovel 100.
  • Screen 1100 includes images 1101 to 1106.
  • Image 110 is an image representing a work target around shovel 100.
  • Image 1102 is an image schematically representing shovel 100, similar to image 1002 in FIG.
  • the image 1103 is an image that represents, in a list format, a plurality of target trajectories of the bucket 6 for one motion to be proposed by the proposal unit 302D among the plurality of candidate motions.
  • the image 1103 includes images 1103A to 1103D representing each row of four target trajectories ("sweeping I" to "sweeping IV") of the bucket 6 regarding the sweeping motion as one of the motions to be proposed. included.
  • the reliability (degree of suitability) of each of the four target trajectories of the bucket 6 is expressed in the images 1103A to 1103D.
  • the operator can select the bucket 6 to be executed by the excavator 100 either by the operator's own operation or automatically, considering the reliability among the four target trajectories of the bucket 6 for one proposed operation (sweeping operation).
  • a target trajectory can be selected.
  • the image 1103 may represent only the target trajectory of the bucket 6 (in this example, "sweep I”), which has the highest degree of reliability (degree of suitability).
  • the proposal unit 302D may suggest to the operator, through the image 1103, only the motion with the highest degree of reliability (degree of suitability) among the plurality of target trajectories for one motion to be proposed in a predetermined task.
  • the image 1103 also includes target trajectories (for example, "Sweeping I” and "Sweeping II”) whose reliability (fitness) is higher than a predetermined standard (for example, 30%) among the plurality of target trajectories of the bucket 6. may be expressed only.
  • the proposal unit 302D may propose to the operator only those target trajectories whose reliability (degree of suitability) is equal to or higher than a predetermined standard among the plurality of target trajectories in the bucket 6 for one motion to be proposed. The same may apply to image 1203A, which will be described later.
  • Image 1104 is an image representing four target trajectories for one motion of the proposal target, which is represented in image 1103. Similar to the image 1004, the image 1104 is displayed in the vicinity of the image 1002, superimposed on the image 1001. As a result, the operator can easily determine multiple target trajectories for one motion (sweeping motion) to be proposed while comparing the image 1101 representing the ground condition of the work site around the shovel 100 and the image 1102 representing the shovel 100. can be grasped. Image 1104 includes images 1104A to 1104D.
  • Image 1104A is an image representing the target trajectory of the sweeping operation, which corresponds to image 1103A ("Sweeping I").
  • Image 1104B is an image representing the target trajectory of the sweeping operation, which corresponds to image 1103B ("Sweeping II").
  • Image 1104C is an image representing the target trajectory of the sweeping operation, which corresponds to image 1103C ("Sweeping III").
  • Image 1104D is an image representing the target trajectory of the sweeping operation, which corresponds to image 1103C ("Sweeping IV").
  • a matte cursor is expressed in the image 1103A corresponding to "sweeping operation I".
  • an image 1104A corresponding to the target trajectory of "sweeping operation I" in the image 1104 is expressed by a thicker line than images 1104B to 1104D corresponding to other target trajectories.
  • a state is expressed in which the "sweeping I" target trajectory is selected among the four target trajectories.
  • the operator can select any one of the four target trajectories by specifying any one of the images 1103A to 1103D using the input device 52.
  • the operator can select any one of the four target trajectories for the bucket 6 by specifying any one of the images 1104A to 1104D using the input device 52. .
  • the image 1105 is an icon for executing one of the proposed operations (sweeping operation) so that the bucket 6 moves on a target trajectory selected by the user (operator) from among a plurality of target trajectories.
  • the operator can cause the excavator 100 to automatically perform one of the proposed operations so that the bucket 6 moves along the selected target trajectory. .
  • the controller 30 determines the plurality of target trajectories of the bucket 6 for one operation related to the ground leveling work based on the reliability ( degree of suitability) on the display device 50A. Thereby, the controller 30 can suggest to the operator a plurality of target trajectories for the bucket 6 that have a relatively high degree of reliability (degree of suitability) for the shape of the work target (terrain shape) around the current excavator 100. .
  • ⁇ 3rd example> 12 and 13 are diagrams showing a third example (screens 1200, 1300) of display contents of the display device 50A regarding the motion suggestion function of the excavator 100.
  • Screen 1200 includes images 1201 to 1206.
  • Image 1201 is an image representing a work target around shovel 100.
  • Image 1202 is an image schematically representing shovel 100, similar to image 1002 in FIG.
  • the image 1203 is an image that represents, in a list format, the motions to be proposed by the proposal unit 302D among the plurality of candidate motions.
  • the image 1203 includes images 1203A to 1203D representing each row of a sweeping motion, a horizontal pulling motion, a rolling motion, and a broom motion from among a plurality of candidate motions as motions to be proposed.
  • the images 1203A to 1203D express the reliability (degree of suitability) for each of the broom operation, sweeping operation, horizontal pulling operation, and rolling operation.
  • the image 1203A is an image that represents a plurality of target trajectories of the bucket 6 in a list format for one motion (sweeping motion) to be proposed.
  • image 1203A shows each of the four target trajectories ("sweep I" to "sweep IV") of bucket 6 for the sweep motion with the highest degree of reliability (fitness) among the multiple candidate motions.
  • Images 1203A1 to 1203A4 representing rows of are included.
  • the reliability (degree of suitability) of each of the four target trajectories of the bucket 6 is expressed in the images 1203A1 to 1203A4.
  • Image 1204 is an image representing the target trajectory for each proposed motion, which is expressed in the image 1203.
  • Image 1204 includes images 1204A to 1204D.
  • Image 1204A is an image representing the target trajectory of the sweeping operation. Specifically, the image represents the target trajectory ("sweeping I") with the highest reliability among the target trajectories ("sweeping I” to "sweeping IV") of the bucket 6 corresponding to the sweeping operation.
  • the images 1204B to 1204D are the same as the images 1004B to 1004D in FIG. 10, so a description thereof will be omitted.
  • a matte cursor is expressed in the image 1003A corresponding to the sweeping operation. Furthermore, in this example, the image 1004A corresponding to the sweeping motion of the image 1004 is expressed by a thicker line than the images 1004B to 1004D corresponding to other motions. Thus, in this example, a state in which the sweeping operation is selected is expressed.
  • the image 1205 is an icon for confirming execution of the action selected by the user (operator) from among the proposed actions.
  • the image 1205 is for transitioning to a screen 1300 for selecting four target trajectories of the bucket 6 corresponding to the sweeping motion in a state where the sweeping motion with the highest reliability among the motions to be proposed is selected.
  • This is the icon. That is, when the image 1205 is operated through the input device 52 while the screen 1200 is displayed, the screen shifts to the screen 1300.
  • Screen 1300 includes images 1301 to 1306.
  • the image 1303 is an image that represents, in a list format, the motions to be proposed by the proposal unit 302D among the plurality of candidate motions.
  • the image 1303 includes images 1203A to 1203D representing each row of a sweeping motion, a horizontal pulling motion, a rolling motion, and a broom motion from among a plurality of candidate motions as motions to be proposed. It will be done.
  • the image 1303A is an image that represents a plurality of target trajectories of the bucket 6 in a list format for one motion (sweeping motion) to be proposed. Specifically, the image 1203A shows four target trajectories ("sweep I" to "sweep IV") of bucket 6 for the sweep motion with the highest degree of reliability (fitness) among the multiple candidate motions. Images 1303A1 to 1303A4 representing each row are included.
  • Image 1304 is similar to image 1104 in FIG. 11, and image 1104 is an image representing four target trajectories for one motion of the proposed object, which is expressed in image 1103. Specifically, image 1304 includes images 1304A to 1304D.
  • Images 1304A to 1304D are the same as images 1104A to 1104D in FIG. 11, respectively, so their description will be omitted.
  • a matte cursor is expressed in the image 1303A1 corresponding to "sweeping operation I". Further, in this example, an image 1304A corresponding to the target trajectory of "sweeping operation I" in the image 1304 is expressed by a thicker line than images 1304B to 1304D corresponding to other target trajectories. As a result, in this example, a state is expressed in which the "sweeping I" target trajectory is selected among the four target trajectories.
  • the image 1305 is an icon for executing one of the proposed operations (sweeping operation) so that the bucket 6 moves on a target trajectory selected by the user (operator) from among a plurality of target trajectories.
  • the controller 30 causes the display device 50A to display a plurality of motions among a plurality of candidate motions related to the ground leveling work together with their reliability, and also displays a plurality of targets for the motion with the highest reliability.
  • the trajectory is displayed on the display device 50A.
  • the controller 30 selects a plurality of operations that have a relatively high degree of reliability with respect to the current shape of the work target (terrain shape) around the excavator 100, and a plurality of target trajectories of the bucket 6 for the most reliable operation. can be proposed to the operator.
  • FIG. 14 is a diagram showing a fourth example (screen 1400) of display content on the display device 50A regarding the motion suggestion function of the shovel 100.
  • Screen 1400 includes images 1401 to 1406.
  • Image 1401 is an image representing a work target around shovel 100.
  • Image 1402 is an image schematically representing shovel 100, similar to image 1002 in FIG.
  • the image 1403 is an image that represents, in a list format, the motions to be proposed by the proposal unit 302D among the plurality of candidate motions.
  • the image 1403 includes images 1403A to 1403D representing each row of a sweeping motion, a horizontal pulling motion, a rolling motion, and a broom motion from among a plurality of candidate motions as motions to be proposed. It will be done.
  • image 1404 is an image representing the target trajectory for each proposed motion expressed in the image 1403. Specifically, image 1404 includes images 1404A to 1404D.
  • Images 1404A to 1404D are the same as images 1004A to 1004D in FIG. 10, respectively, so their description will be omitted.
  • the worker W is shown in the image area where the image 1403A of the image 1401 is displayed in a superimposed manner. Therefore, if the most reliable operation (sweeping operation) is selected, there is a possibility that the attachment AT will come too close to the worker W or that the attachment AT will come into contact with the worker W.
  • a satin-textured cursor is expressed in image 1403B corresponding to the horizontal pulling motion
  • image 1404B corresponding to the horizontal pulling motion is thicker than images 1404A, 1404C, and 1404D corresponding to other motions. It is represented by a line. That is, in this example, the operator selects an operation other than the sweeping operation (horizontal pulling operation) through the input device 52, and is about to have the excavator 100 execute it. Thereby, it is possible to prevent the attachment AT from coming too close to the worker W or coming into contact with the worker W.
  • the image 1405 is the same as the image 1005 in FIG. 10, so its description will be omitted.
  • the operator can compare the target trajectory with the worker W at the work site. It is possible to understand the relationship with obstacles. Therefore, the safety of the shovel 100 can be improved while improving the operator's convenience and the working efficiency of the shovel 100.
  • FIG. 15 is a diagram showing a fifth example (screen 1500) of display content on the display device 50A regarding the motion suggestion function of the shovel 100.
  • Screen 1500 includes images 1501 to 1506.
  • Image 1501 is an image representing a work target around shovel 100.
  • Image 1502 is an image schematically representing shovel 100, similar to image 1002 in FIG.
  • the image 1503 is an image that represents, in a list format, the motions to be proposed by the proposal unit 302D among the plurality of candidate motions.
  • the image 1503 includes images 1503A to 1503D representing each row of a horizontal pulling action, a sweeping action, a rolling action, and a broom action from among a plurality of candidate actions as suggested actions. It will be done.
  • image 1504 is an image representing the target trajectory for each proposed motion expressed in the image 1503. Specifically, image 1504 includes images 1504A to 1404D.
  • Images 1404A to 1404D are the same as images 1004A to 1004D in FIG. 10, respectively, so their description will be omitted.
  • Image 1004A is an image representing the target trajectory of the sweeping operation.
  • Image 1004B is an image representing the target trajectory of the horizontal pulling operation.
  • Image 1004C is an image representing the target trajectory of the rolling operation.
  • Image 1004D is an image representing the target trajectory of the broom motion.
  • the image area in which the images 1504A, 1504C, and 1504D of the image 1501 are displayed in a superimposed manner is included in the area 1501A in which the leveling work has already been completed. Therefore, if the operation with the highest reliability (the leveling operation corresponding to image 1504A) is selected, not only will wasteful work be performed on the area where the leveling work has already been completed, but also the impact of the wasteful work will be You will need to do some work to restore it. As a result, there is a possibility that the working efficiency of the shovel 100 will be reduced and the progress of the work at the work site will be delayed.
  • a matte cursor is expressed in the image 1503B corresponding to the sweeping motion, and the image 1504B corresponding to the sweeping motion is drawn with thicker lines than the images 1504A, 1504C, and 1504D corresponding to other motions. It is expressed. That is, in this example, the operator selects a different operation (horizontal pulling operation) from the sweeping operation and is trying to have the excavator 100 execute it. This can prevent the shovel 100 from performing work on an area where work has already been completed. Therefore, it is possible to suppress a decrease in the work efficiency of the shovel 100 and a delay in the progress of work at the work site.
  • an image (such as an image covering the area 1501A with diagonal lines) indicating that the area 1501A is an area where land leveling work has already been completed is displayed in a superimposed manner. This allows the operator to more reliably understand that the area 1501A is an area where the leveling work has been completed.
  • information regarding the area where the work has been completed at the work site is distributed from the information processing device 200 to the shovel 100, for example.
  • the image 1505 is the same as the image 1005 in FIG. 10, so its description will be omitted.
  • the operator can check the target trajectory and the completion status of the work at the work site. It is possible to understand the relationship between Therefore, the operator can select a more appropriate motion or target trajectory according to the completion status of the work at the work site.
  • the proposal unit 302D proposes only the motion to be proposed out of a plurality of candidate motions that can be performed in a predetermined task, and Proposal of a target trajectory corresponding to the motion may be omitted.
  • the proposal unit 302D only selects target trajectories that have a relatively high degree of reliability (degree of conformity) to the shape of the work target around the shovel 100. Instead, motions or target trajectories with relatively low reliability (fitness) may be intentionally proposed.
  • the motion with relatively low reliability (fitness) is the motion of the excavator 100 or the target trajectory of the bucket 6, which is estimated based on the trained model LM and whose reliability (fitness) is lower than a predetermined standard. .
  • a trained model other than the trained model LM that is, a trained model that has been machine learned using a teacher dataset that includes inappropriate motions and target trajectories.
  • the motion of the shovel 100 or the target trajectory of the bucket 6 estimated based on the model may be used.
  • a log to that effect may be recorded in the auxiliary storage device 30A of the controller 30 or the like. Further, the log may include identification information of the operator.
  • the support device 150 (controller 30) can label an operator who has a high probability of selecting an operation of the shovel 100 or a target trajectory of the bucket 6 that has a relatively low degree of reliability (degree of suitability). I can do it. Therefore, a manager or the like at a work site can manage the usage status of the motion suggestion function for each operator by checking logs, labels, etc. after the fact.
  • part or all of the functions of the support device 150 may be transferred to the remote operation support device 300.
  • the functions of the proposal unit 302D are transferred to the remote operation support device 300.
  • the functions of the estimation section 302C may be transferred to the remote operation support device 300.
  • the functions of the work object shape acquisition section 302B may be transferred to the remote operation support device 300.
  • image data of the imaging device 40 is transmitted from the excavator 100 to the remote operation support device 300.
  • part or all of the functions of the support device 150 may be transferred to the information processing device 200.
  • the work target shape acquisition unit 302B is transferred to the information processing device 200.
  • image data of the imaging device 40 is transmitted from the excavator 100 to the information processing device 200.
  • the function of the estimation unit 302C may be transferred to the information processing device 200.
  • the functions of the proposal section 302D may be transferred to the information processing apparatus 200.
  • the support device 150 may perform one or more of the plurality of candidate motions in a predetermined work of a working machine different from the excavator 100.
  • a plurality of actions may be suggested to the operator.
  • another working machine is a forestry machine with a harvester device.
  • the support device 150 may propose an operation for one or more trees to be operated by the harvester device from among operations for a plurality of candidate trees existing in the field.
  • the support device includes an acquisition unit and a proposal unit.
  • the support device is, for example, the support device 150.
  • the working machine is, for example, a shovel 100.
  • the acquisition unit is, for example, a work target shape acquisition unit 302B.
  • the proposal unit is, for example, the proposal unit 302D. Specifically, the acquisition unit acquires data regarding the shape of the work target (for example, topographic shape) around the working machine. Then, the proposal unit proposes to the user a motion among the plurality of candidate motions of the work machine in a predetermined work based on the data regarding the shape of the work target around the work machine acquired by the acquisition unit.
  • the work machine may include the above-mentioned support device.
  • the support device can, for example, suggest to the operator of the work machine an action that is more suitable for the shape of the work target around the work machine, from among a plurality of candidate actions that can be performed in a predetermined work. Therefore, the work machine can be operated more appropriately. Therefore, the working efficiency of the working machine can be improved.
  • the support device includes an estimator.
  • the estimator is, for example, the estimator 302C.
  • the estimation unit uses a trained model that has been machine-learned using training data related to the operation of the work machine by operations of a relatively highly skilled operator, which is associated with the shape of the work target. Based on the data regarding the shape of the work target around the work machine acquired by the acquisition unit, a motion matching the shape of the work target around the work machine is estimated from among the plurality of candidate motions.
  • the learned model is, for example, the learned model LM.
  • the proposal unit may propose a motion among the plurality of candidate motions based on the estimation result of the estimation unit.
  • the support device can use the learned model to suggest a motion that is more suitable for the shape of the work target around the work machine, from among a plurality of candidate motions that can be performed in a predetermined work.
  • the proposal unit may propose a plurality of motions among the plurality of candidate motions to the user based on the data regarding the shape of the work target around the work machine acquired by the acquisition unit. good.
  • the support device can provide options to the operator and encourage the operator to make a decision based on his or her own will. Therefore, by reflecting the operator's judgment, the work machine can be operated more appropriately.
  • the plurality of motions to be proposed may include motions that have a relatively low degree of adaptation to the shape of the work target around the work machine among the plurality of candidate motions. good.
  • the support device can encourage the operator to make a decision based on his or her own will. Therefore, by reflecting the operator's judgment, the work machine can be operated more appropriately.
  • the proposal unit selects a motion among the plurality of candidate motions based on the data regarding the shape of the work target around the work machine acquired by the acquisition unit. It may be proposed together with the degree of conformity to the shape of the work target.
  • the support device can provide the operator with information to decide whether or not to execute the operation. Therefore, the support device can prompt the operator to make more appropriate decisions and cause the work machine to operate more appropriately.
  • the proposal unit selects a plurality of motions among the plurality of candidate motions based on the data regarding the shape of the work target around the work machine acquired by the acquisition unit. It may be proposed together with the degree of conformity to the shape of the work object around the work machine.
  • the support device can provide the operator with a plurality of options regarding the operation of the work machine, and can also provide material for making decisions regarding the selection. Therefore, the support device can prompt the operator to make more appropriate decisions and cause the work machine to operate more appropriately.
  • the proposal unit selects a motion among the plurality of candidate motions based on the data regarding the shape of the work target around the work machine acquired by the acquisition unit. It may be proposed together with the trajectory of the part.
  • the work part is, for example, the bucket 6.
  • the support device can provide the operator with information to decide whether or not to execute the operation. Therefore, the support device can prompt the operator to make more appropriate decisions and cause the work machine to operate more appropriately.
  • the proposing unit selects a motion among the plurality of candidate motions based on the data regarding the shape of the work target around the work machine acquired by the acquisition unit. It may be proposed along with the orbit of .
  • the support device can provide the operator with multiple options regarding the trajectory of the work part that corresponds to the proposed motion. Therefore, the support device can prompt the operator to make more appropriate decisions and cause the work machine to operate more appropriately.
  • the proposing unit selects a motion among the plurality of candidate motions based on the data regarding the shape of the work target around the work machine acquired by the acquisition unit. It may be proposed together with the trajectory of the object and the degree of conformity to the shape of the work object around the working machine for each of the plurality of trajectories.
  • the support device can provide the operator with a plurality of options for the work part, and can also provide information for making a decision on the selection. Therefore, the support device can prompt the operator to make more appropriate decisions and cause the work machine to operate more appropriately.
  • the support device may include a display section.
  • the display section is, for example, a display device 50A.
  • the proposal unit may display the trajectory of the work part according to the motion to be proposed among the plurality of candidate motions on the display unit, superimposed on the image representing the surroundings of the work machine.
  • the support device can prompt the operator to make more appropriate decisions and cause the work machine to operate more appropriately.
  • the proposal unit displays on the display unit the trajectory portion that contacts the work target in the trajectory due to the proposed motion among the plurality of candidate motions and the other trajectory portions in different manners. You may let them.
  • the support device can assist the sense of perspective on the image representing the surroundings of the work machine, allowing the operator to grasp the trajectory more appropriately.
  • the support device includes a control section.
  • the control unit is, for example, the operation control unit 302E.
  • the control unit may cause the work machine to automatically execute the operation proposed by the suggestion unit in response to an input instruction from the user.
  • the acquisition unit obtains data regarding the shape of the work target around the work machine by predicting the shape of the work target around the work machine after the operation automatically executed by the control unit. may be obtained.
  • the support device can propose the operation of the work machine more smoothly when supporting the work of the work machine while repeatedly proposing the operation of the work machine. Therefore, the working efficiency of the working machine can be further improved.
  • FIG. 16 is a block diagram illustrating a first example of a functional configuration related to generation of a target trajectory of a working part of excavator 100.
  • FIG. 17 is a diagram illustrating an example of a screen (screen 700) related to generation of a target trajectory of a working part of excavator 100, which is displayed on display device 50A.
  • FIG. 18 is a diagram illustrating another example of a screen (screen 800) related to generation of a target trajectory of a working part of excavator 100, which is displayed on display device 50A.
  • FIG. 19 is a diagram showing still another example (screen 900) of a screen related to generation of a target trajectory of a working part of excavator 100, which is displayed on display device 50A.
  • the work area of the shovel 100 is, for example, the toe or back of the bucket 6.
  • the excavator 100 includes a support device 150.
  • the support device 150 provides support regarding the work of the excavator 100.
  • the support device 150 includes an operating device 26, a controller 30, an imaging device 40, and an output device 50. Further, when the excavator 100 is remotely controlled, the support device 150 may include the communication device 60.
  • the controller 30 includes an operation log providing unit 301 and a work support unit 302 as functional units.
  • the controller 30 includes only the former of the operation log providing unit 301 and the work support unit 302, and the excavator 100 that includes only the latter. May exist.
  • the former excavator 100 acquires an operation log of the excavator 100 used for the operator operation support function (a function related to generating the trajectory of the work part) of the latter excavator 100, and provides it to the information processing device 200. It has only a function.
  • the information processing device 200 includes an operation log acquisition unit 2001, an operation log storage unit 2002, a teacher data generation unit 2003, a machine learning unit 2004, a learned model storage unit 2005, and a distribution unit 2006 as functional units. include.
  • the operation log providing unit 301 is a functional unit that acquires the operation log of the shovel 100, which is the original data for realizing the function of generating the target trajectory of the working part of the excavator 100, and provides it to the information processing device 200.
  • a relatively experienced operator hereinafter referred to as an "expert" for convenience
  • who has a long history of operating the excavator 100 obtains an operation log when operating the excavator 100 and stores it in the information processing device 200. provide.
  • the operation log of the shovel 100 includes data regarding the shape of the work target around the shovel 100 and data regarding the operation of the shovel 100 performed on the shape of the work target.
  • the data regarding the shape of the work target around the shovel 100 is, for example, data regarding the topographical shape of the ground at the work site as the work target of the shovel 100.
  • the data regarding the shape of the work target of the shovel 100 is, for example, image data of the imaging device 40 or three-dimensional data of the work target obtained from the image data.
  • the data regarding the operation of the shovel 100 is, for example, data representing the details of the operator's operation.
  • the data representing the contents of the operator's operation may be, for example, the output data of the operating pressure sensor 29 in the case of the hydraulic pilot type operating device 26 or the output data of the operating device 26 (operation signal data) in the case of the electric operating device 26. It is. Furthermore, the data regarding the operation of the shovel 100 may be data representing the operation state of the shovel 100 actually executed in response to an operation by an operator.
  • the data representing the operating state of the shovel 100 is, for example, the output data of the sensors S1 to S5, or the data related to the posture state of the shovel 100 obtained from the output data of the sensors S1 to S5.
  • the operation log providing section 301 includes an operation log recording section 301A, an operation log storage section 301B, and an operation log transmission section 301C.
  • the operation log recording unit 301A acquires the operation log of the shovel 100 and records it in the operation log storage unit 301B. For example, every time an operation of the shovel 100 is executed, the operation log recording unit 301A records data regarding the shape of the work target around the shovel 100 at the start of execution of the operation or immediately before execution, and data regarding the operation of the shovel 100. is recorded in the operation log storage unit 301B.
  • the operation log storage unit 301B stores operation logs of the shovel 100 in an accumulated manner.
  • the operation log storage unit 301B stores data regarding the shape of a work target around the shovel 100 for each operation of the shovel 100 and data regarding the operation of the shovel 100 in a linked form.
  • the operation log storage unit 301B stores record data representing the correspondence relationship between data regarding the shape of the work target around the shovel 100 and data regarding the operation of the shovel 100 for each operation of the shovel 100.
  • a database of logs may be constructed.
  • operation log in the operation log storage unit 301B that has been transmitted to the information processing device 200 by the operation log transmission unit 301C may be deleted after the fact.
  • the operation log transmission unit 301C transmits the operation log of the shovel 100, which is stored in the operation log storage unit 301B, to the information processing device 200 via the communication device 60.
  • the operation log transmitting unit 301C also transmits to the information processing device 200 data regarding the shape of the work target around the shovel 100 for each operation of the shovel 100 and record data representing the correspondence between the data regarding the operations of the shovel 100. You may.
  • the operation log transmitting unit 301C may store the operation log in the operation log storage unit 301B in response to a signal requesting transmission of the operation log of the excavator 100 (hereinafter referred to as a “transmission request signal”) received from the information processing device 200.
  • the operation log of the shovel 100 that has not been sent yet is sent to the information processing device 200.
  • the operation log transmitting unit 301C may automatically transmit the unsent operation log of the shovel 100, which is stored in the operation log storage unit 301B, to the information processing device 200 at a predetermined timing.
  • the predetermined timing is, for example, when the excavator 100 stops operating (the key switch is turned off) or when the excavator 100 starts operating (the key switch is turned on).
  • the operation log acquisition unit 2001 acquires the operation log of the shovel 100, which is received from the shovel 100.
  • the operation log acquisition unit 2001 acquires the operation log of the excavator 100 by transmitting a transmission request signal to the excavator 100 in response to an operation by the user of the information processing device 200 or automatically at a predetermined timing. Further, the operation log acquisition unit 2001 may acquire an operation log of the excavator 100 that is transmitted from the excavator 100 at a predetermined timing.
  • the operation log storage unit 2002 stores operation logs of the shovel 100 acquired by the operation log acquisition unit 2001 in an accumulated manner.
  • the operation log storage unit 2002 as in the case of the operation log storage unit 301B, data regarding the shape of the work object around the shovel 100 for each operation of the shovel 100 and data regarding the operation of the shovel 100 are linked. remembered in form.
  • the teacher data generation unit 2003 generates teacher data for machine learning based on the operation log of the excavator 100 in the operation log storage unit 2002.
  • the teacher data generation unit 2003 may automatically generate the teacher data by batch processing, or may generate the teacher data in response to input from the user of the information processing apparatus 200.
  • the training data includes data regarding the shape of the work object around the shovel 100 as input data, and data representing the trajectory (trajectory) of the work part of the shovel 100 corresponding to the input data as correct answer output data (hereinafter referred to as "correct answer"). data").
  • the data representing the trajectory of the working part of the shovel 100 is generated based on the output data of the sensors S1 to S5, which is included in the data regarding the operation of the shovel 100, for example.
  • the machine learning unit 2004 performs machine learning on the base learning model based on the set of teacher data generated by the teacher data generation unit 2003 to generate a learned model LM.
  • the learned model LM includes, for example, a neural network such as a DNN (Deep Neural Network).
  • the learned model LM outputs data representing the target trajectory of the work part of the shovel 100 and predicted probabilities using, for example, data regarding the type of operation of the shovel 100 and the shape of the work object around the shovel 100 as input conditions. .
  • the trained model LM also includes data representing the target trajectory of the work area of the shovel 100, and the types of movements of the shovel 100, such as digging movement, sweeping movement, horizontal pulling movement, compaction movement, broom movement, etc. include.
  • the sweeping operation is, for example, an operation in which the attachment AT is operated to push the bucket 6 forward along the ground, thereby sweeping out earth and sand forward on the back surface of the bucket 6. In the sweeping operation, for example, the attachment AT lowers the boom 4 and opens the arm 5.
  • the horizontal pulling operation is, for example, an operation of smoothing out unevenness on the surface of the ground by operating the attachment AT and moving the toe of the bucket 6 along the ground substantially horizontally toward the user.
  • the attachment AT performs a raising operation of the boom 4 and a closing operation of the arm 5.
  • the rolling operation is, for example, an operation of operating the attachment AT and pressing the back surface of the bucket 6 against the ground.
  • the compaction operation is performed by pushing the bucket 6 forward along the ground, sweeping the earth and sand to a predetermined position in front with the back of the bucket 6, and then rolling the ground at a predetermined position with the back of the bucket 6. It may also be a pressing motion.
  • the attachment AT lowers the boom 4 when pressing against the ground.
  • the broom operation is, for example, an operation in which the upper rotating body 3 is operated and the bucket 6 is rotated left and right while keeping it along the ground. Further, the broom operation may be, for example, an operation of pushing the bucket 6 forward while operating the attachment AT and the upper rotating body 3 and rotating the bucket 6 alternately left and right while keeping the bucket 6 along the ground.
  • the upper revolving body 3 alternately repeats left and right turning movements.
  • the boom 4 may be lowered and the arm 5 may be opened, as in the case of the sweeping operation.
  • the predicted probability represents the reliability of the target trajectory of the work part. This is because, as described above, the learned model LM reflects the operation log when the shovel 100 is operated by an expert, and it is considered that the higher the prediction probability, the higher the reliability of the target trajectory of the work part. Furthermore, the predicted probability represents the degree of suitability of the target trajectory of the work area to the shape of the work object around the shovel 100 as an input condition.
  • the learned model LM is generated for each task such as land leveling work, slope construction work, and embankment work.
  • the trained model storage unit 2005 stores the trained model LM output by the machine learning unit 2004.
  • the distribution unit 2006 distributes the learned model LM to the excavator 100.
  • the distribution unit 2006 distributes the most recently generated learned model LM to the excavator 100. Further, the distribution unit 2006 may distribute the latest learned model LM in the learned model storage unit 2005 to the shovel 100 in response to a signal received from the excavator 100 requesting distribution of the learned model LM. .
  • the work support unit 302 is a functional unit that supports the work of the shovel 100 operated by the operator.
  • the work support unit 302 includes a learned model storage unit 302A, a work target shape acquisition unit 302B, a motion selection unit 302F, a condition setting unit 302G, a trajectory generation unit 302H, a display processing unit 302I, and a motion control unit 302E. including.
  • the trained model storage unit 302A stores the trained model LM distributed from the information processing device 200 and received through the communication device 60.
  • the work target shape acquisition unit 302B acquires data regarding the shape of the work target (terrain shape) around the shovel 100 based on the output of the imaging device 40 and the distance sensor.
  • the motion selection unit 302F selects (the type of motion) of the shovel 100 from among a plurality of motion candidates in response to input from the user (operator) received through the input device 52. Further, when the excavator 100 is remotely controlled, the motion selection unit 302F selects one of a plurality of motion candidates in response to an input from a user (operator) using the remote operation support device 300, which is received through the communication device 60.
  • the operation of the shovel 100 may be selected from the following.
  • the condition setting unit 302G sets preconditions regarding the generation of the target trajectory of the work area of the excavator 100 in response to input from the user (operator) received through the input device 52.
  • the condition setting unit 302G sets preconditions regarding the target trajectory of the excavator 100 in response to input from the user (operator) using the remote operation support device 300, which is received through the communication device 60. May be set.
  • the condition setting unit 302G may automatically set the preconditions without depending on input from the user.
  • the condition setting unit 302G uses a history of data of combinations of data regarding the shape of the work target and preconditions set for the shape of the work target as a training data set to create a trained model. Based on this, the preconditions may be automatically set. In this case, the condition setting unit 302G may automatically modify the preconditions that have already been set in response to input from the user.
  • the precondition is, for example, a point in the topographical shape around the shovel 100 that is a target during the operation of the shovel 100 (hereinafter referred to as a "target point").
  • the target points include, for example, a target point through which a work part passes when the shovel 100 is in operation, a point corresponding to a place where earth and sand from the bucket 6 is to be discharged when the shovel 100 is in operation, and the like.
  • the preconditions may include the attitude state of the bucket 6 at the target point (the attitude angle of the bucket 6).
  • the trajectory generation unit 302H generates a target shape of the excavator 100 based on the data acquired by the work target shape acquisition unit 302B, the target shape of the work target, the motion selected by the motion selection unit 302F, and the preconditions set by the condition setting unit 302G.
  • the target trajectory of the work part is generated.
  • the target shape of the work target is, for example, a target construction surface representing a flat or curved surface as a construction target, which is formed by work on the work target (the ground at the work site).
  • the target shape of the work object is set, for example, by inputting parameters representing a plane or a curved surface from the user through the input device 52 or the remote operation support device 300 (input device).
  • the target shape of the work object may be distributed to the excavator 100 from an external device such as the information processing device 200, for example.
  • the trajectory generation unit 302H uses as input data the data acquired by the work target shape acquisition unit 302B, the target shape of the work target, the motion selected by the motion selection unit 302F, and the preconditions set by the condition setting unit 302G. Apply the learned model LM.
  • the trajectory generation unit 302H uses the target shape of the work target, the motion selected by the motion selection unit 302F, and the data acquired by the work target shape acquisition unit 302B as input data, and generates a target shape of the work part from the learned model LM.
  • the trajectory may also be output. Then, the trajectory generation unit 302H may generate the target trajectory of the work area by optimizing the output target trajectory of the work area using the preconditions set by the condition setting unit 302G.
  • the display processing unit 302I causes the display device 50A to display a screen related to the generation of the target trajectory of the work area of the excavator 100 (see FIGS. 17 and 18).
  • the user can input operations related to the operation of the shovel 100 selected by the operation selection section 302F and the preconditions set by the condition setting section 302G, for example.
  • the screen related to the generation of the target trajectory of the working part of the shovel 100 includes a screen that displays the target trajectory of the working part of the shovel 100, which is generated by the trajectory generating section 302H.
  • the display processing unit 302I may transmit data related to a screen related to generation of a target trajectory of the work area of the excavator 100 to the remote operation support device 300 via the communication device 60. Thereby, the display processing unit 302I can cause the remote operation support device 300 (display device) to display a screen related to the generation of the target trajectory of the work part of the excavator 100.
  • the display processing unit 302I displays screens 700 and 800 on the display device 50A.
  • the screen 700 includes images TG, CG, SB, and PB1.
  • the image TG is an image representing the topographical shape around the excavator 100.
  • the image TG is generated based on data acquired by the work target shape acquisition unit 302B.
  • the image TG is an image representing the topographic shape around the shovel 100 as seen from a predetermined viewpoint outside the shovel 100.
  • the predetermined viewpoint can be changed, for example, according to input from the user (operator) through the input device 52 or the remote operation support device 300 (input device).
  • the image CG is an image representing the shovel 100.
  • the positional relationship between images TG and CG is set to be the same as the actual positional relationship between the topographical shape around the shovel 100 and the shovel 100.
  • Image SB is an image representing candidate motions that can be selected by motion selection section 302F.
  • the image SB includes images SB1 to SB5 representing operations of the excavator 100 candidate that may be performed during land leveling work.
  • the image SB1 is an operation icon for the user to select a combination of the excavation operation and the earth removal operation of the shovel 100.
  • the image SB2 is an operation icon for the user to select the sweeping operation of the shovel 100.
  • the image SB3 is an operation icon for the user to select the horizontal pulling operation of the shovel 100.
  • the image SB4 is an operation icon for the user to select the broom operation of the shovel 100.
  • the image SB5 is an operation icon for the user to select the rolling operation of the shovel 100.
  • the user can specify any one of the images SB1 to SB5 through the input device 52 or the remote operation support device 300 (input device), and select the operation of the shovel 100 through the operation selection section 302F.
  • the input device 52 or the remote operation support device 300 input device
  • the operation selection section 302F selects the operation of the shovel 100 through the operation selection section 302F.
  • there is a cursor (matte texture in the figure) on image SB1 and a state is represented in which a combination of excavation and earth removal operations of the shovel 100 is selected.
  • image SB may display operation icons for the user to select other actions different from the actions corresponding to images SB1 to SB5. Further, in place of at least one of the images SB1 to SB5, an operation icon for the user to select another operation different from the images SB1 to SB5 may be displayed on the image SB.
  • the image TG includes image regions TG1 and TG2.
  • the image region TG1 represents a convex portion on the ground around (in front of) the shovel 100.
  • the image region TG2 represents a recess on the ground around (in front of) the shovel 100.
  • the screen 700 includes images P1 and P2 corresponding to the target point.
  • Image P1 is displayed superimposed on image region TG1.
  • Image P2 is displayed superimposed on image region TG2.
  • the user can set the target points corresponding to the images P1 and P2 through the condition setting section 302G by specifying the image regions TG1 and TG2 through the input device 52 or the remote operation support device 300 (input device). I can do it.
  • the user may be able to set the target point in the entire range of the image TG through the input device 52 or the remote operation support device 300 (input device), or may be able to set the target point in the entire range of the image TG. It may be possible to set the target point within a range that can be reached. In the former case, when the target point is set in a range within the entire range of the image TG that can reach the work area of the bucket 6, display content indicating an error (warning) is displayed on the screen 700.
  • an image representing a range within the entire range of the image TG that can be reached by the work site of the bucket 6 may be displayed superimposed on the image TG. Further, the user may be able to delete a set target point through the input device 52 or the remote operation support device 300 (input device).
  • images RC1 and RC2 are displayed on the screen 700 so as to accompany images P1 and P2, respectively.
  • the image RC1 is an image representing the preconditions for the attitude angle of the bucket 6 corresponding to the target point corresponding to the image P1.
  • the image RC2 is an image representing the preconditions for the attitude angle of the bucket 6 corresponding to the target point corresponding to the image P2.
  • the user can set the premise of the posture angle of the bucket 6 corresponding to the images RC1 and RC2 through the condition setting unit 302G. Conditions can be set.
  • the image PB1 is an icon for operation to cause the trajectory generation unit 302H to generate the trajectory of the work part of the bucket 6 in accordance with the operation selected on the screen 800 and the preconditions set on the screen 800.
  • the user can generate the target trajectory of the bucket 6 through the trajectory generation unit 302H by operating the image PB1 through the input device 52 or the remote operation support device 300 (input device).
  • the screen 800 like the screen 700, includes images TG, CG, and SB. Further, like the screen 700, the screen 800 includes images P1 and P2. Further, the screen 800 includes images OG, CG1, and PB2.
  • Image OG is an image representing the target trajectory.
  • the image CG1 is an image representing the bucket 6 that is displayed along with the image OG corresponding to the target trajectory.
  • image OG represents a target trajectory for realizing an operation of scooping up earth and sand at a target point corresponding to image P1 by an excavation operation and discharging it to a target point corresponding to image P2.
  • the image OG may be expressed in such a way that the trajectory portion where the working part of the bucket 6 comes into contact with the earth and sand can be distinguished from the other trajectory portions.
  • the track portion where the working part of the bucket 6 comes into contact with earth and sand and the other track portions are displayed in different colors.
  • the image PB2 is an icon for operation to reproduce on the screen 800 with a moving image (animation) the operation of moving the work part of the bucket 6 along the target trajectory corresponding to the image OG.
  • the user can create a video in which the image CG1 corresponding to the bucket 6 moves along the image OG corresponding to the target trajectory.
  • the image can be displayed on screen 800. Therefore, the user can determine whether or not the target trajectory is appropriate by checking the moving image.
  • the shape of the work target (topographic shape) after the operation of the shovel 100 to move the bucket 6 along the target trajectory may be displayed.
  • the screen 800 shows the predicted shape of the work target around the shovel 100 (terrain shape) after the shovel 100 is operated so as to move the bucket 6 along the target trajectory corresponding to the image OG. may be displayed.
  • the user can more appropriately determine whether the target trajectory is appropriate by checking the moving image and the predicted change in topographical shape.
  • the screen 900 like the screen 800, includes images TG, CG, and SB. Further, like the screen 800, the screen 900 includes images P1 and P2. Further, the screen 900 includes images OG and CG1. Further, the screen 900 includes an image PB3.
  • the image PB3 is an operation icon for automatically operating the shovel 100 so as to move the working part of the bucket 6 along the target trajectory corresponding to the image OG.
  • the user can control the shovel so that the bucket 6 moves on a target trajectory corresponding to the image OG through the operation control unit 302E. 100 can be operated automatically.
  • the operation control unit 302E causes the work area of the bucket 6 to move along the target trajectory generated by the trajectory generation unit 302H in response to input from the user (operator) received through the input device 52.
  • the excavator 100 is operated as follows. Specifically, the operation control unit 302E controls the hydraulic control valve 31 while grasping the position of the work area of the bucket 6 from the outputs of the sensors S1 to S5, etc., so as to control the work of the bucket 6 along the target trajectory.
  • the shovel 100 can be operated so that the part moves.
  • the operation control unit 302E operates the shovel 100 so that the working part of the bucket 6 moves along the target trajectory generated by the trajectory generation unit 302H in response to an instruction to execute an operation from the user. .
  • the operation control unit 302E moves the work area of the bucket 6 along the target trajectory generated by the trajectory generation unit 302H in accordance with the operation of the operating device 26 or a remote control signal in a manner that assists the operator's operation.
  • the excavator 100 may be operated as follows.
  • FIG. 20 is a flowchart schematically illustrating an example of processing related to generation of a target trajectory of a working part of the excavator 100.
  • the flowchart in FIG. 20 is repeatedly executed, for example, during operation of a function related to generation of a target trajectory for the work area of the excavator 100.
  • the function related to the generation of the target trajectory of the working part of the excavator 100 is activated (activated) by inputting an instruction from the user, which is received through the input device 52 or the remote operation support device 300 (input device).
  • step S302 an example of an acquisition step
  • the work object shape acquisition unit 302B acquires data regarding the shape of the work object around the shovel 100 from the imaging device 40.
  • step S302 Upon completion of the process in step S302, the controller 30 proceeds to step S304.
  • step S304 an example of a display step
  • the display processing unit 302I displays a setting screen (for example, screen 700) including an image representing the topographical shape on the display device 50A or in a remote control support system based on the data acquired in step S302.
  • the information is displayed on the device 300 (display device).
  • step S304 Upon completion of the process in step S304, the controller 30 proceeds to step S306.
  • step S306 the motion selection unit 302F selects one motion from among the plurality of candidate motions of the shovel 100 in response to the input from the user.
  • step S306 Upon completion of the process in step S306, the controller 30 proceeds to step S308.
  • step S308 the condition setting unit 302G (an example of a setting step) sets preconditions regarding the generation of the target trajectory of the work area of the shovel 100 in response to input from the user.
  • step S308 Upon completion of the process in step S308, the controller 30 proceeds to step S310.
  • steps S306 and S308 may be changed depending on the input from the user.
  • step S310 (an example of a generation step), the trajectory generation unit 302H generates a target trajectory for the work area of the shovel 100 for the operation selected in step S306 under the preconditions set in step S308.
  • step S310 Upon completion of the process in step S310, the controller 30 proceeds to step S312.
  • step S312 the display processing unit 302I displays the image representing the target trajectory generated in step S310 on the display device 50A or the remote operation support device 300 (display device).
  • step S312 Upon completion of the process in step S312, the controller 30 proceeds to step S314.
  • step S314 the controller 30 determines whether an operation input instructing the shovel 100 to perform an operation to move the working part of the bucket 6 along the target trajectory generated in step S312 has been received. .
  • the controller 30 receives an operation input instructing the execution of the operation of the excavator 100, the process proceeds to step S316, and other operations, specifically, an operation for generating a target trajectory again, are accepted. If so, the process returns to step S306.
  • step S316 the operation control unit 302E controls the hydraulic control valve 31 to automatically operate the shovel 100 so that the working part of the bucket 6 moves on the target trajectory generated in the process of the most recent step S310. .
  • step S316 the controller 30 ends the process of the current flowchart.
  • the shovel 100 may be in a state where the working part of the bucket 6 is at the end point of the target trajectory.
  • the posture may be returned to the state before the start of the process in step S314.
  • the support device 150 (controller 30) can generate a target trajectory that matches the shape of the work target. Therefore, the working efficiency of the shovel 100 can be improved.
  • the support device 150 can generate a target trajectory that meets preconditions such as the target point and the attitude angle of the bucket 6. Therefore, a more appropriate target trajectory can be generated by reflecting the user's judgment and intention based on the shape of the work target.
  • the support device 150 can automatically operate the shovel 100 so that the working part of the bucket 6 moves along the generated target trajectory. Therefore, even an inexperienced operator can make the shovel 100 perform appropriate operations, and as a result, the working efficiency of the shovel 100 can be further improved.
  • the trajectory generation unit 302H may generate the target trajectory without using the learned model LM.
  • a trajectory that serves as a reference for a work part is defined in advance for each of a plurality of candidate motions, and the trajectory generation section 302H generates a trajectory that serves as a reference for the motion selected by the motion selection section 302F for a work target around the excavator 100.
  • the target trajectory of the work area may be generated by optimizing data on the shape (topographical shape) of the target area and preconditions.
  • data regarding the shape of the work target around the shovel 100 may be acquired based on data from an imaging device, a distance sensor, etc. installed outside the shovel 100.
  • data from an imaging device or a distance sensor installed at a work site is received by the excavator 100 through the communication device 60, so that the work object shape acquisition unit 302B acquires data regarding the shape of the work object around the shovel 100.
  • data from an imaging device or a distance sensor mounted on a drone flying above the work site is received by the excavator 100 through the communication device 60, so that the work object shape acquisition unit 302B acquires information about the surroundings of the excavator 100. It is possible to obtain data regarding the shape of the work target.
  • the support device 150 may generate a trajectory of a working part of a working machine other than the excavator 100.
  • the support device 150 may have the above-described function of proposing the operation of the working machine in addition to the function of generating the trajectory of the working part of the working machine.
  • part or all of the functions of the support device 150 may be transferred to the remote operation support device 300.
  • part or all of the functions of the support device 150 may be transferred to the information processing device 200.
  • Patent Document 1 it is necessary to set teaching points by operating a shovel and operating a working machine, and as a result, it takes a lot of time and effort to generate the target trajectory of the work part. There is a possibility that it will happen.
  • the support device includes an acquisition section, a display section, a setting section, and a generation section.
  • the support device is, for example, the support device 150.
  • the acquisition unit is, for example, a work target shape acquisition unit 302B.
  • the display section is, for example, a display device 50A.
  • the setting section is, for example, a condition setting section 302G.
  • the generation unit is, for example, a trajectory generation unit 302H.
  • the acquisition unit acquires data regarding the topographic shape of the construction target around the working machine.
  • the working machine is, for example, a shovel 100.
  • the display unit displays an image representing the topographical shape of the construction target based on the data acquired by the acquisition unit.
  • the setting unit sets a point (target point) that is a target during operation of the working machine in the topographical shape of the construction target. Then, the generation unit generates a trajectory of the working part of the working machine based on the data acquired by the acquisition unit, the target shape of the construction target, and the points set by the setting unit.
  • the trajectory of the working part of the working machine can be generated more easily. Further, since the target point is set, it is possible to generate a more appropriate trajectory of the work part that reflects the judgment and intention of the user who visually recognized the shape of the work target around the work machine, for example. Therefore, the working efficiency of the working machine can be improved.
  • the support device may include a selection section.
  • the selection unit may select one motion from among a plurality of candidate motions of the working machine in response to an input from the user.
  • the generation unit may generate a trajectory of the work part due to one operation of the work machine based on the data acquired by the acquisition unit and the points set by the setting unit.
  • the support device can define the operation of the work machine and generate the trajectory of the work part. Therefore, the working efficiency of the working machine can be further improved.
  • the setting unit determines, in response to input from the user, a point (target point) that is a target during operation of the working machine in the topographical shape around the working machine, and a work part corresponding to the point. You may also set the posture of
  • the support device can generate a more appropriate trajectory of the work part that reflects the user's judgment and intention regarding the posture of the work part.
  • the work efficiency of the working machine can be further improved.
  • the display unit may display an image representing the trajectory generated by the generation unit, superimposed on an image representing the topographical shape around the working machine.
  • the display unit may display a moving image of the work part moving along the trajectory generated by the generation unit, superimposed on an image representing the topographical shape around the work machine.
  • the user can more appropriately judge the validity of the generated trajectory by checking the moving image.
  • the display unit may display an image representing the predicted shape of the work target around the work machine after the work part moves on the trajectory generated by the generation unit.
  • the validity of the generated trajectory can be more appropriately determined by checking the shape of the work target after the work part moves along the generated trajectory.
  • the support device may include a control unit that automatically operates the work machine based on the trajectory generated by the generation unit in response to user input.
  • the control unit is, for example, the operation control unit 302E.
  • the working machine may include the above-mentioned operation support device.
  • the work machine can more easily generate the target trajectory and improve work efficiency.

Landscapes

  • Engineering & Computer Science (AREA)
  • Mining & Mineral Resources (AREA)
  • Civil Engineering (AREA)
  • General Engineering & Computer Science (AREA)
  • Structural Engineering (AREA)
  • Mechanical Engineering (AREA)
  • Operation Control Of Excavators (AREA)

Abstract

L'invention concerne une technologie qui permet de faire fonctionner de manière plus appropriée une machine de travail. Selon un mode de réalisation de la présente divulgation, un dispositif de commande (30) comprend une unité d'acquisition de forme cible de travail (302B) qui acquiert des données concernant la forme d'une cible de travail dans l'environnement d'une pelle (100), une unité d'inférence (302C) qui infère des opérations qui sont adaptées à la forme de la cible de travail dans l'environnement de la pelle (100) parmi une pluralité d'opérations candidates pour la pelle (100) pour un travail prescrit sur la base des données concernant la forme de la cible de travail dans l'environnement de la pelle (100) à l'aide d'un modèle entraîné (LM) qui a été entraîné par apprentissage automatique à l'aide de données d'entraînement concernant des opérations de la pelle (100) en fonctionnement par un opérateur relativement hautement compétent associé aux formes de cibles de travail, et une unité de suggestion (302D) qui suggère une ou plusieurs opérations parmi la pluralité d'opérations candidates à un utilisateur sur la base des résultats d'inférence provenant de l'unité d'inférence (302C).
PCT/JP2023/013195 2022-03-31 2023-03-30 Dispositif d'assistance, machine de travail, programme WO2023190877A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
JP2022-058984 2022-03-31
JP2022-060273 2022-03-31
JP2022058984A JP2023150082A (ja) 2022-03-31 2022-03-31 支援装置、作業機械、プログラム
JP2022060273A JP2023150920A (ja) 2022-03-31 2022-03-31 支援装置、作業機械、プログラム

Publications (1)

Publication Number Publication Date
WO2023190877A1 true WO2023190877A1 (fr) 2023-10-05

Family

ID=88202867

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/013195 WO2023190877A1 (fr) 2022-03-31 2023-03-30 Dispositif d'assistance, machine de travail, programme

Country Status (1)

Country Link
WO (1) WO2023190877A1 (fr)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016160718A (ja) * 2015-03-05 2016-09-05 株式会社日立製作所 軌道生成装置および作業機械
WO2017115810A1 (fr) * 2015-12-28 2017-07-06 住友建機株式会社 Pelle
JP2020041354A (ja) * 2018-09-12 2020-03-19 株式会社小松製作所 作業機械、制御装置、および制御方法
JP2020122389A (ja) * 2015-12-28 2020-08-13 住友建機株式会社 ショベル及びショベル用のシステム
JP2021188362A (ja) * 2020-05-29 2021-12-13 株式会社小松製作所 掘削計画作成装置、作業機械および掘削計画作成方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016160718A (ja) * 2015-03-05 2016-09-05 株式会社日立製作所 軌道生成装置および作業機械
WO2017115810A1 (fr) * 2015-12-28 2017-07-06 住友建機株式会社 Pelle
JP2020122389A (ja) * 2015-12-28 2020-08-13 住友建機株式会社 ショベル及びショベル用のシステム
JP2020041354A (ja) * 2018-09-12 2020-03-19 株式会社小松製作所 作業機械、制御装置、および制御方法
JP2021188362A (ja) * 2020-05-29 2021-12-13 株式会社小松製作所 掘削計画作成装置、作業機械および掘削計画作成方法

Similar Documents

Publication Publication Date Title
JP7358349B2 (ja) 掘削機、情報処理装置
US10508417B2 (en) Construction information display device and method for displaying construction information
CN112867831B (zh) 挖土机
JP7301875B2 (ja) ショベル、ショベルの制御装置
EP3489422B1 (fr) Commande d'une machine de terrassement
KR102659076B1 (ko) 쇼벨
WO2021054436A1 (fr) Excavatrice
US11346086B1 (en) Machine learning for optimizing tool path planning in autonomous earth moving vehicles
CN112955610A (zh) 挖土机、信息处理装置、信息处理方法、信息处理程序、终端装置、显示方法、显示程序
JP7439053B2 (ja) ショベル及びショベルの管理装置
US10968601B2 (en) Controlling earthmoving machine
WO2021020464A1 (fr) Excavatrice
JP2023174887A (ja) 作業機械、情報処理装置
WO2023190877A1 (fr) Dispositif d'assistance, machine de travail, programme
US20210312721A1 (en) Reproduction device, analysis assistance system, and reproduction method
JP2023150082A (ja) 支援装置、作業機械、プログラム
JP2023150920A (ja) 支援装置、作業機械、プログラム
WO2024034624A1 (fr) Dispositif d'assistance, engin de chantier, système d'assistance et programme
WO2023190843A1 (fr) Dispositif d'assistance, engin de chantier et programme
WO2024111596A1 (fr) Machine de travail, dispositif de traitement d'informations et programme
WO2020166673A1 (fr) Excavatrice
JP2022154722A (ja) ショベル
JP2024072055A (ja) 作業機械、情報処理装置、作業支援システム、プログラム
JP2024076741A (ja) 作業機械、情報処理装置、プログラム
JP2021155995A (ja) ショベルの支援装置、ショベルの管理装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23780885

Country of ref document: EP

Kind code of ref document: A1