WO2024034624A1 - Dispositif d'assistance, engin de chantier, système d'assistance et programme - Google Patents

Dispositif d'assistance, engin de chantier, système d'assistance et programme Download PDF

Info

Publication number
WO2024034624A1
WO2024034624A1 PCT/JP2023/029009 JP2023029009W WO2024034624A1 WO 2024034624 A1 WO2024034624 A1 WO 2024034624A1 JP 2023029009 W JP2023029009 W JP 2023029009W WO 2024034624 A1 WO2024034624 A1 WO 2024034624A1
Authority
WO
WIPO (PCT)
Prior art keywords
work
trajectory
input
shovel
excavator
Prior art date
Application number
PCT/JP2023/029009
Other languages
English (en)
Japanese (ja)
Inventor
竜次 續木
孝介 原
Original Assignee
住友重機械工業株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 住友重機械工業株式会社 filed Critical 住友重機械工業株式会社
Publication of WO2024034624A1 publication Critical patent/WO2024034624A1/fr

Links

Images

Classifications

    • E—FIXED CONSTRUCTIONS
    • E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02F—DREDGING; SOIL-SHIFTING
    • E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
    • E02F9/20—Drives; Control devices
    • E—FIXED CONSTRUCTIONS
    • E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02F—DREDGING; SOIL-SHIFTING
    • E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
    • E02F9/26—Indicating devices

Definitions

  • the present disclosure relates to a support device and the like.
  • Patent Document 1 For example, a technique has been disclosed that generates a trajectory of a working part of a working machine when performing a predetermined operation according to the surrounding environment of the working machine, a target shape, etc. (see Patent Document 1).
  • Patent Document 1 a trajectory is generated that is a target for the toe of a shovel bucket when performing an excavation operation.
  • an input section that accepts input from the user; Determining specifications regarding combinations of a plurality of operations of different types of the working machine in response to input from the input unit, and generating a trajectory of the working part of the working machine by a composite movement combining the plurality of movements.
  • a display unit that displays a first operation screen of Assistive equipment is provided.
  • an input section that accepts input from the user; Determining specifications regarding combinations of a plurality of different types of motions of the working machine in response to input from the input unit, and generating a trajectory of a working part of the working machine by a composite motion that combines the plurality of motions.
  • a display unit that displays a first operation screen; Working machinery is provided.
  • the support device includes: an input section that accepts input from the user; In response to input from the input unit, specifications regarding a combination of a plurality of different types of operations of the working machine are determined, and a trajectory of a working part of the working machine is generated by a composite operation that combines the plurality of operations.
  • a display unit that displays a first operation screen for Support systems are provided.
  • An information processing device having an input section and a display section, Determining specifications regarding combinations of a plurality of operations of different types of the working machine in response to input from the input unit, and generating a trajectory of the working part of the working machine by a composite movement combining the plurality of movements. displaying a first operation screen of on the display unit; program will be provided.
  • FIG. 1 is a diagram showing an example of an operation support system. It is a top view showing an example of a shovel.
  • FIG. 2 is a diagram showing an example of a configuration related to remote control of an excavator.
  • FIG. 2 is a block diagram showing an example of the hardware configuration of an excavator. It is a diagram showing an example of the hardware configuration of a remote operation support device.
  • FIG. 2 is a functional block diagram illustrating an example of a functional configuration related to generation of a target trajectory of a working part of an excavator in the operation support system.
  • FIG. 2 is a diagram illustrating an example of the structure of a model for extracting feature amounts of trajectories of working parts of an excavator.
  • FIG. 2 is a diagram illustrating an example of the structure of a model for extracting feature amounts of trajectories of working parts of an excavator.
  • FIG. 2 is a diagram illustrating an example of the structure of a model that generates a trajectory of a working part in a compound motion of an excavator.
  • FIG. 2 is a diagram illustrating an example of the structure of a model that generates a trajectory of a working part in a compound motion of an excavator.
  • FIG. 3 is a diagram showing an example of a trajectory of a working part in soil leveling work with an excavator. It is a figure which shows an example of the screen showing the topographical shape around an excavator.
  • FIG. 1 is a diagram illustrating an example of the structure of a model for extracting feature amounts of trajectories of working parts of an excavator.
  • FIG. 2 is a diagram illustrating an example of the structure of a model that generates a trajectory of a working
  • FIG. 3 is a diagram illustrating a first example of a setting screen for specifications regarding a combination of a plurality of basic operations in a compound operation of an excavator. It is a figure which shows the 2nd example of the setting screen of the specification regarding the combination of the several basic operation
  • 2 is a flowchart schematically showing an example of processing related to generating a trajectory of a working part of an excavator.
  • FIG. 1 is a diagram showing an example of the operation support system SYS.
  • the excavator 100 is shown in a left side view.
  • FIG. 2 is a top view showing an example of the shovel 100.
  • FIG. 3 is a diagram showing an example of a configuration related to remote control of the excavator 100.
  • the direction on the shovel 100 or the direction seen from the shovel 100 may be described by defining the direction in which the attachment AT extends (upward direction in FIG. 2) as seen from the top of the shovel 100 as "front".
  • the operation support system SYS includes an excavator 100 and an information processing device 200.
  • the operation support system SYS uses the information processing device 200 to cooperate with the excavator 100 and provides support regarding the operation of the excavator 100.
  • the number of excavators 100 included in the operation support system SYS may be one or multiple.
  • the excavator 100 is a work machine to which operation support is provided in the operation support system SYS.
  • the work machine included in the operation support system SYS and the target of operation support may be another work machine different from the excavator 100.
  • the other work machine is a work machine equipped with a work attachment, specifically a crane, a forklift, a road machine, etc.
  • the road machine is, for example, an asphalt finisher.
  • the excavator 100 includes a lower traveling body 1, an upper rotating body 3, an attachment AT including a boom 4, an arm 5, and a bucket 6, and a cabin 10.
  • the lower traveling body 1 causes the excavator 100 to travel using the crawler 1C.
  • the crawler 1C includes a left crawler 1CL and a right crawler 1CR.
  • the crawler 1CL is hydraulically driven by a travel hydraulic motor 1ML.
  • the crawler 1CL is hydraulically driven by a travel hydraulic motor 1MR.
  • the lower traveling body 1 can self-propel.
  • the upper rotating body 3 is rotatably mounted on the lower traveling body 1 via the rotating mechanism 2.
  • the upper rotating structure 3 turns with respect to the lower traveling structure 1 by hydraulically driving the turning mechanism 2 by the turning hydraulic motor 2M.
  • the boom 4 is attached to the center of the front part of the upper revolving body 3 so that it can be raised and raised about a rotation axis along the left-right direction.
  • the arm 5 is attached to the tip of the boom 4 so as to be rotatable about a rotation axis extending in the left-right direction.
  • the bucket 6 is attached to the tip of the arm 5 so as to be rotatable about a rotation axis extending in the left-right direction.
  • the bucket 6 is an example of an end attachment, and is used, for example, in excavation work or land leveling work.
  • the bucket 6 is attached to the tip of the arm 5 in a manner that it can be replaced as appropriate depending on the work content of the shovel 100. That is, instead of the bucket 6, a bucket of a different type than the bucket 6, such as a relatively large bucket, a slope bucket, a dredging bucket, etc., may be attached to the tip of the arm 5. Further, an end attachment of a type other than a bucket, such as an agitator, a breaker, a crusher, etc., may be attached to the tip of the arm 5. Further, a preliminary attachment such as a quick coupling or a tilt rotator may be provided between the arm 5 and the end attachment.
  • the boom 4, arm 5, and bucket 6 are hydraulically driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, respectively.
  • the cabin 10 is a control room where an operator boards and operates the shovel 100.
  • the cabin 10 is mounted, for example, on the front left side of the upper revolving body 3.
  • the excavator 100 moves the lower traveling body 1 (that is, the pair of left and right crawlers 1CL, 1CR), the upper revolving body 3, the boom 4, the arm 5, the bucket 6, etc. to operate the driven element of.
  • the lower traveling body 1 that is, the pair of left and right crawlers 1CL, 1CR
  • the upper revolving body 3 that is, the pair of left and right crawlers 1CL, 1CR
  • the boom 4 the arm 5, the bucket 6, etc. to operate the driven element of.
  • the shovel 100 may be configured to be remotely controlled from outside the shovel 100.
  • the interior of the cabin 10 may be unmanned. The following description will proceed on the premise that the operator's operations include at least one of an operator's operation on the operating device 26 by an operator in the cabin 10 and a remote control by an external operator.
  • the remote control includes a mode in which the shovel 100 is operated by an operation input regarding the actuator of the shovel 100 performed by the remote control support device 300.
  • the remote operation support device 300 may be provided separately from the information processing device 200 or may be the information processing device 200.
  • the remote operation support device 300 is provided, for example, in a management center or the like that manages the work of the excavator 100 from the outside. Further, the remote operation support device 300 may be a portable operation terminal, in which case the operator can remotely control the excavator 100 while directly checking the working status of the excavator 100 from around the excavator 100. can.
  • the excavator 100 transmits an image representing the surroundings including the front of the excavator 100 (hereinafter referred to as "surrounding image") based on a captured image output by the imaging device 40 (described later) to the remote operation support device through the communication device 60 (described later). 300. Further, the excavator 100 transmits the captured image output by the imaging device 40 to the remote operation support device 300 through the communication device 60, and the remote operation support device 300 processes the captured image received from the excavator 100 and generates a peripheral image. You may. Then, the remote operation support device 300 may cause the display device to display a surrounding image representing the surroundings including the area in front of the excavator 100.
  • various information images (information screens) displayed on the output device 50 (display device 50A) inside the cabin 10 of the excavator 100 may be similarly displayed on the display device of the remote operation support device 300.
  • an operator using the remote operation support device 300 can, for example, remotely operate the shovel 100 while checking the display contents such as an image or information screen showing the surroundings of the shovel 100 displayed on the display device. Can be done.
  • the excavator 100 operates the actuators to operate the lower traveling structure 1, the upper rotating structure 3, and the boom 4 in response to a remote control signal indicating the content of the remote control received from the remote control support device 300 through the communication device 60.
  • arm 5, and bucket 6 may be driven.
  • the remote control may include, for example, a mode in which the shovel 100 is operated by external voice input or gesture input to the shovel 100 by a person (for example, a worker) around the shovel 100.
  • the excavator 100 receives sounds uttered by surrounding workers, etc. through an audio input device (for example, a microphone), a gesture input device (for example, an imaging device), etc. mounted on the excavator 100. Recognizes gestures etc. performed by Then, the excavator 100 operates the actuator according to the content of the recognized voice or gesture, and moves the lower traveling body 1 (left and right crawlers 1C), the upper rotating body 3, the boom 4, the arm 5, the bucket 6, etc.
  • the driven element may also be driven.
  • a remote monitoring support device having the same functions as remote operation support device 300 may be provided.
  • the remote monitoring support device is, for example, the information processing device 200.
  • the supervisor who is the user of the remote monitoring support device can monitor the working status of the excavator 100 while checking the peripheral image displayed on the display device of the remote monitoring support device. For example, if the supervisor determines that it is necessary from a safety perspective, the supervisor may intervene in the operation of the excavator 100 by inputting a predetermined input using the input device of the remote monitoring support device. It can be stopped in an emergency.
  • the information processing device 200 cooperates with the shovel 100 by communicating with the shovel 100, and provides support regarding the operation of the shovel 100.
  • the information processing device 200 is, for example, a server installed in a management office within the work site of the excavator 100 or a management center that manages the operating status of the excavator 100, etc. located at a location different from the work site of the excavator 100. It is a terminal device for equipment and management.
  • the server device may be an on-premises server, a cloud server, or an edge server.
  • the management terminal device may be a stationary terminal device such as a desktop PC (Personal Computer), or a portable terminal device such as a tablet terminal, smartphone, or laptop PC. terminal).
  • workers at the work site, supervisors who supervise work, managers who manage the work site, and the like can carry the portable information processing device 200 and move around the work site.
  • the operator can, for example, bring the portable information processing device 200 into the cabin of the excavator 100.
  • the information processing device 200 acquires data regarding the operating state from the excavator 100, for example. Thereby, the information processing device 200 can grasp the operating state of the shovel 100 and monitor whether there is any abnormality in the shovel 100 or the like. Further, the information processing device 200 can display data regarding the operating state of the excavator 100 for the user to confirm through a display device 208, which will be described later.
  • the information processing device 200 may transmit various data such as programs and reference data used in processing by the controller 30 and the like to the excavator 100.
  • the excavator 100 can perform various processes related to the operation of the excavator 100 using various data downloaded from the information processing device 200.
  • the hardware configuration of the remote operation support device 300 may be the same as that of the information processing device 200. Therefore, illustration and description of the hardware configuration of the remote operation support device 300 will be omitted.
  • FIG. 4 is a block diagram showing an example of the hardware configuration of shovel 100.
  • the path through which mechanical power is transmitted is a double line
  • the path through which high-pressure hydraulic oil that drives the hydraulic actuator flows is a solid line
  • the path through which pilot pressure is transmitted is a broken line
  • the path through which electrical signals are transmitted is shown. Each route is indicated by a dotted line.
  • the excavator 100 includes a hydraulic drive system for hydraulically driving the driven elements, an operation system for operating the driven elements, a user interface system for exchanging information with the user, a communication system for communicating with the outside, a control system for various controls, etc. Contains each component of.
  • the hydraulic drive system of the excavator 100 includes hydraulic pressure for hydraulically driving each of the driven elements such as the lower traveling body 1 (left and right crawlers 1C), the upper rotating body 3, and the attachment AT, as described above. Includes actuator HA. Further, the hydraulic drive system of the excavator 100 according to the present embodiment includes an engine 11, a regulator 13, a main pump 14, and a control valve 17.
  • the hydraulic actuator HA includes travel hydraulic motors 1ML and 1MR, a swing hydraulic motor 2M, a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, and the like.
  • the excavator 100 part or all of the hydraulic actuator HA may be replaced with an electric actuator.
  • the excavator 100 may be a hybrid excavator or an electric excavator.
  • the engine 11 is the prime mover of the excavator 100 and is the main power source in the hydraulic drive system.
  • the engine 11 is, for example, a diesel engine that uses light oil as fuel.
  • the engine 11 is mounted, for example, at the rear of the upper revolving structure 3.
  • the engine 11 rotates at a predetermined target rotation speed under direct or indirect control by a controller 30, which will be described later, and drives the main pump 14 and the pilot pump 15.
  • the regulator 13 controls (adjusts) the discharge amount of the main pump 14 under the control of the controller 30.
  • the regulator 13 adjusts the angle of the swash plate (hereinafter referred to as "tilt angle") of the main pump 14 in accordance with a control command from the controller 30.
  • the main pump 14 supplies hydraulic oil to the control valve 17 through a high-pressure hydraulic line.
  • the main pump 14 is, for example, mounted at the rear of the upper revolving structure 3, like the engine 11.
  • the main pump 14 is driven by the engine 11 as described above.
  • the main pump 14 is, for example, a variable displacement hydraulic pump, and as described above, the stroke length of the piston is adjusted by adjusting the tilt angle of the swash plate by the regulator 13 under the control of the controller 30, and the stroke length of the piston is adjusted.
  • the flow rate and discharge pressure are controlled.
  • the control valve 17 drives the hydraulic actuator HA in accordance with the contents of the operator's operation on the operating device 26 or remote control, or the operation command corresponding to the automatic operation function.
  • the control valve 17 is mounted, for example, in the center of the upper revolving body 3.
  • the control valve 17 is connected to the main pump 14 via a high-pressure hydraulic line, and controls the hydraulic fluid supplied from the main pump 14 according to an operator's operation or an operation command corresponding to an automatic operation function. , selectively supplying each hydraulic actuator.
  • the control valve 17 includes a plurality of control valves (also referred to as "direction switching valves") that control the flow rate and flow direction of the hydraulic oil supplied from the main pump 14 to each of the hydraulic actuators HA.
  • the operating system of the excavator 100 includes a pilot pump 15, an operating device 26, a hydraulic control valve 31, a shuttle valve 32, and a hydraulic control valve 33.
  • the pilot pump 15 supplies pilot pressure to various hydraulic devices via the pilot line 25.
  • the pilot pump 15 is, for example, mounted at the rear of the upper revolving structure 3, like the engine 11.
  • the pilot pump 15 is, for example, a fixed capacity hydraulic pump, and is driven by the engine 11 as described above.
  • pilot pump 15 may be omitted.
  • the relatively high pressure hydraulic oil discharged from the main pump 14 may be reduced in pressure by a predetermined pressure reducing valve, and then the relatively low pressure hydraulic oil may be supplied as pilot pressure to various hydraulic devices.
  • the operating device 26 is provided near the cockpit of the cabin 10 and is used by the operator to operate various driven elements. Specifically, the operating device 26 is used for an operator to operate the hydraulic actuator HA that drives each driven element, and as a result, the operator operates the driven element to be driven by the hydraulic actuator HA. can be realized.
  • the operating device 26 includes a pedal device and a lever device for operating each driven element (hydraulic actuator HA).
  • the operating device 26 is of a hydraulic pilot type. Specifically, the operating device 26 utilizes hydraulic oil supplied from the pilot pump 15 through the pilot line 25 and a pilot line 25A branching from the pilot line 25, and applies pilot pressure according to the operation content to the pilot line 27A on the secondary side. Output to. Pilot line 27A is connected to one inlet port of shuttle valve 32 and connected to control valve 17 via pilot line 27, which is connected to an outlet port of shuttle valve 32. Thereby, a pilot pressure can be input to the control valve 17 via the shuttle valve 32 in accordance with the operation contents regarding various driven elements (hydraulic actuator HA) in the operating device 26. Therefore, the control valve 17 can drive each hydraulic actuator HA according to the operation performed on the operating device 26 by an operator or the like.
  • the operating device 26 may be electrical.
  • the pilot line 27A, shuttle valve 32, and hydraulic control valve 33 are omitted.
  • the operating device 26 outputs an electrical signal (hereinafter referred to as an "operating signal") according to the content of the operation, and the operating signal is taken into the controller 30.
  • the controller 30 outputs a control command according to the content of the operation signal, that is, a control signal according to the content of the operation on the operating device 26 to the hydraulic control valve 31.
  • pilot pressure corresponding to the operation details of the operating device 26 is inputted from the hydraulic control valve 31 to the control valve 17, and the control valve 17 drives each hydraulic actuator HA according to the operation details of the operating device 26. be able to.
  • control valves built into the control valve 17 and driving the respective hydraulic actuators HA may be of an electromagnetic solenoid type.
  • the operation signal output from the operation device 26 may be directly input to the control valve 17, that is, to an electromagnetic solenoid type control valve.
  • part or all of the hydraulic actuator HA may be replaced with an electric actuator.
  • the controller 30 may output a control command according to the operation content of the operating device 26 or the remote control content specified by the remote control signal to the electric actuator or a driver driving the electric actuator.
  • the operating device 26 may be omitted.
  • the hydraulic control valve 31 is provided for each driven element (hydraulic actuator HA) to be operated by the operating device 26 and for each drive direction of the driven element (hydraulic actuator HA) (for example, the raising direction and lowering direction of the boom 4). .
  • two hydraulic control valves 31 are provided for each double-acting hydraulic actuator HA for driving the lower traveling body 1, the upper rotating body 3, the boom 4, the arm 5, the bucket 6, and the like.
  • the hydraulic control valve 31 is provided, for example, in the pilot line 25B between the pilot pump 15 and the control valve 17, and is configured to be able to change its flow path area (that is, the cross-sectional area through which hydraulic oil can flow). good.
  • the hydraulic control valve 31 can output a predetermined pilot pressure to the secondary side pilot line 27B using the hydraulic oil of the pilot pump 15 supplied through the pilot line 25B. Therefore, the hydraulic control valve 31 can indirectly apply a predetermined pilot pressure according to the control signal from the controller 30 to the control valve 17 through the shuttle valve 32 between the pilot line 27B and the pilot line 27. . Therefore, for example, the controller 30 can cause the hydraulic control valve 31 to supply pilot pressure to the control valve 17 according to an operation command corresponding to the automatic operation function, thereby realizing operation of the excavator 100 according to the automatic operation function.
  • the controller 30 may control the hydraulic control valve 31 to realize remote control of the excavator 100. Specifically, the controller 30 outputs to the hydraulic control valve 31 a control signal corresponding to the content of the remote operation specified by the remote operation signal received from the remote operation support device 300, using the communication device 60. Thereby, the controller 30 can cause the hydraulic control valve 31 to supply pilot pressure corresponding to the content of the remote control to the control valve 17, and realize the operation of the shovel 100 based on the operator's remote control.
  • the controller 30 causes the hydraulic control valve 31 to directly supply pilot pressure according to the operation details (operation signal) of the operating device 26 to the control valve 17, and The operation of the excavator 100 based on the above can be realized.
  • the shuttle valve 32 has two inlet ports and one outlet port, and outputs the hydraulic oil having the higher pilot pressure of the pilot pressures input to the two inlet ports to the outlet port.
  • the shuttle valve 32 is provided for each driven element (hydraulic actuator HA) to be operated by the operating device 26 and for each driving direction of the driven element (hydraulic actuator HA).
  • two shuttle valves 32 are provided for each double-acting hydraulic actuator HA for driving the lower traveling body 1, the upper rotating body 3, the boom 4, the arm 5, the bucket 6, and the like.
  • One of the two inlet ports of the shuttle valve 32 is connected to the pilot line 27A on the secondary side of the operating device 26 (specifically, the above-mentioned lever device or pedal device included in the operating device 26), and the other is It is connected to the pilot line 27B on the secondary side of the hydraulic control valve 31.
  • the outlet port of shuttle valve 32 is connected to the pilot port of the corresponding control valve of control valve 17 through pilot line 27 .
  • the corresponding control valve is a control valve that drives the hydraulic actuator HA, which is connected to one inlet port of the shuttle valve 32 and is operated by the above-mentioned lever device or pedal device.
  • these shuttle valves 32 each control the higher of the pilot pressure in the pilot line 27A on the secondary side of the operating device 26 and the pilot pressure on the pilot line 27B on the secondary side of the hydraulic control valve 31, respectively. It can act on the pilot port of the control valve.
  • the controller 30 controls the corresponding control valve by causing the hydraulic control valve 31 to output a pilot pressure higher than the pilot pressure on the secondary side of the operating device 26, regardless of the operator's operation on the operating device 26. be able to. Therefore, the controller 30 controls the operation of the driven elements (lower traveling structure 1, upper rotating structure 3, attachment AT) regardless of the operation state of the operating device 26 by the operator, and realizes an automatic driving function and a remote control function. can do.
  • the hydraulic control valve 33 is provided in the pilot line 27A that connects the operating device 26 and the shuttle valve 32.
  • the hydraulic control valve 33 is configured to be able to change its flow path area, for example.
  • the hydraulic control valve 33 operates according to a control signal input from the controller 30.
  • the controller 30 can forcibly reduce the pilot pressure output from the operating device 26 when the operating device 26 is being operated by the operator. Therefore, even when the operating device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator corresponding to the operation of the operating device 26.
  • the controller 30 can reduce the pilot pressure output from the operating device 26 to be lower than the pilot pressure output from the hydraulic control valve 31, for example, even when the operating device 26 is being operated. Can be done.
  • the controller 30 applies a desired pilot pressure to the pilot port of the control valve in the control valve 17, for example, regardless of the operation details of the operating device 26. It can be made to work reliably. Therefore, by controlling the hydraulic control valve 33 in addition to the hydraulic control valve 31, for example, the controller 30 can more appropriately realize the automatic operation function and remote control function of the excavator 100.
  • the user interface system of excavator 100 includes an operating device 26, an output device 50, and an input device 52.
  • the output device 50 outputs various information to the user of the excavator 100 (for example, the operator in the cabin 10 or an external remote control operator) and the people around the excavator 100 (for example, a worker or a driver of a work vehicle). Output.
  • the output device 50 includes a lighting device that outputs various information in a visual manner, a display device 50A (see FIG. 6), and the like.
  • the lighting equipment is, for example, a warning light (indicator lamp) or the like.
  • the display device 50A is, for example, a liquid crystal display or an organic EL (Electroluminescence) display.
  • lighting equipment and a display device 50A may be provided inside the cabin 10 and output various information visually to an operator inside the cabin 10.
  • the lighting equipment and the display device 50A may be provided, for example, on the side surface of the revolving upper structure 3, and may output various information visually to workers and the like around the excavator 100.
  • the output device 50 may include a sound output device that outputs various information in an auditory manner.
  • the sound output device includes, for example, a buzzer, a speaker, and the like.
  • the sound output device is, for example, provided inside or outside the cabin 10 and outputs various information audibly to the operator inside the cabin 10 and the people (workers, etc.) around the excavator 100. good.
  • the output device 50 may include a device that outputs various information using a tactile method such as vibration of the cockpit.
  • the input device 52 accepts various inputs from the user of the excavator 100, and signals corresponding to the accepted inputs are taken into the controller 30.
  • the input device 52 is provided inside the cabin 10 and receives input from an operator or the like inside the cabin 10.
  • the input device 52 may be provided, for example, on a side surface of the revolving upper structure 3, and may receive input from a worker or the like around the excavator 100.
  • the input device 52 includes an operation input device that accepts mechanical input from the user.
  • the operation input device may include a touch panel mounted on the display device, a touch pad installed around the display device, a button switch, a lever, a toggle, a knob switch provided on the operation device 26 (lever device), etc. .
  • the input device 52 may include a voice input device that accepts voice input from the user.
  • the audio input device includes, for example, a microphone.
  • the input device 52 may include a gesture input device that accepts gesture input from the user.
  • the gesture input device includes, for example, an imaging device that captures an image of a gesture performed by a user.
  • the input device 52 may include a biometric input device that receives biometric input from the user.
  • the biometric input includes, for example, input of biometric information such as a user's fingerprint or iris.
  • the communication system of the excavator 100 includes a communication device 60.
  • the communication device 60 is connected to an external communication line and communicates with a device provided separately from the excavator 100.
  • Devices provided separately from the excavator 100 may include devices external to the excavator 100 as well as portable terminal devices (portable terminals) brought into the cabin 10 by the user of the excavator 100.
  • the communication device 60 may include, for example, a mobile communication module that complies with standards such as 4G ( 4th Generation) and 5G ( 5th Generation). Further, the communication device 60 may include, for example, a satellite communication module. Further, the communication device 60 may include, for example, a WiFi communication module, a Bluetooth (registered trademark) communication module, or the like. Furthermore, the communication device 60 may include a plurality of communication devices depending on the communication lines to be connected.
  • the communication device 60 communicates with external devices such as the information processing device 200 and the remote operation support device 300 in the work site through a local communication line built at the work site.
  • the local communication line is, for example, a local 5G (so-called local 5G) mobile communication line built at a work site or a local area network (LAN) using WiFi 6.
  • the communication device 60 communicates with an information processing device 200, a remote operation support device 300, etc. located outside the work site via a wide area communication line that includes the work site, that is, a wide area network (WAN). It's okay.
  • the wide area network includes, for example, a wide area mobile communication network, a satellite communication network, an Internet network, and the like.
  • control system of excavator 100 includes a controller 30. Further, the control system of the excavator 100 according to the present embodiment includes an operating pressure sensor 29, an imaging device 40, and sensors S1 to S5.
  • the controller 30 performs various controls regarding the shovel 100.
  • the controller 30 may be realized by arbitrary hardware or a combination of arbitrary hardware and software.
  • the controller 30 includes an auxiliary storage device 30A, a memory device 30B, a CPU (Central Processing Unit) 30C, and an interface device 30D, which are connected by a bus BS1.
  • auxiliary storage device 30A a memory device 30B
  • CPU Central Processing Unit
  • interface device 30D an interface device 30D
  • the auxiliary storage device 30A is a non-volatile storage means, and stores installed programs as well as necessary files, data, etc.
  • the auxiliary storage device 30A is, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory) or a flash memory.
  • the memory device 30B loads the program in the auxiliary storage device 30A so that it can be read by the CPU 30C.
  • the memory device 30B is, for example, an SRAM (Static Random Access Memory).
  • the CPU 30C executes a program loaded into the memory device 30B, and implements various functions of the controller 30 according to instructions of the program.
  • the interface device 30D functions as a communication interface for connecting to a communication line inside the excavator 100, for example.
  • the interface device 30D may include a plurality of different types of communication interfaces depending on the type of communication line to be connected.
  • the interface device 30D functions as an external interface for reading data from and writing data to the recording medium.
  • the recording medium is, for example, a dedicated tool that is connected to a connector installed inside the cabin 10 with a detachable cable.
  • the recording medium may be a general-purpose recording medium such as an SD memory card or a USB (Universal Serial Bus) memory.
  • programs for realizing various functions of the controller 30 can be provided by, for example, a portable recording medium and installed in the auxiliary storage device 30A of the controller 30. Further, the program may be downloaded from another computer outside the excavator 100 through the communication device 60 and installed in the auxiliary storage device 30A.
  • controller 30 may be realized by another controller (control device). That is, the functions of the controller 30 may be realized in a distributed manner by a plurality of controllers.
  • the operating pressure sensor 29 detects the pilot pressure on the secondary side (pilot line 27A) of the hydraulic pilot type operating device 26, that is, the pilot pressure corresponding to the operating state of each driven element (hydraulic actuator) in the operating device 26. To detect. A detection signal of pilot pressure corresponding to the operating state of each driven element (hydraulic actuator HA) in the operating device 26 by the operating pressure sensor 29 is taken into the controller 30.
  • the operating device 26 is an electric type, the operating pressure sensor 29 is omitted. This is because the controller 30 can grasp the operating state of each driven element through the operating device 26 based on the operating signal taken in from the operating device 26.
  • the imaging device 40 acquires images around the excavator 100.
  • the imaging device 40 also generates three-dimensional data (hereinafter simply referred to as "the object's three-dimensional shape") representing the position and external shape of the object around the shovel 100 within the imaging range (angle of view) based on the acquired image and distance-related data described below. "original data”) may be obtained (generated).
  • the three-dimensional data of objects around the shovel 100 is, for example, coordinate information data of a point group representing the surface of the object, distance image data, and the like.
  • the imaging device 40 includes a camera 40F that images the front of the upper revolving structure 3, a camera 40B that images the rear of the upper revolving structure 3, and a camera 40L that images the left side of the upper revolving structure 3. , and a camera 40R that images the right side of the upper rotating body 3.
  • the imaging device 40 can image the entire circumference around the shovel 100, that is, the range covering the angular direction of 360 degrees, when the shovel 100 is viewed from above.
  • the operator visually recognizes peripheral images such as captured images of the cameras 40B, 40L, and 40R and processed images generated based on the captured images through the display device 50A and the remote operation support device 300.
  • the operator operates the excavator 100 while checking the operation of the attachment AT by visually checking peripheral images such as images captured by the camera 40F and processed images generated based on the captured images through the remote operation support device 300. Can be controlled remotely.
  • the cameras 40F, 40B, 40L, and 40R may be collectively or individually referred to as "camera 40X.”
  • the camera 40X is, for example, a monocular camera.
  • the camera 40X acquires data regarding distance (depth) in addition to two-dimensional images, such as a stereo camera, a TOF (Time Of Flight) camera, etc. (hereinafter collectively referred to as a "3D camera"). It may be possible.
  • Output data (for example, image data, three-dimensional data of objects around the excavator 100, etc.) of the imaging device 40 (camera 40X) is taken into the controller 30 through a one-to-one communication line or an in-vehicle network.
  • the controller 30 can monitor objects around the excavator 100 based on the output data of the camera 40X.
  • the controller 30 can determine the surrounding environment of the excavator 100 based on the output data of the camera 40X.
  • the controller 30 can determine the posture state of the attachment AT shown in the captured image based on the output data of the camera 40X (camera 40F).
  • the controller 30 can determine the attitude state of the body of the excavator 100 (the upper revolving body 3) based on the output data of the camera 40X, with reference to objects around the excavator 100.
  • a distance sensor may be provided in the upper revolving body 3.
  • the distance sensor is attached to the upper part of the upper revolving body 3, for example, and acquires data regarding the distance and direction of surrounding objects with respect to the shovel 100 as a reference.
  • the distance sensor may acquire (generate) three-dimensional data (for example, coordinate information data of a point group) of objects around the shovel 100 within the sensing range based on the acquired data.
  • the distance sensor is, for example, LIDAR (Light Detection and Ranging).
  • the distance sensor may be, for example, a millimeter wave radar, an ultrasonic sensor, an infrared sensor, or the like.
  • the sensor S1 is attached to the boom 4 and detects the attitude angle (hereinafter referred to as "boom angle") around the rotation axis of the base end corresponding to the connection part of the boom 4 with the upper revolving structure 3.
  • the sensor S1 includes, for example, a rotary potentiometer, a rotary encoder, an acceleration sensor, an angular acceleration sensor, a 6-axis sensor, an IMU (Inertial Measurement Unit), and the like. The same may apply to the sensors S2 to S4 below. Further, the sensor S1 may include a cylinder sensor that detects the extended/contracted position of the boom cylinder 7. The same may apply to the sensors S2 and S3 below.
  • a detection signal of the boom angle by the sensor S1 is taken into the controller 30. Thereby, the controller 30 can grasp the attitude state of the boom 4.
  • the sensor S2 is attached to the arm 5 and detects the posture angle (hereinafter referred to as "arm angle") around the rotation axis of the base end of the arm 5, which corresponds to the connection part with the boom 4.
  • the arm angle detection signal from the sensor S2 is taken into the controller 30. Thereby, the controller 30 can grasp the posture state of the arm 5.
  • the sensor S3 is attached to the bucket 6 and detects the attitude angle (hereinafter referred to as "arm angle") around the rotation axis of the base end corresponding to the connection part with the arm 5 of the bucket 6.
  • a detection signal of the arm angle by the sensor S3 is taken into the controller 30. Thereby, the controller 30 can grasp the attitude state of the bucket 6.
  • the sensor S4 detects the inclination state of the aircraft body (for example, the upper rotating body 3) with respect to a predetermined reference plane (for example, a horizontal plane).
  • the sensor S4 is attached to the revolving upper structure 3, and measures the inclination angle of the excavator 100 (i.e., the revolving upper structure 3) about two axes in the front-rear direction and the left-right direction (hereinafter, "front-rear inclination angle” and “lateral inclination angle”). ”) is detected.
  • a detection signal corresponding to the inclination angle (front/rear inclination angle and left/right inclination angle) detected by the sensor S ⁇ b>4 is taken into the controller 30 . Thereby, the controller 30 can grasp the tilting state of the aircraft body (upper rotating body 3).
  • the sensor S5 is attached to the revolving upper structure 3 and outputs detection information regarding the turning state of the revolving upper structure 3.
  • the sensor S5 detects, for example, the turning angular velocity and turning angle of the upper rotating body 3.
  • the sensor S5 includes, for example, a gyro sensor, a resolver, a rotary encoder, and the like. Detection information regarding the turning state detected by the sensor S5 is taken into the controller 30. Thereby, the controller 30 can grasp the turning state such as the turning angle of the upper rotating body 3.
  • the senor S4 includes a gyro sensor, a 6-axis sensor, an IMU, etc. that can detect angular velocity around three axes
  • the turning state for example, turning angular velocity
  • sensor S5 may be omitted.
  • at least some of the sensors S1 to S5 may be omitted.
  • FIG. 5 is a block diagram showing an example of the hardware configuration of the information processing device 200.
  • the functions of the information processing device 200 are realized by arbitrary hardware or a combination of arbitrary hardware and software.
  • the information processing device 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU 204, a high-speed arithmetic device 205, a communication interface 206, an input device 207, and It includes a display device 208 and a sound output device 209.
  • the external interface 201 functions as an interface for reading data from and writing data to the recording medium 201A.
  • the recording medium 201A includes, for example, a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disc), a BD (Blu-ray (registered trademark) Disc), an SD memory card, a USB memory, and the like.
  • the information processing device 200 can read various data used in processing through the recording medium 201A, store it in the auxiliary storage device 202, and install programs that implement various functions.
  • the information processing device 200 may obtain various data and programs used in processing from an external device through the communication interface 206.
  • the auxiliary storage device 202 stores various installed programs, as well as files, data, etc. necessary for various processes.
  • the auxiliary storage device 202 includes, for example, an HDD (Hard Disc Drive), an SSD (Solid State Disc), a flash memory, and the like.
  • the memory device 203 reads and stores the program from the auxiliary storage device 202 when there is an instruction to start the program.
  • the memory device 203 includes, for example, DRAM (Dynamic Random Access Memory) and SRAM.
  • the CPU 204 executes various programs loaded from the auxiliary storage device 202 to the memory device 203, and implements various functions related to the information processing device 200 according to the programs.
  • the high-speed arithmetic unit 205 works in conjunction with the CPU 204 and performs arithmetic processing at a relatively high speed.
  • the high-speed calculation device 205 includes, for example, a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), and the like.
  • the high-speed calculation device 205 may be omitted depending on the required speed of calculation processing.
  • the communication interface 206 is used as an interface for communicably connecting to an external device. Thereby, the information processing device 200 can communicate with an external device such as the excavator 100, for example, through the communication interface 206. Furthermore, the communication interface 206 may have a plurality of types of communication interfaces depending on the communication method with the connected device.
  • the input device 207 receives various inputs from the user.
  • the input device 207 includes a remote control operating device for remotely controlling the shovel 100.
  • the input device 207 includes, for example, an input device (hereinafter referred to as "operation input device") that accepts mechanical operation input from the user.
  • the operating device for remote control may be an operating input device.
  • the operation input device includes, for example, a button, a toggle, a lever, a keyboard, a mouse, a touch panel mounted on the display device 208, a touch pad provided separately from the display device 208, and the like.
  • the input device 207 may include a voice input device that can accept voice input from the user.
  • the voice input device includes, for example, a microphone that can collect the user's voice.
  • the input device 207 may include a gesture input device that can accept gesture input from the user.
  • the gesture input device includes, for example, a camera that can capture images of the user's gestures.
  • the input device 207 may include a biometric input device that can accept biometric input from the user.
  • the biometric input device includes, for example, a camera that can acquire image data that includes information about a user's fingerprint or iris.
  • the display device 208 displays information screens and operation screens for the user of the information processing device 200.
  • the display device 208 is, for example, a liquid crystal display, an organic EL (Electroluminescence) display, or the like.
  • the sound output device 209 conveys various information to the user of the information processing device 200 through sound.
  • the sound output device 209 is, for example, a buzzer, an alarm, a speaker, or the like.
  • FIG. 6 is a functional block diagram illustrating an example of a functional configuration related to generation of the trajectory of the working part of the excavator 100 in the operation support system SYS.
  • 7 and 8 are diagrams illustrating an example of the structure of a model for extracting the feature amount of the trajectory of the working part of the excavator 100.
  • 9 and 10 are diagrams illustrating an example of the structure of a model that generates a trajectory of a working part in a compound motion of the excavator 100.
  • FIG. 11 is a diagram illustrating an example of the trajectory of the work area of the excavator 100 during land leveling work.
  • trajectory a model for extracting the feature values of the trajectory of the working part of the excavator 100
  • learning model M2 a model for extracting the feature values of the trajectory of the working part of the excavator 100
  • the work site of the shovel 100 is, for example, the toe or back of the bucket 6.
  • the model that generates the trajectory of the work part in the compound motion of the excavator 100 regardless of whether it is in the learning process of before learning, during learning, or after learning, it is referred to as "learning model M3" and learning model M3 is used.
  • learning model M3 learning model M3
  • learning model M3 is used.
  • a trained model it is referred to as a “trained model LM3.”
  • the compound operation of the shovel 100 is an operation that combines two or more different types of predetermined operations (hereinafter referred to as "basic operations") of the shovel 100.
  • the excavator 100 includes a support device 150.
  • the support device 150 provides support regarding the work of the excavator 100.
  • the support device 150 includes an operating device 26, a controller 30, an imaging device 40, an output device 50, and an input device 52. Further, when the excavator 100 is remotely controlled, the support device 150 may include the communication device 60.
  • the controller 30 includes an operation log providing unit 301 and a work support unit 302 as functional units.
  • the controller 30 includes only the former of the operation log providing unit 301 and the work support unit 302, and the excavator 100 that includes only the latter. May exist.
  • the former shovel 100 only has the function of acquiring the operation log of the shovel 100 and providing it to the information processing device 200, which is used for the work support function of the latter shovel 100.
  • the information processing device 200 includes an operation log acquisition unit 2001, an operation log storage unit 2002, a teacher data generation unit 2003, a machine learning unit 2004, a learned model storage unit 2005, and a distribution unit 2006 as functional units. include.
  • the operation log providing unit 301 is a functional unit that acquires the operation log of the shovel 100, which is the original data for realizing the function of generating the trajectory of the working part of the excavator 100, and provides it to the information processing device 200. .
  • a relatively experienced operator hereinafter referred to as an "expert" for convenience
  • who has a long history of operating the excavator 100 obtains an operation log when operating the excavator 100 and stores it in the information processing device 200. provide.
  • the operation log of the shovel 100 includes data regarding the shape of the work target around the shovel 100 and data regarding the operation of the shovel 100 performed on the shape of the work target.
  • the data regarding the shape of the work target around the shovel 100 is, for example, data regarding the topographical shape of the ground at the work site as the work target of the shovel 100.
  • the data regarding the shape of the work target of the shovel 100 is, for example, image data of the imaging device 40 or three-dimensional data of the work target obtained from the image data.
  • the data regarding the operation of the shovel 100 is, for example, data representing the details of the operator's operation.
  • the data representing the contents of the operator's operation may be, for example, the output data of the operating pressure sensor 29 corresponding to the hydraulic pilot operating device 26 or the output data (operating signal data) of the operating device 26 corresponding to the electrical operating device 26. It is. Furthermore, the data regarding the operation of the shovel 100 may be data representing the operation state of the shovel 100 actually executed in response to an operation by an operator.
  • the data representing the operating state of the shovel 100 is, for example, the output data of the sensors S1 to S5, or the data related to the posture state of the shovel 100 obtained from the output data of the sensors S1 to S5.
  • the operation log providing section 301 includes an operation log recording section 301A, an operation log storage section 301B, and an operation log transmission section 301C.
  • the operation log recording unit 301A acquires the operation log of the shovel 100 and records it in the operation log storage unit 301B. For example, every time an operation of the shovel 100 is executed, the operation log recording unit 301A records data regarding the shape of the work target around the shovel 100 at the start of execution of the operation or immediately before execution, and data regarding the operation of the shovel 100. is recorded in the operation log storage unit 301B.
  • the operation log storage unit 301B stores operation logs of the shovel 100 in an accumulated manner.
  • the operation log storage unit 301B stores data regarding the shape of a work target around the shovel 100 for each operation of the shovel 100 and data regarding the operation of the shovel 100 in a linked form.
  • the operation log storage unit 301B stores record data representing the correspondence relationship between data regarding the shape of the work target around the shovel 100 and data regarding the operation of the shovel 100 for each operation of the shovel 100.
  • a database of logs may be constructed.
  • operation log in the operation log storage unit 301B that has been transmitted to the information processing device 200 by the operation log transmission unit 301C may be deleted after the fact.
  • the operation log transmission unit 301C transmits the operation log of the shovel 100, which is stored in the operation log storage unit 301B, to the information processing device 200 via the communication device 60.
  • the operation log transmitting unit 301C also transmits to the information processing device 200 data regarding the shape of the work target around the shovel 100 for each operation of the shovel 100 and record data representing the correspondence between the data regarding the operations of the shovel 100. You may.
  • the operation log transmitting unit 301C transmits unused data stored in the operation log storage unit 301B in response to a signal requesting transmission of the operation log of the excavator 100 from the information processing device 200 (hereinafter referred to as a “transmission request signal”).
  • the operation log of the sending shovel 100 is sent to the information processing device 200.
  • the operation log transmitting unit 301C may automatically transmit the unsent operation log of the shovel 100, which is stored in the operation log storage unit 301B, to the information processing device 200 at a predetermined timing.
  • the predetermined timing is, for example, when the excavator 100 stops operating (the key switch is turned off) or when the excavator 100 starts operating (the key switch is turned on).
  • the operation log acquisition unit 2001 acquires the operation log of the shovel 100, which is received from the shovel 100.
  • the operation log acquisition unit 2001 acquires the operation log of the excavator 100 by transmitting a transmission request signal to the excavator 100 in response to an operation by the user of the information processing device 200 or automatically at a predetermined timing. Further, the operation log acquisition unit 2001 may acquire an operation log of the excavator 100 that is automatically transmitted from the excavator 100 at a predetermined timing.
  • the operation log storage unit 2002 stores operation logs of the shovel 100 acquired by the operation log acquisition unit 2001 in an accumulated manner.
  • the operation log storage unit 2002 as in the case of the operation log storage unit 301B, data regarding the shape of the work object around the shovel 100 for each operation of the shovel 100 and data regarding the operation of the shovel 100 are linked. remembered in form.
  • the teacher data generation unit 2003 generates teacher data for machine learning based on the operation log of the excavator 100 in the operation log storage unit 2002, and outputs a teacher data set that is a collection of a large number of teacher data.
  • the teacher data generation unit 2003 may automatically generate the teacher data by batch processing, or may generate the teacher data in response to input from the user of the information processing apparatus 200.
  • the teacher data generation unit 2003 includes teacher data generation units 2003A to 2003C.
  • the teacher data generation unit 2003A generates teacher data for generating a trained model LM1, which will be described later.
  • the teacher data generation unit 2003A generates input data regarding the shape of the work target around the shovel 100, and correct output data corresponding to the input data representing the trajectory (trajectory) of the work part of the shovel 100 operated by an expert.
  • the teacher data generation unit 2003B generates teacher data for generating the trained model LM2. For example, the teacher data generation unit 2003B generates a teacher who uses the same data representing the trajectory (trajectory) of a working part in a predetermined movement (basic movement) of the excavator 100 operated by an expert as a combination of input data and correct output data. Generate data. At this time, the teacher data generation unit 2003B generates teacher data for each of a plurality of basic movements of different types (hereinafter simply referred to as "a plurality of basic movements"), and outputs a teacher data set for each of the plurality of basic movements. It's fine. Further, the teacher data generation unit 2003B may generate teacher data regardless of the plurality of basic movements, and output a teacher data set in which teacher data corresponding to the plurality of basic movements coexist.
  • the plurality of basic operations of the shovel 100 include at least two of, for example, a sweeping operation, a leveling operation, a compaction operation, a broom operation, an excavation operation, and an earth removal operation, which are used during ground leveling work.
  • the sweeping operation is, for example, an operation in which the attachment AT is operated to push the bucket 6 forward along the ground, thereby sweeping out earth and sand forward on the back surface of the bucket 6.
  • the attachment AT performs a lowering operation of the boom 4 and an opening operation of the arm 5.
  • the horizontal pulling operation is, for example, an operation to smooth out irregularities on the ground (the surface of the terrain) by operating the attachment AT and moving the toe of the bucket 6 along the ground almost horizontally toward you.
  • the attachment AT performs a raising operation of the boom 4 and a closing operation of the arm 5.
  • the rolling operation is, for example, an operation of operating the attachment AT and pressing the back surface of the bucket 6 against the ground. Further, the rolling operation may be an operation of moving the bucket 6 up and down and pressing the back surface of the bucket 6 against the ground.
  • the compaction operation is performed by pushing the bucket 6 forward along the ground, sweeping the earth and sand to a predetermined position in front with the back of the bucket 6, and then rolling the ground at a predetermined position with the back of the bucket 6. It may also be a pressing motion. In the rolling operation, for example, the attachment AT lowers the boom 4 when pressing against the ground.
  • the broom operation is, for example, an operation in which the upper rotating body 3 is operated and the bucket 6 is rotated left and right while keeping it along the ground. Further, the broom operation may be, for example, an operation of pushing the bucket 6 forward while operating the attachment AT and the upper rotating body 3 and rotating the bucket 6 alternately left and right while keeping the bucket 6 along the ground. In the broom movement, for example, the upper revolving body 3 alternately repeats left and right turning movements. In addition, in the broom operation, for example, in addition to the alternating right and left turning operations of the upper revolving structure 3, the attachment AT may perform a lowering operation of the boom 4 and an opening operation of the arm 5, as in the case of the sweeping operation.
  • the excavation operation is, for example, an operation of operating the attachment AT to excavate earth and sand from a certain location on the ground and scooping it into the bucket 6.
  • the earth removal operation is an operation in which the attachment AT is operated to discharge earth and sand scooped into the bucket 6 in the excavation operation to another location on the ground.
  • the shovel 100 may operate the upper revolving structure 3 in addition to the attachment AT.
  • the plurality of basic operations of the shovel 100 include at least two of, for example, an excavation operation, a boom-up rotation operation, a boom-lower rotation operation, an earth removal operation, and a broom operation, which are used during excavation work.
  • the plurality of basic operations of the excavator 100 include, for example, a cutting operation, a compaction operation, etc., which are used during slope construction work.
  • the teacher data generation unit 2003C generates teacher data for generating the learned model LM3.
  • the teacher data generation unit 2003C generates data representing the trajectory (trajectory) of the working part of the shovel 100 operated by an expert.
  • the teacher data set output by the teacher data generation unit 2003C may include trajectories of work parts in different types of basic movements of the excavator 100.
  • the teacher data set output by the teacher data generation unit 2003C includes the trajectory of the work part in the compound movement of the shovel 100, instead of or in addition to the trajectory of the work part in the basic movement of the shovel 100. You may be
  • the data representing the trajectory (trajectory) of the working part of the shovel 100 is generated based on the output data of the sensors S1 to S5, which is included in the data regarding the operation of the shovel 100, for example.
  • the machine learning unit 2004 performs machine learning on the base learning model based on the set of teacher data generated by the teacher data generation unit 2003 to generate a learned model.
  • the trained model includes, for example, a neural network such as a DNN (Deep Neural Network).
  • the machine learning unit 2004 includes machine learning units 2004A to 2004C.
  • the machine learning unit 2004A causes the base learning model M1 to perform machine learning based on the teacher data set output from the teacher data generation unit 2003A.
  • the machine learning unit 2004A receives the data representing the shape of the work target around the shovel 100 as input data, and generates (outputs) data representing the trajectory of the work part in the basic operation of the shovel 100.
  • a model LM1 can be generated.
  • the machine learning unit 2004A generates a learned model LM1 for each of the plurality of basic movements based on the teacher data set for each of the plurality of basic movements output from the teacher data generation unit 2003A.
  • the machine learning unit 2004B performs machine learning on the base learning model M2 based on the teacher data set output from the teacher data generation unit 2003B to generate a learned model LM2.
  • the learned model LM2 is a model for extracting the feature amount of the trajectory of the working part of the excavator 100.
  • the learning model M2 inputs data representing the trajectory of the working part of the excavator 100, repeats downsampling, then repeats upsampling, and generates data representing the trajectory of the working part of the excavator 100.
  • This is a neural network that outputs .
  • data of the same dimension as input data (latent variable) of the learning model M3, which will be described later, is generated by sequential downsampling.
  • data on the trajectory of the working part of the excavator 100 is outputted by sequential upsampling.
  • the machine learning unit 2004B causes the learning model M2 to perform machine learning based on a teacher data set in which input data and output data are a combination of data representing the same trajectory of the working part of the shovel 100. Thereby, the machine learning unit 2004B can generate a learned model LM2 that can output the same output data as the input data, using data representing the trajectory of the working part of the excavator 100 as input data. At this time, the machine learning unit 2004B may perform machine learning so as to reflect the result of downsampling of the corresponding first half in the process of upsampling the second half of the learning model M2. Thereby, the machine learning unit 2004B can machine learn the learning model M2 so that it can appropriately output the same output data as the input data.
  • the same data as the input data is extracted from the intermediate data of the dimension corresponding to the latent variable, that is, the data representing the trajectory of the working part of the excavator 100 is extracted from the intermediate data.
  • the characteristics for generating are reflected. Therefore, various network parameters in the latter half of the learned model LM2 can be extracted as the feature amount of the trajectory of the working part of the excavator 100, which corresponds to the input data of the learned model LM2.
  • the learned model LM2 is used as the feature amount F_A of the trajectory of the work part in the sweeping operation of the shovel 100, and the network of each layer in the latter half of the learned model LM2 is Parameters A1 to A4 can be extracted.
  • the network parameters B1 to B4 of each layer in the latter half of the trained model LM2 are set as the feature amount F_B of the trajectory of the work part during the horizontal pulling operation of the excavator 100. can be extracted.
  • the learned model LM2 may be generated for each of a plurality of basic movements, or one common to a plurality of basic movements may be generated.
  • the machine learning unit 2004C performs machine learning on the base learning model M3 based on the teacher data set output from the teacher data generation unit 2003C to generate a learned model LM3.
  • the learned model LM3 is a model for generating the trajectory of the work part in the compound motion of the shovel 100.
  • the learning model M3 is a neural network that uses the latent variable z as input data and repeatedly performs upsampling to generate data with a dimension corresponding to the trajectory of the working part of the excavator 100.
  • the trained model LM3 corresponds to a generator in a GAN (Generative Adversarial Network).
  • the machine learning unit 2004C causes a generator (learning model M3) and a discriminator to learn adversarially using the teacher data output from the teacher data generating unit 2003C.
  • the discriminator considers the data representing the trajectory of the working part of the shovel 100 by an expert, which is input as training data, as "real" and the data representing the trajectory of the working part of the shovel 100 generated by the generator.
  • the machine learning unit 2004C can generate a learned model LM3 that can generate the trajectory of the working part of the excavator 100 from the latent variable z by alternately learning the discriminator and the generator.
  • the learned model LM3 can generate the trajectory of the working part of the excavator 100 corresponding to the feature by reflecting the feature extracted from the learned model LM2 in FIG. 8 in the learned model LM3. . Therefore, the learned model LM3 reflects the feature amounts extracted from the learned model LM2, which are a mixture of the feature amounts for each different basic motion, as appropriate, so that the trajectory of the work part in the compound motion of the excavator 100 can be adjusted. It is possible to generate data representing
  • the feature value FV obtained by combining the feature values F_A and F_B of the trajectory of the work area in each of the sweeping operation and horizontal pulling operation of the excavator 100 is learned. It will be reflected in the finished model LM3.
  • the feature amount FV includes feature amounts FV1 to FV4 corresponding to network parameters A1 and B1, network parameters A2 and B2, network parameters A3 and B3, and network parameters A4 and B4, respectively.
  • the feature amount FV1 is adjusted in the range between network parameters A1 and B1 by combining network parameters A1 and B1 that correspond to the same layer of the network, and is input to the target layer of the trained model LM3. be done.
  • the feature value FV2 is adjusted in the range between network parameter A2 and network parameter B2 by combining network parameters A2 and B2 that correspond to the same layer of the network, and is adjusted in the range between network parameter A2 and network parameter B2. entered into the hierarchy.
  • the feature value FV3 is adjusted between network parameters A3 and B3 by combining network parameters A3 and B3 that correspond to the same layer of the network, and is adjusted to the target layer of the trained model LM3. is input.
  • the feature value FV4 is adjusted in the range between the network parameter A4 and the network parameter B4 by combining network parameters A4 and B4 corresponding to the same layer of the network, and is adjusted in the range between the network parameter A4 and the network parameter B4, and entered into the hierarchy.
  • the learned model LM3 can generate a trajectory of the work part in a composite operation of the combination of the sweeping operation and the horizontal pulling operation of the shovel 100.
  • the trained model LM3 is able to perform work in a composite operation of the combination of sweeping operation and horizontal pulling operation of the shovel 100 by reflecting (inputting) adjustment parameters in each layer in addition to the feature amount FV. The details of the part's trajectory can be adjusted (see the solid arrow in the figure). Further, the adjustment parameters may be reflected in the output of the learned model LM3, thereby making it possible to adjust the compound trajectory of the excavator 100 itself output from the learned model LM3 (see the broken line arrow in the figure). .
  • the shovel 100 moves the earth and sand in the earth and sand pools 1102 and 1103 to the depression 1101 at once using the trajectory 1100 of the work area in a combined operation that combines the sweeping operation and the horizontal pulling operation, and flattens the ground. It can be evened out.
  • the learned model storage unit 2005 stores the learned models LM1 to LM3 output by the machine learning unit 2004. Further, when the machine learning unit 2004A performs relearning or additional learning of the trained model LM1, the trained model LM1 in the trained model storage unit 2005 is updated. The same applies to cases where the learned models LM2 and LM3 are subjected to relearning or additional learning by the machine learning units 2004B and 2004C.
  • the distribution unit 2006 distributes the data of the learned models LM1 to LM3 to the excavator 100.
  • the distribution unit 2006 distributes the most recently generated or updated learned model LM1 to the excavator 100. Further, the distribution unit 2006 may distribute the latest learned model LM1 in the learned model storage unit 2005 to the excavator 100 in response to a signal received from the excavator 100 requesting distribution of the learned model LM1. . The same may apply to the trained models LM2 and LM3.
  • the work support unit 302 is a functional unit that supports the work of the shovel 100 operated by the operator.
  • the work support unit 302 includes a learned model storage unit 302A, a work object shape acquisition unit 302B, a work selection unit 302C, a trajectory generation unit 302D, a specification setting unit 302E, a trajectory generation unit 302F, and an operation control unit 302G. and a display processing unit 302H.
  • the trained model storage unit 302A stores trained models LM1 to LM3 that are distributed from the information processing device 200 and received through the communication device 60.
  • the work target shape acquisition unit 302B acquires data representing the shape of the work target (terrain shape) around the shovel 100 based on the output of the imaging device 40 and the distance sensor.
  • the work selection unit 302C selects a work to be performed by the excavator 100 from among a plurality of work candidates of different types in response to input from the user (operator) received through the input device 52. Thereby, the user can generate the trajectory of the work part of the shovel 100 in accordance with the type of work performed by the shovel 100.
  • the plurality of work candidates include, for example, ground leveling work, excavation work, slope construction work, and the like.
  • the work selection unit 302C selects the shovel 100 from among a plurality of work candidates in response to an input from a user using the remote operation support device 300, which is received through the communication device 60. may select the work to be performed.
  • Basic operations of the shovel 100 that can be combined as a compound operation are defined in advance for each task selected by the task selection section 302C.
  • basic actions that can be combined as a compound action include broom action, leveling action, compaction action, excavation action, soil removal action, and the like. Therefore, by selecting the type of work, the work selection unit 302C selects the basic motion to be combined as a compound motion of the shovel 100.
  • the trajectory generation unit 302D uses the learned model LM1 to determine the shape of the excavator 100 based on the data regarding the shape of the work target around the excavator 100 for each of the plurality of basic movements corresponding to the work selected by the work selection unit 302C. Generate data representing the trajectory of the work part.
  • the plurality of target basic movements for each of the plurality of candidate tasks are defined in advance.
  • the plurality of basic operations corresponding to the ground leveling operation include, for example, a sweeping operation, a leveling operation, and a rolling operation.
  • the plurality of basic operations corresponding to the earth leveling work may include a digging operation, an earth removal operation, a broom operation, and the like.
  • the specification setting unit 302E sets (adjusts) specifications regarding the combination of a plurality of basic operations in the compound operation of the excavator 100, in response to input from the user (operator) received through the input device 52. Further, when the excavator 100 is remotely controlled, the specification setting unit 302E determines the combination of a plurality of basic operations of the excavator 100 in accordance with input from the user using the remote operation support device 300, which is received through the communication device 60. Specifications may be set.
  • the specification setting section 302E sets (selects) a basic motion to be combined as a compound motion of the shovel 100 from among a plurality of basic motions defined for the task selected by the task selection section 302C. Further, the specification setting unit 302E may adjust the composition ratio (hereinafter referred to as "compound ratio") of each of the plurality of basic operations in the compound operation, with the entire compound operation of the shovel 100 as 100%. Thereby, when generating the trajectory of the work part in the compound motion of the shovel 100, the user can adjust the distribution for each basic motion making up the compound motion. Further, the specification setting unit 302E may set specifications for distribution of feature amounts for each of a plurality of basic operations. At this time, for example, as shown in FIG.
  • the specification setting unit 302E may adjust the order of execution of each of the plurality of basic operations in the composite operation. Thereby, the user can adjust the order of implementation of the basic operations that make up the compound operation when generating the trajectory of the work part in the compound operation of the shovel 100. Further, the specification setting unit 302E may adjust conditions such as passing points of work parts for each of a plurality of basic movements that constitute a composite movement.
  • the user when generating the trajectory of the work part in the compound motion of the shovel 100, the user can perform multiple basic motions while checking the image representing the terrain shape around the shovel 100 displayed on the display device 50A. It is possible to limit the position through which the work part should pass.
  • the trajectory generation unit 302F uses the learned models LM2 and LM3 to calculate the composite operation of the excavator 100 based on the data generated for each of the plurality of basic movements by the trajectory generation unit 302D and the data of the setting results by the specification setting unit 302E. Generate data representing the trajectory of the work part.
  • the trajectory generation unit 302F uses the trained model LM2 to extract feature amounts for each of the plurality of basic movements based on the data generated for each of the plurality of basic movements by the trajectory generation unit 302D. Then, the trajectory generation unit 302F uses the trained model LM3 to generate data representing the trajectory of the work part on the compound trajectory of the excavator 100 based on the feature values for each of the plurality of basic operations and the setting results by the specification setting unit 302E. generate.
  • the trajectory generation unit 302F adjusts the feature quantity FV based on the composite rate for each of the plurality of basic movements set by the specification setting unit 302E, and uses the learned model LM3 to Data representing the trajectory of the work site on the compound trajectory of the excavator 100 is generated. Further, the trajectory generating section 302F may reflect the setting result by the specification setting section 302E in the data representing the trajectory of the work part on the compound trajectory of the excavator 100 by adjusting the adjustment parameters.
  • the operation control unit 302G moves the work area of the bucket 6 along the trajectory corresponding to the data generated by the trajectory generation unit 302F in response to input from the user (operator) received through the input device 52.
  • the shovel 100 is operated so as to move.
  • the operation control unit 302G controls the hydraulic control valve 31 while grasping the position of the working part of the bucket 6 from the outputs of the sensors S1 to S5, etc., so as to move the bucket 6 along the target trajectory.
  • Excavator 100 can be operated so that the work area moves.
  • the motion control section 302G causes the work part of the bucket 6 to move along a trajectory corresponding to data generated by the trajectory generation section 302F in response to an instruction to execute the motion from the user through the input device 52.
  • the excavator 100 is operated as follows. Further, when the excavator 100 is remotely controlled, the operation control unit 302G operates the excavator 100 in response to an instruction to execute an operation from the user using the remote operation support device 300, which is received through the communication device 60. It's okay.
  • the operation control unit 302G performs work on the bucket 6 along the trajectory corresponding to the data generated by the trajectory generation unit 302F in a manner that assists the operator's operation in accordance with the operation of the operating device 26 or the remote control signal.
  • the shovel 100 may be operated so that the part moves.
  • the display processing unit 302H causes the display device 50A to display a screen related to the generation of the target trajectory of the work area of the excavator 100.
  • the screens related to generating the trajectory of the work area of the shovel 100 include, for example, a screen that displays an image representing the shape of the work target around the shovel 100 based on data acquired by the work target shape acquisition unit 302B. Thereby, the user can check the shape of the construction target around the shovel 100 and determine the operation that the shovel 100 needs to perform in accordance with the work content.
  • the image representing the shape of the construction target around the shovel 100 may be, for example, an image captured by the imaging device 40 or a processed image thereof representing the shape of the construction target around the shovel 100, or an image representing the shape of the construction target around the shovel 100. It may be an image of three-dimensional data representing the shape of the construction target.
  • the processed image is, for example, an image obtained by performing viewpoint conversion processing or the like on a captured image acquired by the imaging device 40.
  • an image TG representing the shape of the construction target around the shovel 100 as seen from the viewpoint of the cabin 10 of the shovel 100 is displayed on the screen.
  • the image TG corresponds to an image of three-dimensional data of the construction target around the shovel 100.
  • the image CG representing the shovel 100 (attachment AT) may be displayed on the screen in a manner that matches the positional relationship with the image TG representing the shape of the construction target around the shovel 100.
  • an image representing the shape of the construction target around the shovel 100 as seen from a predetermined viewpoint around the shovel 100 may be displayed on the screen.
  • the viewpoint of the image representing the shape of the construction target around the shovel 100 displayed on the screen may be arbitrarily changeable according to a predetermined input from the user through the input device 52. Thereby, the user can check the image representing the shape of the work target around the shovel 100 from his/her favorite viewpoint.
  • the screen related to the generation of the trajectory of the work part of the shovel 100 includes, for example, an operation input operation for the user to select a target work to be performed by the shovel 100 from among a plurality of works through the work selection section 302C. Includes screen.
  • the user can use the input device 52 to perform an operation to select a target task to be performed by the shovel 100 from among a plurality of tasks.
  • the user inputs an operation to set specifications regarding the combination of a plurality of basic operations in the compound operation of the excavator 100, for example, through the specification setting section 302E.
  • the user can use the input device 52 to perform an operation to set specifications regarding a combination of a plurality of basic operations in the compound operation of the excavator 100.
  • the user can input an operation to cause the trajectory generating part 302F to generate a trajectory of the working part of the excavator 100, for example, according to the specifications set by the specification setting part 302E.
  • the user can use the input device 52 to instruct the trajectory generation section 302F to generate a trajectory for the work area of the shovel 100 according to the set specifications.
  • the screen related to the generation of the trajectory of the working part of the shovel 100 includes a screen that displays the trajectory of the working part of the shovel 100 corresponding to the data generated by the trajectory generating section 302F.
  • the trajectory of the work area of the shovel 100 may be displayed superimposed on an image representing the shape of the construction target around the shovel 100.
  • a moving image of a simulation of the shovel 100 in which the work part moves along the trajectory may be displayed on the screen. Thereby, the user can grasp in advance how the compound operation of the shovel 100 will be performed based on the trajectory of the work part in the compound operation of the shovel 100 that corresponds to the data generated by the trajectory generation unit 302F. be able to.
  • an image representing the predicted shape of the work target after the work part moves along the trajectory may be displayed on the screen.
  • the user can determine how the shape of the work object will be by having the shovel 100 perform a compound operation based on the trajectory of the work part in the compound operation of the shovel 100 that corresponds to the data generated by the trajectory generation unit 302F. It is possible to know in advance whether the situation will change.
  • a screen related to generation of the trajectory of the working part of the shovel 100 includes a screen that automatically operates the shovel 100 so that the working part of the shovel 100 moves along the trajectory corresponding to the data generated by the trajectory generating section 302F.
  • Contains an operation screen for On the operation screen the user can use the input device 52 to issue an instruction to operate the shovel 100 so that the work part moves along the trajectory generated by the trajectory generation section 302F.
  • the motion control section 302G causes the work portion of the bucket 6 to move along the trajectory corresponding to the data generated by the trajectory generation section 302F in response to an instruction to execute the motion from the user, which is output in response to an operation on the operation screen.
  • the shovel 100 is operated to move.
  • the operation screen is common to the screen that displays the trajectory of the working part of the shovel 100 corresponding to the data generated by the trajectory generation unit 302F, and includes icons corresponding to instructions for executing the operation.
  • the user can confirm the trajectory of the working part of the shovel 100 and then input an instruction to execute the operation of the shovel 100 through the input device 52 or the like.
  • the display processing unit 302H may transmit data regarding a screen related to the generation of the trajectory of the working part of the excavator 100 to the remote operation support device 300 via the communication device 60. Thereby, the display processing unit 302H can cause the remote operation support device 300 (display device) to display a screen related to the generation of the target trajectory of the work part of the excavator 100.
  • the function of the display processing section 302H may be provided in the remote operation support device 300.
  • This allows the display device of the remote operation support device 300 to display a screen related to the generation of the trajectory of the work site of the shovel 100. Therefore, the user (operator) of the remote operation support device 300 uses the screen displayed on the display device of the remote operation support device 300 to input data representing the trajectory of the work part on the compound trajectory of the excavator 100 to the trajectory generation unit 302F. can be generated.
  • the excavator 100 when the excavator 100 is remotely controlled, some or all of the functions of the work object shape acquisition section 302B, work selection section 302C, trajectory generation section 302D, specification setting section 302E, trajectory generation section 302F, and operation control section 302G are performed. may be provided in the remote operation support device 300. Further, some or all of the functions of the work object shape acquisition unit 302B, work selection unit 302C, trajectory generation unit 302D, specification setting unit 302E, trajectory generation unit 302F, and operation control unit 302G are transferred to the information processing device 200. It's okay. Thereby, it is possible to reduce the processing load on the shovel 100 and the remote operation support device 300 regarding the processing related to the generation of the trajectory of the work part in the compound operation of the shovel 100 and the control of the operation of the shovel 100.
  • FIG. 13 is a diagram showing a first example (setting screen 1300) of a setting screen for specifications regarding a combination of a plurality of basic operations in a compound operation of the excavator 100.
  • the setting screen 1300 in FIG. 13 is displayed on the display device of the remote operation support device 300 and is configured to be operable through the input device of the remote operation support device 300.
  • the same may apply to the setting screen 1400 in FIG. 14, which will be described later.
  • the settings screen 1300 includes an image 1301, an operation image 1302, and an operation image 1303.
  • Image 1301 is an image that schematically represents the basic operation of shovel 100.
  • Image 1301 includes images 1301A to 1301C.
  • Image 1301A is an image schematically representing the horizontal pulling operation of shovel 100.
  • Image 1301B is an image schematically representing the rolling operation of shovel 100.
  • Image 1301C is an image schematically representing the sweeping operation of shovel 100.
  • Images 1301A to 1301C are arranged vertically to the left of the center of the setting screen 1300 in the horizontal direction.
  • the operation images 1302 and 1303 are images that can be operated through the input device 52.
  • the operation image 1302 is an operation image for adjusting the distribution of the combination of the horizontal pulling operation, the rolling operation, and the sweeping operation.
  • the operation image 1302 includes operation images 1302A to 1302C.
  • the operation image 1302A is a slide bar that can adjust the composition ratio (compound ratio) of the horizontal pulling operation in the composite operation of the shovel 100 between 0% and 100%.
  • the operation image 1302A is placed to the right of the image 1301A.
  • the operation image 1302B is a slide bar that can adjust the composition ratio (compound ratio) of the rolling operation in the composite operation of the shovel 100 between 0% and 100%.
  • the operation image 1302B is placed to the right of the image 1301B.
  • the operation image 1302C is a slide bar that can adjust the composition ratio (compound ratio) of the sweeping operation in the composite operation of the shovel 100 between 0% and 100%.
  • the operation image 1302C is placed to the right of the image 1301C.
  • the operation images 1302A to 1302C are arranged vertically on the right side of the center of the setting screen 1300 in the horizontal direction.
  • the combined ratio of each of the horizontal pulling action, rolling action, and sweeping action is adjusted so that the sum of the combined ratio of the horizontal pulling action, rolling action, and sweeping action is 100%.
  • the slide bars of the operation images 1302A to 1302C are all set to 0% in the initial state, and if any slide bar is specified as a percentage greater than 0%, the other slide bars are set to a total of 100%. Movement is limited to a range not exceeding .
  • the operation image 1303 determines the composite rate of the horizontal pulling operation, compaction operation, and sweeping operation in the combined operation of the shovel 100 specified by the operation image 1302, and represents the trajectory of the work area by the trajectory generation unit 302F. This is an image for operation to execute data generation processing.
  • the slide bar of the operation image 1302B is set to 0%, and the operation images 1302A and 1302B are both set to approximately 50%.
  • the user can control the trajectory of the work area in the combined operation of the shovel 100 by combining the sweeping operation and the horizontal pulling operation in accordance with the combination ratio specified in the operation image 1302. (See FIG. 11).
  • the trajectory generating unit 302F will not perform the compound operation of the excavator 100. , it is possible to generate the trajectory of the work part according to the basic motion.
  • FIG. 14 is a diagram showing a first example (setting screen 1400) of a setting screen for specifications regarding a combination of a plurality of basic operations in a compound operation of the excavator 100.
  • the settings screen 1400 includes an image 1401, an operation image 1402, and an operation image 1403.
  • Image 1401 is an image that schematically represents the basic operation of shovel 100.
  • Image 1401 includes images 1401A to 1401C.
  • Image 1401A is an image schematically representing the horizontal pulling operation of shovel 100.
  • the image 1401A is arranged at the center of the setting screen 1400 in the horizontal direction and above the center in the vertical direction.
  • Image 1401B is an image schematically representing the rolling operation of shovel 100.
  • the image 1401B is arranged to the right of the center in the horizontal direction and below the center in the vertical direction of the setting screen 1400.
  • Image 1401C is an image schematically representing the sweeping operation of shovel 100.
  • the image 1401C is arranged to the left of the center in the horizontal direction and below the center in the vertical direction of the setting screen 1400.
  • Images 1401A to 1401C are arranged on the setting screen 1400 to approximately form an equilateral triangle.
  • the operation images 1402 and 1403 are images that can be operated through the input device 52.
  • the operation image 1402 is an operation image for adjusting the distribution of the combination of the horizontal pulling operation, the rolling operation, and the sweeping operation.
  • the operation image 1402 includes operation images 1402A to 1402D.
  • the operation image 1402A is a circular shape representing a state in which the composition ratio (compound ratio) of the horizontal pulling operation of the shovel 100 in the compound operation of the shovel 100 is 100%, that is, a state in which the shovel 100 performs the horizontal pulling operation instead of the compound operation. This is an image of The operation image 1402A is arranged below and adjacent to the image 1401A.
  • the operation image 1402B is a circular image representing a state in which the composition ratio of the rolling operation in the compound operation of the shovel 100 is 100%, that is, a state in which the shovel 100 performs the rolling operation instead of the compound operation.
  • the operation image 1402B is arranged to the left of the image 1401A.
  • the operation image 1402C is a circular image representing a state in which the composition ratio of the sweeping operation in the compound operation of the shovel 100 is 100%, that is, a state in which the shovel 100 performs the sweeping operation instead of the compound operation.
  • the operation image 1402C is placed to the right of the image 1401C.
  • the operation images 1402A to 1402C are arranged to approximately form an equilateral triangle, and the operation images 1402A, 1402B, the operation images 1402B, 1402C, and the operation images 1402C, 1402A are connected by straight lines.
  • the operation images 1402A to 1402C may be referred to as "operation images 1402X" without distinguishing them individually.
  • the operation image 1402D is an image of a circular cursor that specifies the composition ratio (compound ratio) of the horizontal pulling operation, rolling operation, and sweeping operation in the combined operation of the shovel 100.
  • the operation image 1402D can be moved above or within the triangle formed by the operation images 1402A to 1402C.
  • composition ratio (compound ratio) of the horizontal pulling operation, rolling operation, and sweep-out operation in the composite operation of the shovel 100 is determined by the cursor of the operation image 1402D on or inside the triangle formed by the operation images 1402A to 1402C. It is uniquely specified by the position. Specifically, the smaller the distance between one operation image 1402X and the operation image 1402D is relative to the distance between each of the other two operation images 1402X and the operation image 1402D, the more the one operation The composite rate of the basic motion corresponding to the image is specified to be large.
  • the distance between the first operation image 1402X and the operation image 1402D increases in relation to the distance between the other two operation images 1402X and the operation image 1402D, the distance between the first operation image 1402X and the operation image 1402D increases. is specified so that the compound rate of the basic motion corresponding to is small.
  • the operation image 1403 determines the composite rate of the horizontal pulling operation, compaction operation, and sweeping operation in the combined operation of the shovel 100 specified by the operation image 1402, and represents the trajectory of the work area by the trajectory generation unit 302F. This is an image for operation to execute data generation processing.
  • the composite rate of the rolling operation is designated as 0%.
  • the composite rate of each of the horizontal pulling operation and sweeping operation in the composite operation of the shovel 100 is specified by the position of the operation image 1402D on the line segment (one side) between the operation images 1402A and 1402C.
  • the trajectory generation unit 302F performs a composite operation of the combination of the horizontal pulling operation and the sweeping operation of the shovel 100 in accordance with the composite rate according to the position of the operation image 1402D.
  • Generate data representing the trajectory of the work part As a result, the user can create data representing the trajectory of the work area due to a combination of horizontal pulling, sweeping, and rolling operations of the excavator 100, which is a combination of the horizontal pulling and sweeping operations (Fig. (see 11).
  • the composite rate of the sweeping operation is designated as 0%.
  • the composite rate of each of the horizontal pulling operation and the rolling operation in the composite operation of the shovel 100 is specified by the position of the operation image 1402D on the line segment (one side) between the operation images 1402A and 1402B.
  • the trajectory generation unit 302F performs a composite operation of the combination of the horizontal pulling operation and the rolling operation of the shovel 100 in accordance with the composite rate according to the position of the operation image 1402D.
  • Generate data representing the trajectory of the work part As a result, the user can create data representing the trajectory of the work area due to a combined operation of the horizontal pulling operation, sweeping operation, and rolling operation of the shovel 100, which is a combination of the horizontal pulling operation and the rolling operation.
  • the composite rate of the horizontal pulling operation is specified as 0%.
  • the composite rate of each of the rolling operation and sweeping operation in the composite operation of the shovel 100 is specified by the position of the operation image 1402D on the line segment (one side) between the operation images 1402B and 1402C.
  • the trajectory generation unit 302F performs a composite operation of the combination of the rolling operation and sweeping operation of the excavator 100 in accordance with the composite rate according to the position of the operation image 1402D.
  • Generate data representing the trajectory of the work part Thereby, the user can create data representing the trajectory of the work area by a combination of the horizontal pulling operation, sweeping operation, and rolling operation of the shovel 100, which is a combination of the rolling operation and the sweeping operation.
  • the composite ratios of the horizontal pulling operation, rolling operation, and sweeping operation are all greater than 0%. It is specified. Specifically, the composite rate of each of the horizontal pulling operation, the rolling operation, and the sweeping operation is determined depending on the relative magnitude of the distance between the operation image 1402D and each of the operation images 1402A to 1402C. The total is specified to be 100%.
  • the trajectory generation unit 302F controls the horizontal pulling operation, rolling operation, and sweeping operation of the shovel 100 according to the composite rate according to the position of the operation image 1402D. Generate data representing the trajectory of the work part in the combined compound motion. Thereby, the user can create data representing the trajectory of the work area by the combined operation of all the combinations of the horizontal pulling operation, sweep-out operation, and compaction operation of the shovel 100.
  • the trajectory generation unit 302F can generate the trajectory of the work part based on the basic motion of the shovel 100, rather than the compound motion. .
  • FIG. 15 is a flowchart schematically showing an example of processing related to generating the trajectory of the working part of the shovel 100.
  • the flowchart in FIG. 15 is started when the controller 30 receives a predetermined input from the input device 52 or the remote operation support device 300 for performing processing related to generating the trajectory of the working part of the excavator 100.
  • step S102 the work target shape acquisition unit 302B acquires data of the latest captured image of the imaging device 40, and represents the shape of the work target around the shovel 100 based on the captured image. Get data.
  • step S102 Upon completion of the process in step S102, the controller 30 proceeds to step S104.
  • step S104 the display processing unit 302H displays an image representing the shape of the work target around the shovel 100 on the display device 50A or the display device of the remote operation support device 300, based on the data acquired in step S102 ( (See Figure 12).
  • step S104 Upon completion of the process in step S104, the controller 30 proceeds to step S106.
  • step S106 the display processing unit 302H causes the display device 50A or the display device of the remote operation support device 300 to display a screen for selecting one task from among a plurality of task candidates.
  • the display processing unit 302H displays a plurality of work candidate options, superimposed on an image representing the shape of the work target around the shovel 100.
  • step S106 Upon completion of the process in step S106, the controller 30 proceeds to step S108.
  • step S108 the work selection unit 302C selects work candidates (for example, grading work, excavation work, slope construction work, etc.) in response to a predetermined input on the selection screen in step S106, which is received through the input device 52 or the like. ) Select one task from among them.
  • work candidates for example, grading work, excavation work, slope construction work, etc.
  • step S108 Upon completion of the process in step S108, the controller 30 proceeds to step S110.
  • step S110 the trajectory generation unit 302D uses the learned model LM1 to generate trajectories of the work part for each of the plurality of basic movements corresponding to the work selected in step S106, based on the data acquired in step S102. generate.
  • step S110 Upon completion of the process in step S110, the controller 30 proceeds to step S112.
  • step S112 the display processing unit 302H causes the display device 50A or the display device of the remote operation support device 300 to display a setting screen for specifications regarding the combination of a plurality of basic operations.
  • step S112 Upon completion of the process in step S112, the controller 30 proceeds to step S114.
  • step S114 the specification setting unit 302E sets specifications regarding the combination of a plurality of basic operations in response to a predetermined input on the setting screen in step S112, which is received through the input device 52 or the like.
  • step S114 Upon completion of the process in step S114, the controller 30 proceeds to step S116.
  • step S116 the trajectory generation unit 302F uses the learned models LM2 and LM3 to determine the work area of the excavator 100 due to the compound motion, according to the trajectory data generated in step S110 and the settings in step S114. Generate data representing the trajectory.
  • step S116 Upon completion of the process in step S116, the controller 30 proceeds to step S118.
  • step S118 the display processing unit 302H causes the display device 50A or the display device of the remote operation support device 300 to display an image representing the trajectory of the work area due to the combined motion of the shovel 100 corresponding to the data generated in step S116. .
  • step S118 Upon completion of the process in step S118, the controller 30 proceeds to step S120.
  • step S120 the controller 30 determines whether an input instructing the shovel 100 to perform an operation to move the working part of the bucket 6 along the trajectory corresponding to the data generated in step S118 has been received. judge. If the controller 30 receives an input instructing the shovel 100 to perform the operation, the controller 30 proceeds to step S122. On the other hand, if the controller 30 receives any other input, specifically, if another input is received to generate the trajectory of the work site by the combined motion of the shovel 100, the process returns to step S106.
  • step S122 the operation control unit 302G controls the hydraulic control valve 31 to automatically move the excavator 100 so that the working part of the bucket 6 moves in a trajectory corresponding to the data generated in the process of the most recent step S116. Make it work.
  • step S122 the controller 30 ends the process of the current flowchart.
  • the shovel 100 upon completion of the process in step S122, the shovel 100 (attachment AT) may be in a state where the working part of the bucket 6 is at the end point of the trajectory.
  • the posture may be returned to the state before the start of the process in step S122.
  • the support device 150 (controller 30) is able to generate the trajectory of the work part by the combined motion of the excavator 100 in accordance with the specifications regarding the combination of a plurality of basic motions set by the user. can. Therefore, the working efficiency of the shovel 100 can be improved.
  • the support device includes an input section and a display section.
  • the support device is, for example, the support device 150, the information processing device 200, or the remote operation support device 300.
  • the input unit is, for example, the input device 52 described above or the input device of the remote operation support device 300. Specifically, the input unit receives input from the user. Then, the display unit determines the specifications regarding the combination of multiple operations of different types of the working machine in accordance with the input from the input unit, and displays the trajectory of the working part of the working machine by the composite operation that combines the multiple operations. A first operation screen for generation is displayed.
  • the working machine is, for example, the above-mentioned shovel 100.
  • the working machine may be the above-mentioned crane, forklift, or road machine.
  • the display unit is, for example, the above-described display device 50A or the display device of the remote operation support device 300.
  • the first operation screen is, for example, the settings screen 1300 or the settings screen 1400 described above.
  • the program causes an information processing device having an input section and a display section to execute the display step.
  • the display step specifications regarding combinations of multiple operations of different types of the work machine are determined in accordance with input from the input unit, and the work part of the work machine is determined by a composite operation that combines multiple operations.
  • a first operation screen for generating a trajectory is displayed on the display unit.
  • the display step is, for example, step S112.
  • the user may be able to perform an operation to determine the specifications through the input unit.
  • the user can generate a trajectory of the work part based on a compound motion of the shovel 100, which is a combination of a plurality of motions. Therefore, the user operates the work machine so that the work part moves along the trajectory, or activates the automatic operation function of the work machine so that the work part moves along the track. It is possible to proceed with the work of the working machine. Therefore, it is possible to improve work efficiency when performing work by combining a plurality of basic operations of the work machine.
  • each of the plurality of operations may be an operation performed by the work machine using a work attachment in a predetermined work.
  • the work attachment is, for example, the above-mentioned attachment AT.
  • the user can generate a trajectory of the work part by the combined motion of the work machine in accordance with a predetermined work performed by the work machine.
  • the predetermined work may be earth leveling work, excavation work, or slope construction work.
  • the plurality of operations may include at least two of a leveling operation, a compaction operation, a broom operation, an excavation operation, and an earth removal operation.
  • the plurality of operations may include at least two of the following: a digging operation, a boom-raising turning movement, a boom-lowering turning movement, an earth removal operation, and a broom movement.
  • the plurality of operations may include a cutting operation and a compaction operation when the predetermined operation is a slope construction operation.
  • the user can generate a trajectory of the work site by the combined operation of the work machine in accordance with the earth leveling work, excavation work, or slope construction work.
  • the trajectory of the working part may be the trajectory of a predetermined part set on the end attachment at the tip of the working attachment of the working machine.
  • the end attachment is, for example, the bucket 6.
  • the user can generate a trajectory of a predetermined part of the end attachment at the tip of the work attachment, which is the work part that comes into contact with the work object in the work machine.
  • the above-mentioned specifications may include the distribution of combinations of a plurality of operations in a compound operation of the work machine.
  • the user can generate a more appropriate trajectory for the work part by the combined movement of the excavator 100 in the support device by appropriately adjusting the distribution (for example, composition ratio) of the combination of multiple movements in the combined movement of the work machine. can be done.
  • the distribution for example, composition ratio
  • the display unit displays an image representing the shape of the work target around the work machine, and also displays an image that is superimposed on the image representing the shape of the work target around the work machine, and displays the image on the first operation screen.
  • the trajectory generated through the process may also be displayed.
  • the image representing the shape of the work target may be a captured image of the vicinity of the work machine or a processed image thereof, or an image of three-dimensional data of the work target around the work machine.
  • the display unit displays a moving image of a simulation of the work machine in which the work part moves along a trajectory generated through the first operation screen, superimposed on the image representing the shape of the work target. May be displayed.
  • the user can confirm in advance how the excavator 100 performs a compound operation so that the work part moves along the generated trajectory.
  • the display unit may display an image representing the predicted shape of the work target after the work part moves on the trajectory generated through the first operation screen.
  • the display unit displays a second operation screen for operating the work machine to move the work site along a trajectory generated through the first operation screen in response to an input from the input unit. may be displayed.
  • the user uses the input unit to operate the instruction to operate the work machine so that the work part moves along the trajectory generated through the first operation screen. It may be possible to do so.
  • the user can cause the excavator 100 to automatically perform a compound operation so that the work part moves along the generated trajectory. Therefore, for example, even if the user (operator) is not an expert, the work part of the work machine can be moved along the generated trajectory, and as a result, the work efficiency of the work machine can be further improved. Can be done.
  • the work machine may include the above-mentioned support device.
  • a working machine has an input section that accepts input from the user, determines specifications regarding the combination of multiple operations of different types of the working machine according to the input from the input section, and performs a composite operation that combines multiple operations.
  • the display unit may also include a display unit that displays a first operation screen for generating a trajectory of a working part of the working machine due to the motion.
  • the user (operator) riding the working machine can use the support device to generate a trajectory of the work area due to the combined motion of the shovel 100.
  • the support system may include a work machine and the above-mentioned support device. That is, the support system is, for example, the operation support system SYS described above.
  • a user external to the work machine can use the support device to generate a trajectory of the work area due to the combined motion of the shovel 100.
  • an acquisition unit that acquires data regarding the shape of a work target around the work machine; a first generation unit that generates a trajectory of a working part of the work machine based on the data acquired by the acquisition unit, based on a composite operation that combines a plurality of operations of the work machine; Work support system.
  • the work support system is, for example, the operation support system SYS described above.
  • the working machine is, for example, the above-mentioned shovel 100.
  • the acquisition unit is, for example, the work target shape acquisition unit 302B described above.
  • the first generation unit is the above-mentioned trajectory generation unit 302F.
  • (2) a second generation unit that generates a trajectory of the work part for each of the plurality of operations based on the data acquired by the acquisition unit;
  • the first generation unit generates a trajectory of the work part according to the composite operation based on the trajectory of the work part for each of the plurality of movements, which is generated by the second generation unit.
  • the second generation unit is, for example, the trajectory generation unit 302D described above.
  • (3) comprising a selection unit that selects the plurality of operations from among the operations of the work machine based on the data acquired by the acquisition unit;
  • the selection unit is, for example, the above-mentioned work selection unit 302C.
  • the first generation unit generates a trajectory of the work part by the composite operation in response to an input from the input unit regarding specifications of the combination of the plurality of operations.
  • the work support system described in (2) The work support system described in (2).
  • the input unit is, for example, the input device 52 described above or the input device of the remote operation support device 300.
  • the first generation unit adjusts a blending ratio of the feature amounts for each of the plurality of actions in response to an input from the input unit regarding specifications of the combination of the plurality of actions, and generates a combination ratio of the feature amounts for each of the plurality of actions. generating a trajectory of the working part of the compound motion based on the The work support system described in (4).
  • the extraction unit is, for example, the trajectory generation unit 302F described above.
  • (6) a display unit that displays an image representing the shape of the work target based on the data acquired by the acquisition unit;
  • the display unit displays the trajectory generated by the first generation unit superimposed on an image representing the shape of the work target.
  • the display unit is, for example, the display device 50A or the display device of the remote operation support device 300.
  • the display unit displays a moving image of the work part moving along a trajectory generated by the first generation unit, superimposed on an image representing the shape of the work target.
  • the work support system described in (6) The work support system described in (6).
  • the display unit displays an image representing a predicted shape of the work target after the work part moves on the trajectory generated by the first generation unit.
  • the control unit is, for example, the operation control unit 302G described above.
  • the information processing device is, for example, the above-mentioned controller 30, information processing device 200, or remote operation support device 300.
  • an acquisition unit that acquires data regarding the shape of a work target around the work machine; a first generation unit that generates a trajectory of a working part of the working machine based on the data acquired by the acquisition unit by a composite operation that combines a plurality of operations of the working machine; working machine.

Landscapes

  • Engineering & Computer Science (AREA)
  • Mining & Mineral Resources (AREA)
  • Civil Engineering (AREA)
  • General Engineering & Computer Science (AREA)
  • Structural Engineering (AREA)
  • Operation Control Of Excavators (AREA)

Abstract

L'invention concerne une technologie avec laquelle il est possible d'améliorer l'efficacité de travail lors de la réalisation d'un travail par combinaison de multiples opérations d'une machine. Un dispositif d'assistance (150) selon un mode de réalisation de la présente divulgation comprend : un dispositif d'entrée (52) qui reçoit une entrée d'un utilisateur ; et un dispositif d'affichage (50A) qui affiche un écran de configuration (1300, 1400) destiné à déterminer, sur la base de l'entrée provenant du dispositif d'entrée (52), une spécification pour une combinaison de multiples types différents d'opérations de base d'une excavatrice (100) et pour générer un trajet pour une partie du travail de l'excavatrice (100) sur la base d'une opération combinée obtenue par combinaison des multiples opérations de base.
PCT/JP2023/029009 2022-08-09 2023-08-08 Dispositif d'assistance, engin de chantier, système d'assistance et programme WO2024034624A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2022-127437 2022-08-09
JP2022127437 2022-08-09

Publications (1)

Publication Number Publication Date
WO2024034624A1 true WO2024034624A1 (fr) 2024-02-15

Family

ID=89851874

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/029009 WO2024034624A1 (fr) 2022-08-09 2023-08-08 Dispositif d'assistance, engin de chantier, système d'assistance et programme

Country Status (1)

Country Link
WO (1) WO2024034624A1 (fr)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040267404A1 (en) * 2001-08-31 2004-12-30 George Danko Coordinated joint motion control system
WO2019189260A1 (fr) * 2018-03-27 2019-10-03 住友重機械工業株式会社 Excavatrice
WO2019187519A1 (fr) * 2018-03-30 2019-10-03 住友建機株式会社 Excavateur et dispositif de traitement d'informations
JP2021095718A (ja) * 2019-12-16 2021-06-24 住友重機械工業株式会社 ショベル、情報処理装置
JP2021181732A (ja) * 2020-05-20 2021-11-25 住友重機械工業株式会社 ショベル

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040267404A1 (en) * 2001-08-31 2004-12-30 George Danko Coordinated joint motion control system
WO2019189260A1 (fr) * 2018-03-27 2019-10-03 住友重機械工業株式会社 Excavatrice
WO2019187519A1 (fr) * 2018-03-30 2019-10-03 住友建機株式会社 Excavateur et dispositif de traitement d'informations
JP2021095718A (ja) * 2019-12-16 2021-06-24 住友重機械工業株式会社 ショベル、情報処理装置
JP2021181732A (ja) * 2020-05-20 2021-11-25 住友重機械工業株式会社 ショベル

Similar Documents

Publication Publication Date Title
CN112867831B (zh) 挖土机
JP7358349B2 (ja) 掘削機、情報処理装置
KR102602384B1 (ko) 쇼벨
EP3489422B1 (fr) Commande d'une machine de terrassement
JPWO2019189030A1 (ja) ショベル
CN109837942B (zh) 控制土方机器
JPWO2019189935A1 (ja) ショベル
WO2019189920A1 (fr) Machine de travail et dispositif de traitement d'informations
JP2023174887A (ja) 作業機械、情報処理装置
JPWO2020162428A1 (ja) ショベル
WO2024034624A1 (fr) Dispositif d'assistance, engin de chantier, système d'assistance et programme
JP7404280B2 (ja) ショベル
JP2022154722A (ja) ショベル
FI20180082A1 (en) Control of earthmoving machine
WO2023190877A1 (fr) Dispositif d'assistance, machine de travail, programme
WO2023190843A1 (fr) Dispositif d'assistance, engin de chantier et programme
JP2023034771A (ja) 建設機械の支援システム、建設機械の支援プログラム
JP2023150920A (ja) 支援装置、作業機械、プログラム
JP2023150082A (ja) 支援装置、作業機械、プログラム
WO2024111596A1 (fr) Machine de travail, dispositif de traitement d'informations et programme
JP2024076741A (ja) 作業機械、情報処理装置、プログラム
WO2022196530A1 (fr) Système d'assistance, dispositif de traitement d'informations et programme
JP2021156080A (ja) 施工支援システム及び施工支援装置
JP2023183992A (ja) 支援システム、遠隔操作支援装置、作業機械、プログラム
WO2022210613A1 (fr) Pelle et dispositif de commande de pelle

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23852586

Country of ref document: EP

Kind code of ref document: A1