US20250283308A1 - Working machine, information processing device, and program - Google Patents

Working machine, information processing device, and program

Info

Publication number
US20250283308A1
US20250283308A1 US19/211,957 US202519211957A US2025283308A1 US 20250283308 A1 US20250283308 A1 US 20250283308A1 US 202519211957 A US202519211957 A US 202519211957A US 2025283308 A1 US2025283308 A1 US 2025283308A1
Authority
US
United States
Prior art keywords
working machine
shovel
state
target material
future
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
US19/211,957
Other languages
English (en)
Inventor
Kosuke Hara
Ryuji Tsuzuki
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sumitomo Heavy Industries Ltd
Original Assignee
Sumitomo Heavy Industries Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sumitomo Heavy Industries Ltd filed Critical Sumitomo Heavy Industries Ltd
Assigned to SUMITOMO HEAVY INDUSTRIES, LTD. reassignment SUMITOMO HEAVY INDUSTRIES, LTD. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: TSUZUKI, RYUJI, HARA, KOSUKE
Publication of US20250283308A1 publication Critical patent/US20250283308A1/en
Pending legal-status Critical Current

Links

Images

Classifications

    • E—FIXED CONSTRUCTIONS
    • E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02F—DREDGING; SOIL-SHIFTING
    • E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
    • E02F9/26—Indicating devices
    • E02F9/261—Surveying the work-site to be treated
    • E02F9/262—Surveying the work-site to be treated with follow-up actions to control the work tool, e.g. controller
    • E—FIXED CONSTRUCTIONS
    • E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02F—DREDGING; SOIL-SHIFTING
    • E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
    • E02F9/20—Drives; Control devices
    • E—FIXED CONSTRUCTIONS
    • E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02F—DREDGING; SOIL-SHIFTING
    • E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
    • E02F9/20—Drives; Control devices
    • E02F9/2025—Particular purposes of control systems not otherwise provided for
    • E02F9/205—Remotely operated machines, e.g. unmanned vehicles
    • E—FIXED CONSTRUCTIONS
    • E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02F—DREDGING; SOIL-SHIFTING
    • E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
    • E02F9/26—Indicating devices
    • E—FIXED CONSTRUCTIONS
    • E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02F—DREDGING; SOIL-SHIFTING
    • E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
    • E02F9/26—Indicating devices
    • E02F9/261—Surveying the work-site to be treated

Definitions

  • an information processing device includes an operation planning part configured to determine a future operation of a working machine from among a plurality of operations according to a performance status of an operation of the working machine.
  • a non-transitory computer-readable recording medium having a program embodied therein causes an information processing device to determine a future operation of a working machine from among a plurality of operations according to a performance status of an operation of the working machine.
  • a non-transitory computer-readable recording medium having a program embodied therein a support to perform causes device determining a future operation of a working machine from among a plurality of operations according to a performance status of an operation of the working machine, and notifying an operator of the determined future operation.
  • FIG. 1 is a drawing illustrating an example of an operation support system
  • FIG. 2 is a top view illustrating an example of a shovel
  • FIG. 3 is a drawing illustrating an example of a configuration related to remote control of the shovel
  • FIG. 4 is a drawing illustrating an example of a hardware configuration of the shovel
  • FIG. 5 is a drawing illustrating an example of a hardware configuration of an information processing device
  • FIG. 6 is a functional block diagram illustrating a first example of a functional configuration of the operation support system
  • FIG. 7 is a drawing describing an example of a relationship between a timing of processing in shovel operation planning and a planned operation
  • FIG. 8 is a drawing describing another example of the relationship between the timing of processing in shovel operation planning and the planned operation
  • FIG. 9 is a state transition diagram illustrating an example of a transition of a shovel operation in slope construction work
  • FIG. 10 is a state transition diagram illustrating an example of a transition of a shovel operation in ground leveling work
  • FIG. 11 is a functional block diagram illustrating a second example of the functional configuration of the operation support system
  • FIG. 12 is a flowchart schematically illustrating an example of processing related to a start of an autonomous operation of the shovel
  • FIG. 13 is a main flowchart schematically illustrating an example of processing related to shovel operation planning and bucket path generation
  • FIG. 14 is a drawing illustrating an example of an observation target area
  • FIG. 15 is a sub-flowchart schematically illustrating an example of processing related to the bucket path generation
  • FIG. 16 is a drawing illustrating an example of cost conditions and operation parameters corresponding to a plurality of sections of a shovel excavation operation.
  • FIG. 17 is a flowchart schematically illustrating an example of processing related to shovel operation control.
  • the working machine can perform the more appropriate operation.
  • FIGS. 1 to 3 An outline of an operation support system SYS according to the present embodiment will be described with reference to FIGS. 1 to 3 .
  • FIG. 1 is a drawing illustrating an example of the operation support system SYS.
  • a left side view of a shovel 100 is shown.
  • FIG. 2 is a top view illustrating an example of the shovel 100 .
  • FIG. 3 is a drawing illustrating an example of a configuration related to remote control of the shovel 100 .
  • a direction in the shovel 100 or a direction viewed from the shovel 100 may be described by defining a direction in which an attachment AT extends in the top view of the shovel 100 (upward direction in FIG. 2 ) as “front”.
  • the operation support system SYS includes the shovel 100 , an information processing device 200 , and a sensor group 300 .
  • the operation support system SYS supports the operation of the shovel 100 by coordinating with the shovel 100 using the information processing device 200 .
  • the shovel 100 included in the operation support system SYS may be one or a plurality of shovels.
  • the shovel 100 is a working machine to be supported in the operation support system SYS.
  • the shovel 100 includes a lower traveling body 1 , an upper swivel body 3 , the attachment AT including a boom 4 , an arm 5 , and a bucket 6 , and a cabin 10 .
  • the lower traveling body 1 uses crawlers 1 C to cause the shovel 100 to travel.
  • the crawlers 1 C include a left crawler 1 CL and a right crawler 1 CR.
  • the crawler 1 CL is hydraulically driven by a traveling hydraulic motor 1 ML.
  • the crawler 1 CR is hydraulically driven by a traveling hydraulic motor 1 MR.
  • the lower traveling body 1 can move by itself.
  • the upper swivel body 3 is mounted on the lower traveling body 1 via a swivel mechanism 2 so as to be capable of swiveling (freely swivel).
  • the upper swivel body 3 swivels with respect to the lower traveling body 1 by hydraulically driving the swivel mechanism 2 by a swivel hydraulic motor 2 M.
  • the boom 4 is attached to the front center of the upper swivel body 3 so that it is capable of lifting with respect to a rotation axis along the lateral direction.
  • the arm 5 is rotatably attached to the tip of the boom 4 around a rotation axis oriented laterally.
  • the bucket 6 is rotatably attached to the tip of the arm 5 around a rotation axis oriented laterally.
  • the bucket 6 is an example of an end attachment, and is used, for example, for excavation work, slope work, and ground leveling work.
  • the bucket 6 is attached to the tip of the arm 5 in a manner that allows it to be easily replaced based on work requirements of the shovel 100 . That is, in place of the bucket 6 , a bucket of a different kind from the bucket 6 , for example, a relatively large bucket, a bucket for slope work, a bucket for dredging work, or the like may be attached to the tip of the arm 5 .
  • An end attachment other than a bucket for example, an agitator, a breaker, a crusher, or the like may be attached to the tip of the arm 5 .
  • a spare attachment for example, a quick coupling or a tilt rotator, may be provided between the arm 5 and the end attachment.
  • the boom 4 , the arm 5 , and the bucket 6 are hydraulically driven by the boom cylinder 7 , the arm cylinder 8 , and the bucket cylinder 9 , respectively.
  • the cabin 10 is a control cabin in which an operator rides and operates the shovel 100 .
  • the cabin 10 is mounted, for example, on the front left of the upper swivel body 3 .
  • the shovel 100 includes as equipment a communication device 60 and can communicate with the information processing device 200 through a predetermined communication line NW.
  • the communication line NW includes, for example, a local area network (LAN) of a work site.
  • the communication line NW may also include a wide area network (WAN).
  • the wide area network includes, for example, a mobile communication network terminated at a base station, a satellite communication network using a communication satellite, and an Internet network.
  • the communication line NW may also include, for example, a short-distance communication line based on a wireless communication standard such as WiFi or Bluetooth (registered trademark).
  • the shovel 100 causes driven elements such as the lower traveling body 1 (i.e., a pair of right and left crawlers 1 CL and 1 CR), the upper swivel body 3 , the boom 4 , the arm 5 , and the bucket 6 to operate according to the operation of the operator in the cabin 10 .
  • driven elements such as the lower traveling body 1 (i.e., a pair of right and left crawlers 1 CL and 1 CR), the upper swivel body 3 , the boom 4 , the arm 5 , and the bucket 6 to operate according to the operation of the operator in the cabin 10 .
  • the shovel 100 may be configured to be operable by the operator in the cabin 10 , or may be configured to be remotely operated (remote control) from the outside of the shovel 100 .
  • the interior of the cabin 10 may be unmanned.
  • the shovel 100 is dedicated to remote control, the cabin 10 may be omitted.
  • the description proceeds on the assumption that the operation of the operator includes at least one of the operation of an operating device 26 of the operator in the cabin 10 and the remote control of an external operator.
  • the remote control includes a mode in which the shovel 100 is operated by the operation input related to an actuator of the shovel 100 performed by the remote control support device 400 which can communicate with the shovel 100 through the communication line NW.
  • the remote control support device 400 may be provided separately from the information processing device 200 or may be the information processing device 200 .
  • the remote control support device 400 may be provided, for example, in a management center or the like which manages the work of the shovel 100 from the outside. Further, the remote control support device 400 may be a portable operation terminal, and in this case, the operator can perform the remote control of the shovel 100 while directly confirming the work status of the shovel 100 from surroundings of the shovel 100 .
  • the shovel may 100 transmit an image (hereinafter, “surrounding image”) representing the surrounding state including the front of the shovel 100 based on the captured image output by an imaging device mounted in the shovel to the remote control support device 400 through, for example, the communication device 60 described later. Further, the shovel 100 may transmit the captured image output from the imaging device to the remote control support device 400 through the communication device 60 , and the remote control support device 400 may process the captured image received from the shovel 100 to generate the surrounding image. Then, the remote control support device 400 may display the surrounding images representing the surrounding condition including the front of the shovel 100 on its own display device.
  • surrounding image an image representing the surrounding state including the front of the shovel 100 based on the captured image output by an imaging device mounted in the shovel to the remote control support device 400 through, for example, the communication device 60 described later. Further, the shovel 100 may transmit the captured image output from the imaging device to the remote control support device 400 through the communication device 60 , and the remote control support device 400 may process the captured image received from the shovel
  • various information images (information screens) displayed on an output device 50 (display device) inside the cabin 10 of the shovel 100 may similarly be displayed on the display device of the remote control support device 400 .
  • the operator using the remote control support device 400 can remotely operate the shovel 100 while confirming, for example, the display contents of the information screen and the image representing the surrounding condition of the shovel 100 displayed on the display device.
  • the shovel 100 may operate the actuator to drive the driven elements such as the lower traveling body 1 , the upper swivel body 3 , the boom 4 , the arm 5 , and the bucket 6 according to the remote control signal representing the contents of the remote control received from the remote control support device 400 by the communication device 60 .
  • the remote control may include a mode in which the shovel 100 is operated by, for example, external voice input or gesture input to the shovel 100 by a person (e.g., worker) around the shovel 100 . More specifically, the shovel 100 recognizes voices spoken by surrounding workers, gestures performed by workers, or the like, through a voice input device (e.g., microphone), a gesture input device (e.g., imaging device), or the like mounted in the shovel. Then, the shovel 100 may operate the actuator to drive the driven elements such as the lower traveling body 1 (right and left crawlers 1 C), the upper swivel body 3 , the boom 4 , the arm 5 , and the bucket 6 according to the contents of the recognized voices, gestures, or the like.
  • a voice input device e.g., microphone
  • a gesture input device e.g., imaging device
  • the shovel 100 may automatically operate the actuator regardless of the contents of the operator's operation.
  • the shovel 100 can achieve a function of automatically operating at least a part of the driven elements such as the lower traveling body 1 , the upper swivel body 3 , and the attachment AT, that is, what is called an “automatic operation function” or a “Machine Control (MC) function”.
  • an automated operation function or a “Machine Control (MC) function”.
  • the automatic operation function includes, for example, a semi-automatic operation function (operation-assisted MC function).
  • the semi-automatic operation function is a function of automatically operating driven elements (actuators) other than the driven element (actuator) to be operated in response to the operator's operation.
  • the automatic operation function may also include a fully automatic operation function (fully automatic MC function).
  • the fully automatic operation function is a function of automatically operating at least a part of the plurality of driven elements (hydraulic actuators) on the assumption that there is no operator's operation.
  • the semi-automatic operation function and the fully automatic operation function include, for example, a rule-based automatic operation function.
  • the rule-based automatic operation function is an automatic operation function in which the operation contents of the driven element (actuator) to be operated automatically are determined according to a previously defined rule.
  • the semi-automatic operation function and the fully automatic operation function may include an autonomous operation function.
  • the autonomous operation function is an automatic operation function in which the shovel 100 autonomously makes various determinations, and the operation contents of the driven element (hydraulic actuator) to be operated automatically are determined according to the determination results.
  • the operation of the shovel 100 may be remotely monitored.
  • a remote monitoring support device having the same function as the remote control support device 400 may be provided.
  • the remote monitoring support device is, for example, an information processing device 200 .
  • a monitor who is a user of the remote monitoring support device can monitor the operation of the shovel 100 while checking the surrounding image displayed on the display device of the remote monitoring support device.
  • the monitor determines that it is necessary from the viewpoint of safety, for example, the monitor can intervene in the operation or automatic operation of the shovel 100 by using the input device of the remote monitoring support device and by making a predetermined input, and can make the shovel 100 stop urgently.
  • the information processing device 200 coordinates with the shovel 100 by communicating with each other and supports the operation of the shovel 100 .
  • the information processing device 200 is, for example, a server device or a terminal device for management installed in a management office in the work site of the shovel 100 , or in a management center or the like that manages the operation status of the shovel 100 at a place different from the work site of the shovel 100 .
  • the server device may be an on-premises server, a cloud server, or an edge server.
  • the terminal device for management may be, for example, a stationary terminal device such as a desktop PC (Personal Computer), or a portable terminal device (portable terminal) such as a tablet terminal, a smartphone, or a laptop PC.
  • a worker at the work site, a supervisor who supervises the work, or a manager who manages the work site can carry the portable information processing device 200 and move around the work site.
  • an operator can carry the portable information processing device 200 into the cabin of the shovel 100 , for example.
  • the information processing device 200 acquires data related to the operating state from the shovel 100 , for example.
  • the information processing device 200 can determine the operating state of the shovel 100 and monitor presence or absence of abnormality of the shovel 100 .
  • the information processing device 200 can display data related to the operating state of the shovel 100 through the display device 208 described later, for example, and allow the user to confirm the data.
  • the information processing device 200 can, for example, make a learning model learn the operating state of the shovel 100 and generate a trained model for supporting the operation of the shovel 100 .
  • the information processing device 200 may transmit to the shovel 100 various data such as programs and reference data used in the processing of a controller 30 or the like to the shovel 100 .
  • the shovel 100 can perform various processes related to the operation of the shovel 100 using various data downloaded from the information processing device 200 .
  • the sensor group 300 is installed at the work site of the shovel 100 .
  • Target material includes, for example, earth in the work area around the shovel 100 .
  • the sensor group 300 is provided for each shovel 100 .
  • one sensor group 300 may be shared for the plurality of shovels 100 .
  • the sensor group 300 includes sensors 300 - 1 to 300 -M (M: an integer of 2 or more).
  • the sensors 300 - 1 to 300 -M measure the state of objects in the work site around the shovel 100 and acquire measurement data related to the state.
  • the objects in the work site include, for example, target materials (earth in the work area) around the shovel 100 , working machines such as other shovels and bulldozers, and working vehicles such as trucks for transporting earth around the shovel 100 .
  • the state of the objects includes a shape and characteristic of the objects.
  • the sensors 300 - 1 to 300 -M include, for example, ranging sensors (distance sensors).
  • the ranging sensors include, for example, LIDAR (Light Detecting and Ranging), millimeter wave radar, ultrasonic sensors, infrared sensors, and the like.
  • the sensors 300 - 1 to 300 -M may also include, for example, a stereo camera, a TOF (Time Of Flight) camera, and a 3D camera capable of acquiring data on distance (depth) in addition to two-dimensional images.
  • the sensors 300 - 1 to 300 -M may also include both the ranging sensor and the 3D camera.
  • the sensor group 300 can acquire measurement data representing the shape of the object at the work site around the shovel 100 .
  • a sensor capable of acquiring measurement data representing the shape of the object such as a ranging sensor or a 3D camera, may be conveniently referred to as a “shape sensor”.
  • the sensors 300 - 1 to 300 -M may include a multi-wavelength spectral camera.
  • the multi-wavelength spectral camera includes, for example, a multi-spectral camera or a hyperspectral camera.
  • the sensor group 300 can acquire measurement data representing a characteristic of the object at the work site around the shovel 100 , such as hardness and moisture content of the earth.
  • a sensor capable of acquiring measurement data representing the characteristics of the object such as a multi-wavelength spectral camera, may be conveniently referred to as a “characteristic sensor”.
  • the sensors 300 - 1 to 300 -M include a plurality of shape sensors.
  • the plurality of shape sensors may be provided at different locations in the work site around the shovel 100 , with their sensing ranges overlapping at least one other shape sensor's sensing range.
  • other shape sensors may be able to acquire the measurement data for the part of the object. Therefore, the sensor group 300 can more reliably acquire measurement data representing the shape of an object in the work site around the shovel 100 .
  • the sensors 300 - 1 to 300 -M may include a plurality of characteristic sensors. Further, the plurality of characteristic sensors may be provided at different locations in the work site around the shovel 100 , with their sensing ranges overlapping at least one other characteristic sensor's sensing range. Thus, for example, even if one characteristic sensor's measurement data cannot acquire data representing the shape of a part of an object within the sensing range due to occlusion, other characteristic sensors may be able to acquire the measurement data for the part of the object. Therefore, the sensor group 300 can more reliably acquire measurement data representing the characteristics of an object in the work site around the shovel 100 .
  • the sensors 300 - 1 to 300 -M may include a sensor having both the function of a shape sensor and the function of a characteristic sensor.
  • the sensors 300 - 1 to 300 -M may include a plurality of integrated sensors (hereinafter, “integrated sensor”). Further, the plurality of integrated sensors may be provided at different locations in the work site around the shovel 100 , with their sensing ranges overlapping at least one other integrated sensor's sensing range.
  • the sensor group 300 may simply include only one shape sensor or characteristic sensor.
  • the operation support system SYS may simply include only one sensor capable of acquiring measurement data related to the state of an object in the work site around the shovel 100 .
  • the sensors 300 - 1 to 300 -M may be fixed to the work site around the shovel 100 or mounted on a mobile body movable in the work site around the shovel 100 .
  • the mobile body may include, for example, a working machine or a working vehicle moving in the work site.
  • the mobile body movable in the work site may include, for example, a flying body such as a drone flying over the work site.
  • the output (measured data) of the sensors 300 - 1 to 300 -M is received into the information processing device 200 through the communication line NW.
  • the output of the sensors 300 - 1 to 300 -M is received directly into the information processing device 200 through the communication line NW, for example.
  • the output of the sensors 300 - 1 to 300 -M may be received into the shovel 100 through the communication line NW and received into the information processing device 200 via the shovel 100 .
  • the output of the sensors 300 - 1 to 300 -M may be received into the predetermined device and received into the information processing device 200 from the device.
  • the hardware configuration of the remote control support device 400 may be the same as that of the information processing device 200 . Therefore, illustration and description of the hardware configuration of the remote control support device 400 will be omitted.
  • FIG. 4 is a block diagram illustrating an example of a hardware configuration of a shovel 100 .
  • a path through which mechanical power is transmitted is indicated by a double line
  • paths through which high-pressure hydraulic fluid for driving the hydraulic actuators flows are indicated by solid lines
  • paths through which pilot pressure is transmitted are indicated by broken lines
  • paths through which electrical signals are transmitted are indicated by dotted lines.
  • the shovel 100 includes components such as a hydraulic drive system for hydraulically driving the driven element, an operation system for operating the driven element, user a interface system for exchanging information with the user, a communication system for communicating with the outside, and a control system for various controls.
  • the hydraulic drive system of the shovel 100 includes hydraulic actuators HA for hydraulically driving each of the driven elements such as the lower traveling body 1 (left and right crawlers 1 C), the upper swivel body 3 , the boom 4 , the arm 5 , and the bucket 6 , as described above.
  • the hydraulic drive system of the shovel 100 includes an engine 11 , a regulator 13 , a main pump 14 , and a control valve 17 .
  • the hydraulic actuators HA include such as traveling hydraulic motors 1 ML and 1 MR, a swivel hydraulic motor 2 M, a boom cylinder 7 , an arm cylinder 8 , and a bucket cylinder 9 .
  • the shovel 100 may be a hybrid shovel or an electric shovel.
  • the engine 11 is a motor of the shovel 100 and a main power source in the hydraulic drive system.
  • the engine 11 is, for example, a diesel engine using light oil as fuel.
  • the engine 11 is mounted, for example, at the rear of the upper swivel body 3 .
  • the engine 11 rotates at a predetermined target speed, for example, directly or indirectly controlled by the controller 30 to be described later, and drives the main pump 14 and a pilot pump 15 .
  • another motor e.g., electric motor or the like may be mounted in the shovel 100 .
  • the regulator 13 controls (adjusts) a discharge amount of the main pump 14 under the control of the controller 30 .
  • the regulator 13 adjusts an angle (hereinafter, “tilt angle”) of a swash plate of the main pump 14 according to a control command from the controller 30 .
  • the main pump 14 supplies hydraulic fluid to the control valve 17 through a high-pressure hydraulic line.
  • the main pump 14 is mounted, for example, at the rear of the upper swivel body 3 as well as the engine 11 .
  • the main pump 14 is driven by the engine 11 as described above.
  • the main pump 14 is, for example, a variable displacement hydraulic pump, and the stroke length of a piston is adjusted by adjusting the tilt angle of the swash plate by the regulator 13 under the control of the controller 30 , and the discharge flow rate and discharge pressure are controlled as described above.
  • the control valve 17 drives the hydraulic actuators HA in accordance with the operator's operation or remote control of the operating device 26 or an operation command corresponding to the automatic operation function.
  • the control valve 17 is mounted, for example, in the center of the upper swivel body 3 .
  • the control valve 17 is connected to the main pump 14 via a high-pressure hydraulic line as described above, and selectively supplies hydraulic oil supplied from the main pump 14 to the respective hydraulic actuators HA in accordance with the operator's operation or an operation command corresponding to the automatic operation function.
  • the control valve 17 includes a plurality of controlling valves (direction switching valves) for controlling the flow rate and direction of the hydraulic oil supplied from the main pump 14 to the hydraulic actuators HA.
  • the operation system of the shovel 100 includes a pilot pump 15 , an operating device 26 , a hydraulic controlling valve 31 , a shuttle valve 32 , and a hydraulic controlling valve 33 .
  • the pilot pump 15 supplies the pilot pressure to various hydraulic devices via a pilot line 25 .
  • the pilot pump 15 is mounted, for example, in the rear part of the upper swivel body 3 , similar to the engine 11 .
  • the pilot pump 15 is, for example, a fixed displacement hydraulic pump, and is driven by the engine 11 as described above.
  • the pilot pump 15 may be omitted.
  • the relatively high pressure hydraulic fluid discharged from the main pump 14 is depressurized by a predetermined pressure reducing valve, the relatively low pressure hydraulic fluid may be supplied to various hydraulic devices as the pilot pressure.
  • the operating device 26 is provided near an operator's seat of the cabin 10 , and is used for an operator to operate various driven elements. Specifically, the operating device 26 is used for an operator to operate the hydraulic actuators HA that drives each driven element, and as a result, the operator can operate the driven element to be driven by the hydraulic actuators HA.
  • the operating device 26 includes a pedal device and a lever device for operating each driven element (hydraulic actuator HA).
  • the operating device 26 is a hydraulic pilot type. Specifically, the operating device 26 uses hydraulic fluid supplied from the pilot pump 15 through the pilot line 25 and a branched pilot line 25 A, and outputs a pilot pressure corresponding to the operation contents to a secondary pilot line 27 A.
  • the pilot line 27 A is connected to one inlet port of the shuttle valve 32 , and is connected to the control valve 17 through a pilot line 27 connected to an outlet port of the shuttle valve 32 .
  • the pilot pressure corresponding to the operation contents of various driven elements (hydraulic actuators HA) in the operating device 26 can be input to the control valve 17 through the shuttle valve 32 . Therefore, the control valve 17 can drive the hydraulic actuators HA according to the operation contents of the operating device 26 by the operator or the like.
  • the operating device 26 may be an electric type.
  • the pilot line 27 A, the shuttle valve 32 , and the hydraulic controlling valve 33 are omitted.
  • the operating device 26 outputs an electric signal (hereinafter, “operation signal”) according to the operation contents, and the operation signal is received into the controller 30 .
  • the controller 30 outputs a control command according to the operation contents, that is, a control signal according to the operation contents for the operating device 26 , to the hydraulic controlling valve 31 .
  • the pilot pressure corresponding to the operation contents of the operating device 26 are input from the hydraulic controlling valve 31 to the control valve 17 , and the control valve 17 can drive the respective hydraulic actuators HA according to the operation contents of the operating device 26 .
  • the controlling valve (direction switching valve) built in the control valve 17 and driving the respective hydraulic actuators HA may be an electromagnetic solenoid type.
  • the operation signal output from the operating device 26 may be directly input to the control valve 17 (i.e., to the electromagnetic solenoid type controlling valve).
  • the hydraulic actuators HA may be replaced with electric actuators.
  • the controller 30 may output a control command corresponding to the operation content of the operating device 26 or the remote control content defined by the remote control signal to the electric actuator or a driver for driving the electric actuator.
  • the operating device 26 When the shovel 100 is operated remotely, the operating device 26 may be omitted.
  • the hydraulic controlling valve 31 is provided for each driven element (hydraulic actuator HA) to be operated by the operating device 26 and for each driving direction (e.g., the upward and downward directions of the boom 4 ) of the driven element (hydraulic actuator HA).
  • two hydraulic controlling valves 31 are provided for each double-acting hydraulic actuator HA for driving the lower traveling body 1 , the upper swivel body 3 , the boom 4 , the arm 5 , and the bucket 6 , and the like.
  • the hydraulic controlling valve 31 may be provided, for example, in the pilot line 25 B between the pilot pump 15 and the control valve 17 , and may be configured so that the flow path area (i.e., the cross-sectional area through which the hydraulic fluid can flow) thereof can be changed.
  • the hydraulic controlling valve 31 can output a predetermined pilot pressure to a secondary pilot line 27 B by utilizing the hydraulic fluid of the pilot pump 15 supplied through the pilot line 25 B. Therefore, the hydraulic controlling valve 31 can indirectly apply a predetermined pilot pressure corresponding to a control signal from the controller 30 to the control valve 17 through the shuttle valve 32 between the pilot line 27 B and the pilot line 27 .
  • the controller 30 can cause the hydraulic controlling valve 31 to supply the pilot pressure corresponding to an operation command corresponding to the automatic operation function to the control valve 17 to achieve the operation of the shovel 100 by the automatic operation function.
  • the controller 30 may control the hydraulic controlling valve 31 to achieve the remote control of the shovel 100 .
  • the controller 30 outputs to the hydraulic controlling valve 31 a control signal corresponding to the content of the remote control specified by the remote control signal received from the remote control support device 400 by the communication device 60 .
  • the controller 30 can cause the hydraulic controlling valve 31 to supply the pilot pressure corresponding to the content of the remote control to the control valve 17 to achieve the operation of the shovel 100 based on the remote control of the operator.
  • the controller 30 directly supplies the pilot pressure corresponding to the operation content (operation signal) of the operating device 26 from the hydraulic controlling valve 31 to the control valve 17 to achieve the operation of the shovel 100 based on the operation of the operator.
  • the shuttle valve 32 has two inlet ports and one outlet port, and outputs hydraulic fluid having the higher pilot pressure among the pilot pressures input to the two inlet ports to the outlet port.
  • the shuttle valve 32 is provided for each driven element (hydraulic actuator HA) to be operated by the operating device 26 and for each driving direction of the driven element (hydraulic actuator HA).
  • two shuttle valves 32 are provided for each double-acting hydraulic actuator HA for driving the lower traveling body 1 , the upper swivel body 3 , the boom 4 , the arm 5 , and the bucket 6 .
  • One of the two inlet ports of the shuttle valve 32 is connected to the secondary pilot line 27 A of the operating device 26 (specifically, the above-described lever device and pedal device included in the operating device 26 ), and the other is connected to the secondary pilot line 27 B of the hydraulic controlling valve 31 .
  • the outlet port of the shuttle valve 32 is connected to the pilot port of the corresponding control valve of the control valve 17 through the pilot line 27 .
  • the corresponding control valve is a control valve for driving the hydraulic actuator HA to be operated by the above-described lever device or pedal device connected to one inlet port of the shuttle valve 32 .
  • these shuttle valves 32 can apply the higher of the pilot pressure of the secondary pilot line 27 A of the operating device 26 and the pilot pressure of the secondary pilot line 27 B of the hydraulic controlling valve 31 to the pilot port of the corresponding control valve.
  • the controller 30 can control the corresponding control valve by outputting the pilot pressure higher than the pilot pressure of the secondary side of the operating device 26 from the hydraulic controlling valve 31 regardless of the operator's operation of the operating device 26 . Therefore, the controller 30 can control the operation of the driven elements (lower traveling body 1 , upper swivel body 3 , boom 4 , arm 5 , and bucket 6 ) and achieve the automatic operation function and the remote control function regardless of the operation state of the operator with respect to the operating device 26 .
  • the hydraulic controlling valve 33 is provided in the pilot line 27 A connecting the operating device 26 and the shuttle valve 32 .
  • the hydraulic controlling valve 33 is configured so that the flow path area can be changed, for example.
  • the hydraulic controlling valve 33 operates according to a control signal input from the controller 30 .
  • the controller 30 can forcibly reduce the pilot pressure output from the operating device 26 . Therefore, even when the operating device 26 is operated, the controller 30 can forcibly reduce or stop the operation of the hydraulic actuator HA corresponding to the operation of the operating device 26 .
  • the controller 30 can reduce the pilot pressure output from the operating device 26 to be lower than the pilot pressure output from the hydraulic controlling valve 31 .
  • the controller 30 can surely cause the desired pilot pressure to act on the pilot port of the control valve in the control valve 17 regardless of the operation content of the operating device 26 , for example. Therefore, by controlling the hydraulic controlling valve 33 in addition to the hydraulic controlling valve 31 , the controller 30 can more appropriately achieve the automatic operation function and the remote control function of the shovel 100 .
  • the user interface system of the shovel 100 includes an operating device 26 , an output device 50 , and an input device 52 .
  • the output device 50 outputs various types of information to a user of the shovel 100 (e.g., the operator in the cabin 10 or the external operator of the remote control) or to a person around the shovel 100 (e.g., a worker or a driver of a working vehicle).
  • the output device 50 includes a lighting device and a display device for outputting various types of information in a visual manner.
  • the lighting device is, for example, a warning lamp (indicator lamp).
  • the display device is, for example, a liquid crystal or display an organic EL (Electroluminescence) display.
  • the lighting device and the display device may be provided inside the cabin 10 and output various types of information in a visual manner to an operator or the like inside the cabin 10 .
  • the lighting apparatus and the display device may be provided, for example, on the lateral surface of the upper swivel body 3 , and may output various types of information in a visual manner to a worker or the like around the shovel 100 .
  • the output device 50 may also include a sound output device which outputs various types of information in an auditory method (see FIG. 7 ).
  • the sound output device may include, for example, a buzzer or a speaker.
  • the sound output device may be provided, for example, inside or outside the cabin 10 , and may output various types of information in an auditory method to an operator inside the cabin 10 or to a person (a worker, etc.) around the shovel 100 .
  • the output device 50 may also include a device which outputs various types of information in a tactile manner such as vibration of the operator's seat.
  • the input device 52 receives various types of inputs from the user of the shovel 100 , and signals corresponding to the received inputs are received into the controller 30 .
  • the input device 52 is provided inside the cabin 10 and receives inputs from an operator inside the cabin 10 .
  • the input device 52 may be provided, for example, on the lateral surface of the upper swivel body 3 , and may receive input from a worker or the like around the shovel 100 .
  • the input device 52 may include an operation input device that accepts input by mechanical operation from a user.
  • the operation input device may include a touch panel mounted on the display device, a touch pad installed around the display device, a button switch, a lever, a toggle, and a knob switch provided on the operating device 26 (lever device).
  • the input device 52 may also include a voice input device that accepts voice input from a user.
  • the voice input device may include, for example, a microphone.
  • the input device 52 may also include a gesture input device that accepts gesture input from the user.
  • the gesture input device may include, for example, an imaging device that captures a gesture performed by a user.
  • the input device 52 may also include a biometric input device that accepts biometric input of the user.
  • the biometric input includes, for example, the input of biometric information as such the user's fingerprint or iris.
  • the communication system of the shovel 100 includes a communication device 60 .
  • the communication device 60 is connected to an external communication line NW and communicates with a device provided separately from the shovel 100 .
  • the device provided separately from the shovel 100 may include a device outside the shovel 100 and a portable terminal device (portable terminal) brought into the cabin 10 by the user of the shovel 100 .
  • the communication device 60 may include, for example, a mobile communication module conforming to a standard such as 4G (Fourth Generation) or 5G (Fifth Generation).
  • the communication device 60 may include, for example, a satellite communication module.
  • the communication device 60 may include, for example, a WiFi communication module or a Bluetooth (registered trademark) communication module.
  • the communication device 60 may include a plurality of communication devices according to the type of communication line NW when there are a plurality of communication lines NW that can be connected.
  • the communication device 60 communicates with external devices such as the information processing device 200 and the remote control support device 400 in the work site through a local communication line constructed at the work site.
  • the local communication line may be, for example, a mobile communication line using local 5G (what is called local 5G) constructed at the work site or a local network using WiFi6.
  • the communication device 60 may communicate with the information processing device 200 and the remote control support device 400 located outside the work site through a wide-area communication line including the work site, that is, a wide-area network.
  • the control system of the shovel 100 includes a controller 30 .
  • the control system of the shovel 100 includes an operating pressure sensor 29 , a sensor 40 , and sensors S 1 to S 9 .
  • the controller 30 performs various controls related to the shovel 100 .
  • the controller 30 includes an auxiliary storage device 30 A, a memory device 30 B, a CPU (Central Processing Unit) 30 C, and an interface device 30 D connected by bus BS 1 .
  • auxiliary storage device 30 A a memory device 30 B
  • CPU Central Processing Unit
  • interface device 30 D an interface device 30 D connected by bus BS 1 .
  • the auxiliary storage device 30 A is a nonvolatile storage means and stores programs to be installed, and also stores necessary files, data, and the like.
  • the auxiliary storage device 30 A may be, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory) or a flash memory.
  • the memory device 30 B loads the program of the auxiliary storage device 30 A so that the CPU 30 C can read it, for example, when a program start instruction is given.
  • the memory device 30 B is, for example, an SRAM (Static Random Access Memory).
  • the CPU 30 C performs the program loaded into the memory device 30 B, for example, and implements various functions of the controller 30 according to the instruction of the program.
  • the interface device 30 D functions, for example, as a communication interface for connecting to the communication line inside the shovel 100 .
  • the interface device 30 D may include a plurality of different types of communication interfaces according to the type of communication line to be connected.
  • the interface device 30 D also functions as an external interface for reading data from or writing data to the storage medium.
  • the storage medium is, for example, a dedicated tool connected to a connector installed inside the cabin 10 by a detachable cable.
  • the storage medium may be, for example, a general-purpose storage medium such as an SD memory card or a USB (Universal Serial Bus) memory.
  • a program for realizing various functions of the controller 30 may be provided, for example, by a portable storage medium and installed in the auxiliary storage device 30 A of the controller 30 .
  • the program may be downloaded from another computer (e.g., information processor 200 ) outside the shovel 100 through the communication device 60 and installed in the auxiliary storage device 30 A.
  • controller 30 may be achieved by other controllers (control devices).
  • the functions of the controller 30 may be achieved by a plurality of controllers installed in the shovel 100 in a distributed manner.
  • the operating pressure sensor 29 detects the pilot pressure on the secondary side (pilot line 27 A) of the hydraulic pilot type operating device 26 , that is, the pilot pressure corresponding to the operating state of each driven element (hydraulic actuator HA) in the operating device 26 .
  • a detection signal of the pilot pressure corresponding to the operating state of each driven element (hydraulic actuator HA) in the operating device 26 by the operating pressure sensor 29 is received into the controller 30 .
  • the operating pressure sensor 29 is omitted. This is because the controller 30 can determine the operating state of each driven element through the operating device 26 based on the operating signal received from the operating device 26 .
  • the sensor 40 acquires, for example, measurement data related to the shape of an object around the shovel 100 .
  • the senor 40 is a shape sensor capable of acquiring measurement data representing the shape of an object around the shovel 100 , such as a distance measurement sensor or a 3D camera.
  • the sensor 40 may be an integrated sensor having the function of a characteristic sensor capable of acquiring measurement data representing the characteristic of an object around the shovel 100 , such as a multi-wavelength spectral camera.
  • the sensor 40 includes sensors 40 F, 40 B, 40 L, and 40 R.
  • the sensor 40 F measures the state (shape and characteristics) of the object in front of the upper swivel body 3 .
  • the sensor 40 B measures the state of the object in the upper swivel body 3 .
  • the sensor 40 L measures the state of the object to the left of the upper swivel body 3 .
  • the sensor 40 R measures the state of the object to the right of the upper swivel body 3 .
  • the sensor 40 can measure the state of the object in the whole circumference centering on the shovel 100 , that is, in the angular direction of 360 degrees.
  • the sensors 40 F, 40 B, 40 L, and 40 R may be collectively or individually referred to as “sensor 40 X”.
  • the output data (i.e., measurement data about the state of the object around the shovel 100 ) of the sensor 40 (sensor 40 X) is received into the controller 30 through a one-to-one communication line or an in-vehicle network.
  • the controller 30 can determine the state of the shape and characteristics of the object around the shovel 100 based on the output data of the sensor 40 X.
  • Some or all of the sensors 40 B, 40 L and 40 R may be omitted.
  • the sensor S 1 is attached to the boom 4 and measures an attitude state of the boom 4 .
  • the sensor S 1 outputs measurement data indicating the attitude state of the boom 4 .
  • the attitude state of the boom 4 is, for example, the attitude angle (hereinafter, “boom angle”) around the rotation axis of the base end corresponding to a connecting part of the boom 4 with the upper swivel body 3 .
  • the sensor S 1 includes, for example, a rotary potentiometer, a rotary encoder, an acceleration sensor, an angular acceleration sensor, a six-axis sensor, an IMU (Inertial Measurement Unit), and the like.
  • IMU Inertial Measurement Unit
  • the sensor S 1 may also include a cylinder sensor for detecting the expansion or contraction position of the boom cylinder 7 .
  • the same may apply to the sensors S 2 and S 3 .
  • the output of the sensor S 1 (measurement data representing the attitude state of the boom 4 ) is received into the controller 30 .
  • the controller 30 can determine the attitude state of the boom 4 .
  • the sensor S 2 is attached to the arm 5 and measures the attitude state of the arm 5 .
  • the sensor S 2 outputs measurement data indicating the attitude state of the arm 5 .
  • the attitude state of the arm 5 is, for example, the attitude angle (hereinafter, “arm angle”) around the rotational axis of the base end corresponding to the connecting part of the arm 5 with the boom 4 .
  • the output of the sensor S 2 (measurement data indicating the attitude state of the arm 5 ) is received into the controller 30 .
  • the controller 30 can determine the attitude state of the arm 5 .
  • the sensor S 3 is attached to the bucket 6 and measures the attitude state of the bucket 6 .
  • the sensor S 3 outputs measurement data indicating the attitude state of the bucket 6 .
  • the attitude state of the bucket 6 is, for example, the attitude angle (hereinafter, “arm angle”) around the rotation axis of the base end corresponding to the connecting part with the arm 5 of the bucket 6 .
  • the output of the sensor S 3 (measurement data indicating the attitude state of the bucket 6 ) is received into the controller 30 .
  • the controller 30 can determine the attitude state of the bucket 6 .
  • the sensor S 4 measures the attitude state of the body (e.g., upper swivel body 3 ) of the shovel 100 .
  • the sensor S 4 outputs measurement data indicating the attitude state of the body of the shovel 100 .
  • the attitude state of the body of the shovel 100 is, for example, the inclination state of the body with respect to a predetermined reference plane (e.g., horizontal plane).
  • the sensor S 4 is attached to the upper swivel body 3 , and measures the inclination angle of the shovel 100 about two axes in the longitudinal direction and the lateral direction (hereinafter, “longitudinal inclination angle” and “lateral inclination angle”).
  • the output of the sensor S 4 (measurement data indicating the attitude state of the body of the shovel 100 ) is received into the controller 30 .
  • the controller 30 can determine the attitude state (inclination state) of the body (upper swivel body 3 ).
  • the sensor S 5 is attached to the upper swivel body 3 and measures the swivel state of the upper swivel body 3 .
  • the sensor S 5 outputs measurement data representing the swivel state of the upper swivel body 3 .
  • the sensor S 5 measures, for example, a swivel angular velocity and a swivel angle of the upper swivel body 3 .
  • the sensor S 5 includes, for example, a gyro sensor, a resolver, a rotary encoder, etc.
  • the output of sensor the S 5 (measurement data representing the swivel state of the upper swivel body 3 ) is received into the controller 30 .
  • the controller 30 can determine the swivel state such as the swivel angle of the upper swivel body 3 .
  • the controller 30 can determine (estimate) the position of the tip (bucket 6 ) of the attachment AT based on the outputs of the sensors S 1 to S 5 .
  • the swivel state (e.g., swivel angular velocity) of the upper swivel body 3 may be detected based on the detection signal of the sensor S 4 .
  • the sensor S 5 may be omitted.
  • the sensor S 6 measures the position of the shovel 100 .
  • the sensor S 6 may measure the position in world (global) coordinates or in local coordinates at the work site.
  • the sensor S 6 may be, for example, a GNSS (Global Navigation Satellite System) sensor.
  • the sensor S 6 is a transceiver capable of communicating with a device that serves as a reference for the position at the work site and outputting a signal corresponding to the position relative to the reference.
  • the output of the sensor S 6 is received into the controller 30 .
  • the sensor S 7 measures pressure (cylinder pressure) in an oil chamber of the boom cylinder 7 .
  • the sensor S 7 includes, for example, a sensor for measuring the cylinder pressure (rod pressure) of the oil chamber on the rod side of the boom cylinder 7 and a sensor for measuring the cylinder pressure (bottom pressure) of the oil chamber on the bottom.
  • the output of the sensor S 7 (measurement data of the cylinder pressure of the boom cylinder 7 ) is received into the controller 30 .
  • the sensor S 8 measures the pressure (cylinder pressure) of the oil chamber of the arm cylinder 8 .
  • the sensor S 8 includes, for example, a sensor for measuring the cylinder pressure (rod pressure) of the oil chamber on the rod side of the arm cylinder 8 and a sensor for measuring the cylinder pressure (bottom pressure) of the oil chamber on the bottom side of the arm cylinder 8 .
  • the output of the sensor S 8 (measurement data of the cylinder pressure of the arm cylinder 8 ) is received into the controller 30 .
  • the sensor S 9 measures the pressure (cylinder pressure) of the oil chamber of the bucket cylinder 9 .
  • the sensor S 9 includes, for example, a sensor for measuring the cylinder pressure (rod pressure) of the oil chamber on the rod side of the bucket cylinder 9 and a sensor for measuring the cylinder pressure (bottom pressure) of the oil chamber on the bottom side of the bucket cylinder 9 .
  • the output of the sensor S 9 (measurement data of the cylinder pressure of the bucket cylinder 9 ) is received into the controller 30 .
  • the controller 30 can determine the load state acting on the attachment AT based on the outputs of the sensors S 7 to S 9 .
  • the load acting on the attachment AT includes, for example, the reaction force acting on the bucket 6 from the target material (earth) and the weight of the earth contained in the bucket 6 .
  • FIG. 5 is a drawing illustrating an example of a hardware configuration of the information processing device 200 .
  • the functions of the information processing device 200 are achieved by optional hardware or combinations of optional hardware and software.
  • the information processing device 200 includes an external interface 201 , an auxiliary storage device 202 , a memory device 203 , a CPU 204 , a high speed arithmetic unit 205 , a communication interface 206 , an input device 207 , a display device 208 , and a sound output device 209 . These devices are connected by bus BS 2 .
  • the external interface 201 functions as an interface for reading data from a storage medium 201 A and writing data to the storage medium 201 A.
  • the storage medium 201 A includes, for example, a flexible disk, CD (Compact Disc), DVD (Digital Versatile Disc), BD (Blu-ray (registered trademark) Disc), SD memory card, USB memory, and the like.
  • CD Compact Disc
  • DVD Digital Versatile Disc
  • BD Blu-ray (registered trademark) Disc
  • SD memory card USB memory
  • the information processing device 200 can read various data to be used in processing through the storage medium 201 A, store them in the auxiliary storage device 202 , and install programs to achieve various functions.
  • the information processing device 200 may acquire various data and programs to be used in processing from an external device through the communication interface 206 .
  • the auxiliary storage device 202 stores various installed programs, and also stores files, data, and the like necessary for various processing.
  • the auxiliary storage device 202 includes, for example, hard disc drives (HDDs), solid state discs (SSDs), flash memories, and the like.
  • the memory device 203 reads and stores a program from the auxiliary storage device 202 when a program start instruction is given.
  • the memory device 203 includes, for example, DRAM (Dynamic Random Access Memory) and SRAM.
  • the CPU 204 performs various programs loaded from the auxiliary storage device 202 to the memory device 203 , and implements various functions related to the information processing device 200 according to the programs.
  • the high speed arithmetic unit 205 performs arithmetic processing at a relatively high speed in conjunction with the CPU 204 .
  • the high speed arithmetic unit 205 includes, for example, GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), and FPGA (Field-Programmable Gate Array).
  • the high speed arithmetic unit 205 may be omitted depending on the required arithmetic processing speed.
  • the communication interface 206 is used as an interface to connect to enable to communicate with an external device.
  • the information processing device 200 can communicate with an external device such as a shovel 100 , for example, through the communication interface 206 .
  • the communication interface 206 may have a plurality of types of communication interfaces depending on a communication system or the like with the connected device.
  • the input device 207 receives various inputs from the user.
  • the input device 207 includes an operating device for remote control of the shovel 100 .
  • the input device 207 includes, for example, an input device (hereinafter, “operation input device”) which accepts mechanical operation input from the user.
  • the operating device for remote control may be an operation input device.
  • the operation input device includes, for example, a button, a toggle, a lever, a keyboard, a mouse, a touch panel mounted on the display device 208 , a touch pad provided separately from the display device 208 , and the like.
  • the input device 207 may also include a voice input device capable of receiving voice input from a user.
  • the voice input device may include, for example, a microphone capable of collecting user voice.
  • the input device 207 may also include a gesture input device capable of receiving gesture input from a user.
  • the gesture input device may include, for example, a camera capable of capturing an image of a user's gesture.
  • the input device 207 may also include a biometric input device capable of receiving biometric input from a user.
  • the biometric input device may include, for example, a camera capable of acquiring image data containing information about a user's fingerprint or iris.
  • the display device 208 displays an information screen and an operation screen to the user of the information processing device 200 .
  • the display device 208 is, for example, a liquid crystal display or an organic EL (Electroluminescence) display.
  • the sound output device 209 transmits various types of information by sound to the user of the information processing device 200 .
  • the sound output device 209 is, for example, a buzzer, alarm, speaker, etc.
  • FIG. 6 is a functional block diagram illustrating a first example of the functional configuration of the operation support system SYS.
  • the shovel 100 includes a support device 150 .
  • the support device 150 supports the shovel 100 operated by the autonomous operation function in performing the work.
  • the support device 150 includes a controller 30 , a hydraulic controlling valve 31 , a sensor 40 , an output device 50 , an input device 52 , and sensors S 1 to S 9 . Further, when instructions concerning autonomous operation are input from outside the shovel 100 , the support device 150 may include a communication device 60 instead of or in addition to the input device 52 .
  • the controller 30 includes, as functional parts, an operation log providing part 301 and a work support part 302 .
  • shovel 100 When there are multiple shovels 100 included in the operation support system SYS, there may be a shovel 100 in which the controller 30 includes only the operation log providing part 301 and the work support part 302 , and a shovel 100 in which only the latter is included.
  • the former shovel 100 has only the function of acquiring an operation log of the shovel 100 used for the work support function of the latter shovel 100 and providing it to the information processing device 200 . The same may be applied to the case of the second example described later.
  • the information processing device 200 includes, as functional parts, a log acquisition part 2001 , a simulator part 2002 , a log storage part 2003 , a training data generation part 2004 , a machine learning part 2005 , a trained model storage part 2006 , and a distribution part 2007 .
  • the operation log providing part 301 is a functional part for acquiring the operation log of a predetermined operation of the shovel 100 and providing it to the information processing device 200 .
  • a plurality of (types of) predetermined operations are previously defined.
  • the plurality of predetermined operations include an excavation operation, a boom raising and swivel operation, a boom lowering and swivel operation, an earth removing operation, a broom operation, and the like.
  • the plurality of predetermined operations include an excavation operation, an earth removing operation, a sweeping operation, a horizontal pulling operation, a rolling operation, a broom operation, and the like.
  • the plurality of predetermined operations may include an excavation operation, an earth removing operation, a slope pulling operation, a rolling operation, and the like.
  • the slope pulling operation corresponds to the horizontal pulling operation of the ground leveling work, and involves moving the attachment AT to draw a cutting edge (edge of teeth) of the bucket 6 toward the body (upper swivel body 3 ) along the slope corresponding to the target construction surface.
  • the sweeping operation involves, for example, operating the attachment AT and pushing the bucket 6 forward along the ground to sweep the earth forward at the back of the bucket 6 .
  • the attachment AT performs a lowering operation of the boom 4 and an opening operation of the arm 5 .
  • the horizontal pulling operation involves, for example, operating the attachment AT and moving the edge of teeth of the bucket 6 along the ground drawing toward the front approximately horizontally to level unevenness of the ground (surface of terrain).
  • the attachment AT performs a raising operation of the boom 4 and a closing operation of the arm 5 .
  • the rolling operation involves, for example, operating the attachment AT and pressing the ground against the back of the bucket 6 .
  • the rolling operation may involve pressing the ground by striking the back of the bucket 6 against the ground while moving the bucket 6 up and down.
  • the rolling operation may involve pushing the ground against the back of the bucket 6 forward along the ground so that the earth is swept out to a predetermined position forward by the back of the bucket 6 , and then pressing the ground against the ground with the back of the bucket 6 .
  • the attachment AT performs a lowering operation of the boom 4 when pressing the ground.
  • the broom operation involves, for example, operating the upper swivel body 3 to swivel the bucket 6 left and right while keeping it along the ground.
  • the broom operation may involve operating the attachment AT and the upper swivel body 3 to swivel the bucket 6 left and right while keeping it along the ground and push forward the bucket 6 .
  • the upper swivel body 3 alternately repeats left and right swivel operations.
  • the attachment AT may perform the lowering operation of the boom 4 and the opening operation of the arm 5 .
  • the operation log of the shovel 100 is time-series data representing the operation state of the shovel 100 .
  • the operation log of the shovel 100 includes time-series data representing the operation contents of the operator.
  • the time-series data representing the operation contents of the operator includes, for example, time-series output data of the operating pressure sensor 29 corresponding to the hydraulic pilot type operating device 26 and time-series output data (operation signal data) of the operating device 26 corresponding to the electric type operating device 26 .
  • the operation log of the shovel 100 may be time-series output data of the sensors S 1 to S 5 and time-series data representing the attitude state of the shovel 100 obtained from the output data of the sensors S 1 to S 5 .
  • the operation log providing part 301 may acquire the operation log of the shovel 100 operated by an operator (hereinafter, “skilled person” for convenience) having a long operating history of the shovel 100 and provide the operation log to the information processing device 200 .
  • the trained model LM 3 capable of reproducing the operation of the shovel 100 operated by an expert can be generated by machine learning based on the operation log of the shovel 100 , as described later.
  • the operation log providing part 301 includes an operation log recording part 301 A, an operation log storage part 301 B, and an operation log transmission part 301 C.
  • An operation log recording part 301 A acquires an operation log during a predetermined operation of the shovel 100 and records it in an operation log storage part 301 B. For example, each time a predetermined operation of the shovel 100 is performed, the operation log recording part 301 A records an operation log during the operation in an operation log storage part 301 B.
  • the operation log storage part 301 B stores the operation log of the shovel 100 .
  • the operation log and data of the time (date and time) when the predetermined operation was performed are associated and stored in the operation log storage part 301 B.
  • the data of the time when the predetermined operation was performed includes data of both the start and the end of the predetermined operation of the shovel 100 .
  • the operation log, data of the time when the predetermined operation was performed, and data of identification information of the performed predetermined operation are associated and stored in the operation log storage part 301 B.
  • data linked to the operation log of the shovel 100 may be conveniently referred to as “accompanying data”.
  • record data representing a correspondence between the operation log and the accompanying is data accumulated in the operation log storage part 301 B, so that a database of the operation log at the time of performance of the predetermined operation of the shovel 100 is constructed.
  • the operation log of the operation log storage part 301 B which has already been transmitted to the information processing device 200 by the operation log transmission part 301 C described later may be deleted afterwards.
  • the operation log transmission part 301 C transmits the operation log stored in the operation log storage part 301 B when the shovel 100 performs a predetermined operation and the accompanying data associated with the operation log to the information processing device 200 through the communication device 60 .
  • the operation log transmission part 301 C may also transmit to the information processing device 200 record data showing the correspondence between the operation log of the shovel 100 and the accompanying data for each predetermined operation performed by the shovel 100 .
  • the operation log transmission part 301 C transmits a non-transmitted operation log of the shovel 100 and the accompanying data stored in the operation log storage part 301 B to the information processing device 200 in response to a transmission request of the operation log of the shovel 100 received from the information processing device 200 .
  • the operation log transmission part 301 C may also automatically transmit the non-transmitted operation log of the shovel 100 and the accompanying data stored in the operation log storage part 301 B to the information processing device 200 at a predetermined timing.
  • the predetermined timing is, for example, when the shovel 100 stops operation (key switch is turned off) or starts operation (key switch is turned on).
  • the log acquisition part 2001 acquires a log when the shovel 100 performs a predetermined operation.
  • the log when the shovel 100 performs the predetermined operation includes the operation log when the shovel 100 performs the predetermined operation and the state log of the target material.
  • the state log of the target material includes time-series data showing the state of the target material before, during, and after the performance of the predetermined operation of the shovel 100 .
  • the state of the target material includes a shape of the earth (terrain shape) and a characteristic of the earth of the target material, and the like.
  • the characteristics of the earth may include, for example, hardness of the earth, moisture content of the earth, size of the earth grain (grain size), angle of repose of the earth, and the like.
  • the operation log when the shovel 100 performs the predetermined operation is uploaded from the shovel 100 .
  • the state log of the target material when the shovel 100 performs the predetermined operation is acquired based on the measurement data uploaded from the sensor group 300 and the accompanying data (data at the time when the predetermined operation is performed) uploaded from the shovel 100 .
  • the state log of the target material may be acquired based on the measurement data of the sensor 40 of the shovel 100 .
  • the measurement data acquired by the sensor 40 at the time of the predetermined operation of the shovel 100 is uploaded from the shovel 100 to the information processing device 200 .
  • the sensor group 300 may be omitted.
  • the simulator part 2002 performs computer simulation of a predetermined operation of the shovel 100 using a virtual model of the shovel 100 and the target material (earth).
  • the simulator part 2002 causes the virtual model of the shovel 100 to perform a predetermined operation such as an excavation operation, and can virtually reproduce the whole behavior of the earth of the target material and the reaction forces from the earth in entirety by analyzing the individual operation of the minute particles.
  • a predetermined operation such as an excavation operation
  • the simulator part 2002 acquires data on the path of the work part of the shovel 100 and data on the state of the target material (earth) before, during, and after the performance of the predetermined operation as a log when the shovel 100 performs the predetermined operation by computer simulation.
  • the former data corresponds to the operation log when the shovel 100 performs the predetermined operation by computer simulation
  • the latter data corresponds to the state log of the target material when the shovel 100 performs the predetermined operation by computer simulation.
  • the simulator part 2002 performs computer simulation of a large number of patterns concerning a predetermined operation of the shovel 100 using various conditions of target materials (earth) and paths of various work parts of the shovel 100 .
  • the simulator part 2002 can store logs when the shovel 100 performs a predetermined operation by computer simulation under different conditions in the log storage part 2003 .
  • Logs when the shovel 100 performs a predetermined operation acquired by the log acquisition part 2001 and the simulator part 2002 are stored in the log storage part 2003 in an accumulated form.
  • the log storage part 2003 stores an operation log for each predetermined operation actually performed by the shovel 100 or by computer simulation, a state log of the target material, and accompanying data in a form linked to each other.
  • the logs acquired by the log acquisition part 2001 and the logs acquired by the simulator part 2002 may be stored in a distinguishable manner or may be stored in a mixed manner in an indistinguishable manner.
  • the training data generation part 2004 generates training data for machine learning based on the logs when the shovel 100 performs a predetermined operation stored in the log storage part 2003 , and outputs a training data set which is an aggregate of a large number of training data.
  • the training data generation part 2004 may automatically generate the training data by batch processing, or may generate the training data in response to an input from the user of the information processing device 200 .
  • the training data generation part 2004 includes training data generation parts 2004 A to 2004 C.
  • the training data generation part 2004 A generates a training data set for generating the trained model LM 1 .
  • the trained model LM 1 infers a future state of the target material of the shovel 100 at a predetermined time in the future by using a current state of the target material of the shovel 100 and the path of the work part of the shovel 100 up to a predetermined time in the future as inputs.
  • the training data is a combination of the state of the target material of the shovel 100 at a first time and a path of the work part of the shovel 100 from the first time to a second time after the first time as input data and the state of the target material at the second time as ground truth.
  • the training data set for generating the trained model LM 1 may be generated only from the former log among the log acquired by the log acquisition part 2001 and the log output from the simulator part 2002 . In this case, the simulator part 2002 may be omitted. Similarly, the training data set for generating the trained model LM 1 may be generated only from the latter log among the log acquired by the log acquisition part 2001 and the log output from the simulator part 2002 . In this case, the operation log providing part 301 of the sensor group 300 and the shovel 100 may be omitted. In addition, the training data set for generating the trained model LM 1 may include a base training data set and a final adjustment (fine-tuning) training data set.
  • the base training data set may be generated based on the log output from the simulator part 2002 , and the final adjustment training data set may be generated based on the log acquired by the log acquisition part 2001 .
  • the same may be applied to the trained models LM 2 and LM 3 .
  • the training data generation part 2004 B generates training data for generating the trained model LM 2 .
  • the trained model LM 2 is generated for each of the plurality of tasks.
  • the trained model LM 2 uses data of the state of the target material around the shovel 100 as an input, and infers one predetermined operation suitable for the state of the target material corresponding to the input data from among the plurality of predetermined operations used in the work.
  • the training data is, for example, a combination of the state of the target material before the performance of the predetermined operation of the shovel 100 as input data and the type of the predetermined operation performed by the shovel 100 thereafter as ground truth.
  • the training data may further include the target shape (e.g., target construction surface) of the target material as input data.
  • the training data generation part 2004 B may generate a training data set based on the log obtained by the log acquisition part 2001 when the shovel 100 performs the predetermined operation by a skilled person's operation.
  • the trained model LM 2 can reproduce the way in which the skilled person selects a prescribed operation of the shovel 100 .
  • a training data generation part 2004 C generates training data for generating the trained model LM 3 .
  • the trained model LM 3 is used for inferring a target path of a work part in a prescribed operation of the shovel 100 with data of the state of a target material around the shovel 100 as input.
  • the trained model LM 3 is generated for each (type of) prescribed operation of the shovel 100 .
  • the trained model LM 3 for example, infers an operation parameter that defines a target path of a work part in a prescribed operation of the shovel 100 based on the state of the target material before the performance of the prescribed operation of the shovel 100 .
  • the training data is a combination of the state of the target material before the performance of the prescribed operation of the shovel 100 as input data and the operation parameter corresponding to the path of the work part when the shovel 100 performs the prescribed operation as ground truth.
  • the trained model LM 3 may also infers a target path of a work part in a prescribed operation of the shovel 100 based on the state of the target material before the performance of the prescribed operation of the shovel 100 .
  • the training data is a combination of the state of the target material before the performance of the prescribed operation of the shovel 100 as input data and the path of the work part when the shovel 100 performs the prescribed operation as ground truth.
  • the training data may further include the target shape (e.g., target construction surface) of the target material as input data.
  • the training data generation part 2004 C may generate a training data set based on a log obtained by the log acquisition part 2001 when the shovel 100 performs a predetermined operation by an expert.
  • the trained model LM 3 can reproduce the operation of the shovel 100 by the expert.
  • the machine learning part 2005 causes the base learning model to perform machine learning based on the training data set generated by the training data generation part 2004 , and generates the trained models LM 1 to LM 3 .
  • the trained model includes a neural network such as a DNN (Deep Neural Network), for example.
  • the machine learning part 2005 includes machine learning parts 2005 A to 2005 C.
  • the machine learning part 2005 A causes the base learning model M 1 to perform machine learning based on the training data set output from the training data generation part 2004 A.
  • the machine learning part 2005 A can generate a trained model LM 1 capable of outputting (inferring) the state of the target material of the shovel 100 at a predetermined time in the future by taking data such as the current state of the target material of the shovel 100 and the target path of the work part of the shovel 100 up to a predetermined time in the future as inputs.
  • the machine learning part 2005 A may correct (additionally train) the trained model LM 1 so that an error between the inference result of the trained model LM 1 and the measurement result of an actual sensor 40 is reduced.
  • the inference result of the trained model LM 1 and the data of the measurement result of the actual sensor 40 are uploaded from the shovel 100 to the information processing device 200 .
  • the machine learning part 2005 B causes the base learning model M 2 to perform machine learning based on the training data set output from the training data generation part 2004 B.
  • the machine learning part 2005 B can generate the trained model LM 2 capable of outputting (inferring) one predetermined operation from among a plurality of predetermined operations corresponding to the target operation using data of the state of the target operation around the shovel 100 as an input.
  • the machine learning part 2005 B may generate the trained model LM 2 by performing reinforcement learning using the simulator part 2002 .
  • the machine learning part 2005 C causes the base learning model M 3 to perform machine learning based on the training data set output from the training data generation part 2004 C.
  • the machine learning part 2005 C can generate the trained model LM 3 capable of outputting (inferring) the target path of the target operation in the predetermined operation of the shovel 100 using data of the state of the target operation around the shovel 100 as an input.
  • Trained models LM 1 and LM 2 output by the machine learning part 2005 are stored in the trained model storage part 2006 .
  • the trained model LM 1 in the trained model storage part 2006 is updated.
  • the trained models LM 2 and LM 3 are re-trained or additionally trained by the machine learning parts 2005 B and 2005 C.
  • the distribution part 2007 distributes the data of the trained models LM 1 to LM 3 to the shovel 100 .
  • the distribution part 2007 distributes the most recently generated or updated trained model LM 1 to the shovel 100 .
  • the distribution part 2007 may also distribute the latest trained model LM 1 of the trained model storage part 2006 to the shovel 100 in response to a signal received from the shovel 100 requesting the distribution of the trained model LM 1 . The same may be applied to the trained models LM 2 and LM 3 .
  • the work support part 302 is a functional part for supporting the work of the shovel 100 operating by the autonomous operation function.
  • the work support part 302 includes a trained model storage part 302 A, a target material state prediction part 302 B, an operation planning part 302 C, a target path generation part 302 D, and an operation control part 302 E.
  • the trained model storage part 302 A stores trained models LM 1 and LM 2 distributed from the information processing device 200 received and through the communication device 60 .
  • the target material state prediction part 302 B predicts the state of the target material of the shovel 100 at a predetermined time in the future based on the current state of the target material around the shovel 100 and the target path of the work part of the shovel 100 up to a predetermined time. Specifically, the target material state prediction part 302 B predicts the state of the target material of the shovel 100 at a future time using the trained model LM 1 .
  • the current state of the target material around the shovel 100 is acquired, for example, based on the output of the sensor 40 .
  • the current state of the target material around the shovel 100 may be acquired based on the output of the sensors S 7 to S 9 instead of or in addition to the sensor 40 . This is because the controller 30 can estimate the reaction force from the ground acting on the bucket 6 from the output of the sensors S 7 to S 9 , and can estimate the state (shape and characteristics) of the earth of the target material from the estimated result of the reaction force.
  • the current state of the target material around the shovel 100 may be a prediction result of the state of the target material output by the target material state prediction part 302 B itself at a processing timing earlier than the current time.
  • the target material state prediction part 302 B predicts the state of the target material based on the initial state of the target material at the beginning of the work, and subsequently uses the prediction result as the current state of the target material.
  • the initial state of the target material at the beginning of the work may be provided from the outside of the shovel 100 , or may be previously defined as a fixed state, such as a plane at the same height as the ground on which the lower traveling body 1 of the shovel 100 is grounded.
  • the predetermined time point is, for example, the timing (time t b ) of the start of the predetermined operation that can be modified immediately.
  • a modification of the predetermined operation means, for example, the modification of the type of the predetermined operation that is scheduled to be performed to another type, such as the modification of a future predetermined in operation an undetermined or tentatively determined state or an excavation operation that is scheduled to be performed to a slope pulling operation.
  • the predetermined time point may be the timing (time t s ) at which the path of the bucket 6 and the type of the predetermined operation can be modified, considering the delay time ⁇ s in the process.
  • the delay time ⁇ s includes, for example, the calculation time for the controller 30 to generate the target path of the bucket 6 or to determine the predetermined operation of the shovel 100 , and the interface time to transfer the calculation result to the control part.
  • the time t s is calculated by the following equation (1) using the current time t 1 and the delay time ⁇ s .
  • the delay time ⁇ s may be a fixed value or a variable value.
  • the fixed value is previously defined as the maximum value of the delay time that can be assumed depending on, for example, the processing state of the controller 30 .
  • the delay time ⁇ s is varied according to a predetermined rule depending on, for example, the processing state of the controller 30 such as the CPU load state.
  • the time t b corresponds to a start time of the next predetermined operation of the predetermined operation being performed at the time t s .
  • the time t b corresponds to the start time of the next operation B of the current operation A.
  • the controller 30 cannot modify the next operation B to another type of predetermined operation due to the delay time ⁇ s . Therefore, in this case, the time t b corresponds to the start time of the next operation C of the operation B being performed at the time t s .
  • the operation planning part 302 C plans (determines) the predetermined operation (type) to be performed by the shovel 100 from the time t b based on the prediction result (prediction result of the state of the target material at the time t b ) by the target material state prediction part 302 B.
  • the controller 30 can determine the predetermined operation type to be performed by the shovel 100 in accordance with the predicted state of the target material in the future (time t b ). Therefore, for example, it is not necessary to stop the operation of the shovel 100 to some extent, as in the case of determining the subsequent predetermined operation based on the actual state of the target material at the time of completion of the predetermined operation immediately before the time t b . Therefore, the controller 30 can improve the work efficiency of the shovel 100 .
  • the operation planning part 302 C may adjust the processing timing so as to plan the next predetermined operation after the predetermined operation currently being performed by the shovel 100 . For example, when the remaining time until the end time of the predetermined operation currently being performed by the shovel 100 is not less than the delay time ⁇ s , the operation planning part 302 C plans the next predetermined operation of the shovel 100 .
  • the operation planning part 302 C determines the predetermined operation (type) to be started at the time t b based on the prediction result of the state of the target material at the time t b by using a rule-based method.
  • a plurality of predetermined operations that can be performed are defined for each of the target materials, and transition conditions for each of the previously defined operations that can transition are specified in advance for each of the predetermined operations.
  • the predetermined operations that can transition can include the same predetermined operation. This is because the same predetermined operation may be repeated.
  • the operation planning part 302 C determines the predetermined operation (type of operation) to be started at time t b based on the success or failure of a plurality of transition conditions starting from the predetermined to be performed operation immediately before time t b .
  • the excavation operation ST 1 - 1 , the earth removing operation ST 1 - 2 , and the slope pulling operation ST 1 - 3 are defined as the plurality of predetermined operations that can be performed.
  • the predetermined operations that can be transitioned from a standby state ST 1 - 0 corresponding to before the work start and after the work completion are the excavation operation ST 1 - 1 and the slope pulling operation ST 1 - 3 , and the transition conditions SC 1 - 01 and SC 1 - 03 are defined respectively.
  • the transition conditions SC 1 - 01 and SC 1 - 03 are mutually opposing conditions. For example, if a difference between the shape of the target material and the target shape at the beginning of the work is not more than the predetermined standard, the transition condition SC 1 - 03 is established, and if it exceeds the predetermined standard, the transition condition SC 1 - 01 is established.
  • the predetermined operation to be performed at the beginning of the work is determined to be the predetermined operation corresponding to one of the transition conditions SC 1 - 01 and SC 1 - 03 .
  • the predetermined operation that can transition from the excavation operation ST 1 - 1 is the earth removing operation ST 1 - 2 , and the transition condition SC 1 - 12 is defined. That is, when the predetermined operation performed immediately before time t b is the excavation operation ST 1 - 1 , the transition condition SC 1 - 12 is always established, and the predetermined operation performed from time t b is unambiguously determined to be the earth removing operation ST 1 - 2 .
  • the predetermined operations that can transition from the earth removing operation ST 1 - 2 are the excavation operation ST 1 - 1 and the slope pulling operation ST 1 - 3 , and the transition conditions SC 1 - 21 and SC 1 - 23 are defined respectively.
  • the transition conditions SC 1 - 21 and SC 1 - 23 are mutually opposing conditions. For example, when the difference between the predicted result of the shape of the target material and the target shape at time t b is not more than the predetermined standard, the transition condition SC 1 - 23 is established, and when it exceeds the predetermined standard, the transition condition SC 1 - 21 is established.
  • the predetermined operation to be performed from time t b is determined to be the predetermined operation corresponding to one of the transition conditions SC 1 - 21 and SC 1 - 23 .
  • the predetermined operations that can transition from the slope pulling operation ST 1 - 3 are the excavation operation ST 1 - 1 and the slope pulling operation ST 1 - 3 , and the transition conditions SC 1 - 31 and SC 1 - 33 are defined respectively.
  • the transition conditions SC 1 - 31 and SC 1 - 33 are mutually opposite. If the predetermined operation performed immediately before time t b is the slope pulling operation ST 1 - 3 , the predetermined operation performed from time t b is determined to be the predetermined operation corresponding to one of the transition conditions SC 1 - 31 and SC 1 - 33 .
  • the controller 30 shifts to the standby state without performing the predetermined operation determined in advance by the operation planning part 302 C (see a dashed arrow in the figure).
  • the work completion condition is that the difference between the shape of the target material and the target shape just before time t b is so small that it can be considered zero.
  • the operation planning part 302 C may use the trained model LM 2 to determine the predetermined operation (type of operation) to be started by the shovel 100 at time t b based on the prediction result of the state of the target material at time t b .
  • the operation planning part 302 C can more appropriately determine the predetermined operation to be started by the shovel 100 at time t b .
  • the operation planning part 302 C can determine the predetermined operation to be performed by the shovel 100 from time t b only by the rule-based method, the training data generation part 2004 B and the machine learning part 2005 B are omitted.
  • the target path generation part 302 D generates the target path of the work part in the predetermined operation of the shovel 100 based on the state of the target material around the shovel 100 .
  • the predetermined operation at this time is the type of predetermined operation determined by the operation planning part 302 C.
  • the target path generation part 302 D generates the target path of the work part after time t s using the trained model LM 3 based on the prediction result (predicted state of the target material at time t s , t b ) of the target material state prediction part 302 B.
  • the controller 30 can modify the target path of the work part in the predetermined operation of the shovel 100 in accordance with the state (prediction result) of the target material during performance of the predetermined operation. Therefore, the shovel 100 can progress the work of the shovel 100 more appropriately and efficiently according to the change of the state of the target material.
  • the target path generation part 302 D may generate the target path of the work part in the predetermined operation performed at time t s and the target path of the work part in the predetermined operation started from time t b .
  • the target path generation part 302 D may generate the target path of the work part of the shovel 100 in accordance with the state (prediction result) of the target material around the shovel 100 by applying any known technique instead of the trained model LM 3 .
  • the training data generation part 2004 C and the machine learning part 2005 C may be omitted.
  • the target path generation part 302 D may generate data of the target path of the work part of the shovel 100 by MPC (Model Predictive Control) based on the prediction result (predicted state of the target material at time t s , t b ) of the target material state prediction part 302 B.
  • the target path generation part 302 D may generate data of the target path of the work part of the shovel 100 by optimizing the previously defined reference path of the work part of the shovel 100 based on data concerning the characteristics of earth given in advance.
  • the operation control part 302 E causes the shovel 100 to perform a predetermined operation so that the predetermined part of the shovel 100 moves along the target path generated by the target path generation part 302 D. Specifically, the operation control part 302 E causes the shovel 100 to perform a predetermined operation so that the work part of the shovel 100 moves along the target path by controlling the hydraulic controlling valve 31 while determining the position of the work part from the outputs of the sensors S 1 to S 5 . Thus, the shovel 100 can autonomously advance the work while performing the predetermined operation according to the shape of the target material.
  • the controller 30 determines a future predetermined operation of the shovel 100 from among a plurality of predetermined operations corresponding to the target work according to the performance status of the predetermined operation of the working machine. Specifically, the controller 30 may predict the future state of the target material according to the performance status of the operation of the working machine. Then, the controller 30 may determine a future predetermined operation of the shovel 100 from among a plurality of predetermined operations corresponding to the target work based on the prediction result of the future state of the target material. Thus, the controller 30 can determine in advance the operation of the shovel 100 to be performed in accordance with the future state of the target material. Therefore, idle time when the predetermined operation of the shovel 100 is completed and the next predetermined operation is performed can be reduced, and the work efficiency of the shovel 100 can be improved.
  • the controller 30 after determining the future predetermined operation of the shovel 100 , the controller 30 generates a target path of the work part corresponding to the determined predetermined operation of the shovel 100 .
  • the controller 30 can hierarchically determine the future predetermined operation of the shovel 100 and generate a target path of the work part in the future predetermined operation of the shovel 100 . Therefore, for example, when the operation plan of the shovel 100 and the path plan of the work part of the shovel 100 are performed in parallel, it is possible to control against an occurrence of a situation in which the operation plan and the path plan cannot be performed in a realistic time as a result of enormous conditions and parameters.
  • Some or all of the functions of the target material state prediction part 302 B, the operation planning part 302 C, the target path generation part 302 D, and the operation control part 302 E may be transferred to the information processing device 200 .
  • the processing load of the shovel 100 can be reduced for the processing related to the generation of the target path of the work part of the shovel 100 and the processing related to the control of the operation of the shovel 100 .
  • FIG. 11 is a functional block diagram illustrating a second example of the functional configuration of the operation support system SYS.
  • the shovel 100 includes a support device 150 as in the first example described above.
  • the support device 150 supports a user who operates the semi-automatic shovel 100 and performs work.
  • the support device 150 includes a controller 30 , a hydraulic controlling valve 31 , a sensor 40 , an output device 50 , and sensors S 1 to S 9 .
  • the support device 150 may include a communication device 60 instead of or in addition to the input device 52 .
  • the controller 30 includes, as functional parts, an operation log providing part 301 and a work support part 302 .
  • the operation log providing part 301 includes, as functional parts, an operation log recording part 301 A, an operation log storage part 301 B, and an operation log transmission part 301 C.
  • the information processing device 200 includes, as functional parts, a log acquisition part 2001 , a simulator part 2002 , a log storage part 2003 , a training data generation part 2004 , a machine learning part 2005 , a trained model storage part 2006 , and a distribution part 2007 .
  • the work support part 302 is a functional part for supporting a user who operates the semi-automatic shovel 100 and performs work.
  • the work support part 302 includes a trained model storage part 302 A, a target material state prediction part 302 B, an operation planning part 302 C, a target path generation part 302 D, and an operation control part 302 E.
  • the work support part 302 includes an operation proposal part 302 F.
  • the operation proposal part 302 F proposes predetermined operation (type) of the shovel 100 from time t b determined (planned) by the operation planning part 302 C to the user through the output device 50 and the remote control support device 400 .
  • the user selects a predetermined operation (type) to be performed by the shovel 100 starting from time t b from among a plurality of predetermined operations corresponding to the current work.
  • a predetermined operation to be performed by the shovel 100 starting from time t b using the input device 52 and the remote control support device 400 , for example.
  • the selection result by the user is input to the target path generation part 302 D through the input device 52 and the communication device 60 .
  • the target path generation part 302 D generates the target path of the work part after time t s by using the trained model LM 3 based on the prediction result of the target material state prediction part 302 B (the prediction result of the state of the target material at time t s ).
  • the target path generation part 302 D may generate the target path of the work part in the predetermined operation performed at time t s and the target path of the work part in the predetermined operation started at time t b .
  • the predetermined operation started at time t b is the predetermined operation of the shovel 100 corresponding to the selection result by the user.
  • the controller 30 determines the future predetermined operation of the shovel 100 from among a plurality of predetermined operations corresponding to the target operation based on the prediction result of the state of the target material in the future, and proposes it to the user through the output device 50 and the remote control support device 400 .
  • the user can determine the type of a recommended predetermined operation of the shovel 100 based on the prediction result of the state of the target material in the future before the completion of the predetermined operation immediately before the recommended predetermined operation of the shovel 100 . Therefore, idle time when the predetermined operation of the shovel 100 is completed and the next predetermined operation is performed can be reduced, and the work efficiency of the shovel 100 can be improved.
  • the functions of the operation planning part 302 C and the operation proposal part 302 F may be adopted in a manually operated shovel 100 , whose operations are entirely controlled by the operator.
  • the controller 30 may predict the path of the work part in the predetermined operation of the shovel 100 based on the history of the operation contents of the operator or the output of the sensors S 1 to S 9 , and may predict the future state of the target material based on the predicted path.
  • some or all of the functions of the trained model storage part 302 A, the target material state prediction part 302 B, the operation planning part 302 C, the target path generation part 302 D, the operation control part 302 E, and the operation proposal part 302 F may be provided in the remote control support device 400 .
  • the functions of the target material state prediction part 302 B, the operation planning part 302 C, the target path generation part 302 D, the operation control part 302 E, and the operation may be transferred to the proposal part 302 F information processing device 200 .
  • the processing load of the shovel 100 and the remote control support device 400 can be reduced in the processing related to the generation of the target path of the work part of the shovel 100 and the processing related to the control of the operation of the shovel 100 .
  • FIG. 12 is a flowchart schematically illustrating an example of processing related to a start of autonomous operation of the shovel 100 . This flowchart is performed when a predetermined input related to the start of autonomous operation is made by the user through the input device 52 , the remote control support device 400 , or the remote monitoring support device.
  • the controller 30 selects a predetermined operation to be performed at the beginning of autonomous operation of the shovel 100 in response to a predetermined input from the user through the input device 52 , the remote control support device 400 , or the remote monitoring support device. For example, the controller 30 selects one predetermined operation from among a plurality of predetermined operations defined for the target work such as excavation work, leveling work, and slope work. Further, prior to the step S 102 , the controller 30 may select one operation from among a plurality of operations such as excavation work, leveling work, and slope work in response to a predetermined input from the user through the input device 52 , the remote control support device 400 , or the remote monitoring support device.
  • step S 104 the controller 30 proceeds to a step S 104 .
  • the controller 30 acquires data representing the state (shape and characteristic) of the earth to be worked, based on the output of the sensor 40 .
  • step S 104 When the processing of the step S 104 is completed, the controller 30 proceeds to a step S 106 .
  • the controller 30 (target path generation part 302 D) generates the target path of the bucket 6 for the predetermined operation of the shovel 100 selected in the step S 102 at the beginning of the work, based on the data acquired in the step S 104 .
  • the target path generation part 302 D generates the target path of the work part of the shovel 100 in the same manner as in a step S 208 described later.
  • step S 106 When the processing of the step S 106 is completed, the controller 30 proceeds to the step S 108 .
  • the controller 30 notifies the user that autonomous operation is enabled. For example, the controller 30 notifies the user inside the cabin 10 or the user near the shovel 100 through the output device 50 .
  • the controller 30 may also notify the user using the remote control support device 400 or the remote monitoring support device by transmitting the notification signal to the remote control support device 400 or the remote monitoring support device through the communication device 60 .
  • step S 108 When the processing in the step S 108 is completed, the controller 30 proceeds to a step S 110 .
  • the controller 30 starts the autonomous operation of the shovel 100 in response to an instruction from the user received through the input device 52 , the remote control support device 400 , or the remote monitoring support device.
  • step S 110 When the processing of the step S 110 is completed, the controller 30 ends processing of the present flowchart.
  • the controller 30 can start the autonomous operation of the shovel 100 in response to a predetermined input from the user.
  • FIG. 13 is a main flowchart schematically illustrating an example of processing related to bucket path generation.
  • FIG. 14 is a drawing illustrating an example of an observation target area TA.
  • This flowchart is repeatedly performed every predetermined control cycle after the start of the autonomous operation of the shovel 100 .
  • a step S 202 the controller 30 acquires the future times t s and t b as references for the operation plan of the shovel 100 and the path generation.
  • step S 202 When the processing in the step S 202 is completed, the controller 30 proceeds to a step S 204 .
  • the target material state prediction part 302 B predicts the state of the earth of the target material (ground) at times t s and t b . Specifically, the target material state prediction part 302 B predicts the state of the earth of the target material at a modifiable start time t s based on the state of the earth of the target material at the current time t 1 and the target path of the bucket 6 from the current time t 1 to the time t s .
  • the target material state prediction part 302 B predicts the state of the earth of the target material at time t b based on the state of the earth of the target material at the current time t 1 and the target path of the bucket 6 from the current time t 1 to the time t b .
  • the data of the target path of the bucket 6 used in this step is obtained in the process of a step S 208 in the previous flow chart or in the process of the step S 106 in FIG. 8 .
  • the observation target area TA around the shovel 100 is divided into grid units of a predetermined number N.
  • the observation target area TA is an area around the shovel 100 in which the target material state prediction part 302 B acquires data indicating the state of the earth.
  • the shape h t of the earth and the characteristic ⁇ t of the earth of the work target of the shovel 100 for each grid unit i of the observation target area TA at the current time t 1 are expressed by the following equations (2) and (3).
  • the target path X t1:ts of the bucket 6 from the current time t 1 to the modifiable start time t s and the target path X t1:tb of the bucket 6 from the current time t 1 to the time t b are expressed by the following equations (4) and (5).
  • the target path X t1:ts of the bucket 6 is defined as a set of positions X t of the bucket 6 at each time t which are expressed discretely.
  • the position X t of the bucket 6 at time t is expressed by the following equation (6) as a set of the attitude ⁇ 1, t of the boom 4 , the attitude ⁇ 2, t of the arm 5 , the attitude ⁇ 3, t of the bucket 6 , and the attitude ⁇ 4, t of the upper swivel body 3 .
  • the attitude ⁇ 1, t of the boom 4 is information representing, for example, the position (rod position) of the boom cylinder 7 .
  • the attitude ⁇ 1, t of the boom 4 may be information representing the attitude angle of the boom 4 .
  • the data of the attitude ⁇ 1, t of the boom 4 is acquired based on the output of the sensor S 7 .
  • the attitude ⁇ 2, t of the arm 5 is, for example, information representing the position (rod position) of the arm cylinder 8 .
  • the attitude ⁇ 2, t of the arm 5 may be information representing the attitude angle of the arm 5 .
  • the data of the attitude ⁇ 2, t of the arm 5 is acquired based on the output of the sensor S 8 .
  • the attitude ⁇ 3, t of the bucket 6 is, for example, information representing the position (rod position) of the bucket cylinder 9 .
  • the attitude ⁇ 3, t of the bucket 6 may be information representing the attitude angle of the bucket 6 .
  • the data of the attitude ⁇ 3, t of the bucket 6 is acquired based on the output of the sensor S 9 .
  • the attitude ⁇ 4, t of the upper swivel body 3 is, for example, information representing the swivel angle of the upper swivel body 3 .
  • the data of the attitude ⁇ 4, t of the upper swivel body 3 is acquired based on the output of the sensor S 4 or the sensor S 5 .
  • the position X t of the bucket 6 may include information on the velocity, acceleration, or jerk and the like of each of the boom 4 , the arm 5 , and the bucket 6 .
  • the target material state prediction part 302 B predicts the shape h ts of the earth and the earth kts of the earth at the time t s by using a function g corresponding to the trained model LM 1 .
  • the shape h ts of the earth and the characteristic kts of the earth at the time t s is expressed by the following equation (7).
  • the target material state prediction part 302 B predicts the shape h tb of the earth and the characteristic ⁇ tb of the earth at the time t b by using a function g.
  • the shape h tb of the earth and the characteristic ⁇ tb of the earth at the time t b is expressed by the following equation (8).
  • the function g is based on, for example, a DNN.
  • step S 204 When the processing in the step S 204 is completed, the controller 30 proceeds to a step S 206 .
  • the operation planning part 302 C determines the next predetermined operation (type) v k that can be modified from among a plurality of predetermined operations corresponding to the current operation based on the current predetermined operation (type) v k-1 and the state of the earth predicted in the step S 204 .
  • the operation planning part 302 C determines the next predetermined operation (type) v k that can be modified from among a plurality of predetermined operations corresponding to the current operation using the function f corresponding to the trained model LM 2 .
  • the predetermined operation v k is expressed by the following equation (9) using the function f.
  • the predetermined operations v k and v k-1 are expressed by, for example, a one-hot vector.
  • the function f may be defined in the rule base instead of being given by the trained model LM 2 as described above.
  • step S 206 When the processing of the step S 206 is completed, the controller 30 proceeds to a step S 208 .
  • the target path generation part 302 D generates the target path of the bucket 6 in the predetermined operation of the shovel 100 after the time t s , based on the prediction result (the state of the earth at the time t s ) of the process in step S 204 . Specifically, the target path generation part 302 D generates the target path of the bucket 6 in the predetermined operation of the shovel 100 from the time t s to the time t b , and the target path of the bucket 6 in the predetermined operation of the shovel 100 after the time t b until the predetermined timing.
  • step S 208 When the process in the step S 208 is completed, the controller 30 proceeds to a step S 210 .
  • the target path generation part 302 D writes the data of the target path of the bucket 6 generated in the step S 208 to a predetermined storage area (address) of the memory device 30 B.
  • the operation control part 302 E can access the latest data of the target path of the shovel 100 by accessing the predetermined address of the memory device 30 B.
  • the controller 30 ends the process of the present flowchart.
  • the controller 30 predicts a future shape of the earth, and determines the predetermined future operation of the shovel 100 based on the future shape of the earth.
  • the controller 30 can generate the target path of the bucket 6 based on the predicted future shape of the earth and the determined predetermined future operation.
  • FIG. 15 is a sub-flowchart schematically illustrating an example of processing related to path generation of the bucket 6 .
  • FIG. 16 is a drawing illustrating an example of cost conditions and operation parameters corresponding to a plurality of sections of a shovel 100 excavation operation.
  • the sub-flowchart of FIG. 15 corresponds to the process of the step S 208 in FIG. 13 .
  • the target path generation part 302 D selects a constraint function related to the path of the bucket 6 corresponding to (type of) a predetermined operation of the shovel 100 .
  • the predetermined operation of the shovel 100 is a predetermined operation of the shovel 100 performed at time t s and a predetermined operation of the shovel 100 starting from time t b .
  • the target path generation part 302 D selects a constraint function related to the path of the bucket 6 corresponding to the predetermined operation.
  • the constraint function includes, for example, constraints related to the moving range, speed, and acceleration of the boom cylinder 7 , the arm cylinder 8 , and the bucket cylinder 9 . Further, the constraint function (constraint condition) may include a constraint condition for avoiding collision between an obstacle around the shovel 100 and the bucket 6 .
  • the obstacle around the shovel 100 includes, for example, a person, a working vehicle, another working machine, a feature (e.g., fence, utility pole), and the like, and can be recognized based on the output of the sensor 40 .
  • step S 302 When the processing of the step S 302 is completed, the controller 30 proceeds to a step S 304 .
  • the target path generation part 302 D selects an objective function (cost function) related to the path of the bucket 6 corresponding to (type of) the predetermined operation of the shovel 100 .
  • the predetermined operation of the shovel 100 is the predetermined operation of the shovel 100 performed at time t s and the predetermined operation of the shovel 100 starting from time t b , as in the case of step S 302 .
  • the target path generation part 302 D selects a cost function related to the path of the bucket 6 corresponding to the predetermined operation.
  • the excavation operation as the predetermined operation of the shovel 100 is divided into the operation sections of approach, penetration, horizontal excavation, and scooping up.
  • “Approach” is an operation section in which the bucket 6 is brought close to the ground in order to penetrate the ground.
  • Pornetration is an operation section in which the cutting edge of the bucket 6 is brought into contact with the ground after the operation section of approach and the bucket 6 is penetrated to a certain depth of the ground.
  • Horizontal excavation is an operation section in which the bucket is 6 moved approximately horizontally after the operation section of penetration.
  • Sprizontal excavation is an operation section in which the bucket is 6 moved approximately horizontally after the operation section of penetration.
  • “Scooping up” is an operation section in which earth are stored in the bucket 6 after horizontal excavation and the earth are scooped up on the ground.
  • cost functions related to the speed and acceleration of the bucket 6 , the travel time of the bucket 6 , and the like are defined over all the operation sections from “approach” to “scooping up”.
  • cost functions related to the position of the cutting edge of the bucket 6 , the angle of the cutting edge to a predetermined reference (e.g., a horizontal plane), and the path of the cutting edge are defined at the end of one operation section among “approach”, “penetration”, “horizontal excavation”, and “scooping up”, and in one operation section among the above.
  • step S 304 When the processing of the step S 304 is completed, the controller 30 proceeds to a step S 306 .
  • the target path generation part 302 D estimates an operation parameter that defines the target path of the bucket 6 in a predetermined operation of the shovel 100 using the trained model LM 3 based on the prediction result (state of the earth of the target material at time t s , t b ) in the step S 302 .
  • operation parameters q 1 to q 4 that define the position of the cutting edge of the bucket 6
  • operation parameters ⁇ 12 , ⁇ 23 , and ⁇ 4 that define the angle of the cutting edge of the bucket 6 to a predetermined reference are defined.
  • the operation parameter q 1 is an operation parameter representing the position of the cutting edge of the bucket 6 at the end of “approach” and at the beginning of “penetration”.
  • a cost function corresponding to the condition for determining that the position of the cutting edge of the bucket 6 matches the position of the operation parameter q 1 is defined.
  • the operation parameter q 2 is an operation parameter representing the position of the cutting edge of the bucket 6 at the end of “penetration” and at the beginning of “horizontal excavation”.
  • a cost function corresponding to the condition for determining that the position of the cutting edge of the bucket 6 matches the position of the operation parameter q 2 is defined.
  • a cost function corresponding to the condition for determining that the position (path) of the cutting edge is on the straight line defined by the operation parameters q 1 and q 2 in the operation section of “penetration” is defined.
  • the operation parameter q 3 is an operation parameter representing the position of the cutting edge of the bucket 6 at the end of “horizontal excavation” and at the beginning of “scooping up”.
  • the cost function corresponding to the condition for determining that the position of the edge of the bucket 6 and the position of the operation parameter q 3 match at the end of “horizontal excavation” and at the beginning of “scooping up” is defined.
  • the cost function corresponding to the condition for determining that the position (path) of the edge is on the straight line defined by the operation parameters q 2 and q 3 in the operation section of “horizontal excavation” is defined.
  • the operation parameter q 4 is an operation parameter representing the position of the edge of the bucket 6 at the end of “scooping up”.
  • the cost function corresponding to the condition for determining that the position of the edge of the bucket 6 and the position of the operation parameter q 4 match at the end of “scooping up” is defined.
  • the operation parameter ⁇ 12 is an operation parameter representing the angle of the edge of the bucket 6 with respect to the predetermined reference in the operation section of “penetration”.
  • the constraint function and cost function corresponding to the condition for determining that the angle of the edge of the bucket 6 with respect to the predetermined reference and the angle of the operation parameter ⁇ 12 match in the operation section of “penetration” are defined.
  • the operation parameter ⁇ 23 is an operation parameter representing the angle of the cutting edge of the bucket 6 with respect to a predetermined reference in the operation section of “horizontal excavation”.
  • the constraint function and the cost function corresponding to the condition for determining that the angle of the cutting edge of the bucket 6 with respect to a predetermined reference matches the angle of the operation parameter ⁇ 23 , are defined.
  • the operation parameter ⁇ 4 is an operation parameter representing the angle of the cutting edge of the bucket 6 with respect to a predetermined reference at the end of “scooping up”.
  • the constraint function and the cost function corresponding to the condition for determining that the angle of the cutting edge of the bucket 6 with respect to a predetermined reference matches the angle of the operation parameter ⁇ 4 at the end of “scooping up” are defined.
  • step S 306 When the processing in the step S 306 is completed, the controller 30 proceeds to a step S 308 .
  • the target path generation part 302 D calculates the target path of the bucket 6 in the predetermined operation of the shovel 100 based on the constraint function and the objective function selected in the steps S 302 and S 304 , and the operation parameter of the estimation result in the step S 306 . Specifically, the target path generation part 302 D calculates the target path of the bucket 6 in the predetermined operation of the shovel 100 by solving a constrained nonlinear optimization problem defined by the constraint function and the objective function using a predetermined solver.
  • the controller 30 can generate the target path of the shovel 100 by using the constraint function and the objective function corresponding to the predetermined operation of the shovel 100 .
  • the path of the bucket 6 is represented by a relatively small number of operation parameters. Then, the controller 30 can infer the operation parameters which define the path of the bucket 6 by using the trained model LM 3 based on the prediction result of the state of the earth of the future target material.
  • FIG. 17 is a flowchart schematically illustrating an example of processing related to operation control of the shovel 100 .
  • This flowchart is repeatedly performed for each predetermined processing cycle during performance of the autonomous operation of the shovel 100 , for example.
  • the operation control part 302 E reads the latest data representing the target path of the bucket 6 in the predetermined operation of the shovel 100 from the predetermined address of the memory device 30 B. This data is the data registered in the step S 210 of FIG. 13 .
  • step S 402 When the processing in the step S 402 is completed, the controller 30 proceeds to a step S 404 .
  • the operation control part 302 E controls the operation of the shovel 100 based on the data of the target path of the bucket 6 in the predetermined operation of the shovel 100 read in the step S 402 . Specifically, the operation control part 302 E controls the operation of the shovel 100 while outputting a control command to the hydraulic controlling valve 31 so that the bucket 6 moves along the target path corresponding to the data read in the step S 402 .
  • the controller 30 can control the operation of the shovel 100 so as to move along the target path of the predetermined operation of the shovel 100 .
  • the training data c j is a combination of input data h ⁇ circumflex over ( ) ⁇ j , ⁇ circumflex over ( ) ⁇ j , and X j , and the shape h r j of the earth and the characteristic ⁇ r j of the earth at the time of completion of the path X j as the ground truth.
  • the input data h ⁇ circumflex over ( ) ⁇ j , ⁇ circumflex over ( ) ⁇ j , and X j correspond to the input data of the function g in equations (7) and (8).
  • the training data set D may be generated from the log acquired by the log acquisition part 2001 , the log acquired by the simulator part 2002 , or both logs.
  • the simulator part 2002 for example, as described above, a particle simulation such as DEM is adopted, and the height h r j of the earth is acquired by ray tracing of the shape sensor such as LIDAR which is virtually arranged with respect to the position of the particle.
  • a particle simulation such as DEM is adopted, and the height h r j of the earth is acquired by ray tracing of the shape sensor such as LIDAR which is virtually arranged with respect to the position of the particle.
  • the training data set D may include a base training data set generated from logs acquired by the simulator part 2002 and a fine-tuning training data set generated from logs acquired by the log acquisition part 2001 .
  • the number of training data included in the fine-tuning training data set may be relatively small.
  • the function g has a parameter W, and the machine learning part 2005 A performs machine learning in such a manner that the parameter W is optimized by the training data set D.
  • the function g corresponding to the trained model LM 1 is generated by optimizing the parameter W so that the loss function E(W) in the following equation (12) is minimized.
  • the information processing device 200 generates the training data set D including the training data c j , and can generate the function g corresponding to the trained model LM 1 by machine learning based on the training data set D.
  • the working machine includes an operation planning part.
  • the working machine is, for example, the above-described shovel 100 .
  • the operation planning part is, for example, the above-described operation planning part 302 C. Specifically, the operation planning part determines the future operation of the working machine from among a plurality of according operations to the performance status of the operation of the working machine.
  • the information processing device may include an operation planning part.
  • the information processing device may be, for example, the controller 30 , the information processing device 200 , or the remote control support device 400 described above.
  • the program may cause the information processing device to perform an operation planning step in order to achieve the function of the operation planning part.
  • the operation planning step the future operation of the working machine is determined from among a plurality of operations according to the performance status of the operation of the working machine.
  • the operation planning step is, for example, the step S 206 described above.
  • the working machine or the like can determine the future operation of the working machine from among a plurality of operations in consideration of the state of the target material which changes according to the performance status of the operation of the working machine. Therefore, the working machine can achieve a more appropriate operation.
  • the working machine and the information processing device may include a prediction part.
  • the prediction part is, for example, the target material state prediction part 302 B described above.
  • the prediction part may predict a future state of the target material according to the performance status of the operation of the working machine.
  • the operation planning part may determine the future operation of the working machine from among the plurality of operations, based on the prediction result of the future state of the target material by the prediction part.
  • the program may cause the information processing device to perform the prediction step in order to achieve the function of the prediction part.
  • the prediction step may predict the future state of the target material according to the performance state of the operation of the working machine.
  • the operation planning step may decide the future operation of the working machine from among a plurality of operations based on the prediction result of the future state of the target material by the prediction part.
  • the working machine or the like can determine the future operation of the working machine from among a plurality of operations in accordance with the future state of the target material predicted according to the performance status of the operation of the working machine.
  • the prediction part may predict the future state of the target material, based on the path of the work part by the operation of the working machine.
  • the working machine or the like can predict the future state of the target material considering the change of the state of the target material which changes in accordance with the path of the work part.
  • the prediction part may predict a state of the target material after completion of a current operation of the working machine or a next operation after the current operation, based on the path of the work part by the current operation of the working machine or a next scheduled operation. Then, the operation planning part determines a next operation after the current operation of the working machine or a further next operation after the next scheduled operation from among the plurality of operations, based on the prediction result of the future state of the target material by the prediction part.
  • the working machine or the like can determine the next operation according to the prediction result of the work state after the completion of the operation under performance of the working machine. Furthermore, the working machine or the like can determine the next operation after the next operation scheduled to be performed according to the prediction result of the work state after the completion of the next operation.
  • the working machine or the like may include a generation part and a control part.
  • the generation part is, for example, the target path generation part 302 D described above.
  • the control part is, for example, the operation control part 302 E described above.
  • the generation part may generate the path of the work part by the operation of the working machine, based on the state of the target material.
  • the control part may control the operation of the working machine so that the work part moves along the path generated by the generation part.
  • the prediction part may predict the future state of the target material, based on the current state of the target material and the path of the work part generated by the generation part.
  • the program may cause the information processing device to perform the generation step and the control step in order to achieve the functions of the generation part and the control part.
  • the generation step is, for example, the step S 208 described above.
  • the control step is, for example, the step S 404 described above.
  • the generation step the path of the work part by the operation of the working machine may be generated based on the state of the target material.
  • the control step the operation of the working machine may be controlled so that the work part moves along the path generated in the generation step.
  • the prediction step the future state of the target material may be predicted based on the current state of the target material and the path of the work part generated in the generation step.
  • the working machine or the like can predict the future state of the target material based on the path of the work part generated for automatic operation of the working machine.
  • the prediction part may predict the state of the target material after the elapse of a predetermined time.
  • the generation part may generate the path of the work part after the elapse of the predetermined time, based on the prediction result of the state of the target material after the elapse of the predetermined time by the prediction part.
  • the working machine or the like can generate the path of the work part after the elapse of the predetermined time considering, for example, the delay time from the instruction to control the operation of the working machine to the actual control of the operation.
  • the generation part may generate the path of the work part, based on measurement data of the state of the target material, by using an objective function and a constraint function defined for each of the plurality of operations.
  • the working machine or the like can generate the path of the work part.
  • the path of the work part may be expressed by a predetermined number, which is two or greater, of parameters for each of the plurality of operations. Then, the generation part may generate the path of the work part by determining a predetermined number of parameters, based on the state of the target material.
  • the working machine or the like can generate the path of the work part.
  • the state of the target material may include at least one of the shape and the characteristic of the earth on the surface of the target material.
  • the working machine or the like can predict the future state of the target material in accordance with the shape and the characteristic of the earth of the target material.
  • the working machine or the like may include a notification part for notifying the operator of the operation determined by the operation planning part.
  • the program may cause the support device to perform the operation planning step and the notification step.
  • the support device is, for example, the support device 150 and the remote control support device 400 .
  • the operation planning step the future operation of the working machine may be decided from among a plurality of operations according to the performance status of the operation of the working machine.
  • the notification step the operation decided in the operation planning step may be notified to the operator of the working machine.
  • the working machine or the like can notify the operator of a more appropriate type of operation of the working machine to the operator who operates the working machine. Therefore, the operator can operate the working machine more appropriately. Moreover, the working machine or the like can notify the operator in advance of a more appropriate type of operation of the working machine to the operator who operates the working machine. Therefore, the operator can start the next operation immediately after the completion of a certain operation, and as a result, the work efficiency of the working machine can be improved.

Landscapes

  • Engineering & Computer Science (AREA)
  • Mining & Mineral Resources (AREA)
  • Civil Engineering (AREA)
  • General Engineering & Computer Science (AREA)
  • Structural Engineering (AREA)
  • Operation Control Of Excavators (AREA)
US19/211,957 2022-11-22 2025-05-19 Working machine, information processing device, and program Pending US20250283308A1 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
JP2022186780 2022-11-22
JP2022-186780 2022-11-22
PCT/JP2023/041864 WO2024111596A1 (ja) 2022-11-22 2023-11-21 作業機械、情報処理装置、プログラム

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/041864 Continuation WO2024111596A1 (ja) 2022-11-22 2023-11-21 作業機械、情報処理装置、プログラム

Publications (1)

Publication Number Publication Date
US20250283308A1 true US20250283308A1 (en) 2025-09-11

Family

ID=91196141

Family Applications (1)

Application Number Title Priority Date Filing Date
US19/211,957 Pending US20250283308A1 (en) 2022-11-22 2025-05-19 Working machine, information processing device, and program

Country Status (5)

Country Link
US (1) US20250283308A1 (https=)
JP (1) JPWO2024111596A1 (https=)
CN (1) CN119816642A (https=)
DE (1) DE112023004870T5 (https=)
WO (1) WO2024111596A1 (https=)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2025006866A (ja) * 2023-06-30 2025-01-17 株式会社日立製作所 自律制御システムおよび自律制御方法

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7114302B2 (ja) * 2018-03-31 2022-08-08 住友建機株式会社 ショベル及びショベルの管理装置
JP2021055433A (ja) * 2019-09-30 2021-04-08 住友重機械工業株式会社 ショベル
JP7478590B2 (ja) * 2020-05-20 2024-05-07 住友重機械工業株式会社 ショベル
JP7009600B1 (ja) * 2020-12-07 2022-01-25 日立建機株式会社 作業機械
JP7721305B2 (ja) * 2021-03-31 2025-08-12 住友重機械工業株式会社 建設機械の管理システム

Also Published As

Publication number Publication date
CN119816642A (zh) 2025-04-11
DE112023004870T5 (de) 2025-10-02
JPWO2024111596A1 (https=) 2024-05-30
WO2024111596A1 (ja) 2024-05-30

Similar Documents

Publication Publication Date Title
US12158760B2 (en) Worksite plan execution
JP7726614B2 (ja) ショベル
JP7597022B2 (ja) ショベル
US20250019937A1 (en) Assistance device, work machine, and recording medium
US20230417023A1 (en) Support system, information processing device, and storage medium
US20250179762A1 (en) Assistance device, work machine, and assistance system
US20250283308A1 (en) Working machine, information processing device, and program
US20250163671A1 (en) Autonomous Control Of Powered Earth-Moving Vehicles To Control Ripper Tool Operations
CN115877800A (zh) 机上远程控制
WO2024057961A1 (ja) 作業機械を含むシステム、作業機械のコントローラ、および作業機械の制御方法
US20250215668A1 (en) Work machine, remote operation assisting device, and assisting system
US12442158B1 (en) Automated training and use of predictive models for autonomous control of powered earth-moving vehicles
US20250171983A1 (en) Autonomous Control Of Powered Earth-Moving Vehicles To Control Calibration Operations For On-Vehicle Sensors
JP7798248B2 (ja) 支援装置、作業機械、プログラム
JP7799952B2 (ja) 支援装置、作業機械、プログラム
JP2024099211A (ja) ショベル
JP2024076741A (ja) 作業機械、情報処理装置、プログラム
WO2025115773A1 (ja) 作業機械、情報処理装置、プログラム
US12618223B1 (en) Autonomous control of powered earth-moving vehicles to rectify vehicle slippage
US20250207357A1 (en) Work machine, operation assisting system, information processing device, and program
US20250003197A1 (en) Supporting device, work machine, and program
WO2025115981A1 (ja) 作業機械、情報処理装置、プログラム
US20250163673A1 (en) Autonomous Control Of Powered Earth-Moving Vehicles To Control Steering Operations Using A Blade Tool
US20250163672A1 (en) Autonomous Control Of Powered Earth-Moving Vehicles To Control Blade Tool Loading Operations
EP4403705A1 (en) Shovel

Legal Events

Date Code Title Description
AS Assignment

Owner name: SUMITOMO HEAVY INDUSTRIES, LTD., JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HARA, KOSUKE;TSUZUKI, RYUJI;SIGNING DATES FROM 20250331 TO 20250403;REEL/FRAME:071156/0285

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION