JP7479156B2 - Work machines, data processing devices - Google Patents

Description

This disclosure relates to work machines, etc.

For example, there is known technology that outputs an alarm when a monitored object, such as a worker, is detected within a close range around a work machine such as an excavator (see Patent Document 1).

JP 2019-060228 A

However, depending on the environment around the work machine, etc., the detection accuracy of monitored objects may decrease in at least a part of the monitored area, resulting in erroneous detection or overlooking of monitored objects. For example, when a monitored object is detected based on image information about the surroundings of the work machine, if the appearance of a background object is similar to the characteristics of the monitored object, the background object may be erroneously detected (misrecognized) as the monitored object. In addition, if the color of the background object and the color of the monitored object are similar, the monitored object may blend in with the background object, and the monitored object may not be detected (recognized).

Therefore, in consideration of the above issues, the objective is to provide technology that can further improve the safety of work machines.

In order to achieve the above object, in one embodiment of the present disclosure,
an acquisition unit that acquires information about objects around the work machine;
a detection unit that detects a specific type of object around the work machine based on the information acquired by the acquisition unit;
an alarm output unit that outputs an alarm to an operator when the specific type of object is detected by the detection unit;
and a notification unit that notifies an operator about an area around the work machine when there is an area within an information acquisition range around the work machine by the acquisition unit where the accuracy with which the detection unit can detect the specific type of object is relatively low compared to a predetermined standard due to the influence of other objects that exist around the work machine and correspond to a background seen from the acquisition unit when it is assumed that the specific type of object exists.
A work machine is provided.

In another embodiment of the present disclosure,
an operation unit that receives remote control operations related to the work machine;
a communication unit that transmits the contents of the remote operation received by the operation unit to the work machine;
an alarm output unit that outputs an alarm to an operator performing remote operation when a specific type of object is detected around the work machine by a detection unit of the work machine based on information about objects around the work machine acquired by an acquisition unit of the work machine;
a notification unit that notifies the operator about the area around the work machine when there is an area within an information acquisition range around the work machine by the acquisition unit where the accuracy with which the detection unit can detect the specific type of object is relatively low compared to a predetermined standard due to the influence of other objects that exist around the work machine and correspond to a background seen from the acquisition unit when it is assumed that the specific type of object exists,
An information processing device is provided.

The above-described embodiment provides technology that can further improve the safety of work machines.

FIG. 1 is a schematic diagram illustrating an example of an excavator management system. FIG. 2 is a top view showing an example of a shovel. FIG. 2 is a block diagram showing an example of a configuration of an excavator management system. FIG. 11 is a block diagram showing another example of the configuration of the excavator management system. 10 is a flowchart illustrating an example of a detection accuracy notification process performed by a controller. 10 is a flowchart illustrating an example of a detection accuracy determination process performed by a controller. 11A to 11C are diagrams specifically illustrating a detection accuracy determination process performed by the controller. 11A to 11C are diagrams specifically illustrating a detection accuracy determination process performed by the controller. 11A to 11C are diagrams specifically illustrating a detection accuracy determination process performed by the controller. FIG. 13 is a diagram showing an example of a low detection accuracy notification screen displayed on the display device.

Below, we will explain the form for implementing the invention with reference to the drawings.

[Outline of the excavator management system]
First, an overview of the excavator management system SYS according to this embodiment will be described with reference to Figs. 1 and 2.

Figure 1 is a schematic diagram showing an example of an excavator management system SYS according to this embodiment. Figure 2 is a top view showing an example of an excavator 100 according to this embodiment.

As shown in FIG. 1, the excavator management system SYS includes an excavator 100 and a management device 200.

<Outline of the excavator>
As shown in Figures 1 and 2, a shovel 100 (an example of a work machine) according to this embodiment includes a lower running body 1, an upper rotating body 3 mounted on the lower running body 1 so as to be freely rotatable via a rotating mechanism 2, a boom 4, an arm 5, and a bucket 6 that constitute an attachment (work machine), and a cabin 10.

The undercarriage 1 allows the excavator 100 to travel by a pair of left and right crawlers 1C that are hydraulically driven by left and right traveling hydraulic motors 1M, respectively. That is, the crawlers 1C include a left crawler 1CL and a right crawler 1CR, and the traveling hydraulic motors 1M include a left traveling hydraulic motor 1ML and a right traveling hydraulic motor 1MR.

The upper rotating body 3 rotates relative to the lower traveling body 1 by being driven by the rotating hydraulic motor 2A.

The boom 4 is attached to the center of the front of the upper rotating body 3 so that it can be raised and lowered, an arm 5 is attached to the tip of the boom 4 so that it can rotate up and down, and a bucket 6 is attached to the tip of the arm 5 so that it can rotate up and down as an end attachment. The boom 4, arm 5, and bucket 6 are hydraulically driven by a boom cylinder 7, arm cylinder 8, and bucket cylinder 9, which serve as hydraulic actuators, respectively. The bucket 6 is an example of an end attachment, and instead of the bucket 6, other end attachments such as a slope bucket, dredging bucket, or breaker may be attached to the tip of the arm 5 depending on the work content, etc.

In addition, at least some of the lower traveling body 1, upper rotating body 3, boom 4, arm 5, bucket 6, etc. may be driven by electric actuators instead of hydraulic actuators. For example, the upper rotating body 3 may be electrically driven by an electric motor instead of the swing hydraulic motor 2A.

The cabin 10 is the cab in which the operator sits and is mounted on the front left side of the upper rotating body 3.

The shovel 100 according to this embodiment is equipped with a communication device T1 and is communicatively connected to the management device 200 via a specified communication line NW. The specified communication line may include, for example, a mobile communication network terminated in a base station, a satellite communication network using a communication satellite, the Internet network, and the like. The communication line NW may also be a wireless communication line based on a short-range wireless communication standard such as Bluetooth (registered trademark) or WiFi. This allows the shovel 100 to transmit (upload) information acquired by the shovel 100 to the management device 200.

The excavator 100 operates driven elements such as the lower traveling body 1 (crawlers 1CL, 1CR), upper rotating body 3, boom 4, arm 5, and bucket 6 in response to the operation of an operator in the cabin 10.

In addition to or instead of being configured to be operable by an operator inside the cabin 10, the shovel 100 may be configured to be remotely operated from outside the shovel. When the shovel 100 is remotely operated, the inside of the cabin 10 may be unmanned. In the following description, it is assumed that the operation of the operator includes at least one of the operation of the operating device 26 by the operator inside the cabin 10 and the remote operation by an external operator.

The remote operation includes, for example, a mode in which the shovel 100 is operated by an operation input related to the actuator of the shovel 100 performed by a specified external device. The specified external device is, for example, the management device 200. In this case, the shovel 100 transmits image information (captured image) output by the imaging device 40 described later to the management device 200, and the image information may be displayed on a display device 230A described later provided in the management device 200. In addition, various information images (information screens) displayed on the display device 50 inside the cabin 10 of the shovel 100 may also be displayed on the display device 230A of the management device 200. This allows the operator of the management device 200 to remotely operate the shovel 100 while checking the display contents of the captured image and information screen showing the surroundings of the shovel 100 displayed on the display device 230A, for example. The excavator 100 may operate hydraulic actuators and drive driven elements such as the lower traveling body 1 (crawlers 1CL, 1CR), upper rotating body 3, boom 4, arm 5, and bucket 6 in response to a remote control signal indicating the content of the remote control received from the management device 200 by the communication device T1 described below.

Remote operation may also include a mode in which the shovel 100 is operated by external voice input or gesture input to the shovel 100 by a person (e.g., a worker) around the shovel 100. Specifically, the shovel 100 recognizes voices uttered by surrounding workers and gestures made by workers through a voice input device (e.g., a microphone) or a gesture input device (e.g., an imaging device) mounted on the shovel 100 (its own machine). The shovel 100 may then operate actuators in accordance with the contents of the recognized voices and gestures to drive driven elements such as the lower traveling body 1 (crawlers 1CL, 1CR), upper rotating body 3, boom 4, arm 5, and bucket 6.

The excavator 100 may also automatically operate the actuators regardless of the content of the operator's operation. This allows the excavator 100 to realize a function (so-called "automatic driving function" or "machine control function") that automatically operates at least some of the driven elements such as the lower traveling body 1 (crawlers 1CL, 1CR), upper rotating body 3, boom 4, arm 5, and bucket 6.

The automatic driving function may include a function (so-called "semi-automatic driving function") that automatically operates a driven element (hydraulic actuator) other than the driven element (hydraulic actuator) that is the object of operation, in response to the operator's operation of the operating device 26 or remote operation. The automatic driving function may also include a function (so-called "fully automatic driving function") that automatically operates at least a part of the multiple driven elements (hydraulic actuators) on the premise that the operator does not operate the operating device 26 or remote operation. When the fully automatic driving function is enabled in the shovel 100, the inside of the cabin 10 may be unmanned. The semi-automatic driving function, the fully automatic driving function, etc. may also include a mode in which the operation content of the driven element (hydraulic actuator) that is the object of automatic driving is automatically determined according to a rule that is specified in advance. The semi-automatic driving function, the fully automatic driving function, etc. may also include a mode in which the shovel 100 autonomously makes various judgments and autonomously determines the operation content of the driven element (hydraulic actuator) that is the object of automatic driving in accordance with the judgment results (so-called "autonomous driving function").

<Overview of the management device>
The management device 200 (an example of an information processing device) is communicatively connected to the shovel 100 through a predetermined communication line NW, and manages (monitors) the state, operation, and the like of the shovel 100 based on various information received from the shovel 100. In addition, the management device 200 supports remote operation of the shovel 100.

The management device 200 is, for example, a cloud server installed in a management center outside the work site of the shovel 100. The management device 200 may also be, for example, an edge server installed in a location relatively close to the shovel 100 (for example, a management office in the work site, or a wireless base station or station building relatively close to the work site). The management device 200 may also be a terminal device (for example, a desktop computer terminal) installed (fixed) in a management office or the like in the work site of the shovel 100. The management device 200 may also be a mobile terminal (for example, a smartphone, tablet terminal, laptop computer terminal, etc.) that can be carried by the manager of the shovel 100, etc.

[Excavator management system configuration]
Next, a specific configuration of the excavator management system SYS according to this embodiment will be described with reference to Figs. 3 and 4 in addition to Figs. 1 and 2.

Figures 3 and 4 are block diagrams each showing an example and another example of the configuration of the excavator management system SYS according to this embodiment. Figures 3 and 4 have the same configuration of the management device 200, and only the configuration of the excavator 100 is different.

In addition, in Figures 3 and 4, mechanical power lines, hydraulic oil lines, pilot lines, and electrical signal lines are indicated by double lines, thick solid lines, thin solid lines, dashed lines, and dotted lines, respectively.

<Excavator configuration>
3 and 4 , as described above, the hydraulic drive system of the excavator 100 according to this embodiment includes hydraulic actuators such as the traveling hydraulic motors 1ML, 1MR, the swing hydraulic motor 2A, the boom cylinder 7, the arm cylinder 8, and the bucket cylinder 9. The hydraulic drive system of the excavator 100 according to this embodiment also includes an engine 11, a regulator 13, a main pump 14, and a control valve 17.

The engine 11 is the main power source in the hydraulic drive system. The engine 11 is, for example, a diesel engine that uses light oil as fuel. The engine 11 is, for example, mounted at the rear of the upper rotating body 3, and rotates at a constant speed at a preset target speed under direct or indirect control of the controller 30, driving the main pump 14 and the pilot pump 15.

The regulator 13 adjusts the discharge volume of the main pump 14. For example, the regulator 13 adjusts the angle (tilt angle) of the swash plate of the main pump 14 in response to a control command from the controller 30.

The main pump 14 is mounted, for example, at the rear of the upper rotating body 3, similar to the engine 11, and supplies hydraulic oil to the control valve 17 through a high-pressure hydraulic line. The main pump 14 is driven by the engine 11 as described above. The main pump 14 is, for example, a variable displacement hydraulic pump, and as described above, under the control of the controller 30, the tilt angle of the swash plate is adjusted by the regulator 13 to adjust the stroke length of the piston and control the discharge flow rate.

The control valve 17 is a hydraulic control device that controls the hydraulic drive system in response to the operation of the operator. The control valve 17 is mounted, for example, in the center of the upper rotating body 3. The control valve 17 selectively supplies hydraulic oil supplied from the main pump 14 to multiple hydraulic actuators in response to the operation of the operating device 26 or the remote operation. The control valve 17 includes multiple control valves (also called directional control valves) that control the flow rate and flow direction of the hydraulic oil supplied from the main pump 14 to each of the multiple hydraulic actuators.

Each of the multiple control valves (directional control valves) is, for example, a spool valve that supplies hydraulic oil supplied from the main pump 14 to the corresponding hydraulic actuator and discharges hydraulic oil discharged by the corresponding hydraulic actuator to a hydraulic oil tank. Specifically, each control valve is configured so that the spool can move in two opposing directions from a neutral position, and the direction of flow of the hydraulic oil in the hydraulic actuator, i.e., the operating direction of the hydraulic actuator, may be determined by the direction of movement of the spool.

As shown in Figures 3 and 4, the operating system of the excavator 100 according to this embodiment includes a pilot pump 15, an operating device 26, a controller 30, and a hydraulic control valve 31. Also, as shown in Figure 3, when the operating device 26 is of the hydraulic pilot type, the operating system of the excavator 100 according to this embodiment includes a shuttle valve 32 and a hydraulic control valve 33.

The pilot pump 15 is mounted, for example, at the rear of the upper rotating body 3 and supplies pilot pressure to various hydraulic devices such as the operating device 26 via a pilot line 25. The pilot pump 15 is, for example, a fixed displacement hydraulic pump, and is driven by the engine 11 as described above.

The operating device 26 is provided near the driver's seat of the cabin 10, and is used by the operator to operate each of the multiple driven elements of the excavator 100. The multiple driven elements include, for example, the lower traveling body 1, the upper rotating body 3, the boom 4, the arm 5, and the bucket 6. In other words, the operating device 26 is used by the operator to operate the multiple hydraulic actuators that drive each of the driven elements. The multiple hydraulic actuators include, for example, the traveling hydraulic motors 1ML, 1MR, the swing hydraulic motor 2A, the boom cylinder 7, the arm cylinder 8, and the bucket cylinder 9, as described above.

The operating device 26 includes, for example, a lever device that operates each of the pair of left and right crawlers (travel hydraulic motors 1ML, 1MR) of the lower traveling body 1, the boom 4 (boom cylinder 7), the arm 5 (arm cylinder 8), the bucket 6 (bucket cylinder 9), and the upper rotating body 3 (slewing hydraulic motor 2A).

As shown in FIG. 3, the operating device 26 is, for example, a hydraulic pilot type that outputs hydraulic oil having a pilot pressure corresponding to the operation content. Specifically, the operating device 26 uses hydraulic oil supplied from the pilot pump 15 through the pilot line 25 and the pilot line 25A branched from the pilot line 25 to output a pilot pressure corresponding to the operation content to the secondary pilot line 27A. The pilot line 27A is connected to the inlet port of the shuttle valve 32 and is connected to the control valve 17 via the pilot line 27 connected to the outlet port of the shuttle valve 32. As a result, pilot pressure corresponding to the operation content related to various driven elements (i.e., hydraulic actuators) in the operating device 26 can be input to the control valve 17 via the shuttle valve 32. Therefore, the control valve 17 can drive each hydraulic actuator according to the operation content of the operating device 26 by the operator, etc.

As shown in FIG. 4, the operating device 26 is, for example, electric. Specifically, the operating device 26 outputs an electric signal (hereinafter, "operation signal") according to the operation content, and the operation signal is input to the controller 30. The controller 30 then outputs a control command according to the operation signal, that is, a control signal according to the operation content for the operating device 26, to the hydraulic control valve 31. As a result, a pilot pressure according to the operation content of the operating device 26 is input from the hydraulic control valve 31 to the control valve 17, and the control valve 17 can drive each hydraulic actuator according to the operation content of the operating device 26.

The control valves (directional control valves) built into the control valve 17 that drive the respective hydraulic actuators may be electromagnetic solenoid type spool valves. In this case, the operation signal output from the operating device 26 may be input directly to the control valve 17, i.e., the electromagnetic solenoid type control valve.

The hydraulic control valve 31 is provided for each driven element (hydraulic actuator) to be operated by the operating device 26. That is, the hydraulic control valve 31 is provided for each of the crawler 1CL (traveling hydraulic motor 1ML), crawler 1CR (traveling hydraulic motor 1MR), upper rotating body 3 (swing hydraulic motor 2A), boom 4 (boom cylinder 7), arm 5 (arm cylinder 8), and bucket 6 (bucket cylinder 9). The hydraulic control valve 31 is provided, for example, in the pilot line 25B between the pilot pump 15 and the control valve 17. The hydraulic control valve 31 may be configured to change its flow area (i.e., the cross-sectional area through which the hydraulic oil can flow), for example. As a result, the hydraulic control valve 31 can output a predetermined pilot pressure to the secondary pilot line 27B by using the hydraulic oil of the pilot pump 15 supplied through the pilot line 25B. Therefore, as shown in FIG. 3, the hydraulic control valve 31 can indirectly apply a predetermined pilot pressure corresponding to the control signal from the controller 30 to the control valve 17 through the shuttle valve 32 between the pilot line 27B and the pilot line 27. Also, as shown in FIG. 4, unlike the case of FIG. 3, the pilot line 27A and the shuttle valve 32 are omitted, and the hydraulic control valve 31 can directly apply a predetermined pilot pressure corresponding to the control signal from the controller 30 to the control valve 17 through the pilot line 27B and the pilot line 27. Therefore, the controller 30 can supply a pilot pressure corresponding to the operation of the electric operating device 26 from the hydraulic control valve 31 to the control valve 17, thereby realizing the operation of the excavator 100 based on the operation of the operator.

More specifically, for example, two hydraulic control valves 31 are provided for each of a plurality of driven elements (hydraulic actuators). The two hydraulic control valves 31 are connected to two pilot ports of a control valve (directional control valve) for moving the spool in the first direction and the second direction, respectively, via the shuttle valve 32 and the pilot line 27. As a result, the controller 30 can operate the hydraulic actuator to be operated in a desired direction by outputting a control signal to one of the two hydraulic control valves 31. Hereinafter, the description will be given on the premise that the movement of the spool in the first direction corresponds to the operation of the hydraulic actuator (driven element) in the first direction and the operation of the hydraulic actuator (driven element) in the first direction. The same applies to the movement of the spool in the second direction. In addition, the respective pilot ports to which pilot pressures for moving the spool of the control valve (directional control valve) in the first direction and the second direction are supplied may be referred to as the "first pilot port" and the "second pilot port".

The controller 30 also controls, for example, the hydraulic control valve 31 to realize remote operation of the shovel 100. Specifically, the controller 30 outputs a control signal to the hydraulic control valve 31 corresponding to the content of the remote operation specified by a remote operation signal or the like received from the management device 200. As a result, the controller 30 causes the hydraulic control valve 31 to supply a pilot pressure corresponding to the content of the remote operation to the control valve 17, thereby realizing the operation of the shovel 100 based on the remote operation of the operator.

The controller 30 may also control the hydraulic control valve 31, for example, to realize an automatic driving function. Specifically, the controller 30 outputs a control signal corresponding to an operation command related to the automatic driving function to the hydraulic control valve 31, regardless of whether the operation device 26 is operated or remotely operated. As a result, the controller 30 can supply pilot pressure corresponding to the operation command related to the automatic driving function from the hydraulic control valve 31 to the control valve 17, thereby realizing the operation of the excavator 100 based on the automatic driving function.

3, the shuttle valve 32 has two inlet ports and one outlet port, and outputs hydraulic oil having the higher pilot pressure of the two pilot pressures input to the inlet ports to the outlet port. The shuttle valve 32 is provided for each driven element (hydraulic actuator) to be operated by the operating device 26. That is, the shuttle valve 32 is provided for each of the crawler 1CL (traveling hydraulic motor 1ML), crawler 1CR (traveling hydraulic motor 1MR), upper rotating body 3 (swing hydraulic motor 2A), boom 4 (boom cylinder 7), arm 5 (arm cylinder 8), and bucket 6 (bucket cylinder 9). One of the two inlet ports of the shuttle valve 32 is connected to the pilot line 27A on the secondary side of the operating device 26 (specifically, the above-mentioned lever device included in the operating device 26), and the other is connected to the pilot line 27B on the secondary side of the hydraulic control valve 31. The outlet port of the shuttle valve 32 is connected to the pilot port of the corresponding control valve of the control valve 17 through the pilot line 27. The corresponding control valve refers to a control valve that drives a hydraulic actuator that is the object of operation of the lever device connected to one inlet port of the shuttle valve 32. Therefore, each of these shuttle valves 32 can apply the higher of the pilot pressure of the pilot line 27A on the secondary side of the operating device 26 (lever device) and the pilot pressure of the pilot line 27B on the secondary side of the hydraulic control valve 31 to the pilot port of the corresponding control valve. In other words, the controller 30 can control the corresponding control valve regardless of the operator's operation of the operating device 26 by outputting a pilot pressure higher than the pilot pressure of the pilot line 27A on the secondary side of the operating device 26 from the hydraulic control valve 31. Therefore, the controller 30 can control the operation of the driven elements (crawlers 1CL, 1CR, upper rotating body 3, boom 4, arm 5, and bucket 6) regardless of the operating state of the operating device 26 by the operator, thereby realizing the automatic operation function and remote operation function of the excavator 100.

More specifically, for example, two shuttle valves 32 are provided for each of a plurality of driven elements (hydraulic actuators). One inlet port of each of the two shuttle valves 32 is connected to each of the two pilot lines 27A corresponding to the operation of the lever device in the first and second directions, which are opposite to each other. The other inlet ports of each of the two shuttle valves 32 are connected to the secondary pilot lines 27B of the two hydraulic control valves 31, as described above. The outlet ports of each of the two shuttle valves 32 are connected to the first pilot port and the second pilot port of the control valve, respectively, through the pilot line 27. As a result, pilot pressure is supplied from one of the two shuttle valves 32 to either the first pilot port or the second pilot port of the control valve, and the hydraulic actuator to be operated can be operated in the desired direction (first direction or second direction).

3, the hydraulic control valve 33 is provided in the pilot line 27A connecting the operating device 26 (lever device) and the shuttle valve 32. The hydraulic control valve 33 is configured to be able to change its flow area, for example. The hydraulic control valve 33 operates in response to a control signal input from the controller 30. As a result, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26 when the operating device 26 is operated by the operator. Therefore, even when the operating device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator corresponding to the operation of the operating device 26. In addition, the controller 30 can reduce the pilot pressure output from the operating device 26 to be lower than the pilot pressure output from the hydraulic control valve 31, for example, even when the operating device 26 is being operated. Therefore, the controller 30 can reliably apply a desired pilot pressure to the pilot port of the control valve in the control valve 17, for example, regardless of the operation content of the operating device 26, by controlling the hydraulic control valve 31 and the hydraulic control valve 33. Therefore, for example, the controller 30 can more appropriately realize the automatic operation function, remote control function, etc. of the excavator 100 by controlling the hydraulic control valve 33 in addition to the hydraulic control valve 31.

More specifically, for example, two hydraulic control valves 33 are provided for each of a plurality of driven elements (hydraulic actuators). The two hydraulic control valves 33 are provided in each of the two pilot lines 27A corresponding to the operation of the lever device in the opposing first and second directions. This allows the two hydraulic control valves 33 to reduce the pilot pressure of the corresponding pilot line 27A regardless of whether the lever device is operated in the opposing first or second direction.

The hydraulic control valve 33 may be omitted. The controller 30 may control the hydraulic control valve 31 corresponding to the second direction operation of the hydraulic actuator in order to suppress or stop the first direction operation of the driven element (hydraulic actuator) based on the operation of the operator, regardless of the presence or absence of the hydraulic control valve 33. As a result, pilot pressure is supplied from the hydraulic control valve 31 to the second pilot port of the control valve corresponding to the hydraulic actuator via the shuttle valve 32. Therefore, pilot pressure can be applied to the second pilot port of the control valve in a manner that counteracts the pilot pressure acting on the first pilot port of the control valve in response to the operation of the operator. Thus, the spool of the control valve can be brought closer to a neutral state, and the operation of the hydraulic actuator can be suppressed. Similarly, the hydraulic control valve 31 corresponding to the first direction operation of the hydraulic actuator may be controlled in order to suppress or stop the second direction operation of the driven element (hydraulic actuator) based on the operation of the operator, regardless of the presence or absence of the hydraulic control valve 33. For example, the hydraulic control valve 33 of FIG. 3 may be provided in the pilot line 27B of FIG. 4. This allows the controller 30 to forcibly reduce the pilot pressure output from the hydraulic control valve 31 when the operating device 26 is being operated by the operator. Therefore, even when the pilot pressure corresponding to the operation of the operating device 26 is being output from the hydraulic control valve 31, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator corresponding to the operation of the operating device 26.

As shown in Figures 3 and 4, the control system of the excavator 100 according to this embodiment includes a controller 30, an imaging device 40, a surrounding object information acquisition device 45, a display device 50, a sound output device 52, an input device 54, and a communication device T1. Also, as shown in Figure 3, the control system of the excavator 100 according to this embodiment includes an operating pressure sensor 29 when the operating device 26 is of a hydraulic pilot type.

The controller 30 is provided, for example, in the cabin 10 and performs various controls related to the excavator 100. The functions of the controller 30 may be realized by any hardware or any combination of hardware and software. For example, the controller 30 is mainly configured with a computer including a CPU (Central Processing Unit), a memory device such as a RAM (Random Access Memory), a non-volatile auxiliary storage device such as a ROM (Read Only Memory), and an interface device related to input and output with the outside. The controller 30 may also include a high-speed calculation circuit such as a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array) that works in conjunction with the CPU. The controller 30 includes, for example, a remote operation control unit 301, an object detection unit 302, a safety control unit 303, an evaluation value calculation unit 304, a detection accuracy determination unit 305, and a low detection accuracy notification unit 306 as functional units realized by executing various programs installed in the auxiliary storage device on the CPU.

The imaging device 40 is attached to the top of the upper rotating body 3, and captures images of the surroundings of the shovel 100 from areas relatively close to the shovel 100 to areas relatively far from the shovel 100 to obtain captured images. The imaging device 40 includes cameras 40F, 40B, 40L, and 40R. Hereinafter, cameras 40F, 40B, 40L, and 40R may be collectively or individually referred to as "camera 40X."

Camera 40F, camera 40B, camera 40L, and camera 40R are attached to the upper front end, upper rear end, upper left end, and upper right end of the upper rotating body 3, respectively, and capture the front, rear, left side, and right side of the upper rotating body 3. For example, camera 40X is a monocular camera (i.e., a wide-angle camera) with a very wide angle of view. Also, for example, camera 40X may be a stereo camera, a distance image camera, a depth camera, etc. Camera 40F captures an imaging range in front of the upper rotating body 3, for example, an imaging range in the horizontal direction (i.e., the circumferential direction as seen from the shovel 100) extending from the left front to the right front. Also, camera 40B captures an imaging range in the rear of the upper rotating body 3, for example, an imaging range in the horizontal direction (i.e., the circumferential direction as seen from the shovel 100) extending from the left rear to the right rear. Camera 40L captures, for example, an imaging range on the left side of upper rotating body 3, for example, an imaging range in the horizontal direction (circumferential direction as seen from shovel 100) extending from the left front to the left rear of upper rotating body 3. Camera 40R captures, for example, an imaging range on the right side of upper rotating body 3, for example, an imaging range in the horizontal direction (circumferential direction as seen from shovel 100) extending from the right front to the right rear of upper rotating body 3. Camera 40X is attached to the top of upper rotating body 3 with its optical axis facing diagonally downward, and captures an imaging range in the vertical direction including the ground near shovel 100 to the far side of shovel 100.

The camera 40X outputs captured images at a predetermined period (e.g., 1/30th of a second) between the start of the shovel 100 (i.e., key switch ON) and the stop of the shovel 100 (i.e., key switch OFF). The captured images output from the camera 40X are captured by the controller 30. The captured images output from the camera 40X may also be transmitted (uploaded) from the controller 30 to the management device 200 via the communication device T1.

The surrounding object information acquisition device 45 is attached to the top of the upper rotating body 3 and acquires information about objects in the vicinity of the excavator 100. The surrounding object information acquisition device 45 includes sensors 45BL, 45BR, 45L, and 45R. Hereinafter, the sensors 45BL, 45BR, 45L, and 45R may be collectively or individually referred to as "sensor 45X."

Sensor 45BL, sensor 45BR, sensor 45L, and sensor 45R are attached to the upper left rear end, upper right rear end, upper left end, and upper right end of the upper rotating body 3, respectively, and acquire information on the conditions on the left rear, right rear, left side, and right side of the upper rotating body 3. For example, sensor 45X is a LIDAR (Light Detection and Ranging). Also, for example, sensor 45X may be a millimeter wave radar, ultrasonic sensor, etc. The following explanation will focus on the case where sensor 45X is a LIDAR.

The sensor 45X, for example, emits infrared rays in a certain direction and receives reflected light from objects in that direction to obtain information about objects around the shovel 100, specifically, information about the received reflected light (hereinafter, "received light information"). The sensor 45X is, for example, a scanning LIDAR, and is a three-dimensional laser scanner that can scan the irradiation direction of the infrared laser in the up-down and left-right directions. The sensor 45X may also be a so-called flash-type LIDAR that emits infrared rays from a light-emitting module over a wide three-dimensional area and captures the reflected light (infrared rays) with a three-dimensional distance image element.

The light reception information includes information on the time from infrared irradiation to the reception of reflected light (TOF: Time Of Flight) for each direction of infrared irradiation (hereinafter, "TOF information"), and information on the intensity of reflected light received for each direction of infrared irradiation (hereinafter, "received light intensity information").

The sensor 45BL is configured to irradiate infrared rays to an irradiation range on the left rear side of the upper rotating body 3, for example, an irradiation range in the horizontal direction (i.e., the circumferential direction as seen from the shovel 100) extending from the left rear to the rear of the upper rotating body 3. The sensor 45BR is configured to irradiate infrared rays to an irradiation range on the right rear side of the upper rotating body 3, for example, an irradiation range in the horizontal direction (circumferential direction as seen from the shovel 100) extending from the right rear to the rear of the upper rotating body 3. The sensor 45L is configured to irradiate infrared rays to an irradiation range on the left side of the upper rotating body 3, for example, an irradiation range in the horizontal direction (circumferential direction as seen from the shovel 100) extending from the left front to the left rear of the upper rotating body 3. The sensor 45R is configured to irradiate infrared rays to an irradiation range on the right side of the upper rotating body 3, for example, an irradiation range from the right front to the right rear of the upper rotating body 3. The sensor 45X is mounted on the upper part of the upper rotating body 3 so that its optical axis (i.e., the reference axis for the direction of infrared radiation) faces diagonally downward, and has an infrared radiation range in the vertical direction centered on a part of the ground relatively close to the shovel 100.

Each of the sensors 45X outputs light reception information at a predetermined cycle from when the shovel 100 is started to when it is stopped. The light reception information output from the sensor 45X is input to the controller 30.

The display device 50 is provided in a location that is easily visible to the operator seated in the cabin 10, and displays various information images under the control of the controller 30. The display device 50 is, for example, a liquid crystal display or an organic EL (electroluminescence) display. This allows the display device 50 to notify the operator of visual information.

The display device 50 displays an image (hereinafter, "surrounding image") showing the surroundings of the excavator 100 based on, for example, an image captured by the imaging device 40. The surrounding image may be the image information of the surroundings of the excavator 100 captured by the imaging device 40, or may be a processed image generated by applying known image processing (e.g., viewpoint conversion processing) to the image information. The surrounding image may include, for example, at least one of the left side, right side, and rear side of the upper rotating body 3. This allows the operator of the cabin 10 to check the surrounding image on the display device 50 to confirm the safety of the left side, right side, and rear side of the upper rotating body 3, which are areas that are likely to become blind spots during normal operation of the excavator 100 (hereinafter, "blind spot areas").

The sound output device 52 is provided inside the cabin 10 and outputs sound under the control of the controller 30. The sound output device 52 is, for example, a buzzer or a speaker. This allows the sound output device 52 to notify the operator of auditory information based on the content of the sound it outputs (tone, sound pressure, sound pattern, voice content, etc.).

The input device 54 is provided within reach of the operator seated in the cabin 10, accepts various inputs from the operator, and outputs a signal corresponding to the input to the controller 30. For example, the input device 54 includes an operation input device that accepts operation input from the operator. The operation input device may include, for example, a touch panel mounted on the display of the display device 50. The operation input device may also include, for example, a touch pad, a button switch, a lever, a toggle, etc., that are installed around the display device 50. The operation input device may also include, for example, a knob switch that is provided at the tip of the operation device 26 (lever device). For example, the input device 54 may also include a voice input device or a gesture input device that accepts voice input or gesture input from the operator. The voice input device includes, for example, a microphone. The gesture input device includes, for example, an imaging device that images the operator in the cabin 10. A signal corresponding to the input content to the input device 54 is taken into the controller 30.

The communication device T1 communicates with an external device (e.g., the management device 200) through the communication line NW. The communication device T1 is, for example, a mobile communication module that connects to a mobile communication network that ends at a base station. The communication device T1 may also be, for example, a satellite communication module that connects to a satellite communication network that uses a communication satellite. The communication device T1 may also be a WiFi communication module or a Bluetooth communication module that performs short-range communication.

The operating pressure sensor 29 detects the secondary pilot pressure (operating pressure) corresponding to the operation of the operating device 26. The output of the operating pressure sensor 29 is input to the controller 30. This allows the controller 30 to obtain the operation of the operating device 26.

The remote operation control unit 301 performs control related to the remote operation of the shovel 100. Specifically, as described above, the remote operation control unit 301 controls the hydraulic control valve 31 or the hydraulic control valves 31, 33 in accordance with the content of the remote operation specified by the remote operation signal received from the management device 200 by the communication device T1, thereby realizing the remote operation of the shovel 100.

The object detection unit 302 detects monitored objects (hereinafter simply "monitored objects") within the monitoring area surrounding the shovel 100 based on the output of the imaging device 40 and the surrounding object information acquisition device 45. Monitored objects include people such as workers working around the shovel 100 and work site supervisors. Monitored objects can also include any obstacles other than people, such as materials temporarily stored at the work site, stationary obstacles such as temporary offices at the work site, and moving obstacles such as vehicles including trucks.

The object detection unit 302 detects a monitoring target in a specified monitoring area (hereinafter, for convenience, "first monitoring area") around the excavator 100 (upper rotating body 3) based on, for example, the output of the imaging device 40, i.e., the image captured by the imaging device 40.

The object detection unit 302 may detect a monitoring target in a first monitoring area extending in a horizontal direction as seen from the shovel 100 (hereinafter simply referred to as the "horizontal direction"), that is, in a direction along the plane (hereinafter referred to as the "work plane") on which the shovel 100 is working (the lower running body 1 is on the ground). Specifically, the object detection unit 302 may detect a monitoring target in a first monitoring area in which the horizontal distance D from the shovel 100 (upper rotating body 3) is within a predetermined distance Dth1 (e.g., 5 meters).

For example, the object detection unit 302 recognizes the monitoring target in the captured image by arbitrarily applying various known image processing methods and machine learning-based classifiers including artificial intelligence (AI). Specifically, the object detection unit 302 cuts out a partial image of a size equivalent to the size of the monitoring target including the target pixel for each pixel included in the captured image, and acquires local image features of the partial image. Examples of local image features include histograms of oriented gradients (HOG) and scaled invariance feature transform (SIFT). The object detection unit 302 may then use a classifier trained with multiple teacher data consisting of a combination of an image including the monitoring target and the local image features of the image to determine whether the monitoring target is included in the partial image based on the acquired local image features.

In addition, the object detection unit 302 determines (estimates) the position (e.g., the foot position) (hereinafter, "actual position") of the recognized surveillance target (person) captured in the image captured by the monocular imaging device 40 by applying various known techniques.

For example, the object detection unit 302 estimates the horizontal distance (hereinafter, "horizontal distance") from the shovel 100 based on the size of the recognized monitoring target in the captured image (e.g., the size in the height direction in the captured image). This is because there is a correlation in which the size of the recognized monitoring target in the captured image becomes smaller as the monitoring target moves away from the shovel 100. Specifically, since there is a range of expected sizes for the monitoring target (e.g., the range of expected human heights), the correlation between the horizontal distance of the monitoring target included in the expected size range as seen from the shovel 100 and its size in the captured image can be specified in advance. Therefore, the object detection unit 302 can estimate the horizontal distance of the monitoring target from the shovel 100 based on, for example, a map or conversion formula that indicates the correlation between the size of the monitoring target in the captured image and the horizontal distance as seen from the shovel 100, which is stored in advance in an internal memory such as an auxiliary storage device of the controller 30. Furthermore, the object detection unit 302 can estimate the direction in which the monitored object is located as seen from the excavator 100 (camera 40X) based on the lateral (left-right) position on the captured image.

Also, for example, the object detection unit 302 can estimate the actual position (e.g., foot position) of the monitored object by projective transformation (homography) of the captured image onto the plane, assuming that the monitored object exists on the same plane as the shovel 100 (specifically, the lower traveling body 1). In this case, a certain part (a certain point) constituting the captured image is associated with a certain position on the same plane as the shovel 100.

The object detection unit 302 also detects a monitoring target in a specified monitoring area (hereinafter, for convenience, "second monitoring area") around the excavator 100 (upper rotating body 3) based on, for example, the output (i.e., light reception information) of the surrounding object information acquisition device 45.

The object detection unit 302 may detect a monitoring target in a second monitoring area extending horizontally, that is, in a direction along the work plane. Specifically, the object detection unit 302 may detect a monitoring target in a second monitoring area in which the horizontal distance D from the shovel 100 (upper rotating body 3) is within a predetermined distance Dth2. The predetermined distances Dth1 and Dth2 may be the same or different. That is, the first monitoring area and the second monitoring area may be the same or different. For example, the first monitoring area may include a range relatively far from the shovel 100, and the second monitoring area may be limited to a range closer to the shovel 100 than the first monitoring area.

For example, the object detection unit 302 recognizes the presence and position of surrounding objects based on TOF information from the light reception information acquired from the surrounding object information acquisition device 45. The object detection unit 302 may also recognize the type of surrounding object by recognizing the shape and size of the object based on the light reception information (TOF information) corresponding to the reflected light received from multiple irradiation directions, and determine whether the object corresponds to a monitoring target. The object detection unit 302 may also recognize the type of object by recognizing the retroreflectivity and reflectance of the surrounding object based on the light reception intensity information from the light reception information, and determine whether the object corresponds to a monitoring target.

Hereinafter, devices that acquire information about objects around the excavator 100, including the imaging device 40 (camera 40X) and the surrounding object information acquisition device 45, and are used by the object detection unit 302 to detect monitoring targets, may be referred to as "information sources."

In addition, for example, when the object detection unit 302 detects a monitoring target within the monitoring area while the shovel 100 is being remotely operated, the object detection unit 302 transmits a signal indicating that the monitoring target has been detected within the monitoring area (hereinafter, a "monitoring target detection signal") to the management device 200 via the communication device T1. This enables the management device 200 to recognize that a monitoring target is present within the monitoring area around the shovel 100.

The function of the object detection unit 302 may be switched between ON (enabled) and OFF (disabled) in response to a predetermined operation by a user, such as an operator, on the input device 54. In this case, the function of detecting a monitoring target based on the output of the imaging device 40 and the function of detecting a monitoring target based on the output of the surrounding object information acquisition device 45 may each be switchable between ON (enabled) and OFF (disabled).

The object detection unit 302 may detect a monitoring target in the monitoring area around the shovel 100 based on only the output of either the imaging device 40 or the surrounding object information acquisition device 45. Also, when the object detection unit 302 detects a monitoring target in the monitoring area around the shovel 100 based only on the output of the imaging device 40, the surrounding object information acquisition device 45 may be omitted.

The safety control unit 303 controls the safety functions of the shovel 100. Specifically, the safety control unit 303 controls the safety functions to avoid contact between the shovel 100 and objects around it.

The safety control unit 303 activates a predetermined safety function (hereinafter, for convenience, the "first safety function"), for example, when the object detection unit 302 detects a monitoring target within the monitoring area.

The first safety function includes, for example, a function to notify an operator, etc. in the cabin 10 or workers, etc. around the excavator 100 that a monitored object has been detected, such as by outputting an alarm at least one of the inside and outside of the cabin 10 (hereinafter, the "alert function").

The safety control unit 303 activates the alarm function, for example, when the object detection unit 302 detects a monitoring target within a predetermined range (hereinafter, "alarm range") included in the monitoring area. The alarm range may be the same as the monitoring area, or may be set so that its outer edge is relatively closer to the shovel 100 than the monitoring area. This allows the operator inside the cabin 10 and workers around the shovel 100 to be aware that a monitoring target is present within a predetermined range around the shovel 100.

The safety control unit 303, for example, controls the sound output device 52 to activate a sound (i.e., auditory) alarm function for at least one of the inside and outside of the cabin 10. This allows the sound output device 52 to realize an auditory alarm function under the control of the controller 30. At this time, the safety control unit 303 may vary the pitch, sound pressure, tone color, blowing period when the sound is blown periodically, content of the audio information, etc. of the output sound according to various conditions.

The safety control unit 303 may also control the display device 50 to activate a notification function by displaying image information inside the cabin 10. Specifically, the safety control unit 303 may display an image indicating that a monitoring target has been detected on the surrounding image displayed on the display device 50. The safety control unit 303 may also highlight the monitoring target shown in the surrounding image displayed on the display device 50 or a position on the surrounding image corresponding to the position of the detected monitoring target as seen from the excavator 100. More specifically, the safety control unit 303 may superimpose and display a frame surrounding the monitoring target shown on the surrounding image, or superimpose and display a marker on a position on the surrounding image corresponding to the actual position of the detected monitoring target. In this way, the display device 50 can realize a visual notification function for the operator under the control of the controller 30.

The safety control unit 303 may also activate a visual notification function for workers and supervisors around the shovel 100, for example, by controlling a headlight or an external display device provided in the house of the upper rotating body 3. The safety control unit 303 may also activate a tactile notification function for the operator in the cabin 10, for example, by controlling a vibration generating device that vibrates the cockpit in which the operator sits. This allows the controller 30 to make the operator, workers and supervisors around the shovel 100 aware that there is a monitoring target (for example, a person such as a worker) around the shovel 100. Therefore, the controller 30 can prompt the operator to check the safety around the shovel 100, and can prompt workers in the monitoring area to evacuate from the monitoring area.

In addition, the safety control unit 303 may vary the notification mode (i.e., the method of notification) depending on the positional relationship between the monitored object detected within the notification range and the excavator 100.

For example, when a monitoring target detected within the notification range by the object detection unit 302 is located relatively far from the shovel 100, the safety control unit 303 may output an alarm with a relatively low level of alert (hereinafter, a "low alert level alarm") that alerts the operator or the like to pay attention to the monitoring target. On the other hand, when a monitoring target detected within the notification range by the object detection unit 302 is located relatively close to the shovel 100, the safety control unit 303 may output an alarm with a relatively high level of alert (hereinafter, a "high alert level alarm") that notifies the operator that the monitoring target is approaching the shovel 100 and that the danger is increasing.

In this case, the safety control unit 303 may change the pitch, sound pressure, tone, blowing period, content of the audio information, etc. of the sound output from the sound output device 52 between the warning level alarm and the alert level alarm. In addition, the safety control unit 303 may change the color, shape, size, presence or absence of blinking, blinking period, etc. of the image indicating that the monitored object has been detected and the image (e.g., frame, marker, etc.) that highlights the monitored object or the position of the monitored object displayed on the surrounding image displayed on the display device 50 between the low alert level alarm and the high alert level alarm. In this way, the controller 30 can allow the operator, etc. to grasp the urgency, in other words, the proximity of the monitored object to the excavator 100, based on the difference in the alarm sound (alarm sound) output from the sound output device 52 and the alarm image displayed on the display device 50.

The first safety function may include, for example, a function (hereinafter, "operation restriction function") that restricts or prohibits the operation of the shovel 100 in response to the operation of the driven element (hydraulic actuator) (for example, operation of the operating device 26 or remote operation). The operation restriction function includes at least one of an operation deceleration function that slows down the operation speed of the shovel 100 in response to the operation of the hydraulic actuator below normal, and an operation stop function that stops the operation of the shovel 100 and maintains the stopped state regardless of the operation of the hydraulic actuator.

The safety control unit 303 activates the operation restriction function, for example, when the object detection unit 302 detects a monitoring target within a predetermined range (hereinafter, "operation restriction range") included in the monitoring area. The operation restriction range may be the same as the monitoring area, or may be set so that its outer edge is relatively closer to the shovel 100 than the monitoring area. The operation restriction range also includes at least one of an operation deceleration range that slows the operation speed of the shovel 100 in response to the operation of the hydraulic actuator below normal, and an operation stop range that stops the operation of the shovel 100 and maintains the stopped state regardless of the operation of the actuator. This allows the controller 30 to slow down or stop the operation of the shovel 100 when a monitoring target is present around the shovel 100. Therefore, the controller 30 can suppress contact between the shovel 100 and the monitoring target around the shovel 100. For example, when the motion limit range includes both the motion deceleration range and the motion stop range, the motion stop range is, for example, a range of the motion limit range that is close to the shovel 100, and the motion deceleration range is a range of the motion limit range that is set outside the motion stop range.

For example, when the operating device 26 is of a hydraulic pilot type, the safety control unit 303 may activate the operation restriction function by controlling a predetermined hydraulic control valve provided in the pilot line 25. Specifically, the safety control unit 303 may activate the operation deceleration function by reducing the pilot pressure in the pilot line 25 using the hydraulic control valve. The safety control unit 303 may also activate the operation stop function by putting the pilot line 25 into a non-communicating state using the hydraulic control valve.

In addition, for example, when the operating device 26 is electric, the safety control unit 303 may activate the operation restriction function by adjusting the control signal output from the controller 30 to the hydraulic control valve 31 depending on the operation of the operating device 26 (lever device) or the content of remote operation. Specifically, the safety control unit 303 may activate the operation deceleration function by outputting a control signal to the hydraulic control valve 31 that makes the operating speed of the hydraulic actuator relatively slower than the content of the operation. In addition, the safety control unit 303 may activate the operation stop function by not outputting the control signal itself to the hydraulic control valve 31.

The safety control unit 303 may also activate the operation restriction function by, for example, controlling the hydraulic control valve 33. Specifically, the safety control unit 303 may activate the operation deceleration function by reducing the pilot pressure in the pilot line 27B on the secondary side of the operating device 26 (lever device) using the hydraulic control valve 33. The safety control unit 303 may also activate the operation stop function by putting the pilot line 27B in a disconnected state using the hydraulic control valve 33.

The safety control unit 303 may also activate the operation limiting function by controlling, for example, one of the two hydraulic control valves 31 provided for each driven element (hydraulic actuator), which corresponds to the operation of the hydraulic actuator in the opposite direction to the operation direction of the hydraulic actuator. Specifically, the safety control unit 303 may activate the operation limiting function by applying pilot pressure from the hydraulic control valve 31 to the second pilot port of the control valve in opposition to the pilot pressure acting on the first pilot port of the control valve in response to the operator's operation. Similarly, the safety control unit 303 may activate the operation limiting function by applying pilot pressure from the hydraulic control valve 31 to the first pilot port of the control valve in opposition to the pilot pressure acting on the second pilot port of the control valve in response to the operator's operation.

The safety control unit 303 may also activate an operation restriction function (operation stop function), for example, by stopping the engine 11.

In addition, the safety control unit 303 may change the control mode for the first safety function depending on whether the monitoring target is detected based on the output (captured image) of the imaging device 40 or the output of the surrounding object information acquisition device 45.

For example, when the object detection unit 302 detects a monitoring target from the output of the imaging device 40, the safety control unit 303 activates only the notification function and the safety confirmation function of the first safety function. In other words, when the object detection unit 302 detects a monitoring target from the output of the imaging device 40, the safety control unit 303 does not activate the operation restriction function regardless of the distance between the monitoring target and the shovel 100, etc. Specifically, when the object detection unit 302 detects a monitoring target within a notification range (hereinafter, for convenience, "first notification range") from the image captured by the imaging device 40, the safety control unit 303 activates the notification function and the safety confirmation function, and does not set a motion restriction range (hereinafter, for convenience, "first motion restriction range") corresponding to the detection of the monitoring target from the image captured by the imaging device 40.

On the other hand, the safety control unit 303 activates the safety confirmation function and the operation restriction function when the object detection unit 302 detects a monitoring target within the operation restriction range (hereinafter, for convenience, the "second operation restriction range") from the output of the surrounding object information acquisition device 45. The safety control unit 303 may also activate the notification function when the object detection unit 302 detects a monitoring target within the notification range (hereinafter, for convenience, the "second notification range") from the output of the surrounding object information acquisition device 45. The first notification range and the second notification range may be the same or different.

For example, if the detection accuracy of the monitored object is relatively low, but the operation restriction function is activated based on the detection result, there is a higher possibility that the operator will feel uncomfortable or the work efficiency of the excavator 100 will decrease than if the alarm function is activated.

In contrast, in this example, the controller 30 allows the operation restriction function to be activated only when the monitored object is detected from the output of the peripheral object information acquisition device 45 out of the imaging device 40 and the peripheral object information acquisition device 45. This is because there is a tendency for detection accuracy to be relatively higher when the monitored object is detected from the light reception information of the peripheral object information acquisition device 45 than when the monitored object is detected by image recognition of the image captured by the imaging device 40. In this way, the controller 30 can improve the safety of the shovel 100 while suppressing discomfort felt by the operator and a decrease in the work efficiency of the shovel 100.

Furthermore, the safety control unit 303 may, for example, after activating the first safety function, deactivate the first safety function and stop the first safety function when the object to be monitored is no longer detected by the object detection unit 302. Furthermore, the safety control unit 303 may, for example, after activating the first safety function, deactivate the first safety function and stop the first safety function when a predetermined input requesting deactivation of the first safety function is received from the operator via the input device 54.

The evaluation value calculation unit 304 calculates an evaluation value (hereinafter, simply "evaluation value") indicating the degree to which information on objects around the shovel 100 acquired by the information source represents the monitored object when no alarm is output, i.e., when the monitored object is not detected. The information source includes, for example, the imaging device 40 (camera 40X) and the sensor 45X, as described above. Specifically, the evaluation value calculation unit 304 may calculate an evaluation value of the information of the information source corresponding to each of a plurality of positions (hereinafter, "evaluation positions") included in the information acquisition range (hereinafter, "information acquisition range") of the information source around the shovel 100. The information acquisition range is determined based on the angular range in the vertical and horizontal directions (for example, the angle of view of the camera 40X and the angular range of the irradiation direction of the infrared laser of the sensor 45X) with the information source mounted on the shovel 100 (upper rotating body 3) as a reference. For example, the information acquisition range by the information source corresponds to the area around the shovel 100 included in this angular range as seen from the information source.

For example, the evaluation value calculation unit 304 calculates an evaluation value indicating the degree to which the evaluation image, which is set to include the partial image of the target, represents the monitoring target (e.g., a person) for each of multiple partial images (e.g., pixels) included in the captured image of the camera 40X corresponding to the multiple evaluation positions. In this case, the evaluation image is set to match the size of the monitoring target when it is assumed that the monitoring target exists on the partial image of the captured image, for example, and the evaluation value calculation unit 304 may calculate the evaluation value based on the local image feature amount of the evaluation image. In addition, the evaluation value calculation unit 304 calculates an evaluation value indicating the degree to which the light reception information obtained from the irradiation direction represents the monitoring target for each of multiple irradiation directions of the infrared laser by the sensor 45X corresponding to the multiple evaluation positions. In this case, the evaluation value calculation unit 304 may calculate the evaluation value based on, for example, received light intensity information included in the received light information.

The detection accuracy determination unit 305 determines whether or not there is a region (hereinafter, "low detection accuracy region") around the shovel 100 where the detection accuracy of the monitored object by the object detection unit 302 is relatively low. The low detection accuracy region includes, for example, a region where the characteristics of an object corresponding to the background (hereinafter, "background object") included in the captured image of the camera 40X or the received light information of the sensor 45X are similar to the characteristics of the monitored object, and the background object may be erroneously detected (misrecognized) as the monitored object. The low detection accuracy region also includes a region where the characteristics of the background object included in the captured image of the camera 40X or the received light information of the sensor 45X are similar to the characteristics of the monitored object, and the monitored object may be assimilated into the background object, and the monitored object may not be detected (recognized). The detection accuracy determination unit 305 may determine, for example, whether or not there is a low detection accuracy region around the shovel 100 where the detection accuracy of the monitored object based on the image information of the imaging device 40 (camera 40X) by the object detection unit 302 is relatively low. Furthermore, the detection accuracy determination unit 305 determines, for example, whether or not there is a low detection accuracy area around the shovel 100 where the detection accuracy of the monitored object based on the light reception information of the sensor 45X by the object detection unit 302 is relatively low. Furthermore, if there are multiple types of monitored objects, the detection accuracy determination unit 305 may determine, for example, whether or not there is a low detection accuracy area around the shovel 100 for each of the multiple types of monitored objects.

The detection accuracy determination unit 305 performs the above process (hereinafter, "detection accuracy determination process") when the shovel 100 is started (e.g., when the key switch is turned ON). The detection accuracy determination unit 305 also performs the above process when a change occurs or there is a possibility that a change occurs in the environment around the shovel 100 as viewed by the imaging device 40 (camera 40X) or the surrounding object information acquisition device 45 (sensor 45X), etc.

The detection accuracy determination process of the detection accuracy determination unit 305 will be described in detail later (see Figures 5 to 9).

When the detection accuracy determination unit 305 determines that a low detection accuracy area exists around the shovel 100, the low detection accuracy notification unit 306 outputs a notification regarding the low detection accuracy area around the shovel 100 to the operator. For example, when the detection accuracy determination unit 305 determines that a low detection accuracy area exists around the shovel 100, the low detection accuracy notification unit 306 notifies the operator that a low detection accuracy area exists around the shovel 100. Furthermore, when the detection accuracy determination unit 305 determines that a low detection accuracy area exists around the shovel 100, the low detection accuracy notification unit 306 may notify (instruct) the operator of the low detection accuracy area around the shovel 100.

Specifically, the low detection accuracy notification unit 306 may control the display device 50 to display information indicating that a low detection accuracy area exists around the shovel 100, information identifying the low detection accuracy area around the shovel 100, etc. on the display device 50. The low detection accuracy notification unit 306 may also control the sound output device 52 to output a sound indicating that a low detection accuracy area exists around the shovel 100, audio information identifying the low detection accuracy area around the shovel 100, etc. from the sound output device 52.

In addition, when the shovel 100 is being remotely operated, the low detection accuracy notification unit 306 may control the communication device T1 to transmit a signal including information indicating that a low detection accuracy area exists around the shovel 100, information identifying the low detection accuracy area around the shovel 100, etc. (hereinafter, "low detection accuracy notification signal") to the management device 200. This allows the controller 30 to notify the operator remotely operating the shovel 100 that a low detection accuracy area exists around the shovel 100 through the management device 200.

Details on how to notify the operator of the existence of low detection accuracy areas and the low detection accuracy areas themselves will be described later (see Figure 10).

In addition, when the detection accuracy determination unit 305 determines that a low detection accuracy area is present around the shovel 100, the control mode of the safety control unit 303 may be changed compared to when the detection accuracy determination unit 305 determines that a low detection accuracy area is not present around the shovel 100.

For example, when the object detection unit 302 detects a monitored object in a state where the detection accuracy determination unit 305 has not determined that a low detection accuracy area is present around the shovel 100, the safety control unit 303 may output an alarm indicating a relatively low level of alert (alarm of a low alert level). On the other hand, when the object detection unit 302 detects a monitored object in a state where the detection accuracy determination unit 305 has determined that a low detection accuracy area is present around the shovel 100, the safety control unit 303 may output an alarm indicating a relatively high level of alert (alarm of a high alert level).

As a result, the shovel 100 can encourage the operator to be more vigilant around the shovel 100 in situations where there are areas of low detection accuracy around the shovel 100 and where erroneous detection or oversight of monitored objects is likely to occur.

The safety control unit 303 may also output a relatively high alert level alarm only when the detection accuracy determination unit 305 has determined that a low detection accuracy area is present around the excavator 100 and the object detection unit 302 detects a monitored object in the low detection accuracy area.

This improves the safety of the excavator 100 while reducing the burden on the operator caused by the output of a relatively high level of alert.

The safety control unit 303 may also prohibit the operation of the alarm function and not output an alarm when the object detection unit 302 detects a monitoring target in a low detection accuracy area while the detection accuracy determination unit 305 has determined that the low detection accuracy area is located around the excavator 100. As described above, the low detection accuracy notification unit 306 notifies the operator of the specific location of the low detection accuracy area, making it possible to leave monitoring of the low detection accuracy area to the operator. In this case, it is desirable to notify the operator in advance that the alarm function will not be activated even if the object detection unit 302 detects a monitoring target in the low detection accuracy area.

<Configuration of Management Device>
The management device 200 includes a control device 210 , a communication device 220 , an output device 230 , and an input device 240 .

The control device 210 performs various controls related to the management device 200. The functions of the control device 210 may be realized by any hardware, or any combination of hardware and software. For example, the control device 210 is mainly configured with a computer including a CPU, a memory device such as RAM, a non-volatile auxiliary storage device such as ROM, and an interface device for input and output with the outside. The control device 210 may also include a high-speed calculation circuit such as a GPU, ASIC, or FPGA that works in conjunction with the CPU. The control device 210 includes, for example, a remote operation support unit 2101, a safety control unit 2102, and a low detection accuracy notification unit 2103 as functional units realized by executing a program installed in the auxiliary storage device on the CPU.

The communication device 220 communicates with an external device (e.g., the excavator 100) through the communication line NW. The communication device 220 is, for example, a modem or an ONU (Optical Network Unit). The communication device 220 may also be, for example, a mobile communication module that connects to a mobile communication network that ends at a base station. The communication device T1 may also be, for example, a satellite communication module that connects to a satellite communication network that uses a communication satellite. The communication device T1 may also be, for example, a WiFi communication module or a Bluetooth communication module that performs short-range communication.

Under the control of the control device 210, the output device 230 outputs various information to the administrator of the management device 200, workers, operators performing remote operations, etc.

The output device 230 includes, for example, a display device that visually outputs various information related to the management device 200 (e.g., various information received from the excavator 100) to an administrator or worker of the management device 200. The display device is, for example, a liquid crystal display or an organic EL display.

The output device 230 also includes a sound output device that audibly outputs various information related to the management device 200 to, for example, an administrator or an operator of the management device 200. The sound output device is, for example, a buzzer or a speaker.

The output device 230 also includes a display device 230A that is used by an operator who remotely operates the shovel 100.

The display device 230A displays various information images that support remote operation of the shovel 100 under the control of the control device 210. The display device 230A displays, for example, a surrounding image that shows the state around the shovel 100. The surrounding image includes image information that shows the state in front of the shovel 100, as in the case of the display device 50 of the shovel 100, and image information that shows at least one state of the left side, right side, and rear of the shovel 100.

The input device 240 accepts input from users such as the administrator of the management device 200, workers, and operators performing remote operations, and outputs the accepted input content to the control device 210.

The input device 240 includes, for example, an operation input device that accepts operation input from an administrator or worker of the management device 200. The operation input device includes, for example, a keyboard, a mouse, a touch panel, etc.

The input device 240 may also include, for example, a voice input device that accepts voice input from an administrator or worker of the management device 200, or a gesture input device that accepts gesture input. The voice input device includes, for example, a microphone. The gesture input device includes, for example, an imaging device that captures gestures made by an administrator or worker of the management device 200.

The input device 240 also includes, for example, an operating device 240A that is used by an operator who remotely operates the shovel 100.

The operating device 240A (an example of an operating unit) is used to remotely operate multiple driven elements of the shovel 100, i.e., hydraulic actuators. The operating device 240A may be configured in the same form as the operating device 26 of the shovel 100, i.e., centered around a lever device. The operating device 240A (lever device) outputs an electrical signal (hereinafter, "remote operation signal") corresponding to the operation content, i.e., the remote operation content, and the remote operation signal is taken into the control device 210.

The remote operation support unit 2101 supports the operator of the management device 200 in remotely operating the excavator 100.

The remote operation support unit 2101, for example, causes the display device 230A to display an image of the surroundings of the shovel 100 based on an image captured by the imaging device 40 (camera 40X) received from the shovel 100 by the communication device 220. This allows the operator of the management device 200 to remotely operate the shovel 100 while checking the surroundings of the shovel 100 displayed on the display device 230A.

When the surrounding image displayed on the display device 230A is a processed image, the processed image generated by the shovel 100 may be transmitted from the shovel 100 to the management device 200 instead of the image captured by the imaging device 40.

The remote operation support unit 2101 also transmits a remote operation signal, which is input from the operation device 240A and indicates the content of the remote operation, to the shovel 100 via the communication device 220 (an example of a communication unit). This allows the operation content of the operation device 240A to be reflected in the operation of the shovel 100, thereby realizing remote operation of the shovel 100.

The safety control unit 2102 controls the safety functions of the shovel 100. Specifically, similar to the safety control unit 303, the safety control unit 2102 controls the safety functions to avoid contact between the shovel 100 and objects around it.

The safety control unit 303 activates a specified safety function (hereinafter, for convenience, the "second safety function"), for example, when a monitoring target detection signal is received from the excavator 100 by the communication device 220, i.e., when a monitoring target is detected within the monitoring area by the object detection unit 302.

The second safety function includes, for example, a notification function that notifies the operator remotely operating the excavator 100 that a monitored object has been detected, such as by outputting an alarm, as in the case of the first safety function.

The safety control unit 2102, for example, activates the alarm function when the object detection unit 302 detects a monitoring target within the alarm range included in the monitoring area, similar to the case of the safety control unit 303.

The safety control unit 2102, for example, activates the sound notification function by controlling the sound output device included in the output device 230, similar to the case of the safety control unit 303. This allows the output device 230 to realize an auditory notification function under the control of the control device 210.

Furthermore, the safety control unit 2102, for example, in the same manner as the safety control unit 303, controls the display device included in the output device 230 to activate a notification function by displaying image information. In this way, the output device 230 can realize a visual notification function for the operator under the control of the control device 210.

When the communication device 220 receives a low detection accuracy notification signal from the shovel 100, the low detection accuracy notification unit 2103 notifies the operator that a low detection accuracy area exists around the shovel 100. In addition, when the detection accuracy determination unit 305 determines that an area in which the detection accuracy of the monitored object is relatively low exists around the shovel 100, the low detection accuracy notification unit 306 may notify (instruct) the operator of the area in which the detection accuracy of the monitored object is relatively low around the shovel 100.

The low detection accuracy notification unit 2103 may control a display device included in the output device 230 to display information indicating the presence of a low detection accuracy area around the shovel 100, information identifying the low detection accuracy area around the shovel 100, etc. on the display device. The low detection accuracy notification unit 2103 may also control a sound output device included in the output device 230 to output a sound indicating the presence of a low detection accuracy area around the shovel 100, audio information identifying the low detection accuracy area around the shovel 100, etc. from the sound output device. Details of the method of notifying the operator of the presence of a low detection accuracy area and the low detection accuracy area itself will be described later (see FIG. 10).

[Control process for notifying an operator of an area with low detection accuracy]
Next, a specific example of a control process (hereinafter, "low detection accuracy area notification process") related to a notification of a low detection accuracy area to an operator by the controller 30 of the shovel 100 will be described with reference to FIG.

Figure 5 is a flowchart that shows an example of low detection accuracy area notification processing by the controller 30. This flowchart is repeatedly executed at predetermined time intervals from the start to the stop of the excavator 100 (e.g., when the key switch is turned OFF).

As shown in FIG. 5, in step S102, the controller 30 determines whether the frequency of at least one of the rotation operation of the upper rotating body 3 and the movement operation of the shovel 100 (i.e., the traveling operation of the lower traveling body 1) occurring from the start of the shovel 100 is relatively high. The frequency is, for example, the number of occurrences per unit time. The controller 30 may determine, for example, whether the frequency of at least one of the rotation operation of the upper rotating body 3 and the traveling operation of the lower traveling body 1 occurring from the start of the shovel 100 is equal to or higher than a predetermined threshold. If the frequency of at least one of the rotation operation of the upper rotating body 3 and the traveling operation of the lower traveling body 1 occurring from the start of the shovel 100 is not relatively high (low), the controller 30 proceeds to step S104, and if the frequency is relatively high, the controller 30 proceeds to step S108.

In step S104, the controller 30 determines whether or not at least one of the rotational movement of the upper rotating body 3 and the traveling movement of the lower running body 1 is being performed. If at least one of the rotational movement of the upper rotating body 3 and the traveling movement of the lower running body 1 is being performed, the controller 30 proceeds to step S106, and if neither movement is being performed, the controller 30 ends the processing of this flowchart.

In step S106, the controller 30 determines whether the rotation of the upper rotating body 3 and the running of the lower running body 1 have stopped. The controller 30 may determine that the rotation of the upper rotating body 3 and the running of the lower running body 1 have stopped, for example, when the state in which both the rotation of the upper rotating body 3 and the running of the lower running body 1 are stopped continues for a certain period of time (e.g., 10 seconds) or more. If both the rotation of the upper rotating body 3 and the running of the lower running body 1 have stopped, the controller 30 proceeds to step S110, and otherwise repeats the processing of step S106 until both the rotation of the upper rotating body 3 and the running of the lower running body 1 have stopped.

On the other hand, in step S108, the controller 30 determines whether or not a predetermined time (e.g., several minutes) has elapsed since the previous detection accuracy determination process (e.g., its completion). If the predetermined time has elapsed since the previous detection accuracy determination process, the controller 30 proceeds to step S110, and if the predetermined time has not elapsed, the controller 30 ends the processing of this flowchart.

In step S110, the controller 30 (safety control unit 303) determines whether or not an alarm based on the detection of a monitored object is being output through the display device 50 and/or the sound output device 52. If the controller 30 is not outputting an alarm, the process proceeds to step S112. If an alarm is being output, the controller 30 cancels the alarm and repeats the process of step S110 until the alarm is stopped.

In step S112, the controller 30 (detection accuracy determination unit 305) performs detection accuracy determination processing. When the processing of step S112 is completed, the controller 30 proceeds to step S114.

In step S114, the controller 30 (detection accuracy determination unit 305) determines whether or not there is a low detection accuracy area around the shovel 100. In this case, the low detection accuracy area may be, for example, an area where at least one of the detection accuracy of the monitored object based on the captured image of the imaging device 40 and the detection accuracy of the monitored object based on the light reception information of the sensor 45X is relatively low. The low detection accuracy area may also be, for example, an area where both the detection accuracy of the monitored object based on the captured image of the imaging device 40 and the detection accuracy of the monitored object based on the light reception information of the sensor 45X are relatively low. The low detection accuracy area may also be an area where one of the detection accuracy of the monitored object based on the captured image of the imaging device 40 and the detection accuracy of the monitored object based on the light reception information of the sensor 45X, which is relatively high under normal circumstances, is relatively low.

If there is a low detection accuracy area around the shovel 100, the controller 30 proceeds to step S116, and if there is no low detection accuracy area around the shovel 100, the controller 30 proceeds to step S118.

In step S116, the controller 30 (low detection accuracy notification unit 306) outputs a notification regarding the low detection accuracy area around the shovel 100 to the operator. Specifically, as described above, the controller 30 may output a notification regarding the low detection accuracy area around the shovel 100 to the operator in the cabin 10 through the display device 50 or the sound output device 52. In addition, when the shovel 100 is being remotely operated, the controller 30 may transmit a low detection accuracy notification signal to the management device 200 through the communication device T1. This allows the controller 30 to output a notification regarding the low detection accuracy area around the shovel 100 to the operator remotely operating the shovel 100 through the management device 200 (low detection accuracy notification unit 2103).

When the controller 30 completes processing of step S116, it completes processing of this flowchart.

In this manner, in this embodiment, the display device 50 and sound output device 52 (hereinafter, "display device 50, etc.") (an example of an alarm output unit) output an alarm to the operator under the control of the controller 30 when a monitoring target is detected around the shovel 100 based on image information acquired by the imaging device 40 (an example of an acquisition unit) and light reception information acquired by the surrounding object information acquisition device 45 (sensor 45X) (an example of an acquisition unit). Then, when a low detection accuracy area is present around the shovel 100, the display device 50, etc. (an example of a notification unit) notifies the operator about the low detection accuracy area around the shovel 100 under the control of the controller 30.

Similarly, in this embodiment, when the shovel 100 is being remotely operated, the output device 230 (an example of an alarm output unit) outputs an alarm to the operator remotely operating the shovel 100 under the control of the control device 210 if a monitoring target is detected around the shovel 100 based on the image captured by the imaging device 40 acquired by the shovel 100 and the light reception information of the sensor 45X, etc. Then, when there is a low detection accuracy area around the shovel 100, the output device 230 (an example of a notification unit) notifies the operator remotely operating the shovel 100 about the low detection accuracy area around the shovel 100 under the control of the control device 210.

This allows the shovel 100 and the management device 200 to make the operator aware of the situation in which the object detection unit 302 is likely to erroneously detect or overlook a monitored object. Therefore, the shovel 100 and the management device 200 can urge the operator to be vigilant and pay closer attention to the situation around the shovel 100. Thus, the function of the object detection unit 302 of the shovel 100 to detect monitored objects is complemented by the operator's awareness of a relatively high level of vigilance regarding the situation around the shovel 100, thereby improving the safety of the shovel 100.

In addition, in this embodiment, if there is a low detection accuracy area around the shovel 100, the display device 50 etc. may notify the operator of the low detection accuracy area around the shovel 100.

Similarly, in this embodiment, when the shovel 100 is being remotely operated, if there is a low detection accuracy area around the shovel 100, the output device 230 of the management device 200 may, under the control of the control device 210, notify the operator remotely operating the shovel 100 of the low detection accuracy area around the shovel 100.

This allows the shovel 100 and the management device 200 to indicate the location of low detection accuracy areas where the object detection unit 302 is likely to erroneously detect or overlook monitored objects. Therefore, the shovel 100 and the management device 200 can urge the operator to pay particular attention to low detection accuracy areas. This further improves the safety of the shovel 100.

In addition, in this embodiment, when there is an area around the shovel 100 where at least one of the detection accuracy of the monitored object based on the captured image (an example of first information) of the imaging device 40 (an example of a first acquisition unit) and the detection accuracy of the monitored object based on the received light information (an example of second information) of the sensor 45X (an example of a second acquisition unit) is relatively low, the display device 50, etc., under the control of the controller 30, may notify the operator of the low detection accuracy area around the shovel 100.

Similarly, in this embodiment, when the shovel 100 is being remotely operated, if there is an area around the shovel 100 where at least one of the detection accuracy of the monitored object based on the image captured by the imaging device 40 and the detection accuracy of the monitored object based on the light reception information of the sensor 45X is relatively low, the output device 230 of the management device 200 may, under the control of the control device 210, notify the operator remotely operating the shovel 100 of the low detection accuracy area around the shovel 100.

As a result, when there is an area where the detection accuracy of the monitored object based on any one of the information from multiple information sources is relatively low, the shovel 100 and the management device 200 can notify the operator of information regarding the low detection accuracy area even if the detection accuracy of the monitored object based on other information is relatively high. Therefore, when there is even a slight decrease in the detection accuracy of the monitored object, the shovel 100 and the management device 200 can urge the operator to pay close attention to the situation around the shovel 100. This can further improve the safety of the shovel 100.

In addition, in this embodiment, when there is an area around the shovel 100 where one of the detection accuracies of the monitored object based on the image captured by the imaging device 40 and the detection accuracies of the monitored object based on the light reception information of the sensor 45X, which are normally relatively high, is relatively low, the display device 50 etc. may notify the operator of the low detection accuracy area around the shovel 100.

Similarly, when the shovel 100 is being remotely operated, if there is an area around the shovel 100 where one of the detection accuracies of the monitored object based on the image captured by the imaging device 40 and the detection accuracies of the monitored object based on the light reception information of the sensor 45X, which are normally relatively high, is relatively low, the output device 230 of the management device 200 may notify the operator of the low detection accuracy area around the shovel 100.

As a result, when an area occurs in which the detection accuracy of the monitored object based on information from an information source that normally provides relatively high detection accuracy among information from each of a plurality of information sources is relatively low, the shovel 100 and the management device 200 can notify the operator of information regarding the low detection accuracy area even when the detection accuracy of the monitored object based on other information is relatively high. Therefore, when the detection accuracy of the monitored object decreases even slightly, the shovel 100 and the management device 200 can urge the operator to pay close attention to the situation around the shovel 100. This can further improve the safety of the shovel 100.

In addition, in this embodiment, the detection accuracy determination unit 305 (an example of a determination unit) determines whether or not there is a low detection accuracy area around the shovel 100 when at least one of the rotation operation of the upper rotating body 3 and the movement of the shovel 100 (the traveling operation of the lower traveling body 1) occurs. Then, when the detection accuracy determination unit 305 determines that there is a low detection accuracy area around the shovel 100, the display device 50, etc., under the control of the controller 30, may notify the operator of the low detection accuracy area around the shovel 100.

As a result, the shovel 100 can check whether there are any low detection accuracy areas around the shovel 100, etc., every time the upper rotating body 3 rotates or the lower traveling body 1 travels, and the situation around the shovel 100 changes as seen by the imaging device 40 and sensor 45X. Therefore, the shovel 100 can notify the operator of low detection accuracy areas around the shovel 100 in accordance with changes in the situation around the shovel 100 as seen by the imaging device 40 and sensor 45X. This can further improve the safety of the shovel 100.

In addition, in this embodiment, the display device 50 etc. cancels the alarm based on the detection of the monitored object when the monitored object is no longer detected or when a predetermined input to cancel the alarm is received through the input device 54. Then, when at least one of the rotation operation of the upper rotating body 3 and the movement of the shovel 100 (the traveling operation of the lower traveling body 1) occurs while the alarm is output, the detection accuracy determination unit 305 may determine whether or not there is an area around the shovel 100 where the detection accuracy is relatively low when the alarm is canceled.

As a result, the shovel 100 can determine whether or not there is a low detection accuracy area around the shovel 100 once it is determined that there is no monitored object. Therefore, the shovel 100 can more appropriately check whether or not there is a low detection accuracy area around the shovel 100.

In addition, in this embodiment, the detection accuracy determination unit 305 may periodically (at predetermined intervals) determine whether or not there is a low detection accuracy area around the shovel 100 when the frequency of at least one of the rotational movement of the upper rotating body 3 and the movement of the shovel 100 (the traveling movement of the lower traveling body 1) is relatively high.

This allows the shovel 100 to suppress a situation in which the frequency of determining whether or not there is a low detection accuracy area around the shovel 100 becomes relatively high in a situation in which the situation around the shovel 100 as seen by the imaging device 40 and the sensor 45X changes frequently. Therefore, the shovel 100 can suppress an increase in the processing load of the controller 30.

[Detection accuracy determination process by controller]
Next, the detection accuracy determination process performed by the controller 30 will be described in detail with reference to FIGS.

Figure 6 is a flow chart outlining an example of the detection accuracy determination process by the controller 30 (i.e., the process of step S112 in Figure 5), and Figures 7 to 9 are diagrams each explaining the detection accuracy determination process by the controller 30. Specifically, Figures 7 to 9 are diagrams each explaining the detection accuracy determination process regarding the detection accuracy of a monitored object based on an image captured by the imaging device 40 (camera 40X).

As shown in FIG. 6, in step S202, the evaluation value calculation unit 304 calculates, for each of a plurality of evaluation positions within the information acquisition range around the excavator 100, an evaluation value that indicates the likelihood that the information from the information source corresponding to that position is a monitoring target. When the process of step S202 is completed, the controller 30 proceeds to step S204.

For example, as shown in FIG. 7, the evaluation value calculation unit 304 sets a rectangular evaluation frame (e.g., evaluation frames 710, 720, 730, 740) for each of the multiple partial images of the captured image IMG1 to define an evaluation image including the target partial image. In this case, the size of the evaluation frame is defined so that the lower the target partial image is located, since the object that is captured lower in the vertical direction of the captured image IMG1 represents an object closer to the shovel 100 (camera 40X). After calculating the evaluation value of the evaluation image divided by the evaluation frame, the evaluation value calculation unit 304 may calculate the evaluation values of the evaluation images corresponding to the leftmost partial image to the rightmost partial image by moving the evaluation frame to the right by an amount corresponding to the adjacent partial image to the right and calculating the evaluation value of the next evaluation image (see the arrow in the figure).

Returning to FIG. 6, in step S204, the evaluation value calculation unit 304 generates data that associates evaluation positions and evaluation values for the entire information acquisition range of the information source based on the calculation results of step S202. When the processing of step S204 is completed, the process proceeds to step S206.

For example, as shown in FIG. 8, data (heat map image IMG2) is generated that visualizes the high and low evaluation values for each of the multiple partial images of the captured image IMG1 by associating them with colors. In this example, light colors are set for partial images (pixels) with relatively high evaluation values, and dark colors are set for partial images (pixels) with relatively low evaluation values.

Returning to FIG. 6, in step S206, the detection accuracy determination unit 305 extracts a group of adjacent evaluation positions with relatively high evaluation values. When the process of step S206 is completed, the controller 30 proceeds to step S208.

In step S208, the detection accuracy determination unit 305 determines, for each extracted collection of evaluation positions, whether the collection of evaluation positions covers a relatively wide range. The detection accuracy determination unit 305 determines, for example, whether the collection of evaluation targets covers a range wider than the range corresponding to the size of the monitored target. If there is at least one collection of evaluation positions that covers a relatively wide range, the detection accuracy determination unit 305 determines that there is a low detection accuracy area around the shovel 100 and proceeds to step S210. On the other hand, if there is not a single collection of evaluation positions that covers a relatively wide range, the detection accuracy determination unit 305 determines that there is no low detection accuracy area around the shovel 100 and ends the processing of this flowchart.

In addition, in step S208, it may be determined whether or not a collection of adjacent evaluation positions with relatively high evaluation values has been extracted. In this case, when a collection of adjacent evaluation positions with relatively high evaluation values has been extracted, the detection accuracy determination unit 305 determines that there is a low detection accuracy area around the shovel 100 regardless of the range. Also, step S206 may be omitted, and in step S208, it may be determined whether or not there is an evaluation position with a relatively high evaluation value among the multiple evaluation positions. In this case, the detection accuracy determination unit 305 determines that there is a low detection accuracy area around the shovel 100 when there is an evaluation position with a relatively high evaluation value.

In step S210, the detection accuracy determination unit 305 sets an area corresponding to a collection of evaluation positions that covers a relatively wide range as a low detection accuracy area. When the process of step S210 is completed, the controller 30 ends the process of this flowchart.

For example, as shown in FIG. 9, the detection accuracy determination unit 305 may extract an image area from the heat map image IMG2 that corresponds to a collection of partial images of colors corresponding to relatively high evaluation values (relatively light colors in this example) (step S206). The detection accuracy determination unit 305 may determine whether the area of each extracted image area exceeds a predetermined threshold value corresponding to the size of the monitored object (step S208). Then, the detection accuracy determination unit 305 may set image areas (e.g., image areas 910, 920, 930) in the heat map image IMG2 that correspond to relatively high evaluation values and have areas that exceed the predetermined threshold value as low detection accuracy areas (step S210).

In this manner, in this embodiment, the evaluation value calculation unit 304 (an example of a calculation unit) calculates an evaluation value indicating the degree to which the information acquired from the information source represents the monitored object when no alarm is output. Then, when an area in which information with a relatively high evaluation value is acquired from the information source is present around the shovel 100, the display device 50 or the like notifies the operator of the low detection accuracy area around the shovel 100.

Similarly, when the shovel 100 is being remotely operated, if there is an area around the shovel 100 from which information with a relatively high evaluation value is obtained from an information source, the output device 230 of the management device 200 notifies the operator remotely operating the shovel 100 of the area around the shovel 100 that has low detection accuracy.

For example, when an information source acquires information about a background object around the shovel 100 that is not a monitored object, which indicates to a relatively high degree that the object is a monitored object, the object detection unit 302 may mistakenly recognize the background object as a monitored object. Also, when an information source acquires information about a background object around the shovel 100 that is not a monitored object, which indicates to a relatively high degree that the object is a monitored object, the monitored object may blend into the background objects, and the object detection unit 302 may not be able to detect the monitored object and may miss it.

In contrast, in this example, the excavator 100 and the management device 200 can determine that an area where information with a relatively high evaluation value is obtained from an information source is a low detection accuracy area, and notify the operator of the low detection accuracy area.

In addition, in this embodiment, the evaluation value calculation unit 304 calculates an evaluation value for each of the multiple partial images included in the image captured by the imaging device 40 when no warning is output. Then, when a partial image with a relatively high evaluation value is present among the multiple partial images, the display device 50, etc., under the control of the controller 30, notifies the operator of areas around the work machine where detection accuracy is relatively low.

Similarly, when the shovel 100 is being remotely operated, if there is a partial image with a relatively high evaluation value among the multiple partial images, the output device 230 of the management device 200, under the control of the control device 210, notifies the operator remotely operating the shovel 100 of areas of low detection accuracy around the shovel 100.

This allows the excavator 100 and the management device 200 to specifically notify about low detection accuracy areas where the detection accuracy based on the image captured by the imaging device 40 (camera 40X) is relatively low.

In addition, in this embodiment, if there is a partial image with a relatively high evaluation value among the multiple partial images of the image captured by the imaging device 40 (camera 40X) and the area occupied by the collection of partial images with relatively high evaluation values in the captured image exceeds a predetermined standard, the display device 50 etc. notifies the operator of the low detection accuracy area around the excavator 100.

Similarly, when the shovel 100 is being remotely operated, if there is a partial image with a relatively high evaluation value among the multiple partial images of the image captured by the imaging device 40 (camera 40X) and the area occupied by the collection of partial images with relatively high evaluation values in the captured image exceeds a predetermined standard, the output device 230 of the management device 200 notifies the operator remotely operating the shovel 100 of the low detection accuracy area around the shovel 100.

As a result, the excavator 100 and the management device 200 can determine that an image area (a collection of partial images) that is large enough to be mistaken for a monitored object or to blend in with the monitored object is a low detection accuracy area, and can notify the operator about the low detection accuracy area.

[Method of notifying an operator about an area of low detection accuracy]
Next, the method of notifying about a low detection accuracy area will be described in detail with reference to FIG.

Figure 10 is a diagram showing an example of a notification method regarding low detection accuracy areas. Specifically, Figure 10 is a diagram showing an example of a notification screen regarding low detection accuracy areas (hereinafter, "low detection accuracy notification screen") displayed on the display device 50.

As shown in FIG. 10, the display device 50 displays the captured image IMG1 of the imaging device 40 (camera 40X) under the control of the controller 30. Also, under the control of the controller 30, the display device 50 displays frames 1010, 1020, and 1030 representing image areas corresponding to low detection accuracy areas around the shovel 100, superimposed on the captured image IMG1. The frames 1010, 1020, and 1030 correspond to the image areas 910, 920, and 930 in FIG. 9, respectively. This allows the display device 50 to highlight and display the image areas corresponding to the low detection accuracy areas around the shovel 100 on the captured image of the imaging device 40 (camera 40X). Therefore, the operator of the cabin 10 can visually recognize the presence of low detection accuracy areas around the shovel 100 and their specific locations.

The low detection accuracy areas around the shovel 100 may also be emphasized by methods other than the frames 1010, 1020, and 1030. For example, under the control of the controller 30, the display device 50 may display other types of markers in the image areas corresponding to the low detection accuracy areas, or may increase the brightness of the image areas corresponding to the low detection accuracy areas relative to other image areas.

For example, the frames 1010, 1020, and 1030 may be constantly displayed between the output of the determination result by the detection accuracy determination unit 305 and the output of the next determination result (i.e., the determination result is updated). In addition, the timing at which the frames 1010, 1020, and 1030 are displayed may be limited. This allows the controller 30 to urge the operator to pay attention to the low detection accuracy area around the excavator 100 while suppressing the annoyance felt by the operator due to the display of the frames 1010, 1020, and 1030. For example, the frames 1010, 1020, and 1030 may be displayed for a predetermined period (e.g., several minutes) after the determination result by the detection accuracy determination unit 305 is output. In addition, for example, the frames 1010, 1020, and 1030 may be periodically switched between a displayed state and a non-displayed state between the output of the determination result by the detection accuracy determination unit 305 and the output of the next determination result. Also, for example, frames 1010, 1020, and 1030 may be displayed when a monitoring target is detected by object detection unit 302 between the time when a determination result is output by detection accuracy determination unit 305 and the time when the next determination result is output. For example, when a monitoring target is actually detected, if the detected monitoring target moves into a low detection accuracy area, the monitoring target may not be detected and the alarm may be canceled.

In addition, under the control of the controller 30, a pop-up window 1040 is displayed on the display device 50 so as to be superimposed on the captured image IMG1. Specifically, the pop-up window 1040 is displayed so as to be superimposed on the bottom edge of the captured image IMG1.

The pop-up window 1040 contains text information indicating a notification regarding the low detection accuracy area. Specifically, the pop-up window 1040 contains text information (warning message) indicating that the detection accuracy of the monitored object is relatively low in the low detection accuracy areas corresponding to the frames 1010, 1020, and 1030. In this example, the pop-up window 1040 displays a warning message stating, "The detection function may not operate normally within the frame. Please be careful." This allows the controller 30 to urge the operator to pay more attention to the area around the excavator 100 (the low detection accuracy area).

For example, the pop-up window 1040 may be constantly displayed from the time when the determination result by the detection accuracy determination unit 305 is output until the next determination result is output (i.e., the determination result is updated). Also, as in the case of the frames 1010, 1020, and 1030, the timing when the pop-up window 1040 is displayed may be limited.

Furthermore, as described above, instead of or in addition to the display device 50, a notification regarding the low detection accuracy area may be provided through the sound output device 52. For example, the sound output device 52 may output audio information corresponding to the text information included in the pop-up window 1040.

For example, the audio information corresponding to the character information included in the pop-up window 1040 may be repeatedly and continuously (continuously) output between the output of the determination result by the detection accuracy determination unit 305 and the output of the next determination result. Also, the number of times that the audio information corresponding to the character information included in the pop-up window 1040 is output may be limited. For example, the audio information corresponding to the character information included in the pop-up window 1040 may be periodically output at a predetermined interval between the output of the determination result by the detection accuracy determination unit 305 and the output of the next determination result. Also, for example, the audio information corresponding to the character information included in the pop-up window 1040 may be output only a predetermined number of times (for example, once) after the output of the determination result by the detection accuracy determination unit 305. In this way, the controller 30 can suppress the annoyance felt by the operator due to the output of the audio information while urging the operator to pay attention to the low detection accuracy area around the excavator 100.

In addition, in the management device 200, notification regarding low detection accuracy areas may be provided in a similar manner through the output device 230 under the control of the control device 210.

In this manner, in this embodiment, when there is a low detection accuracy area around the shovel 100, the display device 50 (an example of a display unit) may display a surrounding image showing the state of the area around the shovel 100, and may highlight and display an image area in the surrounding image that corresponds to the low detection accuracy area.

Similarly, when the shovel 100 is being remotely operated, if there is a low detection accuracy area around the shovel 100, the output device 230 (display device) of the management device 200 may display a surrounding image showing the state of the area around the shovel 100, and may highlight and display an image area in the surrounding image that corresponds to the low detection accuracy area.

This allows the shovel 100 and management device 200 to visually notify the operator of areas of low detection accuracy around the shovel 100.

In addition, in this embodiment, when there is a low detection accuracy area around the shovel 100, the display device 50 may display a surrounding image showing the state of the area around the shovel 100, and may highlight and display an image area in the surrounding image that corresponds to the low detection accuracy area only for a predetermined period of time, periodically, or when a monitored object is detected.

Similarly, when the shovel 100 is being remotely operated, if there is a low detection accuracy area around the shovel 100, the output device 230 (display device) of the management device 200 may display a surrounding image showing the state of the area around the shovel 100, and may highlight and display an image area in the surrounding image that corresponds to the low detection accuracy area only for a specified period of time, periodically, or when a monitored object is detected.

This allows the shovel 100 and the management device 200 to limit the number of times and the time for which visual information to notify the operator of low detection accuracy areas around the shovel 100 is displayed. As a result, the shovel 100 and the management device 200 can improve the safety of the shovel 100 while reducing the inconvenience felt by the operator.

In addition, in this embodiment, when there is a low detection accuracy area around the excavator 100, the sound output device 52 (an example of an audio output unit) may output audio information indicating that the detection accuracy of the monitored object is relatively low in the area around the work machine that corresponds to the highlighted image area in the surrounding image.

Similarly, when the shovel 100 is being remotely operated, if there is a low detection accuracy area around the shovel 100, the output device 230 (sound output device) of the management device 200 may output audio information indicating that the detection accuracy of the monitored object is relatively low in the area around the work machine corresponding to the highlighted image area in the surrounding image.

This allows the shovel 100 and management device 200 to notify the operator of information regarding low detection accuracy areas around the shovel 100 using audio information in addition to visual information.

In addition, in this embodiment, the sound output device 52 may output audio information a predetermined number of times or periodically if there is a low detection accuracy area around the shovel 100.

Similarly, when the shovel 100 is being remotely operated, the output device 230 (sound output device) of the management device 200 may output audio information a predetermined number of times or periodically if there is a low detection accuracy area around the shovel 100.

This allows the shovel 100 and the management device 200 to limit the time and number of times that audio information is output. Therefore, the shovel 100 and the management device 200 can improve the safety of the shovel 100 while reducing the annoyance felt by the operator.

[Transformations/Changes]
Although the above describes in detail the form for carrying out the present invention, the present invention is not limited to such specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

For example, in the above embodiment, a function for prompting attention and safety checks in areas with low detection accuracy around the shovel 100 was described, but a similar function may be adopted in other work machines other than the shovel 100. Other work machines may include, for example, bulldozers, wheel loaders, mobile cranes, etc.

1 Lower traveling body 3 Upper rotating body 30 Controller 40 Imaging device (acquisition unit)
40B, 40F, 40L, 40R Camera 45 Peripheral object information acquisition device (acquisition unit)
45BL, 45BR, 45L, 45R Sensor 50 Display device (alarm output unit, notification unit)
52 Sound output device (alarm output unit, notification unit)
54 Input device 100 Shovel (working machine)
200 Management device (information processing device)
210 Control device 220 Communication device 230 Output device (notification unit, alarm output unit)
230A Display device 240 Input device 240A Operation device 301 Remote operation control unit 302 Object detection unit 303 Safety control unit 304 Evaluation value calculation unit (calculation unit)
305 Detection accuracy determination unit (determination unit)
306 Low detection accuracy notification unit 2101 Remote operation support unit 2102 Safety control unit 2103 Low detection accuracy notification unit T1 Communication device

Claims (16)

an acquisition unit that acquires information about objects around the work machine;
a detection unit that detects a specific type of object around the work machine based on the information acquired by the acquisition unit;
an alarm output unit that outputs an alarm to an operator when the specific type of object is detected by the detection unit;
and a notification unit that notifies an operator about an area around the work machine when there is an area within an information acquisition range around the work machine by the acquisition unit where the accuracy with which the detection unit can detect the specific type of object is relatively low compared to a predetermined standard due to the influence of other objects that exist around the work machine and correspond to a background seen from the acquisition unit when it is assumed that the specific type of object exists.
Working machinery. the notification unit notifies the operator of the area around the work machine when the area is within the acquisition range.
2. The work machine of claim 1. A plurality of the acquisition units are provided,
the plurality of acquisition units include a first acquisition unit that acquires first information regarding objects around the work machine, and a second acquisition unit that acquires second information regarding objects around the work machine, the second information being different from the first information;
the detection unit detects the specific type of object based on the first information, and detects the specific type of object based on the second information;
the notification unit notifies the operator about the area around the work machine when at least one of the following is true: within the acquisition range of the first information by the first acquisition unit around the work machine, there is an area where the accuracy at which the detection unit can detect the specific type of object based on the first information is relatively low compared to a predetermined standard; and within the acquisition range of the second information by the second acquisition unit around the work machine, there is an area where the accuracy at which the detection unit can detect the specific type of object based on the second information is relatively low compared to a predetermined standard.
A work machine according to claim 1 or 2. a calculation unit that calculates an evaluation value indicating a degree to which the information acquired by the acquisition unit represents the specific type of object,
the notification unit notifies the operator about the area around the work machine when the warning has not been output and information in which the evaluation value is relatively high with respect to a predetermined standard is acquired by the acquisition unit within the acquisition range.
A work machine according to any one of claims 1 to 3. The detection unit calculates a feature amount representing a feature of the specific type of object in the information acquired by the acquisition unit, based on the information acquired by the acquisition unit, and detects the specific type of object based on the feature amount;
The calculation unit calculates the evaluation value based on the feature amount.
5. A work machine according to claim 4. the acquisition unit is a camera that acquires a captured image showing a state around the work machine,
the calculation unit calculates the evaluation value for each of a plurality of partial images included in the same captured image;
the notification unit notifies the operator about the area around the work machine when the warning has not been output and a partial image having the evaluation value relatively high with respect to a predetermined standard is present among the plurality of partial images.
A work machine according to claim 4 or 5. the notification unit notifies the operator about the area around the work machine when there is a partial image among the plurality of partial images whose evaluation value is relatively high with respect to a predetermined standard, and when an area occupied by a collection of partial images whose evaluation value is relatively high with respect to the predetermined standard in the captured image including the plurality of partial images is relatively large with respect to the predetermined standard.
7. A work machine according to claim 6. The notification unit includes a display unit,
the display unit displays image information representing the surroundings of the work machine when the area is within the acquisition range, and highlights an image area in the image information that corresponds to the area.
A work machine as claimed in any one of claims 1 to 7. the display unit displays the image information representing the surroundings of the work machine when the area is within the acquisition range, and highlights and displays the image area only for a predetermined period of time, periodically, or when the specific type of object is detected.
9. A work machine according to claim 8. The notification unit includes an audio output unit,
when the area is within the acquisition range, the audio output unit outputs audio information indicating that the accuracy is relatively low with respect to a predetermined standard in the area around the work machine corresponding to the image area.
A work machine according to claim 8 or 9. the audio output unit outputs the audio information a predetermined number of times or periodically when the area is within the acquisition range.
11. A work machine according to claim 10. the warning output unit outputs, when the specific type of object is detected in the area by the detection unit, the first warning of a first warning indicating a relatively high level of alert to the operator and a second warning indicating a relatively low level of alert to the operator;
A work machine as claimed in any one of the preceding claims. a determination unit that determines whether or not the area is within the acquisition range when the work machine is started up, and determines whether or not the area is within the acquisition range when at least one of an operation of a part of the work machine to which the acquisition unit is attached and a movement of the work machine occurs,
the notification unit notifies the operator of the area around the work machine when the determination unit determines that the area is within the acquisition range.
A work machine as claimed in any one of the preceding claims. the alarm output unit cancels the alarm when the specific type of object is no longer detected or when a predetermined input for canceling the alarm is received;
the determination unit, when at least one of an operation of a part of the work machine to which the acquisition unit is attached and a movement of the work machine occurs while the alarm is being output, determines whether or not the area is within the acquisition range when the alarm is released.
14. A work machine according to claim 13. the determination unit periodically determines whether or not the area is within the acquisition range when the number of occurrences per unit time of at least one of the operation of the part of the work machine and the movement of the work machine is relatively high with respect to a predetermined standard.
A work machine according to claim 13 or 14. an operation unit that receives remote control operations related to the work machine;
a communication unit that transmits the contents of the remote operation received by the operation unit to the work machine;
an alarm output unit that outputs an alarm to an operator performing remote operation when a specific type of object is detected around the work machine by a detection unit of the work machine based on information about objects around the work machine acquired by an acquisition unit of the work machine;
a notification unit that notifies the operator about the area around the work machine when there is an area within an information acquisition range around the work machine by the acquisition unit where the accuracy with which the detection unit can detect the specific type of object is relatively low compared to a predetermined standard due to the influence of other objects that exist around the work machine and correspond to a background seen from the acquisition unit when it is assumed that the specific type of object exists,
Information processing device.

Images