JP2023063992A - Shovel - Google Patents

Abstract

To provide a technique for more appropriately detecting an object to be detected from a picked-up image around a shovel.SOLUTION: A shovel 100 includes an undercarriage 1, a super structure 3 turnably mounted on the undercarriage 1, an image pick-up device 40 mounted on the super structure 3 for picking up an image around the super structure 3, an irradiation device 70 for irradiating an image pick-up range of the image pick-up device 40 with infrared light, and an object detection device 302 for recognizing a highly visible safety wear RV on the basis of the image picked-up by the image pick-up device 40 to detect a person around the shovel 100.SELECTED DRAWING: Figure 15

Description

The present disclosure relates to excavators.

Conventionally, a predetermined object (e.g., a person) within a predetermined range around the excavator (upper revolving structure) is detected from an image captured by an imaging device mounted on the upper revolving structure, and a safety function (e.g., outputting an alarm) is performed. is known (see, for example, Patent Document 1).

JP 2014-181508 A

However, a non-existent object may be detected and the safety function may be erroneously activated, or a real object may not be detected and the required safety function may not be activated. Therefore, it is desirable to further improve the accuracy of object detection around the shovel.

Therefore, in view of the above problems, it is an object of the present invention to provide a technique capable of further improving the accuracy of object detection around the excavator.

To achieve the above objectives, in one embodiment of the present disclosure,
a lower running body;
an upper rotating body rotatably mounted on the lower traveling body;
an imaging device mounted on the upper revolving body for imaging the periphery of the upper revolving body;
an irradiation device that irradiates an imaging range of the imaging device with infrared light;
a detection unit that detects people around the excavator by recognizing high-visibility safety clothing based on the captured image of the imaging device;
A shovel is provided.

According to the above-described embodiments, it is an object of the present invention to provide a technique capable of further improving the accuracy of object detection around the shovel.

It is a side view which shows an example of a shovel. It is a top view which shows an example of a shovel. It is a figure which shows an example of an excavator management system. It is a figure which shows an example of the hardware constitutions of an excavator. It is a figure which shows an example of the hardware constitutions of a management apparatus. It is a functional block diagram showing an example of a functional configuration of a shovel. It is a figure explaining the example of the object detection method. FIG. 4 is a diagram showing a first example of an image of training data; FIG. 10 is a diagram showing a second example of an image of training data; FIG. 11 is a diagram showing a third example of an image of training data; FIG. 12 is a diagram showing a fourth example of an image of teacher data; FIG. 12 is a diagram showing a fifth example of an image of teacher data; FIG. 16 is a diagram showing a sixth example of an image of teacher data; It is a figure which shows an example of a worker. It is a figure which shows another example of a worker. It is a figure which shows an example of the captured image of an imaging device. FIG. 10 is a diagram showing an example of an image after correction (corrected captured image) with respect to the captured image of the imaging device; FIG. 4 is a diagram showing a first example of a captured image in which only a part of the whole (whole body) of a monitored object (person) is shown; FIG. 10 is a diagram showing a second example of a captured image in which only a part of the whole (whole body) of a monitored object (person) is shown; FIG. 11 is a diagram showing a third example of a captured image in which only a part of the whole (whole body) of a monitored object (person) is shown; FIG. 12 is a diagram showing a fourth example of a captured image in which only a part of the whole (whole body) of a monitored object (person) is shown; FIG. 10 is a diagram showing an example of a detection state of a monitored object (person); FIG. 10 is a diagram showing another example of a detection state of a monitored object (person); FIG. 10 is a diagram showing still another example of a detection state of a monitored object (person); FIG. 3 is a functional block diagram showing another example of the functional configuration of the shovel;

Embodiments will be described below with reference to the drawings.

[Overview of Excavator]
An outline of the excavator will be described with reference to FIGS. 1 to 3. FIG.

1 and 2 are a side view and a top view showing an example of a shovel 100 according to this embodiment. FIG. 3 is a diagram showing an example of the excavator management system SYS.

As shown in FIGS. 1 and 2, the excavator 100 includes a lower traveling body 1, an upper revolving body 3 mounted on the lower traveling body 1 so as to be rotatable via a revolving mechanism 2, and attachments for performing various operations. An AT and a cabin 10 are provided. Below, in front of the excavator 100 (upper revolving body 3), when the excavator 100 is viewed along the revolving shaft of the upper revolving body 3 from directly above in plan view (top view), the attachment to the upper revolving body 3 extends. Corresponds to the output direction.

The undercarriage 1 includes, for example, a pair of left and right crawlers 1C. The lower traveling body 1 causes the excavator 100 to travel by hydraulically driving the respective crawlers 1C by the left traveling hydraulic motor 1ML and the right traveling hydraulic motor 1MR.

The upper revolving structure 3 revolves with respect to the lower traveling structure 1 by hydraulically driving the revolving mechanism 2 with a revolving hydraulic motor 2M.

Attachment AT includes boom 4, arm 5, and bucket 6 as driven elements.

The boom 4 is attached to the center of the front part of the upper rotating body 3 so as to be able to be raised. An arm 5 is attached to the tip of the boom 4 so as to be vertically rotatable. possible to be installed.

Bucket 6 is an example of an end attachment. The bucket 6 is used, for example, for excavation work or the like.

Further, another end attachment may be attached to the tip of the arm 5 instead of the bucket 6, depending on the type of work and the like. Other end attachments may be other types of buckets such as, for example, large buckets, slope buckets, dredging buckets, and the like. Other end attachments may also be types of end attachments other than buckets, such as agitators, breakers, grapples, and the like. Further, a spare attachment such as a quick coupling or a tilt rotator may be interposed between the arm 5 and the end attachment.

Also, the bucket 6 may be attached with a hook for crane work. The hook is pivotally connected at its proximal end to a bucket pin that connects between the arm 5 and the bucket 6 . Thereby, the hook is accommodated in the space formed between the two bucket links when work other than crane work (hanging work) such as excavation work is performed.

The boom 4, arm 5, and bucket 6 are hydraulically driven by boom cylinders 7, arm cylinders 8, and bucket cylinders 9 as hydraulic actuators, respectively.

In the excavator 100, some or all of the various hydraulic actuators may be replaced with electric actuators. That is, the excavator 100 may be a hybrid excavator or an electric excavator.

The cabin 10 is a cockpit in which an operator boards, and is mounted on the front left side of the upper swing body 3, for example.

Note that when an operator does not operate the excavator 100 and the excavator 100 operates exclusively by remote control or fully automatic operation function as described later, the cabin 10 may be omitted.

Also, as shown in FIG. 3, the excavator 100 may be included in the excavator management system SYS together with the management device 200, and may be able to communicate with the management device 200 through a predetermined communication line NW. . As a result, the excavator 100 can transmit (upload) various information to the management device 200 and receive various signals (for example, information signals and control signals) from the management device 200 .

The communication line NW includes, for example, a wide area network (WAN). A wide area network may include, for example, a mobile communication network terminating at a base station. The wide area network may also include, for example, a satellite communication network that uses communication satellites over the excavator 100 . The wide area network may also include, for example, the Internet network. Also, the communication line NW may include, for example, a local network (LAN: Local Area Network) such as a facility where the management device 200 is installed. The local network may be a wireless line, a wired line, or a line containing both. Also, the communication line NW may include, for example, a short-range communication line based on a predetermined wireless communication method such as WiFi or Bluetooth (registered trademark).

The excavator management system SYS manages the excavator 100 and supports the excavator 100 in the management device 200 .

For example, the excavator management system SYS may manage (monitor) the operating state and operation status of the excavator 100 based on various information uploaded from the excavator 100 in the management device 200 . Further, the excavator management system SYS may support remote control of the excavator 100 in the management device 200 as described later. As will be described later, when the excavator 100 performs the work by fully automatic operation, the excavator management system SYS may support remote monitoring of the work by the fully automatic operation of the excavator 100 in the management device 200, for example. In this case, the excavator management system SYS may support the operation of the excavator 100 by the supervisor's intervention for the fully automated driving function in the management device 200 .

The excavator 100 included in the excavator management system SYS may be one or plural. Also, the number of management devices 200 included in the excavator management system SYS may be one or plural. That is, the plurality of management devices 200 may distribute and implement the processing related to the excavator management system SYS. For example, the plurality of management devices 200 mutually communicate with some of the excavators 100 that are in charge of all the excavators 100 included in the excavator management system SYS. You may perform a process to

The management device 200 may be, for example, an on-premises server or a cloud server installed in a management center or the like outside the work site where the excavator 100 works. Also, the management device 200 may be, for example, an edge server arranged in a work site where the excavator 100 works, or in a place relatively close to the work site. Also, the management device 200 may be a stationary terminal device or a portable terminal device (portable terminal) arranged in a management office or the like in the work site of the excavator 100 . The stationary terminal device may include, for example, a desktop PC (Personal Computer). Portable terminal devices may include, for example, smartphones, tablet terminals, laptop PCs, and the like.

The excavator 100 may operate actuators (for example, hydraulic actuators) to drive driven elements such as the lower traveling body 1, the upper revolving body 3, and the attachment AT in accordance with the operation of an operator on board the cabin 10. .

Also, the excavator 100 may be configured to be remotely controlled (remotely controlled) from a predetermined external device (for example, the management device 200). When the excavator 100 is remotely controlled, the interior of the cabin 10 may be unmanned. The following description is based on the premise that the operator's operation includes both the operation of the operating device 26 by the operator of the cabin 10 and the remote operation by the outside operator.

Specifically, the excavator 100 is operated by a user (operator)'s input regarding the actuator of the excavator 100 performed by the management device 200 . In this case, the excavator 100 may display an image representing the surroundings of the excavator 100 (hereinafter referred to as a “peripheral image”) such as an image captured by the imaging device 40 described later or a processed image based on the captured image (for example, a viewpoint conversion image). ) to the external device. The peripheral image is displayed on a remote control display device (hereinafter referred to as a “remote control display device”) provided in the management device 200 . Various information images (information screens) displayed on the display device 50A (see FIG. 6) in the cabin 10 of the excavator 100 may also be displayed on the remote control display device of the management device 200 in the same manner. As a result, the operator can remotely operate the excavator 100 while confirming the display contents such as the peripheral image representing the surroundings of the excavator 100 and various information images displayed on the remote control display device. . Then, the excavator 100 operates the actuators according to the remote control signal representing the content of the remote control received from the management device 200, and drives the driven elements such as the lower traveling body 1, the upper revolving body 3, and the attachment AT. can be driven.

In addition, the excavator 100 may automatically operate the actuator regardless of the content of the operator's operation. As a result, the excavator 100 has a function of automatically operating at least a part of the driven elements such as the lower traveling body 1, the upper revolving body 3, and the attachment AT, that is, the so-called "automatic driving function" or "machine control (Machine Control)." : MC) function" can be realized.

The automatic operation function includes a function to automatically operate a driven element (actuator) other than the driven element (actuator) to be operated according to the operation of the operator, that is, the so-called "semi-automatic operation function" or "operation support type 'MC function' may be included. In addition, the automatic operation function includes a function that automatically operates at least a part of a plurality of driven elements (hydraulic actuators) on the premise that there is no operator's operation, that is, a so-called "fully automatic operation function" or "fully automatic type". 'MC function' may be included. In the excavator 100, when the fully automatic operation function is enabled, the interior of the cabin 10 may be in an unmanned state. In addition, the semi-automatic operation function, the fully automatic operation function, and the like may include a mode in which the operation contents of the driven elements (actuators) to be automatically operated are automatically determined according to predetermined rules. In the semi-automatic operation function, the fully automatic operation function, etc., the excavator 100 autonomously makes various judgments, and according to the judgment results, the driven elements (hydraulic actuators) to be automatically operated autonomously operate. So-called "autonomous driving functions", whose contents are determined, may also be included.

[Excavator hardware configuration]
Next, a hardware configuration of the excavator 100 will be described with reference to FIG. 4 .

FIG. 4 is a diagram showing an example of the hardware configuration of the excavator 100. As shown in FIG.

In FIG. 4, the path through which mechanical power is transmitted is double lines, the path through which high-pressure hydraulic oil that drives the hydraulic actuator flows is solid lines, and the path through which pilot pressure is transmitted is dashed lines, and electrical signals are transmitted. Paths are indicated by dotted lines respectively.

The excavator 100 includes a hydraulic drive system for hydraulically driving the driven elements, an operation system for operating the driven elements, a user interface system for exchanging information with the user, a communication system for communication with the outside, a control system for various controls, and the like. including each component of

<Hydraulic drive system>
As shown in FIG. 4, the hydraulic drive system of the excavator 100 according to the present embodiment includes the lower traveling body 1 (left and right crawlers 1C), the upper revolving body 3, and the driven elements such as the attachment AT, as described above. includes a hydraulic actuator HA that hydraulically drives the . Also, the hydraulic drive system of the excavator 100 according to this embodiment includes an engine 11 , a regulator 13 , a main pump 14 and a control valve 17 .

As shown in FIG. 6, the hydraulic actuator HA includes travel hydraulic motors 1ML and 1MR, a turning hydraulic motor 2M, a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, and the like.

The engine 11 is a prime mover of the excavator 100 and a main power source in the hydraulic drive system. The engine 11 is, for example, a diesel engine that uses light oil as fuel. The engine 11 is mounted, for example, on the rear portion of the upper revolving body 3 . The engine 11 rotates at a preset target speed under direct or indirect control by a controller 30 to be described later, and drives the main pump 14 and the pilot pump 15 .

Note that, instead of or in addition to the engine 11, another prime mover (for example, an electric motor) or the like may be mounted on the excavator 100. FIG.

The regulator 13 controls (adjusts) the discharge amount of the main pump 14 under the control of the controller 30 . For example, the regulator 13 adjusts the angle of the swash plate of the main pump 14 (hereinafter referred to as “tilt angle”) according to a control command from the controller 30 .

The main pump 14 supplies hydraulic fluid to the control valve 17 through a high pressure hydraulic line. The main pump 14 is mounted, for example, on the rear portion of the upper rotating body 3, similar to the engine 11. As shown in FIG. The main pump 14 is driven by the engine 11 as described above. The main pump 14 is, for example, a variable displacement hydraulic pump, and as described above, under the control of the controller 30, the regulator 13 adjusts the tilting angle of the swash plate, thereby adjusting the stroke length of the piston and discharging. The flow rate (discharge pressure) is controlled.

The control valve 17 is a hydraulic control device that controls the hydraulic actuator HA according to the content of the operator's operation on the operation device 26 or remote control, or an operation command relating to the automatic operation function output from the controller 30 . The operation command corresponding to the automatic driving function may be generated by the controller 30, or may be generated by another control device (arithmetic device) that performs control related to the automatic driving function. The control valve 17 is mounted, for example, in the central portion of the upper revolving body 3 . As described above, the control valve 17 is connected to the main pump 14 via the high-pressure hydraulic line, and supplies hydraulic oil supplied from the main pump 14 according to an operator's operation or an operation command corresponding to the automatic operation function. , selectively feeding the respective hydraulic actuators. Specifically, the control valve 17 includes a plurality of control valves (also referred to as “direction switching valves”) that control the flow rate and flow direction of hydraulic oil supplied from the main pump 14 to each hydraulic actuator HA.

<Operation system>
As shown in FIG. 4 , the operating system of the excavator 100 according to this embodiment includes a pilot pump 15 , an operating device 26 , a hydraulic control valve 31 , a shuttle valve 32 and a hydraulic control valve 33 .

Pilot pump 15 supplies pilot pressure to various hydraulic devices via pilot line 25 . The pilot pump 15 is mounted, for example, on the rear portion of the upper revolving body 3 in the same manner as the engine 11 . The pilot pump 15 is, for example, a fixed displacement hydraulic pump, and is driven by the engine 11 as described above.

Incidentally, the pilot pump 15 may be omitted. In this case, relatively high-pressure hydraulic fluid discharged from the main pump 14 is decompressed by a predetermined pressure reducing valve, and then relatively low-pressure hydraulic fluid is supplied as pilot pressure to various hydraulic devices.

The operating device 26 is provided near the cockpit of the cabin 10 and is used by the operator to operate various driven elements. In other words, the operating device 26 is used by the operator to operate the hydraulic actuators HA that drive the respective driven elements. The operating device 26 includes a pedal device and a lever device for operating each driven element (hydraulic actuator HA).

For example, as shown in FIG. 4, the operating device 26 is hydraulically piloted. Specifically, the operating device 26 utilizes the hydraulic oil supplied from the pilot pump 15 through the pilot line 25 and the pilot line 25A branched therefrom, and applies the pilot pressure according to the operation content to the secondary side pilot line. 27A. The pilot line 27A is connected to one inlet port of the shuttle valve 32 and connected to the control valve 17 via the pilot line 27 connected to the outlet port of the shuttle valve 32 . As a result, a pilot pressure can be input to the control valve 17 through the shuttle valve 32 according to the operation details of various driven elements (hydraulic actuators) in the operating device 26 . Therefore, the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operating device 26 by an operator or the like.

Note that the operating device 26 may be of an electric type. In this case, the pilot line 27A, shuttle valve 32 and hydraulic control valve 33 are omitted. Specifically, the operation device 26 outputs an electric signal (hereinafter referred to as “operation signal”) corresponding to the content of the operation, and the operation signal is captured by the controller 30 . Then, the controller 30 outputs to the hydraulic control valve 31 a control command corresponding to the content of the operation signal, that is, a control signal corresponding to the content of the operation on the operating device 26 . As a result, a pilot pressure corresponding to the operation content of the operation device 26 is input from the hydraulic control valve 31 to the control valve 17, and the control valve 17 drives each hydraulic actuator HA according to the operation content of the operation device 26. be able to. Also, the control valves (direction switching valves) built in the control valve 17 that drive the respective hydraulic actuators may be of the electromagnetic solenoid type. In this case, the operation signal output from the operation device 26 may be directly input to the control valve 17, that is, to the electromagnetic solenoid type control valve. Further, when the excavator 100 is remotely controlled or operates by an automatic driving function, the operating device 26 may be omitted.

The hydraulic control valve 31 is provided for each driven element (hydraulic actuator HA) to be operated by the operating device 26 . The hydraulic control valve 31 is provided, for example, in the pilot line 25B between the pilot pump 15 and the control valve 17, and is configured such that its passage area (that is, the cross-sectional area through which hydraulic oil can flow) can be changed. good. As a result, the hydraulic control valve 31 can output a predetermined pilot pressure to the secondary side pilot line 27B using the hydraulic fluid of the pilot pump 15 supplied through the pilot line 25B. Therefore, as shown in FIG. 4, the hydraulic control valve 31 indirectly controls a predetermined pilot pressure according to the control signal from the controller 30 through the shuttle valve 32 between the pilot lines 27B and 27B. 17. Further, as shown in FIG. 5, the hydraulic control valve 31 can directly apply a predetermined pilot pressure to the control valve 17 according to the control signal from the controller 30 through the pilot line 27B and the pilot line 27. can. Therefore, the controller 30 can cause the control valve 17 to supply the pilot pressure corresponding to the operation content of the electric operation device 26 from the hydraulic control valve 31, and can realize the operation of the excavator 100 based on the operator's operation.

Also, the controller 30 may control the hydraulic control valve 31, for example, to implement an automatic operation function. Specifically, the controller 30 outputs a control signal corresponding to an operation command related to the automatic operation function to the hydraulic control valve 31 regardless of whether or not the operation device 26 is operated. As a result, the controller 30 can cause the hydraulic control valve 31 to supply the control valve 17 with the pilot pressure corresponding to the operation command relating to the automatic operation function, thereby realizing the operation of the excavator 100 based on the automatic operation function.

Further, the controller 30 may control the hydraulic control valve 31 to realize remote control of the excavator 100, for example. Specifically, the controller 30 outputs to the hydraulic control valve 31 through the communication device 60 a control signal corresponding to the content of the remote operation designated by the remote operation signal received from the management device 200 . As a result, the controller 30 causes the hydraulic control valve 31 to supply the pilot pressure corresponding to the content of the remote operation to the control valve 17, thereby realizing the operation of the excavator 100 based on the operator's remote operation.

The shuttle valve 32 has two inlet ports and one outlet port, and outputs to the outlet port the hydraulic fluid having the higher pilot pressure among the pilot pressures input to the two inlet ports. The shuttle valve 32 is provided for each driven element (hydraulic actuator HA) to be operated by the operating device 26 . A shuttle valve 32 is provided for each moving direction of the driven element (for example, the raising direction and the lowering direction of the boom 4). One of the two inlet ports of the shuttle valve 32 is connected to the pilot line 27A on the secondary side of the operating device 26 (specifically, the above-described lever device and pedal device included in the operating device 26), and the other is connected to the pilot line 27A. It is connected to the pilot line 27B on the secondary side of the hydraulic control valve 31 . The outlet port of shuttle valve 32 is connected through pilot line 27 to the corresponding control valve pilot port of control valve 17 . The corresponding control valve is a control valve that drives the hydraulic actuator that is the object of operation of the above-described lever device or pedal device that is connected to one inlet port of the shuttle valve 32 . Therefore, these shuttle valves 32 correspond to the higher one of the pilot pressure of the pilot line 27A on the secondary side of the operating device 26 and the pilot pressure of the pilot line 27B on the secondary side of the hydraulic control valve 31. It can act on the pilot port of the control valve. That is, the controller 30 causes the hydraulic control valve 31 to output a pilot pressure higher than the pilot pressure on the secondary side of the operation device 26, thereby controlling the corresponding control valve regardless of the operator's operation of the operation device 26. be able to. Therefore, the controller 30 can control the operation of the driven elements (the lower traveling body 1, the upper revolving body 3, and the attachment AT) regardless of the operating state of the operating device 26 by the operator, and can realize the automatic driving function. .

The hydraulic control valve 33 is provided in a pilot line 27A that connects the operating device 26 and the shuttle valve 32 . The hydraulic control valve 33 is configured, for example, so that its flow passage area can be changed. The hydraulic control valve 33 operates according to control signals input from the controller 30 . Thereby, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26 when the operating device 26 is operated by the operator. Therefore, even when the operation device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator corresponding to the operation of the operation device 26 . Further, for example, even when the operating device 26 is being operated, the controller 30 reduces the pilot pressure output from the operating device 26 to be lower than the pilot pressure output from the hydraulic control valve 31. can be done. Therefore, the controller 30 controls the hydraulic control valve 31 and the hydraulic control valve 33 to apply a desired pilot pressure to the pilot port of the control valve in the control valve 17, regardless of the operation content of the operating device 26, for example. can work reliably. Therefore, by controlling the hydraulic control valve 33 in addition to the hydraulic control valve 31, the controller 30 can realize the automatic operation function and the remote control function of the excavator 100 more appropriately.

<User interface system>
As shown in FIG. 4 , the user interface system of the excavator 100 according to this embodiment includes an operation device 26 , an output device 50 and an input device 52 .

The output device 50 outputs various kinds of information to a user of the excavator 100 (for example, an operator of the cabin 10, a work vehicle around the excavator 100, etc.).

For example, the output device 50 includes lighting equipment, a display device 50A (see FIG. 6), and the like that output various information in a visual manner. The lighting equipment is, for example, a warning light or the like. The display device 50A is, for example, a liquid crystal display or an organic EL (Electroluminescence) display. The lighting equipment and the display device 50A may be provided inside the cabin 10, for example, and may output various kinds of information to an operator or the like inside the cabin 10 in a visual manner. Further, the lighting device and the display device 50A may be provided on the side surface of the upper revolving body 3, for example, and output various information to workers around the excavator 100 in a visual manner.

Also, for example, the output device 50 includes a sound output device 50B (see FIG. 6) that outputs various information in an auditory manner. The sound output device 50B includes, for example, a buzzer, a speaker, and the like. The sound output device 50B is provided, for example, at least one of the interior and exterior of the cabin 10, and outputs various types of information to the operator inside the cabin 10 and people (workers, etc.) around the excavator 100 in an auditory manner. you can

Also, for example, the output device 50 may include a device that outputs various information in a tactile manner such as vibration of the cockpit.

The input device 52 receives various inputs from the user of the excavator 100 , and signals corresponding to the received inputs are captured by the controller 30 . The input device 52 is provided, for example, inside the cabin 10 and receives input from an operator or the like inside the cabin 10 . Further, the input device 52 may be provided, for example, on the side surface of the upper revolving body 3 or the like, and may receive an input from a worker or the like around the excavator 100 .

For example, the input device 52 includes an operation input device that receives operation input. The operation input device may include a touch panel mounted on the display device, a touch pad installed around the display device, a button switch, a lever, a toggle, a knob switch provided on the operation device 26 (lever device), and the like. .

Also, for example, the input device 52 may include a voice input device that receives a user's voice input. Audio input devices include, for example, microphones.

Also, for example, the input device 52 may include a gesture input device that receives gesture input from the user. The gesture input device includes, for example, an imaging device that captures an image of a user's gesture.

Also, for example, the input device 52 may include a biometric input device that receives biometric input from the user. The biometric input includes, for example, input of biometric information such as a user's fingerprint and iris.

<Communication system>
As shown in FIG. 4 , the communication system of the excavator 100 according to this embodiment includes a communication device 60 .

The communication device 60 is connected to the communication line NW and communicates with a device provided separately from the excavator 100 (for example, the management device 200). Devices provided separately from the excavator 100 may include devices outside the excavator 100 as well as portable terminal devices brought into the cabin 10 by the user of the excavator 100 . The communication device 60 may include, for example, a mobile communication module conforming to standards such as 4G ( ^4th Generation) and 5G ( ^5th Generation). Communication device 60 may also include, for example, a satellite communication module. The communication device 60 may also include, for example, a WiFi communication module, a Bluetooth (registered trademark) communication module, and the like.

<Control system>
As shown in FIG. 4 , the control system of the excavator 100 according to this embodiment includes a controller 30 . Further, the control system of the excavator 100 according to the present embodiment includes an operation pressure sensor 29, an imaging device 40, an irradiation device 70, a boom angle sensor S1, an arm angle sensor S2, a bucket angle sensor S3, and an aircraft attitude. It includes a sensor S4 and a turning angle sensor S5.

The controller 30 performs various controls related to the excavator 100 .

The functions of the controller 30 may be implemented by any hardware, or any combination of hardware and software. For example, as shown in FIG. 4, the controller 30 includes an auxiliary storage device 30A, a memory device 30B, a CPU (Central Processing Unit) 30C, and an interface device 30D, which are connected via a bus B1.

The auxiliary storage device 30A is non-volatile storage means, stores programs to be installed, and stores necessary files, data, and the like. The auxiliary storage device 30A is, for example, a flash memory or the like.

The memory device 30B loads the program in the auxiliary storage device 30A so that the CPU 30C can read it, for example, when a program activation instruction is received. The memory device 30B is, for example, an SRAM (Static Random Access Memory).

The CPU 30C, for example, executes a program loaded into the memory device 30B and realizes various functions of the controller 30 according to instructions of the program.

The interface device 30D is used as an interface for connecting to a communication line inside the excavator 100, for example. The interface device 30D may include a plurality of different types of interface devices according to the type of communication line to be connected.

A program that implements various functions of the controller 30 is provided, for example, by a portable recording medium. In this case, the interface device 30D functions as an interface for reading data from the recording medium and writing data to the recording medium. The recording medium is, for example, a dedicated tool that is connected to a connector installed inside the cabin 10 with a detachable cable. Also, the recording medium may be, for example, a general-purpose recording medium such as an SD memory card or a USB (Universal Serial Bus) memory. Alternatively, the program may be downloaded from another computer (for example, the management device 200) outside the excavator 100 through a predetermined communication line and installed in the auxiliary storage device 30A.

Note that part of the functions of the controller 30 may be implemented by another controller (control device). In other words, the functions of the controller 30 may be distributed and implemented by a plurality of controllers. For example, a function related to image processing of an image captured by the imaging device 40, a function related to detection of objects around the excavator 100, and a function related to securing the safety of the excavator 100 may be implemented by different controllers. Functions related to image processing of captured images of the imaging device 40 include, for example, functions of a display processing unit 301 and an image correction unit 306, which will be described later. Functions related to detection of objects around the excavator 100 include, for example, functions of an object detection unit 302, a detection determination unit 307, and the like, which will be described later. The functions related to ensuring the safety of the excavator 100 include, for example, functions of the safety control unit 304, which will be described later. In addition, the function related to the detection of objects around the excavator 100 is distributed between a controller that implements a function related to image processing of the image captured by the imaging device 40 and a controller that implements a function related to ensuring the safety of the excavator 100. may be mounted in Specifically, among the functions related to the detection of objects around the excavator 100, there is a high relevance to image processing, such as the function of recognizing an object from the captured image of the imaging device 40 (the function of the object detection unit 302). Functionality may reside in the former controller. On the other hand, among the functions related to the detection of objects around the excavator 100, the function of finally determining whether or not a recognized object has been detected (the function of the detection determination unit 307) is used to ensure the safety of the excavator 100. More relevant functionality may reside in the latter controller.

The operation pressure sensor 29 detects the pilot pressure on the secondary side (pilot line 27A) of the hydraulic pilot type operation device 26, that is, the pilot pressure corresponding to the operation state of each driven element (hydraulic actuator) in the operation device 26. To detect. A pilot pressure detection signal corresponding to the operation state of each driven element (hydraulic actuator HA) in the operation device 26 by the operation pressure sensor 29 is taken into the controller 30 .

Incidentally, if the operating device 26 is of an electric type, the operating pressure sensor 29 is omitted. This is because the controller 30 can grasp the operating state of each driven element through the operating device 26 based on the operating signal received from the operating device 26 .

The imaging device 40 acquires an image around the excavator 100 for interpolating a blind spot or a location that is difficult for the operator to visually recognize. The output (captured image) of the imaging device 40 is taken into the controller 30 .

The imaging device 40 is, for example, a monocular camera, a stereo camera, a depth camera, or the like. In addition, based on the captured image, the imaging device 40 acquires three-dimensional data (for example, point cloud data or surface data) representing the position and outline of an object around the excavator 100 within a predetermined imaging range (angle of view). may

For example, as shown in FIGS. 1 and 2, the imaging device 40 includes a camera 40F for imaging the front of the upper rotating body 3, a camera 40B for imaging the rear of the upper rotating body 3, and an image to the left of the upper rotating body 3. and a camera 40R for imaging the right side of the upper rotating body 3 . As a result, the operator can visually recognize peripheral images, such as images captured by the cameras 40B, 40L, and 40R and processed images generated based on the captured images, through the display device 50A and the display device for remote control. You can see the left, right, and rear views. In addition, the operator can visually confirm the operation of the attachment AT including the bucket 6 by visually recognizing the image captured by the camera 40F and the peripheral image such as the processed image generated based on the captured image through the remote control display device. , the excavator 100 can be operated remotely. Hereinafter, the cameras 40F, 40B, 40L, and 40R may be collectively or individually referred to as a "camera 40X."

The imaging devices 40 (cameras 40X) each include an imaging device 41 and an image processing engine 42 . The camera 40X generates a captured image by the image processing engine 42 based on the output (electrical signal) of the image sensor 41, and outputs the generated image (captured image). The image processing engine 42 is, for example, a computer dedicated to image processing including a CPU, a memory device, an auxiliary storage device, etc. Various image processing is realized by executing a program installed in the auxiliary storage device on the CPU. do. Hereinafter, the function of generating a captured image of the image processing engine 42 will be referred to as a captured image generating function, and the captured image of the imaging device 40 means an image generated by the captured image generating function.

The two-dot chain line in FIG. 2 represents the angle of view (imaging range) of the cameras 40F, 40B, 40L, and 40R when viewed from above.

The irradiation device 70 irradiates the imaging range of the imaging device 40 with predetermined light under the control of the controller 30 . The predetermined light is, for example, visible light. Also, the predetermined light may be, for example, infrared light. As long as the entire imaging range of the imaging device 40 can be irradiated, the number of the irradiation devices 70 may be one or plural. For example, the irradiation device 70 is provided for each camera 40X and installed on the upper rotating body 3 so as to be close to the camera 40X.

Note that the irradiation device 70 may be omitted.

The boom angle sensor S1 acquires detection information regarding the attitude angle of the boom 4 (hereinafter referred to as "boom angle") with respect to a predetermined reference (for example, a horizontal plane or one of the two ends of the movable angle range of the boom 4). The boom angle sensor S1 may include, for example, a rotary encoder, an acceleration sensor, an angular velocity sensor, a hexaaxial sensor, an IMU (Inertial Measurement Unit), and the like. Also, the boom angle sensor S1 may include a cylinder sensor capable of detecting the telescopic position of the boom cylinder 7 .

The arm angle sensor S2 detects the posture angle of the arm 5 (hereinafter referred to as the "arm angle ”). Arm angle sensor S2 may include, for example, a rotary encoder, an acceleration sensor, an angular velocity sensor, a hexaaxial sensor, an IMU, or the like. Also, the arm angle sensor S2 may include a cylinder sensor capable of detecting the extension/retraction position of the arm cylinder 8 .

The bucket angle sensor S3 detects the attitude angle of the bucket 6 (hereinafter referred to as "bucket angle ”). Bucket angle sensor S3 may include, for example, a rotary encoder, an acceleration sensor, an angular velocity sensor, a hexaaxial sensor, an IMU, and the like. Also, the bucket angle sensor S3 may include a cylinder sensor capable of detecting the expansion/contraction position of the bucket cylinder 9 .

The machine body posture sensor S4 acquires detection information regarding the body posture of the excavator 100 including the lower running body 1 and the upper revolving body 3 . The machine body attitude sensor S4 is mounted on the upper revolving structure 3, for example, and detects information related to the tilt angle of the upper revolving structure 3 with respect to the horizontal plane and the attitude angle around the revolving axis (that is, the orientation of the upper revolving structure 3 with respect to the ground). to get The airframe attitude sensor S4 may include, for example, an acceleration sensor (tilt sensor), an angular velocity sensor, a hexaaxial sensor, an IMU, and the like.

The turning angle sensor S5 acquires detection information regarding the turning angle of the upper turning body 3 with respect to the lower traveling body 1 (that is, the orientation of the upper turning body 3). The turning angle sensor S5 includes, for example, a potentiometer, rotary encoder, resolver, and the like.

Also, for example, the excavator 100 may include a positioning device capable of positioning its own absolute position. The positioning device is, for example, a GNSS (Global Navigation Satellite System) sensor. Thereby, the estimation accuracy of the posture state of the excavator 100 can be improved.

Also, for example, the excavator 100 may include a distance sensor that detects the distance to objects around the excavator 100 in addition to the imaging device 40 . Distance sensors include, for example, LIDAR (Light Detecting and Ranging), millimeter wave radar, ultrasonic sensors, infrared sensors, distance image sensors, and the like. Thereby, the controller 30 can detect an object around the excavator 100 using, for example, the output of the distance sensor in addition to the output of the imaging device 40 .

A part or all of the boom angle sensor S1, the arm angle sensor S2, the bucket angle sensor S3, the body attitude sensor S4, and the turning angle sensor S5 may be omitted. This is because, for example, when the remote control function and the automatic driving function are not employed, it may not be necessary to estimate the attitude state of the attachment AT of the excavator 100 or the machine body (upper rotating body 3). This is also because, for example, it may be possible to estimate the posture state of the excavator 100 from information around the excavator 100 acquired by the imaging device 40, a distance sensor described later, or the like. Specifically, information about the surroundings of the excavator 100 acquired by the imaging device 40, a distance sensor described later, or the like includes information about the positions and shapes of surrounding objects and attachments viewed from the machine body (upper rotating body 3). may be included. In this case, the controller 30 can estimate the attitude state of the attachment AT and the body (upper rotating body 3) from the information, depending on the accuracy required.

[Hardware configuration of management device]
Next, a hardware configuration of the management device 200 will be described with reference to FIG.

FIG. 5 is a diagram showing an example of the hardware configuration of the management device 200 according to this embodiment.

The functions of the management device 200 are realized by arbitrary hardware or a combination of arbitrary hardware and software. For example, as shown in FIG. 5, the management device 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU 204, a communication interface 206, an input device 207, and a display device 208, which are connected via a bus B2. .

The external interface 201 functions as an interface for reading data from the recording medium 201A and writing data to the recording medium 201A. The recording medium 201A includes, for example, a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disc), a BD (Blu-ray (registered trademark) Disc), an SD memory card, a USB memory, and the like. As a result, the management device 200 can read various data used in processing through the recording medium 201A, store the data in the auxiliary storage device 202, and install programs that implement various functions.

Note that the management device 200 may acquire various data and programs from an external device through the communication interface 206 .

The auxiliary storage device 202 stores various installed programs, as well as files and data required for various processes. The auxiliary storage device 202 includes, for example, an HDD (Hard Disc Drive), SSD (Solid State Drive), flash memory, and the like.

The memory device 203 reads out and stores the program from the auxiliary storage device 202 when a program activation instruction is received. The memory device 203 includes, for example, a DRAM (Dynamic Random Access Memory) and an SRAM (Static Random Access Memory).

The CPU 204 executes various programs loaded from the auxiliary storage device 202 to the memory device 203, and implements various functions related to the management device 200 according to the programs.

The high-speed arithmetic device 205 works in conjunction with the CPU 204 and performs arithmetic processing at a relatively high speed. The high-speed arithmetic unit 205 includes, for example, a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), and the like.

Note that the high-speed arithmetic unit 205 may be omitted depending on the required arithmetic processing speed.

A communication interface 206 is used as an interface for communicably connecting to an external device. Thereby, the management device 200 can communicate with an external device such as the excavator 100 through the communication interface 206 . Also, the communication interface 206 may have a plurality of types of communication interfaces depending on the communication method with the connected device.

The input device 207 receives various inputs from the user. For example, the input device 207 includes an input device (operating device for remote control) for remote control by the operator.

The input device 207 includes, for example, an operation input device that receives mechanical operation input from the user. Operation input devices include, for example, buttons, toggles, levers, and the like. Further, the operation input device includes, for example, a touch panel mounted on the display device 208, a touch pad provided separately from the display device 208, and the like.

Also, the input device 207 includes, for example, a voice input device capable of accepting voice input from the user. The voice input device includes, for example, a microphone capable of collecting the user's voice.

Also, the input device 207 includes, for example, a gesture input device capable of accepting gesture input from the user. The gesture input device includes, for example, a camera capable of capturing an image of a user's gesture.

Also, the input device 207 includes, for example, a biometric input device capable of accepting biometric input from the user. A biometric input device includes, for example, a camera capable of acquiring image data containing information about a user's fingerprint or iris.

The display device 208 displays information screens and operation screens for the user. For example, display device 208 may include the remote display device described above. The display device 208 is, for example, a liquid crystal display or an organic EL (Electroluminescence) display.

[Example of excavator functional configuration]
Next, an example of the functional configuration of the excavator 100 will be described with reference to FIG. 6 .

FIG. 6 is a functional block diagram showing an example of the functional configuration of the shovel 100 according to this embodiment.

As shown in FIG. 6, the controller 30 includes a display processing unit 301, an object detection unit 302, a position estimation unit 303, a safety control unit 304, and an irradiation control unit 305 as functional units. Also, the controller 30 may include an image correction unit 306 as a functional unit. The functions of the display processing unit 301, the object detection unit 302, the position estimation unit 303, the safety control unit 304, and the irradiation control unit 305 are such that, for example, a program installed in the auxiliary storage device 30A is loaded into the memory device 30B and executed on the CPU 30C. It is realized by executing

The display processing unit 301 causes the display device 50A inside the cabin 10 to display a peripheral image based on the captured image input from the imaging device 40 .

The display processing unit 301 may display a peripheral image corresponding to the imaging range of any one of the cameras 40F, 40B, 40L, and 40R on the display device 50A, or may display any one of the cameras 40F, 40B, 40L, and 40R. Alternatively, the imaging ranges of two or more cameras may be displayed on the display device 50A. For example, the display processing unit 301 displays on the display device 50A a peripheral image including at least the imaging range of the cameras 40B and 40R among the cameras 40F, 40B, 40L and 40R. This is because the rear and right sides of the upper rotating body 3, which correspond to the imaging ranges of the cameras 40B and 40R, tend to be blind spots when viewed from the operator of the cabin 10. FIG.

The object detection unit 302 (an example of a detection unit) detects a predetermined object (hereinafter referred to as “monitoring object”) among the monitoring objects around the excavator 100 based on the output (captured image) of the imaging device 40 . Specifically, the object detection unit 302 may recognize the monitored object from the captured image of the imaging device 40 and specify the position (area) where the monitored object is captured. Further, the object detection unit 302 recognizes a monitoring object from an image (hereinafter referred to as a “corrected captured image”) after the captured image of the imaging device 40 (camera 40X) has been corrected by the image correction unit 306, and detects the monitoring object. You may specify the position (area) where the object is reflected. In other words, detection of a monitored object means recognizing a monitored object appearing in an image captured by the imaging device 40 and specifying a position (region) in an input image containing the monitored object. good. Hereinafter, an image input to the object detection unit 302 (a captured image of the output itself of the imaging device 40 or a corrected captured image output from the image correction unit 306) may be referred to as an "input image" for convenience. .

Observed objects may include people such as workers working around the excavator 100 and supervisors at work sites. Objects to be monitored may also include any objects (obstacles) other than people on the work site. Obstacles other than people at the work site include, for example, specific terrain such as holes, ditches, and mountains of earth and sand, road cones, fences, utility poles, temporary materials, temporary offices at the work site, etc. That is, obstacles that cannot move by themselves) may be included. Obstacles other than people at the work site may include, for example, movable obstacles such as other work machines and work vehicles. The number of types of monitored objects to be detected by the object detection unit 302 may be one or plural. The following description will focus on the case where the object to be monitored is a person.

The details of the object detection method by the object detection unit 302 will be described later.

The function of the object detection unit 302 may be switched between ON (enabled) and OFF (disabled) according to a predetermined input by an operator or the like to the input device 52 . Alternatively, the function of the object detection unit 302 may be transferred to the camera 40X, and the detection result may be transmitted to the controller 30. FIG. Alternatively, the function of the object detection unit 302 may be transferred to the display device 50A, and the detection result may be transferred to the controller 30. FIG. In this case, the image captured by camera 40X is directly input to display device 50A. Therefore, in this case, the function of the display processing unit 301 may also be transferred to the display device 50A. Also, the function of the object detection unit 302 may be transferred to the imaging device 40 . In this case, the function of the object detection unit 302 is divided into object detection units that detect monitored objects based on the images captured by the cameras 40F, 40B, 40L, and 40R, respectively. (See FIG. 25, which will be described later).

When the object detection unit 302 detects a monitored object from the captured image of the imaging device 40, the position estimation unit 303 monitors the position where the detected monitored object actually exists, that is, the surroundings of the excavator 100. Estimate the position of an object (hereinafter, “actual position”).

Specifically, the position estimation unit 303 estimates the actual position of the monitored object based on the detected position (detection area) of the monitored object in the captured image specified by the object detection unit 302 . Also, when a plurality of monitored objects are detected by the object detection unit 302, the position estimation unit 303 estimates the actual position of each of the plurality of monitored objects.

For example, if the camera 40X is a monocular camera, the position estimating unit 303 identifies a reference point (detection position) in the detection area of the monitored object in the input image, and based on the horizontal position of the reference point in the input image. , the direction viewed from the excavator 100 (upper revolving body 3). Further, the position estimation unit 303 determines the distance from the excavator 100 to the monitored object (specifically, the distance from the excavator 100 to the monitored object based on the size of the detection area of the monitored object in the input image (for example, the size in the vertical direction). Estimate the distance in the direction along the work plane where it is located. This is because there is a correlation that the size of the recognized monitored object on the input image decreases as the monitored object moves away from the excavator 100 (upper revolving body 3). Specifically, since a monitored object has an assumed size range (for example, an assumed range of human height), from the excavator 100 of the monitored object included in the assumed size range, A correlation between the viewing position and the size on the input image can be predefined. For this reason, the object detection unit 302 uses, for example, a map, a conversion formula, or the like, which is stored in advance in the auxiliary storage device 30A or the like, to represent the correlation between the size of the monitored object on the input image and the distance seen from the upper swing structure 3. Based on this, the actual position of the recognized surveillance object can be estimated. Therefore, the position estimating unit 303 can estimate the recognized position of the monitored object by estimating the direction and distance of the monitored object as seen from the excavator 100 (upper rotating body 3).

Further, for example, when the camera 40X is a monocular camera, the position estimating unit 303, on the premise that the monitored object exists on the same plane as the lower traveling body 1, performs projective transformation of the input image onto the plane, and so on. A real location may be estimated. Specifically, the position estimating unit 303 identifies a reference point (detection position) representing the contact point of the monitored object with the ground, and determines the installation position of the camera 40X with respect to the upper revolving body 3 and The actual position of the monitored object may be calculated by projective transformation defined according to the installation angle. In this case, a certain portion (for example, a certain pixel) constituting the input image is associated one-to-one with a certain position on the same plane as the excavator 100 (undercarriage 1).

Further, for example, when the camera 40X is a stereo camera, the actual position of the monitored object is estimated based on the deviation (parallax) of the reference position in the detection area of the monitored object for each of the two captured images.

A safety control unit 304 (an example of a control unit) performs control related to functional safety of the excavator 100 .

The safety control unit 304 activates the safety function, for example, when the object detection unit 302 detects a monitored object within a predetermined range around the excavator 100 . Specifically, the safety control unit 304 may activate the safety function when the actual position (estimated value) of the monitored object estimated by the position estimation unit 303 is within a predetermined range around the excavator 100. .

The safety function includes, for example, a notification function that outputs an alarm or the like to at least one of the inside of the cabin 10, the outside of the cabin 10, and the remote operator or monitor of the excavator 100 to notify the detection of the monitored object. can be As a result, an operator inside the cabin 10, a worker around the excavator 100, an operator or an observer who performs remote operation or remote monitoring of the excavator 100, or the like, is notified that there is an object to be monitored in the surveillance area around the excavator 100. You can call attention to what you are doing. Hereinafter, the notification function to the inside (operator, etc.) of the cabin 10 is "internal notification function", the notification function to the outside (workers, etc.) of the excavator 100 is "external notification function", and the remote operation and remote monitoring of the excavator 100 are described. The notification function to the operator or the monitor who performs the remote notification is sometimes referred to as a "remote notification function" and may be distinguished.

Further, the safety function may include, for example, an operation restriction function that restricts the operation of the excavator 100 in response to an operation command corresponding to the operation of the operation device 26, remote operation, or automatic driving function. As a result, the operation of the excavator 100 can be forcibly restricted, and the possibility of the excavator 100 approaching or contacting surrounding objects can be reduced. The motion restriction function may include a motion deceleration function that slows down the motion speed of the excavator 100 in response to an operation command corresponding to the operation of the operating device 26, the remote control, or the automatic driving function. In addition, the operation restriction function includes an operation stop function that stops the operation of the excavator 100 and maintains the stopped state regardless of operation commands corresponding to the operation of the operating device 26, remote operation, or automatic operation function. good too.

For example, the safety control unit 304 activates the notification function when the object detection unit 302 detects a monitored object in a predetermined range around the excavator 100 (hereinafter, “report range”). The notification range is, for example, a range in which the distance D from a predetermined portion of the shovel 100 is equal to or less than the threshold value Dth1. The predetermined portion of the excavator 100 is, for example, the upper revolving body 3 . Also, the predetermined portion of the shovel 100 may be, for example, the bucket 6 or the hook at the tip of the attachment AT. The threshold value Dth1 may be constant regardless of the direction viewed from the predetermined portion of the excavator 100, or may vary depending on the direction viewed from the predetermined portion of the excavator 100. FIG.

The safety control unit 304, for example, controls the sound output device 50B to activate an internal notification function and an external notification function using sound (that is, auditory method) for at least one of the inside and outside of the cabin 10. At this time, the safety control unit 304 may vary the pitch, sound pressure, tone color of the sound to be output, the sounding period when the sound is sounded periodically, the content of the sound, etc., according to various conditions. .

Safety control unit 304 also activates an internal notification function, for example, by visual means. Specifically, the safety control unit 304 controls the display device 50A inside the cabin 10 through the display processing unit 301, so that the display device 50A displays an image indicating that the monitored object is being detected together with the peripheral image. can be displayed. In addition, the safety control unit 304, through the display processing unit 301, detects the monitored object appearing in the peripheral image displayed on the display device 50A inside the cabin 10 and the position on the peripheral image corresponding to the detected monitored object. may be emphasized. More specifically, the safety control unit 304 superimposes a frame surrounding the detected monitored object on the peripheral image displayed on the display device 50A inside the cabin 10, or displays the detected monitored object. A marker may be superimposed and displayed at a position on the peripheral image corresponding to the actual position of the monitored object. Thereby, the display device 50A can realize a visual notification function for the operator. Further, the safety control unit 304 may use a warning light, a lighting device, or the like inside the cabin 10 to notify the operator or the like inside the cabin 10 that the monitored object has been detected.

In addition, the safety control unit 304 controls the output device 50 (for example, a lighting device such as a headlight or a display device 50A) provided on the side surface of the housing portion of the upper rotating body 3, for example, to provide a visual An external notification function by the method may be activated. The safety control unit 304 may also activate the internal notification function in a tactile manner, for example, by controlling a vibration generator that vibrates the cockpit on which the operator sits. As a result, the controller 30 can notify the operator, workers and supervisors around the excavator 100 that an object to be monitored (for example, a person such as a worker) exists in a relatively close place around the excavator 100. can be recognized. Therefore, the controller 30 can prompt the operator to check the safety situation around the excavator 100, and can prompt the worker or the like in the monitoring area to evacuate from the monitoring area.

Further, the safety control unit 304 may activate the remote notification function by transmitting a command signal indicating activation of the notification function to the management device 200 via the communication device 60, for example. In this case, when the command signal is received from the excavator 100 through the communication interface 206, the management device 200 may output a warning visually or audibly through the display device 208 or the like. As a result, an operator or a monitor who performs remote operation or remote monitoring of the excavator 100 through the management device 200 can recognize that a monitored object has entered the notification range around the excavator 100 .

Note that the remote notification function of the safety control unit 304 may be transferred to the management device 200 . In this case, the management device 200 receives, from the excavator 100 , information about the detection status of the monitored object by the object detection unit 302 and the estimation result of the actual position of the monitored object by the position estimation unit 303 . Based on the received information, the management device 200 determines whether or not the object to be monitored enters the notification range, and activates the external notification function when the object to be monitored exists within the notification range.

In addition, the safety control unit 304 may vary the notification mode (that is, the notification method) according to the positional relationship between the monitored object detected within the notification range and the upper rotating body 3 .

For example, when the monitored object detected by the object detection unit 302 within the reporting range is located at a position relatively far from a predetermined portion of the excavator 100, the safety control unit 304 may warn of the monitored object. may output a relatively low-urgency warning (hereinafter, “caution-level warning”). Hereinafter, the range in which the distance from the predetermined part of the excavator 100 is relatively long, ie, the range corresponding to the caution level warning, may be referred to as the "caution notification range" for convenience. On the other hand, when the monitored object detected by the object detection unit 302 within the reporting range is relatively close to a predetermined portion of the excavator 100 , the safety control unit 304 determines that the monitored object is located at a predetermined position of the excavator 100 . A relatively high-urgency alarm (hereinafter referred to as "warning level alarm") may be output to notify that the site is approaching and the degree of danger is increasing. Hereinafter, a range in which the distance to a predetermined part of the excavator 100 is relatively close, ie, a range corresponding to an alert level warning, may be referred to as an "alert alert range".

In this case, the safety control unit 304 may vary the pitch, sound pressure, tone, sounding period, etc. of the sound output from the sound output device 50B between the caution level warning and the caution level warning. In addition, the safety control unit 304 controls the image indicating that the monitored object is being detected displayed on the display device 50A and the peripheral image displayed on the display device 50A between the caution level warning and the caution level warning. The color, shape, size, presence/absence of blinking, blinking cycle, etc. of the monitored object on the image or the image (for example, frame, marker, etc.) that emphasizes the position of the monitored object may be varied. As a result, the controller 30 informs the operator or the like of the degree of urgency, in other words, the excavator 100 of the object to be monitored, according to the difference between the notification sound (warning sound) output from the sound output device 50B and the notification image displayed on the display device 50A. It is possible to grasp the degree of proximity to a predetermined site of.

The safety control unit 304 may stop the notification function when the monitored object detected by the object detection unit 302 is no longer detected within the notification range after the operation of the notification function is started. Further, the safety control unit 304 may stop the notification function when a predetermined input for canceling the operation of the notification function is received through the input device 52 after the activation of the notification function is started.

Also, the safety control unit 304 activates the operation restriction function, for example, when the object detection unit 302 detects a monitored object within a predetermined range (hereinafter referred to as “operation restriction range”) around the excavator 100 . The operation limit range is set, for example, to be the same as the notification range described above. Also, the operation restriction range may be set to a range in which the outer edge is relatively closer to a predetermined portion of the excavator 100 than the notification range, for example. As a result, for example, when the monitored object enters the notification range from the outside, the safety control unit 304 first activates the notification function, and then when the monitored object enters the inner operation restriction range, the operation restriction function is activated. can be activated. Therefore, the controller 30 can activate the notification function and the operation restriction function step by step in accordance with the inward movement of the monitored object within the monitored area.

Specifically, the safety control unit 304 activates the operation restriction function when the monitored object is detected within the operation restriction range in which the distance D from the predetermined portion of the excavator 100 is within the threshold value Dth2 (≦Dth1). good. The threshold value Dth2 may be constant regardless of the direction viewed from the predetermined portion of the excavator 100, or may vary depending on the direction viewed from the predetermined portion of the excavator 100. FIG.

In addition, the operation limitation range includes an operation deceleration range in which the operation speed of the excavator 100 is slower than normal in response to an operation command corresponding to the operation of the operation device 26, remote operation, and automatic driving function, and an operation deceleration range of operation of the operation device 26, remote operation. , and at least one of an operation stop range in which the operation of the excavator 100 is stopped and the stop state is maintained regardless of the operation command corresponding to the automatic operation function. For example, when the motion deceleration range and the motion stop range are included in the motion limit range, the motion stop range is a range close to a predetermined portion of the excavator 100 within the motion limit range. The motion deceleration range is a range set outside the motion stop range in the motion limit range.

The safety control unit 304 operates an operation restriction function that restricts the operation of the excavator 100 by controlling the hydraulic control valve 31 . In this case, the safety control unit 304 may limit the operation of all driven elements (that is, the corresponding hydraulic actuators), or may limit the operation of some driven elements (hydraulic actuators). . As a result, the controller 30 can decelerate or stop the operation of the excavator 100 when there is an object to be monitored around the excavator 100 . Therefore, the controller 30 can suppress the occurrence of contact between the monitored objects around the excavator 100 and the excavator 100 and the suspended load. Further, the safety control unit 304 may operate an operation limiting function (operation stopping function) by controlling an electromagnetic switching valve (not shown) of the pilot line 25 to shut off the pilot line 25 .

Further, the safety control unit 304 may stop the operation restriction function when the monitored object detected by the object detection unit 302 is no longer detected within the operation restriction range after the activation of the operation restriction function. Further, the safety control unit 304 may stop the operation restriction function when a predetermined input for canceling the operation of the operation restriction function is received through the input device 52 after the operation of the operation restriction function is started. The content of the input to the input device 52 for deactivation of the notification function and the content of the input for deactivation of the motion restriction function may be the same or different.

Further, the function of the safety control unit 304 may be switched between ON (enabled) and OFF (disabled) according to a predetermined input by an operator or the like to the input device 52 .

The irradiation control unit 305 (an example of a determination unit) controls the irradiation device 70 .

Incidentally, as described above, when the irradiation device 70 is omitted, the irradiation control section 305 is also omitted as a matter of course.

The image correction unit 306 performs predetermined correction on the output (captured image) of the imaging device 40 (camera 40X) and outputs the corrected captured image to the object detection unit 302 .

[Outline of object detection method]
Next, with reference to FIGS. 7 to 15, an outline of an object detection method by the object detection section will be described.

FIG. 7 is a diagram for explaining a specific example of the object detection method. 8 to 13 are diagrams showing first to sixth examples (images 800 to 1300) of images of teacher data. Specifically, FIGS. 8 to 13 show images captured by a camera of the same type as the camera 40X installed at the same position of an excavator of the same model as the excavator 100. On the front side (lower end) of the captured image, The side surface of the upper revolving body 3 is shown. 14 and 15 are diagrams showing an example and another example of the worker W. FIG.

The object detection unit 302 detects the monitored object from the image captured by the camera 40X by simply applying image processing techniques such as shape detection and pattern recognition (template matching).

Further, as shown in FIG. 7, the object detection unit 302 detects the monitored object from the captured image of the camera 40X, for example, by applying machine learning in addition to the image processing technology. Specifically, the object detection unit 302 uses a machine-learned model LM to determine the features of the monitored object appearing in the input image, from the input image (the captured image of the camera 40X or its corrected captured image), A rectangular frame (hereinafter referred to as a “detection frame”) representing the area in which the monitored object is captured and a label representing the type of the monitored object are output. Labels include, for example, a label indicating that no monitored object exists and a label set for each type of monitored object. Only one label may be set for each type of monitored object, or a plurality of labels may be set as described later. Also, when there are multiple types of monitored objects, the single trained model LM is configured to output labels for all types of monitored objects, that is, to detect all types of monitored objects. Alternatively, a plurality of trained models LM capable of detecting only some types of all types of monitored objects may be provided. For example, a trained model LM exists for each type of monitored object, and a label for each trained model LM indicates that a certain type of monitored object exists and a label that indicates that a monitored object of that type does not exist. It may be composed only of the label to indicate.

A trained model LM is generated by applying supervised learning to the base learning model. Specifically, a trained model LM is obtained by subjecting a learning model based on machine learning to a collection of teacher data (teacher data set) by combining an image as an input and a correct answer (detection frame and label) as an output. is generated. Also, the trained model LM may be generated (updated) by additionally learning a new teacher data set to the existing trained model LM. As an input image included in the training data set, naturally, both an image including (showing) the monitored object and an image not including (not showing) the monitored object are adopted.

The learned model LM is generated, for example, by an external device such as the management device 200, and is written from a predetermined recording medium to the auxiliary storage device 30A through the interface device 30D when the excavator 100 is manufactured. Also, the learned model LM may be downloaded from an external device such as the management device 200 to the excavator 100 via a predetermined communication line and registered in the auxiliary storage device 30A of the controller 30 . As a result, the object detection unit 302 can detect the monitored object using the learned model LM registered in the auxiliary storage device 30A.

Also, the learned model LM may be updated by installing update data in the auxiliary storage device 30A from a predetermined recording medium through the interface device 30D. Also, the learned model LM may be updated by downloading update data from an external device such as the management device 200 to the excavator 100 through a predetermined communication line and installing the updated data in the auxiliary storage device 30A. Thereby, the object detection unit 302 can detect the monitored object using the latest updated learned model LM.

For example, the object detection unit 302 detects the surveillance object from the input image using a support vector machine (SVM) that has undergone machine learning of the tendency of the image feature amount of the surveillance object in the image. In this case, the trained model LM includes, as pre-processing, a processing unit that extracts image feature amounts from the image captured by the camera 40X. The image feature quantity is, for example, a HOG (Histogram of Oriented Gradients) feature quantity.

Further, for example, the object detection unit 302 detects a monitored object from an input image using machine learning using a deep neural network (DNN), that is, a trained model LM based on deep learning. . Specifically, the object detection unit 302 may detect a monitored object from an input image using a trained model LM based on deep learning using a convolutional neural network (CNN). The CNN is configured by connecting multiple combinations of convolution layers that perform convolution processing and pooling layers that perform pooling processing using activation functions. conduct. The activation function is, for example, ReLU (Rectified Linear Unit). As a result, the trained model LM can handle the input image as it is without preprocessing.

For example, the object detection unit 302 uses the CNN-based trained model LM to generate regions of candidates for monitored objects from the input image, and classifies these candidates into labels to detect monitored objects. . That is, the CNN-based trained model LM treats, for example, the detection of a monitored object from an input image as a classification problem, generates regions of candidate monitored objects from the image captured by the camera 40X, and labels these candidates It may be a classification model that classifies into Classification models are, for example, R (Region-based)-CNN and its derivatives (Fast R-CNN, Faster R-CNN, etc.).

Further, for example, the object detection unit 302 detects a monitored object by simultaneously recognizing the monitored object and specifying its position (area) from the input image using a CNN-based trained model LM. do. That is, the CNN-based trained model LM, for example, treats the detection of a monitored object from an image captured by the camera 40X as a regression problem, and simultaneously recognizes and identifies the position (area) of the monitored object from the input image. It may be a regression model of the aspect to do. The regression model is, for example, YOLO (You Only Look Once) or SSD (Single Shot Detector).

The learned model LM is generated, for example, by performing machine learning so that the base learning model or the existing learned model LM can detect surveillance objects in different postures.

For example, as shown in FIG. 8, a teacher data image 800 shows a worker W in an upright posture facing the camera. Further, as shown in FIG. 9, an image 900 of the training data shows a worker W who faces the camera and is in a state of crouching. Further, as shown in FIG. 10, an image 1000 of the training data shows the worker W in a crouching posture facing the camera. The trained model LM is machine-learned using a teacher data set including images showing monitored objects with different postures, such as images 800 to 1000, so that monitored objects with different postures (in this example, human ) can be detected from the input image.

Also, the trained model LM may be generated, for example, by performing machine learning so that the base learning model or the existing trained model LM can detect surveillance objects in different directions.

For example, as shown in FIG. 11, an image 1100 of training data shows a worker W in an upright posture facing sideways (to the right) with respect to the camera. In addition, as shown in FIG. 12, an image 1200 of the training data shows the worker W in a state of crouching with his back facing the camera. Further, as shown in FIG. 13, an image 1300 of training data shows a worker W in a squatting posture facing sideways (to the left) with respect to the camera. The learned models LM are machine-learned using a teacher data set including images showing monitored objects in different directions, such as images 800 and 1100, images 900 and 1200, and images 1000 and 1300, so that they differ from each other. The orientation of the monitored object can be detected from the input image.

Also, the trained model LM may be generated, for example, by performing machine learning so that the base learning model or the existing trained model LM can detect surveillance objects that differ in at least one of orientation and orientation from each other. . Specifically, the trained model LM is machine-learned using a teacher data set including images showing monitored objects with different orientations and orientations, such as images 800 to 1300, so that the orientations and orientations of the learned models LM are machine-learned. Different monitored objects can be detected from the input image.

In addition, when the object to be monitored is a person, the learned model LM can detect (recognize) a person who wears clothing that is worn relatively frequently by workers in the vicinity of the excavator 100 and the clothing worn by the worker. It may be generated by machine learning as follows. As a result, the controller 30 can appropriately recognize the characteristics of the wearable item, detect the person and the wearable item, and The existence of a person on the shovel 100 can be grasped.

For example, as shown in FIG. 14, the learned model LM may be generated by machine learning to detect (recognize) a person wearing a helmet (worker W) and the helmet HMT worn by the person. good. Specifically, the trained model LM is machine-learned using a teacher data set containing a large number of images showing a worker W wearing a helmet HMT. It can be detected from the input image.

Further, for example, as shown in FIG. 15, the learned model LM detects (recognizes) a person (worker W) wearing the high-visibility safety clothing RV and the high-visibility safety clothing RV worn by the person (worker W). may be generated by machine learning. The high-visibility safety clothing RV is configured to have relatively high visibility from the wearer's surroundings. For example, a retroreflective material is attached to the high-visibility safety clothing RV. For the high-visibility safety clothing RV, for example, fluorescent fabric colored with fluorescent yellow or fluorescent green is used. Specifically, the learned model LM is machine-learned using a teacher data set that includes many images of a worker W wearing high-visibility safety clothing RV. It is possible to detect a person who has been injured and the high-visibility safety clothing they are wearing from the input image.

Further, for example, as shown in FIG. 15, the learned model LM includes a person (worker W) who wears both the helmet HMT and the high-visibility safety clothing RV, and the high-visibility safety clothing RV and helmet worn by the person (worker W). It may be machine-learned to detect (recognize) both HMTs. In this case, the trained model LM may be capable of outputting both a label representing the presence of the helmet and a label representing the presence of the high-visibility safety clothing. Specifically, the trained model LM is machine-learned using a teacher data set including a large number of images showing a worker W wearing a helmet HMT and high-visibility safety clothing RV. It is possible to detect a person wearing a safety suit and the helmet and high-visibility safety suit worn by the person from the input image.

In addition, the teacher data set includes, for example, an image in which a monitored object is captured by a camera of the same model as the camera 40X. In addition, as shown in FIGS. 8 to 13, for example, the training data set includes images taken by a camera of the same model as the camera 40X, which is installed at approximately the same position and in approximately the same posture as the excavator of the same model as the excavator 100. Also, an image in which the monitored object is shown may be included. As a result, the trained model LM is machine-learned so that the monitored object can be detected from the image captured by the camera 40X, taking into account how the monitored object appears in the image captured by the camera 40X. Therefore, the object detection unit 302 can detect the monitored object with higher accuracy.

In addition, the teacher data set includes, for example, images in which a monitored object appears in each of a plurality of types of backgrounds different from each other. For example, multiple types of backgrounds include dirt grounds, asphalt grounds, forests, houses, urban buildings, and the like. As a result, the trained model LM is machine-learned so that the monitored object can be detected from the input image, taking into account how the monitored object looks due to different backgrounds. Therefore, the object detection unit 302 can detect the monitored object with higher accuracy.

In addition, the teacher data set includes, for example, images in which a monitored object is captured, each of which is captured in a plurality of different time periods. The plurality of time periods include, for example, morning time period (for example, 6:00 to 10:00), afternoon time period (for example, 10:00 to 15:00), evening time period (for example, 16:00 to 19:00). , and nighttime hours (eg, 19:00 to 6:00 the next day). As a result, the trained model LM is machine-learned so that the monitored object can be detected from the input image, taking into consideration how the monitored object looks depending on the time period. Therefore, the object detection unit 302 can detect the monitored object with higher accuracy.

In addition, the teacher data set includes, for example, images showing monitored objects, which are captured under different lighting conditions during night hours. The illumination state includes, for example, the level of illuminance of illumination, the color of illumination, and the like. As a result, the trained model LM is machine-learned so that the monitored object can be detected from the input image, taking into account how the monitored object looks under different lighting conditions. Therefore, the object detection unit 302 can detect the monitored object with higher accuracy.

In addition, the training data set includes, for example, images in which surveillance objects are captured under different types of weather conditions. Multiple types of weather conditions include, for example, sunny, cloudy, rainy, snowy, and the like. As a result, the trained model LM is machine-learned so that the monitored object can be detected from the input image, taking into consideration how the monitored object looks depending on the weather. Therefore, the object detection unit 302 can detect the monitored object with higher accuracy.

In addition, the teacher data set includes, for example, a plurality of images in which a monitored object is shown, with different positional relationships between the light source and the imaging range. The light source is, for example, the sun or night lighting. The plurality of images includes images in which the positional relationship between the light source and the imaging range corresponds to front light, semi-front light, side light, backlight, and the like. Also, the plurality of images includes, for example, images with different heights (elevation angles) from the ground of the sun under the same conditions such as front light, side light, and backlight. As a result, the training data set is machine-learned so that the monitored object can be detected from the input image, taking into consideration how the monitored object looks due to the difference in the position of the light source. Therefore, the object detection unit 302 can detect the monitored object with higher accuracy.

[Specific example of object detection method]
Next, a specific example of a monitoring object detection method by the object detection unit 302 will be described with reference to FIGS. 16 to 18. FIG.

<First example>
In this example, the object detection unit 302 detects people (workers) around the excavator 100 by detecting (recognizing) high-visibility safety clothing from the input image. This makes it easier for the object detection unit 302 to detect a person, for example, even from an input image in which it is difficult to distinguish between the work clothes of people around the excavator 100 and the background. It can be detected with high accuracy.

Specifically, the object detection unit 302 may detect (recognize) a person wearing high-visibility safety clothing from the input image, or detect (recognize) the high-visibility safety clothing itself worn by the person. recognition), or both of them may be detected (recognized). Accordingly, the object detection unit 302 can determine that a person as a monitoring object has been detected when a person wearing high-visibility safety clothing or a high-visibility safety clothing (worn by a person) is detected.

For example, the object detection unit 302 recognizes at least one of the shape and color (e.g., fluorescent yellow or fluorescent green) of the high-visibility safety clothing, and the brightness of the retroreflective material, thereby detecting the presence of people around the excavator 100 . to detect Specifically, the object detection unit 302 detects the shape and color of the high-visibility safety clothing and the brightness of the retroreflective material by applying shape detection, pattern recognition, and the like. , and the brightness of the retroreflector. In addition, as described above, the object detection unit 302 uses a trained model LM based on a teacher data set containing many images of people (workers) wearing high-visibility safety clothing to achieve high-visibility At least one of the shape and color of the safety clothing and the brightness of the retroreflector may be recognized.

Further, in this example, the imaging device 40 (camera 40X) may acquire a captured image of the periphery of the excavator 100 in a state where visible light is emitted from the irradiation device 70 . In this case, under the control of the controller 30 (irradiation control unit 305), the irradiation device 70 may always irradiate visible light while the camera 40X is in operation (while the power is on), or may be synchronized with the imaging timing of the camera 40X. and visible light may be irradiated. As a result, when there is a person (worker) wearing high-visibility safety clothing in the imaging range of the imaging device 40, the retroreflective material and fluorescent color of the high-visibility safety clothing are more clearly captured in the image captured by the camera 40X. reflected in Therefore, the object detection unit 302 can easily detect (recognize) the high-visibility safety clothing from the input image (the captured image of the camera 40X or the captured image corrected by the image correction unit 306). A person can be detected more accurately.

Further, in this example, the imaging device 40 (camera 40X) may acquire a captured image around the excavator 100 while the irradiation device 70 is emitting infrared light. In this case, under the control of the controller 30 (irradiation control unit 305), the irradiation device 70 may always irradiate the infrared light while the camera 40X is in operation (while the power is on), or at the imaging timing of the camera 40X. Synchronously, infrared light may be intermittently (periodically) irradiated. As a result, when there is a person (worker) wearing high-visibility safety clothing within the imaging range of the imaging device 40, the outline of the retroreflective material of the high-visibility safety clothing becomes clearer in the captured image of the camera 40X. reflected. Therefore, the object detection unit 302 can easily detect (recognize) the high-visibility safety clothing from the input image, and as a result, can detect a person as a surveillance object with high accuracy. In addition, unlike visible light, infrared light cannot be directly detected by the human eye, so that the impact on people around the shovel 100 can be suppressed.

Further, in this example, under the control of the controller 30, the irradiation device 70 emits infrared light constantly (continuously) and intermittently in synchronization with the imaging timing of the camera 40X. You may switch between the state of For example, the irradiation control unit 305 determines whether the peripheral image displayed on the display device 50A or the display device 208 is affected by the infrared light. Then, the irradiation control unit 305 may switch the irradiation state of the infrared light from the irradiation device 70 according to the determination result. The effect of infrared light is, for example, when the camera 40X captures near-infrared light reflected from a subject (for example, high-visibility safety clothing), the peripheral image of the display device 50A becomes reddish. means that The irradiation control unit 305 may determine the presence or absence of the influence of infrared light by image analysis. Further, the irradiation control unit 305 may determine that there is an influence of infrared light by receiving a predetermined input from a user (operator or monitor) through the input device 52 or the communication device 60 . Accordingly, when the operator or the observer recognizes that the peripheral image of the display device 50A or the display device 208 is in a reddish state, the state can be corrected by performing a predetermined input through the input device 52 or the input device 207. Controller 30 can be notified.

Specifically, the irradiation control unit 305 normally emits infrared light from the irradiation device 70 in a state in which peripheral images displayed on the display device 50A and the display device 208 are not affected by the infrared light. irradiate. On the other hand, when the peripheral image displayed on the display device 50A or the display device 208 is affected by the infrared light, the irradiation control unit 305 intermittently irradiates the infrared light in synchronization with the imaging timing of the camera 40X. Let Thereby, it is possible to achieve a balance between suppressing the control load of the irradiation device 70 by the controller 30 and suppressing the influence of the infrared light on the peripheral image of the display device 50A or the like.

<Second example>
FIG. 16 is a diagram showing an example of an image captured by the imaging device 40 (camera 40X). FIG. 17 is a diagram showing an example of an image after correction (corrected captured image) for the imaging device 40 (camera 40X) by the image correction unit 306. As shown in FIG.

In this example, the object detection unit 302 detects a monitoring object using an image (corrected captured image) corrected with respect to the captured image of the imaging device 40 by the image correction unit 306 as an input image.

For example, when viewed from the operator of the cabin 10, it is difficult to directly visually recognize the rear, left and right sides of the upper swing structure 3 . Further, for example, a remote operator cannot directly see the surroundings including the front of the cabin 10 . Therefore, the four cameras 40F, 40B, 40L, and 40R need to cover a 360-degree peripheral range when the excavator 100 is viewed from above. As a result, for example, as shown in FIG. 2, the four cameras 40F, 40B, 40L, and 40R may be wide-angle cameras in which the angle of view (viewing angle) in the horizontal direction is set to be extremely large. Wide-angle distortion (volume distortion) may occur in the image.

Further, for example, as shown in FIGS. 1 and 2, the camera 40X is installed on the upper portion of the upper rotating body 3. As shown in FIG. Therefore, the camera 40X needs to detect monitored objects including those near the ground from a relatively high position, and the optical axis of the camera 40X is set obliquely downward. As a result, perspective distortion may occur in the image captured by the camera 40X.

For example, as shown in FIG. 16, workers W1 to W3 are shown at the center, left end, and right end of the image captured by camera 40X. The worker W1 in the central portion of the captured image of the camera 40X is shown such that the vertical axis for the subject substantially coincides with the vertical axis of the captured image. “Substantially” is intended to allow manufacturing errors of the excavator 100 and errors during installation of the camera 40X. On the other hand, the workers W2 and W3 in the image captured by the camera 40X have their vertical axes tilted leftward and rightward with respect to the vertical axis, respectively. This is because the image captured by the camera 40X is distorted such that the vertical axis of the subject at the left and right ends is tilted toward the end with respect to the vertical axis.

On the other hand, in this example, the image correction unit 306 corrects the image captured by the camera 40X so that the difference between the vertical axis of the image captured by the camera 40X and the vertical axis for the subject is small. Output the captured image.

For example, as shown in FIG. 17, the image correction unit 306 inclines only the latter image area out of the central portion and the left and right end portions of the image captured by the camera 40X by a predetermined amount. Generate a corrected captured image. As a result, the workers W2 and W3 in the corrected captured image have a relatively small difference between the vertical axis in the image and the vertical axis for the subject.

Further, the image correction unit 306 may correct the distortion itself of the captured image of the camera 40X using a known method such as projective transformation. Thereby, the image correction unit 306 can reduce the difference between the vertical axis in the corrected captured image and the vertical axis for the subject.

In this example, the object detection unit 302 can detect only the former of the monitored object with a relatively small difference between the vertical axis of the input image and the vertical axis for the subject and the monitored object with a relatively large difference. configured to Objects to be monitored with a relatively large difference between the vertical axis of the image and the vertical axis of the subject, such as workers W2 and W3 in FIG. is a surveillance object. In addition, a monitored object with a relatively small difference between the vertical axis in the image and the vertical axis of the subject, such as worker W1 in FIG. is a surveillance object. In addition, surveillance objects with a relatively small difference between the vertical axis in the image and the vertical axis depending on the subject, such as workers W1 to W3 in FIG. is a surveillance object. Accordingly, the object detection unit 302 can detect the monitored object using the corrected captured image as the input image.

Specifically, the object detection unit 302 applies shape detection, pattern recognition, or the like to a monitored object whose difference between the vertical axis in the input image and the vertical axis for the subject is relatively small, thereby detecting the input image. may detect such surveillance objects from. In addition, the object detection unit 302 detects an image including a monitoring object with a relatively small difference between the vertical axis of the input image and the vertical axis for the subject, and an image including a relatively large monitoring object. A trained model LM based on a teacher data set containing only the former may be used. As a result, the object detection unit 302 needs only to recognize from the input image (corrected captured image) only the monitored object whose difference between the vertical axis of the image and the vertical axis of the subject is relatively small. , the monitored object can be detected more accurately. Also, when the trained model LM is used, only the feature of the monitored object having a relatively small difference between the vertical axis of the image and the vertical axis of the subject can be learned from the image (corrected captured image). Because it is good, it is possible to improve the efficiency of machine learning.

Also, when the trained model LM is used, as described above, for example, an image showing a monitored object captured by a camera of the same model as the camera 40X is used as the teacher data set. Further, as described above, the teacher data set includes, for example, as shown in FIGS. An image captured by a camera showing the monitored object may be used. In this case, an image captured by a camera of the same type as the camera 40X is corrected by the same function as the image correction unit 306, and the corrected image may be included in the teacher data set. As a result, the trained model LM can detect the monitored object by considering how the monitored object appears in the captured image of the camera 40X, and the distance between the vertical axis of the image and the vertical axis of the subject. Machine learning is performed so that only monitored objects with relatively small differences can be detected. Therefore, the object detection unit 302 can detect the monitored object with higher accuracy. Further, in this case, the teacher data set includes, for example, both the captured image in which the monitored object is captured in the center in the horizontal direction and the captured image in which the monitored object is captured at least one end in the horizontal direction. may include the corrected image corresponding to . As a result, the learned model LM can detect both a monitoring object without correction in the left and right central portions of the input image (corrected captured image) and a monitoring object with correction in the left and right ends of the input image. It is machine-learned as follows. Therefore, the object detection unit 302 can detect the monitored object with higher accuracy.

<Third example>
18 to 21 are diagrams showing examples of captured images in which only a portion of the entire (whole body) of a monitored object (person) is shown.

For example, as shown in FIGS. 18 to 21, in the image captured by the camera 40X, the lower part (lower body) of the whole (whole body) of a person (worker W) serving as a monitored object located at both ends of the imaging range in the horizontal direction. ) may be shown. Specifically, in FIG. 18 (first example), a part of the lower body and upper body (waist part) of the worker W is placed at a position relatively close to the upper revolving body 3 on the right end of the image captured by the camera 40X. is reflected. In addition, in FIG. 19 (second example), a part of the lower body and upper body of the worker W (thorax excluding the head, waist, , arms, and hands) are shown. In FIG. 20 (third example), a part of the lower body (lower legs and feet) of the worker W appears at a position relatively close to the upper revolving body 3 on the right end of the image captured by the camera 40X. there is In FIG. 21 (fourth example), a part of the lower body (lower legs and feet) of the worker W appears at a position relatively distant from the upper revolving body 3 on the left end of the image captured by the camera 40X. ing. This is because the camera 40X has an optical axis that extends obliquely downward from the upper surface of the upper revolving body 3 toward the ground, and the upper body of the worker W (person) as a monitored object is out of the three-dimensional imaging range of the camera 40X. is.

In addition, when an obstacle exists between the monitored object and the camera 40X, the lower part (lower body) of the monitored object (person) is hidden by the obstacle, and the captured image of the camera 40X includes Only a part including the upper part of the body (upper body) may be shown.

On the other hand, in this example, the object detection unit 302 detects (recognizes) only a predetermined portion of the whole (whole body) of the monitored object (person) from the input image, thereby detecting (recognizing) the monitored object (person). to detect (recognize) The predetermined part is, for example, the lower part (lower body) or upper part (upper body) of the object (person) to be monitored, as described above. Thereby, the object detection unit 302 can detect the monitored object even when the input image shows only a portion including a predetermined portion of the entire monitored object. Therefore, the object detection unit 302 can detect the monitored object with higher accuracy.

Specifically, the object detection unit 302 may detect a predetermined portion of the monitored object from the input image by applying shape detection, pattern recognition, or the like to the predetermined portion of the monitored object. Further, the object detection unit 302 uses a trained model LM based on a teacher data set including an image showing only a portion including a predetermined part of the entire monitored object to detect a predetermined part of the monitored object. may be detected.

Further, the object detection unit 302 may be configured to detect (recognize) both the entire monitored object and a predetermined portion of the entire monitored object. Accordingly, the object detection unit 302 can determine that the monitored object has been detected, for example, when either the entire monitored object or a predetermined part of the monitored object is detected. Therefore, the object detection unit 302 can detect the monitored object with higher accuracy.

For example, the object detection unit 302 uses a trained model LM based on a teacher data set including an image showing the entire monitored object and an image showing only a part including a predetermined portion of the entire monitored object. is used to detect both the entire monitored object and predetermined parts of the monitored object. In this case, the object detection unit 302 detects the monitored object from among a plurality of labels including a label representing the entire monitored object and a label representing a predetermined portion of the entire monitored object. You can print the label. Then, the object detection unit 302 determines from the learned model LM that the label of the detected monitored object is either a label representing the entire monitored object or a label representing a predetermined portion of the entire monitored object. When output, detection of a monitored object may be determined.

Also, when the trained model LM is used, the teacher data set includes, for example, a camera of the same model as the camera 40X, and only a part including the lower part of the entire monitored object is at least the left end and the right end. The image that appears on one side is used. In addition, the teacher data set includes, for example, the lower part of the entire monitored object captured by a camera of the same model as the camera 40X, which is installed at approximately the same position and in approximately the same posture as the excavator of the same model as the excavator 100. An image in which at least one of the left and right ends may be used. As a result, the learned model LM is machine-learned so that the lower part of the entire monitored object (for example, the lower body of a person) can be detected in consideration of how the monitored object appears in the captured image of the camera 40X. be. Therefore, the object detection unit 302 can more accurately detect a predetermined part of the entire monitored object.

Further, the object detection unit 302 detects the entire monitored object based on the image at the center in the horizontal direction of the input image, and detects the lower part of the entire monitored object based on the images at both ends in the horizontal direction of the input image. (For example, only the lower body of a person) may be detected. This is because, as described above, in the input image, at both ends in the left and right direction, only the lower part of the entire monitored object (for example, the lower body of a person) may appear.

Further, the object detection unit 302 detects a predetermined part of the monitored object in both of the partial areas where the imaging ranges overlap, based on the captured images of the two cameras 40X whose imaging ranges partially overlap each other. In the case of (recognizing), the surveillance object in that area may be detected. The two cameras 40X whose imaging ranges partially overlap each other correspond to, for example, any one combination of the cameras 40F and 40R, the cameras 40R and 40B, the cameras 40B and 40L, and the cameras 40L and 40F. As a result, even if the object detection unit 302 erroneously detects a predetermined portion of the monitored object based on one camera 40X in a partial area of the overlapping imaging ranges of the two cameras 40X, the other camera 40X Based on this, it can be determined that a predetermined portion of the monitored object has not been detected. Therefore, the object detection unit 302 can suppress erroneous detection when detecting a predetermined portion of the entire monitored object.

[Specific example of method for estimating actual position of monitored object]
Next, a specific example of a method for estimating the actual position of a detected monitored object will be described with reference to FIGS. 22 to 24. FIG.

22 to 24 are diagrams showing one example, another example, and still another example of the detection state of the monitored object (person) by the object detection unit 302. FIG. Specifically, FIGS. 22 to 24 are diagrams showing a detection frame FR representing the detection range of the monitored object in the input image when the monitored object (worker W) is detected from the input image by the object detection unit 302. FIG. is.

22 to 24 may be displayed on the display device 50A or the remote control display device, for example, when the object detection unit 302 detects the monitored object (worker W). As a result, the operator or the monitor can grasp the detection status of the monitored object (worker W) around the excavator 100 while viewing the peripheral image (the image captured by the camera 40X).

In the situations of FIGS. 22 and 23, the worker W is present in the same place relatively close to the upper swing body 3 in an upright state facing the camera 40X. In addition, in the situation of FIG. 24 , the worker W is present in a position relatively distant from the upper rotating body 3 and facing the camera 40X in an upright state.

For example, in the situation of FIG. 22, the object detection unit 302 detects (recognizes) the entire worker W appearing in the input image (for example, the image captured by the camera 40X) as a monitoring object (person). Therefore, the detection frame FR is expressed as a rectangular shape surrounding the worker W. As shown in FIG.

On the other hand, for example, in the situation of FIG. 23, the object detection unit 302 detects (recognizes) only the upper part in the vertical direction (upper body part) of the entire worker W shown in the input image as a monitoring object (person). )are doing. Therefore, the detection frame FR is expressed as a rectangular shape that surrounds only the upper part of the worker W including the head on which the helmet is worn and the torso on which the high-visibility safety clothing is worn.

For example, when the object detection unit 302 detects (recognizes) a person by detecting (recognizing) a person's wearing object such as a helmet or high-visibility safety clothing, the object detection unit 302 detects a part of the input image corresponding to the person's wearing object. There is a possibility that only the area of is judged as the detection range of the monitored object. In addition, depending on the background of the monitored object and the weather at that time, features such as the shape of the monitored object in the input image may become unclear, and the object detection unit 302 may be able to recognize only a part of the entire monitored object as the monitored object. . Therefore, the object detection unit 302 may detect (recognize) only a portion of the entire monitored object in the vertical direction on the input image.

For example, in the situation of FIG. 23, as described above, the actual position of the worker W is estimated based on the size of the monitored object detected (recognized) on the input image (the size of the detection frame FR in the vertical direction). Then, it may be determined that the operator exists at a position farther from the upper revolving body 3 than it actually is. Specifically, although the worker W exists at the same position as in the situation of FIG. 22, in the situation of FIG. Position may be estimated.

Also, for example, in the situation of FIG. 23 and the situation of FIG. , the lower end position of the detection range (detection frame FR) is substantially the same. Therefore, as described above, when the actual position of the monitored object is estimated by projective transformation or the like based on the detected position (reference point) of the monitored object on the input image, the worker W detected in the situation of FIG. Similarly, a real position relatively distant from the upper revolving structure 3 may be estimated.

On the other hand, in this example, the position estimating unit 303 is based on information regarding the detection position of the monitoring object in the vertical direction on the input image and information regarding the size and shape of the detection range of the monitoring object on the input image. , to estimate the real position of the monitored object. The information about the detection position of the monitoring object in the vertical direction on the input image is the coordinate position representing the detection range of the monitoring object on the input image. , the top edge, and at least one coordinate of the centroid. Information about the size of the detection range on the input image of the monitored object includes, for example, information about the height (vertical dimension) of the detection range, information about the width (horizontal dimension), information about the aspect ratio, diagonal line Information about the area, information about the area, etc. are included. Information about the shape of the detection range of the monitored object on the input image includes, for example, information about the aspect ratio.

Specifically, the position estimating unit 303 may determine whether or not there is consistency in the shape and size of the detection range on the input image of the monitored object with respect to the vertical detection position of the monitored object on the input image. . Assuming that the whole monitored object is detected in the input image, the shape and size range of the assumed detection range of the monitored object are determined by the vertical detection position of the monitored object on the input image. Therefore, the position estimating unit 303 determines whether the shape and size of the detection range of the monitored object on the input image fall within this assumed range with respect to the detected position of the monitored object in the vertical direction on the input image. Whether or not the whole monitored object is detected can be determined based on the presence or absence of such consistency.

The assumed range of the shape and size of the detection range of the monitored object on the input image may be defined uniformly according to the vertical detection position of the monitored object on the input image. In addition to the detection position of the monitored object in the vertical direction on the input image, other conditions may be considered for the assumed range of the shape and size of the detection range of the monitored object on the input image. . For example, the assumed range of the shape and size of the detection range of the monitored object on the input image includes the vertical detection position of the monitored object on the input image, as well as the posture and orientation of the detected monitored object. may be varied by This is because the appearance of the monitored object on the input image differs depending on the posture and orientation of the monitored object, and as a result, the shape and size of the detection range of the monitored object on the input image change. The orientation and orientation of the monitored object are determined, for example, so that the trained model LM can detect the monitored object for each different orientation and orientation of the monitored object, that is, for each of a plurality of labels corresponding to different orientations and orientations. can be obtained by machine learning so that the can be detected. In addition, the assumed range of the shape and size of the detection range on the input image (captured image) of the monitored object is not limited to the vertical detection position on the input image (captured image) of the monitored object, for example. , may be defined taking into account the distortion due to the lens at that detection position.

The position estimating unit 303 detects the entire monitored object when there is consistency in the shape and size of the detection range of the monitored object on the input image with respect to the detected vertical position of the monitored object on the input image. determined to be In this case, the position estimation unit 303 may estimate the actual position of the monitored object based on the detected position and detection range of the monitored object on the input image, as described above.

On the other hand, if there is no consistency in the shape and size of the detection range on the input image of the monitored object with respect to the detected position of the monitored object in the vertical direction on the input image, the position estimation unit 303 can estimate the entire monitored object. It is determined that only a part of them are detected. In this case, the position estimation unit 303 may correct the detected position or detection range of the monitored object on the input image, and estimate the actual position of the monitored object based on the corrected detected position or detection range.

For example, if the width of the detection range of the monitored object on the input image deviates from the assumed range with respect to the vertical detection position of the monitored object on the input image, the position estimation unit 303 It is determined that only the upper part of the entire surveillance object in the input image is detected (see FIG. 23).

Further, for example, the position estimation unit 303 determines the horizontal-to-horizontal dimension of the range assumed with respect to the vertical detection position of the monitored object on the input image. deviates in the direction of increasing, it is determined that only a portion of the whole monitoring object in the input image has been detected in the vertical direction. Furthermore, when the object detection unit 302 detects only a predetermined portion of the entire monitored object or only the clothing of the monitored object, the position estimation unit 303 considers the predetermined portion or the wearing part of the clothing. Then, it may be determined whether only the upper part or the lower part in the vertical direction of the whole monitoring object in the input image is detected. For example, when the object detection unit 302 detects (recognizes) a helmet or high-visibility safety clothing, the position estimation unit 303 determines whether the helmet or the high-visibility safety clothing is part of the entire monitored object (person) in the input image. It is determined that only a part of the upper side including the wearing part is detected (see FIG. 23). Further, when the position estimating unit 303 determines that only a part of the entire monitored object in the input image is detected in the vertical direction, the position estimating unit 303, giving priority to safety, uniformly detects the entire monitored object in the vertical direction. may be considered to be detected only in part.

In this case, the position estimating unit 303, for example, based on information about the height and width of the detection range on the input image of the monitored object, adjusts the position of the monitored object so that it assumes the shape (aspect ratio) of the monitored object. Correction may be performed to extend the detection range on the input image downward. The assumed shape (aspect ratio) of the monitored object is defined, for example, according to the detected position of the monitored object on the input image. In addition, the shape (aspect ratio) assumed for the monitored object is determined according to the detected position of the monitored object on the input image, the direction and posture of the monitored object, and the distortion caused by the lens at the detected position. may be specified. Further, the position estimating unit 303 calculates the detection position of the monitoring object so as to match the width of the detection range on the input image of the monitoring object, for example, based on the information about the width of the detection range on the input image of the monitoring object. may be moved downward, or the detection range of the monitored object may be extended downward. At this time, the position estimating unit 303 may correct the detection position and detection range of the monitored object in consideration of, for example, the orientation and posture of the monitored object, distortion caused by the lens at the detected position, and the like. Then, the position estimation unit 303 may estimate the actual position of the monitored object based on the corrected detection range and detected position of the monitored object.

Thus, in this example, the position estimation unit 303 detects a state in which only a portion of the entire monitored object (in particular, the upper portion in the vertical direction) is detected by the object detection unit 302 on the input image. , and considering its state, the actual position of the monitored object can be estimated. Therefore, the position estimator 303, for example, suppresses a situation in which the actual position of the monitored object is estimated as a position relatively distant from the upper revolving structure 3, and more appropriately estimates the actual position of the monitored object. can do. Therefore, the controller 30 (safety control section 304) can operate the safety function more appropriately, and as a result, the safety of the excavator 100 can be improved.

[Another example of excavator functional configuration]
Next, another example of the functional configuration of the excavator 100 will be described with reference to FIG. 25 . The following description will focus on the parts that differ from the example described above (FIG. 6), and descriptions that are the same as or correspond to the example described above may be omitted.

FIG. 25 is a functional block diagram showing another example of the functional configuration of the shovel 100 according to this embodiment.

As shown in FIG. 25, the controller 30 includes a display processing unit 301, an object detection unit 302, a position estimation unit 303, a safety control unit 304, and an irradiation control unit 305, as in the above example. , the image correction unit 306 may be included. Further, the controller 30 includes a detection determination section 307 unlike the example described above.

The imaging device 40 includes an object detection section 402 . The imaging device 40 may also include an image correction unit 404 .

An object detection unit 402 (an example of a detection unit) detects a monitored object based on an image captured by the imaging device 40, similar to the object detection unit 302. FIG. Object detection unit 402 includes object detection units 402F, 402B, 402L, and 402R. Hereinafter, the object detection units 402F, 402B, 402L, and 402R may be collectively referred to as an "object detection unit 402X", or any one of them may be individually referred to as an "object detection unit 402X".

The object detection unit 402F is mounted on the camera 40F and, like the object detection unit 302, detects a monitored object based on the captured image of the camera 40F. Specifically, the object detection unit 402F may recognize the monitored object from the captured image of the camera 40F and specify the position (area) where the monitored object is captured. Further, the object detection unit 402F detects the monitored object from the image (corrected captured image) after the captured image of the camera 40F (that is, the captured image generated by the captured image generation function) is corrected by the image correction unit 404F. The position (area) where the monitored object appears may be specified. The function of the object detection unit 402F is realized by arbitrary hardware or a combination of arbitrary hardware and software. For example, the function of the object detection unit 402F is realized by loading a program installed in the auxiliary storage device of the image processing engine 42 of the camera 40F into the memory device and executing it on the CPU. The same applies to the functions of the object detection units 402B, 402L, and 402R below.

The object detection unit 402B is mounted on the camera 40B, and similar to the object detection unit 302, detects a monitored object based on the captured image of the camera 40B. Specifically, the object detection unit 402B may recognize the monitored object from the captured image of the camera 40B and specify the position (area) where the monitored object is captured. Further, the object detection unit 402B detects the monitored object from the image (corrected captured image) after the captured image of the camera 40B (that is, the captured image generated by the captured image generation function) is corrected by the image correction unit 404B. The position (area) where the monitored object appears may be specified.

The object detection unit 402L is mounted on the camera 40L and, like the object detection unit 302, detects a monitored object based on the captured image of the camera 40L. Specifically, the object detection unit 402L may recognize the monitored object from the captured image of the camera 40L and specify the position (area) where the monitored object is captured. Further, the object detection unit 402L detects the monitored object from the image (corrected captured image) after the captured image of the camera 40L (that is, the captured image generated by the captured image generation function) is corrected by the image correction unit 404L. The position (area) where the monitored object appears may be specified.

The object detection unit 402R is mounted on the camera 40R and, like the object detection unit 302, detects a monitored object based on the captured image of the camera 40R. Specifically, the object detection unit 402R may recognize the monitored object from the captured image of the camera 40R and specify the position (area) where the monitored object is captured. Further, the object detection unit 402R detects the monitored object from the image (corrected captured image) after the captured image of the camera 40R (that is, the captured image generated by the captured image generation function) is corrected by the image correction unit 404R. The position (area) where the monitored object appears may be specified.

Like the image correction unit 306 , the image correction unit 404 performs predetermined correction on the output (captured image) of the imaging device 40 (camera 40X) and outputs the corrected captured image to the object detection unit 402 . It includes image correction units 404F, 404B, 404L, and 404R.

The image correction unit 404F is mounted on the camera 40F, performs predetermined correction on the captured image of the camera 40F, and outputs the corrected captured image to the object detection unit 402F, similarly to the image correction unit 306 described above.

Note that the camera 40F may output to the controller 30 a corrected captured image corrected by the image correction unit 404F in addition to the captured image generated by the captured image generation function. The same may be applied to cameras 40B, 40L, and 40R below. In this case, the image correction unit 306 of the controller 30 may be omitted, and the object detection unit 302 may detect the monitored object based on the corrected captured image input from the camera 40X.

The image correction unit 404B is mounted on the camera 40B, performs predetermined correction on the captured image of the camera 40B, and outputs the corrected captured image to the object detection unit 402B, like the image correction unit 306 described above.

The image correction unit 404L is mounted on the camera 40L, performs predetermined correction on the captured image of the camera 40L, and outputs the corrected captured image to the object detection unit 402L, similarly to the image correction unit 306 described above.

The image correction unit 404R is mounted on the camera 40R, performs predetermined correction on the captured image of the camera 40R, and outputs the corrected captured image to the object detection unit 402R, like the image correction unit 306 described above.

The object detection units 402X use the same algorithm to detect the monitored object from the input image. On the other hand, the object detection unit 402X uses an algorithm different from that of the object detection unit 302 to detect the monitored object from the input image. As a result, for example, even if one of the object detection units 302 and 402X cannot detect the monitored object, the possibility of the other detecting the monitored object can be increased. As a result, it is possible to suppress the occurrence of a situation in which the monitoring object around the excavator 100 cannot be detected. Also, for example, even if one of the object detection units 302 and 402X detects a non-existing monitored object, the other does not detect the monitored object, so that the monitored object is not detected. can also

For example, one of the object detection units 302 and 402X uses only the former of the image processing technology and the machine learning technology, and the other uses both the image processing technology and the machine learning technology, that is, the trained model LM may be used to detect the monitored object.

Also, for example, one of the object detection units 302 and 402X uses an SVM that has undergone machine learning of the image feature amount of the monitored object as a trained model LM, and the other uses a trained model LM based on deep learning. Then, the monitored object may be detected from the input image.

Also, the object detection units 302 and 402X may detect the monitored object from the input image using, for example, trained models LM based on different networks (DNN). For example, one of the object detection units 302 and 402X uses a CNN-based classification model as a trained model LM, and the other uses a CNN-based regression model as a trained model LM. , the monitored object may be detected from the input image. Specifically, one of the object detection units 302 and 402X uses a trained model LM based on Faster R-CNN, and the other uses a trained model LM based on YOLO or SSD. A surveillance object may be detected. Also, for example, the object detection units 302 and 402X use CNN-based trained models LM, and each trained model LM has at least the following: one may be different from the other. Properties of the candidate regions may include aspect ratios, fixed or variable differences between the candidate regions, and the like. Specifically, one of the object detection units 302 and 402X uses the learned model LM by YOLO and the other uses the learned model LM by SSD to detect the monitored object from the input image. may Also, for example, the object detection units 302 and 402X use the same SSD trained model LM, and each trained model LM has a network structure, the number of layers of the network, the structure of the feature map, the characteristics of the candidate region, and the like. At least one of the parameters may be different from each other.

Also, for example, the object detection units 302 and 402X may detect a monitored object from an input image using a learned model LM machine-learned with different teacher data sets.

For example, the object detection units 302 and 402X detect a monitored object from an input image using a trained model LM machine-learned with a teacher data set of images in which at least one of the posture and orientation of the monitored object is different from each other. good. For example, one of the learned models LM of the object detection units 302 and 402X is machine-learned with a teacher data set of images showing a person standing upright, and the other is an image showing a person crouching or squatting. It may be machine-learned with a teacher dataset by Also, for example, the trained models of the object detection units 302 and 402X are machine-learned using a teacher data set of images showing a person with one of them facing the camera facing forward or backward, and the other one facing the camera. It may be machine-learned with a training data set of images showing people in landscape orientation. As a result, even if one of the object detection units 302 and 402X cannot properly detect the monitored object due to the influence of the direction and posture of the monitored object appearing in the input image, the other can detect the monitored object from the input image. can increase the possibility of properly detecting Therefore, the object detection units 302 and 402X as a whole can more reliably detect the monitored object.

Also, for example, one of the object detection units 302 and 402X detects (recognizes) the entirety including the shape of the monitored object (person), and the other detects (recognizes) the object worn by the monitored object (person). ), the monitored object may be detected from the input image. For example, one of the object detection units 302 and 402X detects the entire person from the input image, and the other detects (recognizes) the helmet or safety vest worn by the person, thereby recognizing the person from the input image. may be detected. As a result, for example, even if one of the object detection units 302 and 402X cannot properly detect (recognize) the whole person for some reason, the other can detect the monitored object (person) more reliably. can be detected.

Further, for example, the object detection units 302 and 402X may detect the monitored object from the input image by detecting different clothing worn by the monitored object (person). For example, one of the object detection units 302 and 402X detects (recognizes) a helmet worn by a person from the input image, and the other detects (recognizes) a safety vest worn by a person from the input image. , may detect people from the input image. As a result, the object detection units 302 and 402X detect that even if one of the workers cannot recognize the one wearing item because, for example, the worker has not worn the one wearing item, the other wears the other wearing item. By recognizing , the monitored object (person) can be detected more reliably.

Further, for example, the object detection units 302 and 402X detect a monitored object from an input image by using a trained model LM machine-learned with a teacher data set of images with different background types of the monitored object. good too. For example, one of the trained models LM of the object detection units 302 and 402X is machine-learned with a teacher data set based on an image of the earth and sand ground in the background, and the other is based on an image of the ground such as asphalt in the background. It may be machine-learned with a teacher dataset. As a result, even if one of the object detection units 302 and 402X cannot properly detect the monitored object due to the background of the input image, the other can detect the monitored object appropriately from the input image. can be enhanced. Therefore, the object detection units 302 and 402X as a whole can more reliably detect the monitored object.

Further, for example, the object detection units 302 and 402X mutually detect a monitored object from an input image by using a trained model LM machine-learned with a teacher data set of images acquired (captured) in different time zones. You may For example, one of the learned models of the object detection units 302 and 402X is machine-learned using a teacher data set based on images captured in the morning or afternoon, and the other is captured in the evening or nighttime. may be machine-learned on a training dataset with the images obtained. As a result, the controller 30 can selectively use the object detection units 302 and 402X according to, for example, the time period during which the excavator 100 is working, and can make a final decision regarding the detection of the monitored object based on the detection results of the object detection units 302 and 402X. You can switch methods.

Further, when the excavator 100 is used at night, for example, the object detection units 302 and 402X each use a learned model LM machine-learned with a teacher data set of images with different lighting conditions at night to input A surveillance object may be detected from an image. For example, one of the learned models LM of the object detection units 302 and 402X is machine-learned with a teacher data set based on images acquired at relatively high illuminance, and the other is machine-learned with relatively low illuminance. It may be machine-learned with a training data set based on the acquired images. Further, for example, the trained models LM of the object detection units 302 and 402X may be machine-learned using a teacher data set of images acquired under lighting conditions of different colors. As a result, even if one of the object detection units 302 and 402X cannot properly detect the monitored object due to the lighting conditions, the other can increase the possibility that the other can properly detect the monitored object. Therefore, the object detection units 302 and 402X as a whole can more reliably detect the monitored object.

Further, for example, the object detection units 302 and 402X detect a monitored object from an input image by using a learned model LM machine-learned with a teacher data set of images acquired (captured) under different weather conditions. may be detected. For example, one of the object detection units 302 and 402X, for example, the learned model LM of the object detection units 302 and 402X, is machine-learned with a teacher data set based on images acquired in a sunny or cloudy state. The other may be machine-learned using a teacher data set based on images acquired in rain or snow conditions. As a result, even if one of the object detection units 302 and 402X cannot properly detect the monitored object due to weather, for example, the other can increase the possibility that the other can properly detect the monitored object. . Therefore, the object detection units 302 and 402X as a whole can more reliably detect the monitored object.

Further, for example, the object detection units 302 and 402X use a trained model LM machine-learned with a teacher data set of images acquired with different positional relationships between the light source and the imaging range, from the input image. A surveillance object may be detected. For example, one of the trained models LM of the object detection units 302 and 402X is machine-learned with a teacher data set using images corresponding to front light, semi-front light, or side light, and the other is an image corresponding to backlight. It may be machine-learned with a teacher dataset by As a result, even if one of the object detection units 302 and 402X cannot properly detect the monitored object due to the influence of the positional relationship between the light source and the imaging range, the other can properly detect the monitored object. can increase the likelihood of Therefore, the object detection units 302 and 402X as a whole can more reliably detect the monitored object.

The detection determination unit 307 detects the monitored object based on the detection result of the object detection unit 302 and the object detection unit 402X. Specifically, the detection determination unit 307 makes a final determination regarding detection of the monitored object based on the detection result of the object detection unit 302 and the detection result of the object detection unit 402X. The details of the detection method of the monitored object by the detection determination unit 307 will be described later.

The position estimator 303 estimates the actual position of the detected monitored object when the detection determiner 307 detects the monitored object. Moreover, when a plurality of monitored objects are detected by the detection determination unit 307, the position estimation unit 303 estimates the actual position of each of the plurality of monitored objects.

For example, if a monitored object is detected by one of the object detection units 302 and 402X, the above method is arbitrarily applied based on the detection range on the input image specified by one of them. By doing so, the position of the monitored object is estimated.

Further, for example, when the same monitored object is detected by both the object detection units 302 and 402X, the actual position of the monitored object is estimated for each of the detection results of the object detection units 302 and 402X, and The actual position of the monitored object may be estimated from the estimation result. For example, the actual position of the detected monitoring object may be estimated by averaging the estimation results of the actual position of the monitoring object for each of the detection results of the object detection units 302 and 402X.

The safety control unit 304 activates the safety function when, for example, the detection determination unit 307 detects a monitored object within a predetermined range around the excavator 100 . Specifically, the safety control unit 304 may activate the safety function when the actual position (estimated value) of the monitored object estimated by the position estimation unit 303 is within a predetermined range around the excavator 100. .

Note that the function of either one of the object detection units 302 and 402 may be transferred to the display device 50A. Further, in addition to the object detection units 302 and 402, the excavator 100 may be provided with one or more object detection units that detect a monitored object from an input image. The object detection unit to be added may be provided in the display device 50A, may be provided in the controller 30, or may be provided in a controller different from the controller 30. FIG. In this case, the three or more object detection units including the additional ones may use algorithms different from each other to detect the monitored object from the input image.

[Specific example of object detection method]
Next, a method of detecting a monitored object by the detection/judgment unit 307, that is, a method of final determination regarding detection of a monitored object will be described.

For example, when a monitored object is detected by at least one of the object detection unit 302 and the object detection unit 402X, the detection determination unit 307 determines that the monitored object exists around the excavator 100, and detects the monitored object. you can As a result, for example, even if one of the object detection units 302 and 402X does not detect an existing monitored object, if the other detects the monitored object, the controller 30 detects the monitored object. can be detected. Therefore, the controller 30 more reliably detects the actual monitored object in a situation where detection failure of the monitored object is likely to occur or in a situation where it is highly necessary to prevent detection failure of the monitored object. With the operation of the safety function, safety can be further improved.

Further, for example, when a monitored object is detected by both the object detection unit 302 and the object detection unit 402X, the detection determination unit 307 determines that the monitored object exists around the excavator 100, and detects the monitored object. You may As a result, for example, even if one of the object detection units 302 and 402X erroneously detects a non-existing monitored object, if the other does not detect the monitored object, the controller 30 detects the monitored object. can be made undetectable. Therefore, the controller 30 can suppress erroneous detection of a monitored object in a situation where erroneous detection of a monitored object is likely to occur or in a situation where it is highly necessary to prevent a decrease in work efficiency due to erroneous detection of a monitored object. .

As described above, when three or more object detection units are provided in the excavator 100, the detection determination unit 307 selects, for example, a predetermined number (integer of two or more) of all the object detection units. A monitored object may be detected if the monitored object has been detected. The predetermined number may be fixed or variable. For example, the predetermined number may be varied according to a predetermined input received from the user (operator or supervisor) through the input device 52 or communication device 60 .

Further, the detection determination unit 307 may switch between the above two methods according to a predetermined input from the user (operator or supervisor) received through the input device 52 or the communication device 60 .

Further, the detection determination unit 307 detects the monitored object based on the object detection unit 302 and the object detection unit 402X, for example, using the importance defined in the detection results of the object detection unit 302 and the object detection unit 402X. You may Specifically, the detection determination unit 307 detects the monitored object when the monitored object is detected by one of the object detection unit 302 and the object detection unit 402X whose detection result is relatively more important. .

The importance may be set (variable) according to a predetermined input from the user (operator or supervisor) received through the input device 52 or the communication device 60, for example. As a result, the user can compare, for example, the peripheral image displayed on the display device 50A or the remote control display device with the actual detection status of the monitored object by the object detection units 302 and 402X, and the accuracy is considered to be higher. The importance of one detection result can be set high. Also, the importance may be automatically set (variable) according to the surrounding conditions of the excavator 100 . For example, if there is a difference in the detection accuracy of the monitored object between the object detection units 302 and 402X depending on the environmental conditions in which the excavator 100 is placed, the importance of one detection result with high detection accuracy is higher than that of the other. It may be set higher than the importance of the detection result. The environmental conditions in which the excavator 100 is placed may include the conditions of objects around the excavator 100 that appear as the background in the captured image of the camera 40X (for example, types such as earth and sand on the ground, asphalt, etc.). Further, the environmental conditions in which the excavator 100 is placed include, for example, the time period during which work is being performed, the lighting conditions during nighttime work, the weather conditions, the positional relationship between the light source and the imaging range of the camera 40X, and the like. can be As a result, the controller 30 relatively increases the importance of one of the object detection units 302 and 402X, which is assumed to have higher detection accuracy, according to the situation in which the excavator 100 is placed. Can be set.

As described above, when three or more object detection units are provided in the shovel 100, the detection determination unit 307 determines that the detection result of each object detection unit is the final determination as the importance of the detection result of each object detection unit becomes relatively higher. A monitored object may be detected so as to be more likely to be reflected in the result of . For example, when the object to be monitored is detected by some object detection units out of all the object detection units, the detection determination unit 307 determines that the sum of the degrees of importance of the detection results of the part of the object detection units is If it is greater than the sum of the significance levels of the detection results of the other remaining object detection units, the monitored object may be detected. Further, for example, when a monitoring object is detected by some object detection units out of all object detection units, the detection determination unit 307 determines the degree of importance of each detection result of the part of the object detection units. If the sum is greater than or equal to a predetermined threshold, the monitored object may be detected.

[Transformation/change]
Although the embodiments have been described in detail above, the present disclosure is not limited to such specific embodiments, and various modifications and changes are possible within the scope of the claims.

For example, in the above-described embodiment, the method of detecting a monitored object based on the image captured by the imaging device 40 mounted on the excavator 100 has been described. can be applied to the surveillance object detection method based on Other working machines are, for example, liftmag machines, bulldozers, forestry machines (for example, harvesters), road machines (for example, asphalt finishers), etc., in which a lifting magnet is attached to the tip of the attachment AT in place of the bucket 6 .

1 lower traveling body 1C crawler 1ML traveling hydraulic motor 1MR traveling hydraulic motor 2 turning mechanism 2M turning hydraulic motor 3 upper turning body 4 boom 5 arm 6 bucket 7 boom cylinder 8 arm cylinder 9 bucket cylinder 10 cabin 11 engine 13 regulator 14 main pump 15 Pilot pump 17 Control valve 25 Pilot line 25A Pilot line 25B Pilot line 26 Operation device 27 Pilot line 27A Pilot line 27B Pilot line 29 Operation pressure sensor 30 Controller 30A Auxiliary storage device 30B Memory device 30C CPU
30D interface device 31 hydraulic control valve 32 shuttle valve 33 hydraulic control valve 40 imaging device 40B camera 40F camera 40L camera 40R camera 50 output device 50A display device 50B sound output device 52 input device 60 communication device 70 irradiation device 100 excavator 200 management device 201 External Interface 201A Recording Medium 202 Auxiliary Storage Device 203 Memory Device 204 CPU
205 high-speed arithmetic unit 206 communication interface 207 input device 208 display device 301 display processing unit 302 object detection unit (detection unit)
303 position estimation unit 304 safety control unit (control unit)
305 irradiation control unit (determining unit)
306 image correction unit 307 detection determination unit 402, 402B, 402F, 402L, 402R object detection unit (detection unit)
404, 404B, 404F, 404L, 404R Image Corrector AT Attachment HA Hydraulic Actuator HMT Helmet LM Learned Model NW Communication Line RV High Visibility Safety Clothes S1 Boom Angle Sensor S2 Arm Angle Sensor S3 Bucket Angle Sensor S4 Aircraft Attitude Sensor S5 Rotation Angle sensor SYS Excavator management system W, W1 to W3 Operator

Claims (8)

a lower running body;
an upper rotating body rotatably mounted on the lower traveling body;
an imaging device mounted on the upper revolving body for imaging the periphery of the upper revolving body;
an irradiation device that irradiates an imaging range of the imaging device with infrared light;
a detection unit that detects people around the excavator by recognizing high-visibility safety clothing based on the captured image of the imaging device;
Excavator. Recognizing the high-visibility safety clothing is recognizing the high-visibility safety clothing worn by a person, or recognizing the person wearing the high-visibility safety clothing.
Shovel according to claim 1 . The irradiation device irradiates the imaging range of the imaging device with infrared light periodically in synchronization with the imaging cycle of the imaging device, or continuously while the imaging device is in operation.
A shovel according to claim 1 or 2. a display device for displaying the captured image or a processed image generated based on the captured image;
a determination unit that determines whether display content of the captured image or the processed image on the display device is affected by the infrared light,
When the determination unit determines that the display content of the captured image or the processed image on the display device is not affected by the infrared light, the irradiation device continues the imaging while the imaging device is in operation. When the imaging range of the device is irradiated with the infrared light, and it is determined by the determination unit that the display content of the captured image or the processed image on the display device is affected by the infrared light, irradiating the imaging range of the imaging device with the infrared light periodically in synchronization with an imaging cycle;
Shovel according to claim 3. The determination unit determines that the display content of the captured image or the processed image on the display device is affected by the infrared light when a predetermined input from a user is received.
Shovel according to claim 4. The detection unit detects people around the excavator by recognizing at least one of the shape and color of the high-visibility safety clothing and the brightness of the retroreflective material based on the captured image.
Shovel according to any one of claims 1 to 5. The detection unit machine-learns at least one of the shape and color of the high-visibility safety clothing, and the brightness of the retroreflection material, based on an image of training data showing a person wearing the high-visibility safety clothing. Detect people around the excavator using a pre-existing model,
Shovel according to claim 6. a control unit that decelerates or stops the operation of the excavator when the detection unit detects a person within a predetermined range around the excavator;
Shovel according to any one of claims 1 to 7.

Images