EP4506515A1

EP4506515A1 - System for monitoring work machine and method for monitoring work machine

Info

Publication number: EP4506515A1
Application number: EP23839292.2A
Authority: EP
Inventors: Masaaki Imaizumi; Yuya Murakami; Shun KAWAMOTO; Ryohei EMOTO; Shoji SONOYAMA; Shota HIRAMA
Original assignee: Komatsu Ltd
Current assignee: Komatsu Ltd
Priority date: 2022-07-13
Filing date: 2023-05-18
Publication date: 2025-02-12
Also published as: WO2024014115A1; JP2024011049A; EP4506515A4

Abstract

A monitoring system of a work machine includes: a measurement data acquisition unit mounted on a work machine and configured to acquire measurement data of a measurement device that measures a target; a vehicle body data acquisition unit configured to acquire vehicle body data indicating a state of the work machine; a state determination unit configured to determine the state of the work machine based on the vehicle body data; an algorithm selection unit configured to select a specific algorithm for processing the measurement data based on the state of the work machine determined by the state determination unit; and a processing unit configured to process the measurement data and recognize the target based on the specific algorithm selected by the algorithm selection unit.

Description

Field

The present disclosure relates to a work machine monitoring system and a work machine monitoring method.

Background

In a technical field related to work machines, a work vehicle as disclosed in Patent Literature 1 is known.

Citation List

Patent Literature

Patent Literature 1: JP 2021-080790 A

Summary

Technical Problem

There is a possibility of a work machine capable of smoothly performing work by monitoring a situation and a work environment of the work machine during the work. On the other hand, when both the situation and the work environment of the work machine are constantly monitored, a burden required for monitoring increases. Therefore, a technique capable of efficiently monitoring a plurality of targets is desired.
An object of the present disclosure is to efficiently monitor a plurality of targets.

Solution to Problem

In order to achieve an aspect of the present invention, a monitoring system of a work machine comprises: a measurement data acquisition unit mounted on a work machine and configured to acquire measurement data of a measurement device that measures a target; a vehicle body data acquisition unit configured to acquire vehicle body data indicating a state of the work machine; a state determination unit configured to determine the state of the work machine based on the vehicle body data; an algorithm algorithm selection unit configured to select a specific algorithm for processing the measurement data based on the state of the work machine determined by the state determination unit; and a processing unit configured to process the measurement data and recognize the target based on the specific algorithm selected by the algorithm selection unit.

Advantageous Effects of Invention

According to the present disclosure, it is possible to efficiently monitor a plurality of targets.

Brief Description of Drawings

FIG. 1 is a side view illustrating a work machine according to an embodiment.
FIG. 2 is a top view illustrating the work machine according to the embodiment.
FIG. 3 is a front view illustrating the work machine according to the embodiment.
FIG. 4 is a diagram illustrating a drive room of a drive cab according to the embodiment.
FIG. 5 is an enlarged front view illustrating a part of the work machine according to the embodiment.
FIG. 6 is a diagram illustrating a drive system of the work machine according to the embodiment.
FIG. 7 is a diagram illustrating an operation of the working equipment according to the embodiment.
FIG. 8 is a diagram illustrating an operation of the work machine according to the embodiment.
FIG. 9 is a side view illustrating the work machine that performs loading work according to the embodiment.
FIG. 10 is a top view illustrating the work machine that performs the loading work according to the embodiment.
FIG. 11 is a diagram illustrating an example of image data captured by a first imaging device on the left according to the embodiment.
FIG. 12 is a diagram illustrating an example of image data captured by a second imaging device on the left according to the embodiment.
FIG. 13 is a functional block diagram illustrating a monitoring system according to the embodiment.
FIG. 14 is a diagram illustrating a defect determination method of the working equipment according to the embodiment.
FIG. 15 is a diagram illustrating the defect determination method of the working equipment according to the embodiment.
FIG. 16 is a diagram illustrating the defect determination method of the working equipment according to the embodiment.
FIG. 17 is a diagram illustrating the defect determination method of the working equipment according to the embodiment.
FIG. 18 is a diagram illustrating the defect determination method of the working equipment according to the embodiment.
FIG. 19 is a diagram illustrating an example of a soundness determination frame according to the embodiment.
FIG. 20 is a diagram illustrating an example of a defect determination frame according to the embodiment.
FIG. 21 is a diagram illustrating an example of a symbol indicating a bucket tooth displayed on a display device according to the embodiment.
FIG. 22 is a diagram illustrating a relationship between a defect determination frame and a warning form according to the embodiment.
FIG. 23 is a diagram illustrating a determination reference according to the embodiment.
FIG. 24 is a diagram illustrating a warning reference according to the embodiment.
FIG. 25 is a diagram illustrating an example of a setting screen of a determination reference and a warning reference according to the embodiment.
FIG. 26 is a flowchart illustrating a monitoring method according to the embodiment.
FIG. 27 is a diagram illustrating an example of an output device according to the embodiment.
FIG. 28 is a diagram illustrating an example of the output device according to the embodiment.
FIG. 29 is a block diagram illustrating a computer system according to the embodiment.
FIG. 30 is a view illustrating an operation of a work machine according to another embodiment.
FIG. 31 is a diagram illustrating a monitoring system according to still another embodiment.

Description of Embodiments

Hereinafter, embodiments according to the present disclosure will be described with reference to the drawings, but the present disclosure is not limited to the embodiments. Constituent elements in the embodiments to be described below can be combined as appropriate. Some of the constituent elements may not be used in some cases.

[Work Machine]

FIG. 1 is a side view illustrating a work machine 1 according to an embodiment. FIG. 2 is a top view illustrating the work machine 1 according to the embodiment. FIG. 3 is a front view illustrating the work machine 1 according to the embodiment.
The work machine 1 works at a work site. In the embodiment, the work machine 1 is a wheel loader which is a kind of articulated work machine. The work machine 1 performs an excavation operation of excavating an excavation target and a loading operation of loading an excavated object excavated by the excavation operation onto a loading target.
As illustrated in FIGS. 1, 2, and 3, the work machine 1 includes a vehicle body 2, a drive cab 3, a traveling apparatus 4, a front fender 7, a support member 8, a housing 9, working equipment 10, and an angle sensor 90.
The vehicle body 2 includes a front vehicle body 2F and a rear vehicle body 2R. The front vehicle body 2F and the rear vehicle body 2R are connected via a joint mechanism 2A.
The drive cab 3 is supported by the vehicle body 2. A drive room is provided in the drive cab 3. A drive seat is provided in the drive room. The work machine 1 is operated by a driver getting on the drive cab 3.
The traveling apparatus 4 supports the vehicle body 2 and travels on a ground 200 of a work site. The traveling apparatus 4 includes wheels 5 and tires 6. The tires 6 are installed in the wheels 5. The wheels 5 include front wheels 5F supported by the front vehicle body 2F and rear wheels 5R supported by the rear vehicle body 2R. The tires 6 include front tires 6F attached to the front wheels 5F and rear tires 6R attached to the rear wheels 5R.
The front wheels 5F and the front tires 6F are rotatable about a rotation axis FX. The rear wheel 5R and the rear tire 6R are rotatable about the rotation axis RX.
In the embodiment, a direction parallel to the rotation axis FX of the front wheel 5F is appropriately referred to as a vehicle width direction. A direction orthogonal to a ground contact surface of the front tire 6F coming into contact with the ground 200 is appropriately referred to as an up-down direction. A direction orthogonal to both the vehicle width direction and the up-down direction is appropriately referred to as a front-rear direction. When the vehicle body 2 of the work machine 1 travels straight, the rotation axes FX and RX are parallel to each other.
In the embodiment, a position or a direction close to the center CL of the work machine 1 in the vehicle width direction of the work machine 1 is appropriately referred to as an inner side in the vehicle width direction, and a position or a direction away from the center CL is appropriately referred to as an outer side in the vehicle width direction.
One side of the center CL of the work machine 1 in the vehicle width direction is the left, and the opposite side of the left is the right. A position or a direction close to the working equipment 10 when a drive seat of the drive cab 3 in the front-rear direction is a reference is the front, and the opposite side of the front is the rear. A position or a direction close to the ground contact surface of the front tire 6F in the up-down direction is the lower side, and the opposite side of the lower side is the upper side.
The rear vehicle body 2R is disposed behind the front vehicle body 2F. The front vehicle body 2F is bent leftward and rightward with respect to the rear vehicle body 2R. The rear wheel 5R is disposed behind the front wheel 5F. The rear tire 6R is disposed behind the front tire 6F. The front wheel 5F and the front tire 6F are disposed on the left and right sides of the center CL of the work machine 1 in the vehicle width direction of the work machine 1. The rear wheel 5R and the rear tire 6R are disposed on the left and right sides of the center CL of the work machine 1 in the vehicle width direction of the work machine 1.
The tire 6 is a rotating member that rotates while coming into contact with the ground 200. The work machine 1 travels on the ground 200 by the rotation of the tires 6. The front tire 6F is a front rotating member that rotates while coming into contact with the ground 200. The rear tire 6R is a rear rotating member that is disposed behind the front tire 6F and rotates while coming into contact with the ground 200.
The front fender 7 prevents earth and sand scattered from the ground 200 during traveling of the work machine 1 from hitting the vehicle body 2 and the drive cab 3. A part of the front fender 7 is disposed above the front tire 6F. A part of the front fender 7 is disposed on the rear side of the front tire 6F. The front fender 7 is disposed on each of the left and right sides of the center CL of the work machine 1 in the vehicle width direction of the work machine 1. The front fender 7 includes a front fender 7L disposed on the left side of the center CL and a front fender 7R disposed on the right side of the center CL. The left front fender 7L is attached to a left portion of the front vehicle body 2F. The right front fender 7R is attached to a right portion of the front vehicle body 2F.
The support member 8 is a rodlike member. The lower end of the support member 8 is fixed to the front vehicle body 2F. The support member 8 is inclined upward toward the front side and outward in the vehicle width direction. The support member 8 is disposed on each of the left and right sides of the center CL of the work machine 1 in the vehicle width direction of the work machine 1. The support member 8 includes a support member 8L disposed on the left side of the center CL and a support member 8R disposed on the right side of the center CL. The lower end of the left support member 8L is fixed to the left portion of the front vehicle body 2F. The left support member 8L is inclined frontward and leftward toward the upper side. The lower end of the right support member 8R is fixed to the right portion of the front vehicle body 2F. The right support member 8R is inclined to the front and right sides upwards.
The housing 9 is fixed to the upper end of the support member 8. The housing 9 is supported by the front vehicle body 2F via the support member 8. In the front-rear direction, the housing 9 is disposed between the joint mechanism 2A and the front end of the front vehicle body 2F. In the up-down direction, the housing 9 is disposed between the upper end of the drive cab 3 and the upper end of the front vehicle body 2F. The housing 9 is disposed on each of the left and right sides of the center CL of the work machine 1 in the vehicle width direction of the work machine 1. The housing 9 includes a housing 9L disposed on the left side of the center CL and a housing 9R disposed on the right side of the center CL. The left housing 9L is fixed to the upper end of the left support member 8L. The right housing 9R is fixed to the upper end of the right support member 8R. The left housing 9L is disposed above the left front fender 7L. The right housing 9R is disposed above the right front fender 7R.
The working equipment 10 operates in excavation work and loading work. The working equipment 10 is connected to the front vehicle body 2F. At least a part of the working equipment 10 is disposed in front of the front wheel 5F. The working equipment 10 includes a boom 11, bucket 12, a bell crank 15, and a bucket link 16.
The boom 11 is rotatably connected to the front vehicle body 2F. The boom 11 connects the front vehicle body 2F and the bucket 12. The boom 11 is operated by a driving force generated by a boom cylinder 13. The boom cylinder 13 is a hydraulic cylinder. One end of the boom cylinder 13 is connected to the front vehicle body 2F. The other end of the boom cylinder 13 is connected to the boom 11. The boom cylinder 13 is disposed on each of the left and right sides of the center CL of the work machine 1 in the vehicle width direction of the work machine 1.
The bucket 12 excavates an excavation target. The bucket 12 is rotatably connected to a distal end of the boom 11. The bucket 12 is disposed in front of the front wheel 5F. The bucket 12 is operated by a driving force generated by a bucket cylinder 14. The bucket cylinder 14 is a hydraulic cylinder. A central portion of the bell crank 15 is rotatably connected to the boom 11. One end of the bucket cylinder 14 is connected to the front vehicle body 2F. The other end of the bucket cylinder 14 is connected to one end of the bell crank 15. The other end of the bell crank 15 is connected to the bucket 12 via a bucket link 16. One bucket cylinder 14 is disposed at the center in the vehicle width direction.
The bucket 12 includes a bucket body 17, bucket teeth 18, and inter-tooth protectors 19.
The bucket body 17 holds an excavated object. The bucket body 17 includes a bottom plate portion 17A, an upper plate portion 17B, a left plate portion 17C, and a right plate portion 17D. An accommodation space in which an excavated object is accommodated is formed by the bottom plate portion 17A, the upper plate portion 17B, the left plate portion 17C, and the right plate portion 17D. The distal end of the bottom plate portion 17A and the distal end of the upper plate portion 17B extend in the vehicle width direction. Each of the distal end of the left plate portion 17C and the distal end of the right plate portion 17D extends in the up-down direction or the front-rear direction. An opening 12M of the bucket 12 is defined by the distal end of the bottom plate portion 17A, the distal end of the upper plate portion 17B, the distal end of the left plate portion 17C, and the distal end of the right plate portion 17D. The excavated object can enter the accommodation space of the bucket 12 through the opening 12M.
As illustrated in FIGS. 2 and 3, the ends 12E on both sides in the vehicle width direction of the bucket body 17 are disposed outside of the tire 6 in the vehicle width direction of the bucket body 17. The distance in the vehicle width direction between the left end 12E and the right end 12E is larger than the distance in the vehicle width direction between the left side surface of the left tire 6 and the right side surface of the right tire 6.
The bucket teeth 18 configure a blade edge of the bucket 12. The bucket teeth 18 are attached to the bucket body 17. The bucket teeth 18 are attached to the distal end of the bottom plate portion 17A. The plurality of bucket teeth 18 are attached to the bucket body 17. The plurality of bucket teeth 18 are disposed at intervals in the vehicle width direction.
The bucket teeth 18 are exchangeable members which are exchangeably attached to the bucket body 17. The bucket teeth 18 are fixed to the bucket body 17 by bolts, for example. When the fixing by the bolt is released, the bucket teeth 18 can be separated from the bucket body 17.
In the embodiment, eight bucket teeth 18 are disposed at intervals in the vehicle width direction. The bucket teeth 18 include a bucket tooth 181 disposed on the leftmost side, a bucket tooth 182 disposed on the left side following the bucket tooth 181, a bucket tooth 183 disposed on the left side following the bucket tooth 182, a bucket tooth 184 disposed on the left side following the bucket tooth 183, a bucket tooth 185 disposed on the left side following the bucket tooth 184, a bucket tooth 186 disposed on the left side following the bucket tooth 185, a bucket tooth 187 disposed on the left side following the bucket tooth 186, and a bucket tooth 188 disposed on the rightmost side.
The inter-tooth protectors 19 protect the leading end of the bottom plate portion 17A. The inter-tooth protectors 19 are attached to the bucket body 17. The inter-tooth protectors 19 are disposed between pairs of bucket teeth 18 adjacent to each other. The inter-tooth protectors 19 are attached to the leading end of the bottom plate portion 17A. The plurality of inter-tooth protectors 19 are attached to the bucket body 17. The plurality of inter-tooth protectors 19 are disposed at intervals in the vehicle width direction.
The inter-tooth protector 19 is a replacement member that is exchangeably attached to the bucket body 17. The inter-tooth protector 19 is fixed to the bucket body 17 with, for example, a bolt. When the fixing by the bolt is released, the inter-tooth protector 19 can be separated from the bucket body 17.
In the embodiment, seven inter-tooth protectors 19 are disposed at intervals in the vehicle width direction. The inter-tooth protectors 19 include an inter-tooth protector 191 disposed on the leftmost side, an inter-tooth protector 192 disposed on the left side following the inter-tooth protector 191, an inter-tooth protector 193 disposed on the left side following the inter-tooth protector 192, an inter-tooth protector 194 disposed on the left side following the inter-tooth protector 193, an inter-tooth protector 195 disposed on the left side following the inter-tooth protector 194, an inter-tooth protector 196 disposed on the left side following the inter-tooth protector 195, and an inter-tooth protector 197 disposed on the rightmost side.
The angle sensor 90 is an attitude sensor that detects an attitude of the working equipment 10. The attitude of the working equipment 10 includes an angle of the working equipment 10. The angle sensor 90 is included in the working equipment 10. The angle sensor 90 includes a boom angle sensor 91 that detects an angle of the boom 11 and a bucket angle sensor 92 that detects an angle of the bucket 12. The boom angle sensor 91 detects, for example, an angle of the boom 11 with respect to a reference axis of a vehicle body coordinate system defined in the front vehicle body 2F. The bucket angle sensor 92 detects an angle of the bucket 12 with respect to the boom 11. A potentiometer is exemplified as the angle sensor 90. The attitude sensor that detects an attitude of the working equipment 10 may be a stroke sensor that detects a stroke of a hydraulic cylinder (the boom cylinder 13 and the bucket cylinder 14).

[Drive Room]

FIG. 4 is a diagram illustrating a drive room of the drive cab 3 according to the embodiment. FIG. 4 is a view schematically illustrating a scene viewed from a drive seated on a drive seat in a drive cab.
In the drive room, a driving operation device 20, an operation panel 21, a monitor device 22, a rear view monitor device 23, an output device 24, and an input device 25 are disposed. The driving operation device 20 is operated by a driver. The driving operation device 20 includes an accelerator pedal 20A and a pair of brake pedals 20B. Although not illustrated, the driving operation device 20 includes a steering lever, a shift lever, a forward or backward switching lever, and a work lever. When the accelerator pedal 20A is operated, the traveling speed of the work machine 1 increases. When the brake pedal 20B is operated, the traveling speed of the work machine 1 decreases or the traveling of the work machine 1 stops. When the steering lever is operated, the work machine 1 swings. When the shift lever is operated, a speed stage of the work machine 1 is changed. When the forward or backward switching lever is operated, the traveling direction of the work machine 1 is switched between the forward direction and the backward direction. When the working lever is operated, the working equipment 10 is operated.
The output device 24 supplies output data to the driver. The output data includes a warning. The output device 24 includes a display device 24A and a sounding device 24B.
The display device 24A supplies display data as output data to the driver. As the display device 24A, a flat panel display such as a liquid crystal display (LCD) or an organic electroluminescence display (OELD) is exemplified.
The sounding device 24B supplies sound data as output data to the driver. As the sounding device 24B, a buzzer or a speaker is exemplified.
The input device 25 is operated by the driver to generate input data. As the input device 25, a button or a keyboard is exemplified. Note that the input device 25 may include, for example, a touch panel.
A windshield is provided in a front portion of the drive room. The windshield is attached to a pillar 27 of the drive room. In the embodiment, the output device 24 is supported by the pillar 27.

[Measurement Device]

FIG. 5 is an enlarged front view illustrating a part of the work machine 1 according to the embodiment. As illustrated in FIGS. 3 and 5, the work machine 1 includes a headlight 28, a blinker lamp 29, a first measurement device 31, and a second measurement device 32.
The headlight 28 emits illumination light forward to illuminate an illumination area in front of the front vehicle body 2F. The headlight 28 is held by the housing 9. The headlight 28 is disposed on each of the left and right sides of the center CL of the work machine 1 in the vehicle width direction of the work machine 1. The headlight 28 includes a headlight 28L disposed on the left side of the center CL and a headlight 28R disposed on the right side of the center CL. The left headlight 28L is held by the left housing 9L. The right headlight 28R is held by the right housing 9R.
The blinker lamps 29 are turned on or off to display the turning direction of the work machine 1. The blinker lamps 29 are held by the housing 9. The blinker lamps 29 are disposed on the left and right sides of the center CL of the work machine 1 in the vehicle width direction of the work machine 1. The blinker lamps 29 include a blinker lamp 29L disposed on the left side of the center CL and a blinker lamp 29R disposed on the right side of the center CL. The left blinker lamp 29L is held by the left housing 9L. The right blinker lamp 29R is held by the right housing 9R. The blinker lamps 29 are disposed outside of the headlights 28 in the vehicle width direction of the work machine 1.
The first measurement device 31 measures a predetermined target. A target to be measured by the first measurement device 31 includes the working equipment 10. The first measurement device 31 measures the working equipment 10. The first measurement device 31 acquires measurement data of the working equipment 10. In the embodiment, the first measurement device 31 is an imaging device that images the working equipment 10. The measurement data of the working equipment 10 includes image data of the working equipment 10. In the following description, the first measurement device 31 is appropriately referred to as a first imaging device 31.
The second measurement device 32 measures a predetermined target. A target to be measured by the second measurement device 32 includes the ground 200 on which the traveling apparatus 4 travels. The second measurement device 32 measures the ground 200 on which the work machine 1 travels. The second measurement device 32 acquires measurement data of the ground 200. In the embodiment, the second measurement device 32 is an imaging device that images the ground 200. The measurement data of the ground 200 includes image data of the ground 200. In the following description, the second measurement device 32 is appropriately referred to as a second imaging device 32.
Each of the first imaging device 31 and the second imaging device 32 includes an optical system and an image sensor on which light having passed through the optical system is incident. Examples of the image sensor include a couple charged device (CCD) image sensor and a complementary metal oxide semiconductor (CMOS) image sensor.
The first imaging device 31 and the second imaging device 32 are each disposed in the housing 9. In the vehicle width direction of the work machine 1, the first imaging device 31 and the second imaging device 32 are each disposed outside of the headlights 28. In the housing 9, the second imaging device 32 is disposed above the first imaging device 31.
The first imaging device 31 is disposed on each of the left and right sides of the center CL of the work machine 1 in the vehicle width direction of the work machine 1. The first imaging device 31 is disposed in each of the left and right housings 9. The first imaging device 31 includes a first imaging device 31L disposed on the left side of the center CL and a first imaging device 31R disposed on the right side of the center CL.
The second imaging device 32 is disposed on each of the left and right sides of the center CL of the work machine 1 in the vehicle width direction of the work machine 1. The second imaging device 32 is disposed in each of the left and right housings 9. The second imaging device 32 includes a second imaging device 32L disposed on the left side of the center CL and a second imaging device 32R disposed on the right side of the center CL.

[Drive System]

FIG. 6 is a diagram illustrating a drive system 40 of the work machine 1 according to the embodiment. The drive system 40 includes an engine 41, a fuel injection device 42, a power take-off (PTO) 43, a transmission 44, a front axle 45F, a rear axle 45R, a hydraulic pump 46, a control valve 47, and a drive controller 48.
The engine 41 is, for example, a diesel engine. The fuel injection device 42 injects fuel into a cylinder of the engine 41. A driving force of the engine 41 is adjusted by adjusting an amount of fuel injected from the fuel injection device 42 to the engine 41.
The power take-off 43 distributes the driving force of the engine 41 to the transmission 44 and the hydraulic pump 46. The driving force of the engine 41 is transmitted to the transmission 44 and the hydraulic pump 46 via the power take-off 43.
The transmission 44 includes an input shaft to which the driving force of the engine 41 is input and an output shaft that changes a speed of the driving force input to the input shaft and outputs the changed speed of the driving force. The input shaft of the transmission 44 is connected to the power take-off 43. The output shaft of the transmission 44 is connected to each of the front axle 45F and the rear axle 45R. The driving force of the engine 41 is transmitted to each of the front axle 45F and the rear axle 45R via the power take-off 43 and the transmission 44.
The transmission 44 switches a traveling direction of the work machine 1 between the forward direction and the backward direction. The transmission 44 includes a forward gear 44F and a reverse gear 44R. When the forward gear 44F is engaged, the traveling direction of the work machine 1 is determined to be the forward direction. When the reverse gear 44R is engaged, the traveling direction of the work machine 1 is determined to be the reverse direction.
The front axle 45F transmits the driving force transmitted from the transmission 44 to the front wheel 5F. The front wheel 5F is rotated based on the driving force transmitted from the front axle 45F.
The rear axle 45R transmits the driving force transmitted from the transmission 44 to the rear wheel 5R. The rear wheel 5R is rotated based on the driving force transmitted from the rear axle 45R.
When the forward gear 44F is engaged, the work machine 1 enters a forward movable state. In a state where the forward gear 44F is engaged, the accelerator pedal 20A is operated, and the driving force of the engine 41 is transmitted to each of the front axle 45F and the rear axle 45R via the transmission 44, and thus the work machine 1 moves forward. Even in the state where the forward gear 44F is engaged, the work machine 1 does not move forward, for example, in a state where the brake pedal 20B is operated.
When the reverse gear 44R is engaged, the work machine 1 enters a reverse movable state. In the state where the reverse gear 44R is engaged, the accelerator pedal 20A is operated, and the driving force of the engine 41 is transmitted to each of the front axle 45F and the rear axle 45R via the transmission 44, and thus the work machine 1 moves reversely. Even in the state where the reverse gear 44R is engaged, the work machine 1 does not move reversely, for example, in a state where the brake pedal 20B is operated.
The hydraulic pump 46 discharges hydraulic oil. The hydraulic pump 46 is a variable displacement hydraulic pump. The hydraulic pump 46 is driven based on a driving force of the engine 41. The hydraulic oil discharged from the hydraulic pump 46 is supplied to at least one of the boom cylinder 13 and the bucket cylinder 14 via the control valve 47.
The control valve 47 controls a flow rate and a direction of the hydraulic oil supplied to each of the boom cylinder 13 and the bucket cylinder 14. The working equipment 10 is operated with the hydraulic oil supplied from the hydraulic pump 46 via the control valve 47.
The drive controller 48 controls the work machine 1 based on an operation signal output from the driving operation device 20. The drive controller 48 includes a computer system.

[Operation of Working Equipment]

FIG. 7 is a diagram illustrating an operation of the working equipment 10 according to the embodiment. In the embodiment, the working equipment 10 is a front-loading type working equipment in which the opening 12M of the bucket 12 faces forward in the excavation work.
When the boom cylinder 13 extends and contracts, the boom 11 performs a raising operation or a lowering operation.
The raising operation of the boom 11 is an operation in which the boom 11 performs a rotating operation so that the distal end of the boom 11 is separated from the ground 200. When the boom cylinder 13 extends, the boom 11 performs the raising operation.
The lowering operation of the boom 11 is an operation in which the boom 11 performs a rotating operation so that the distal end of the boom 11 approaches the ground 200. When the boom cylinder 13 contracts, the boom 11 performs the lowering operation.
When the bucket cylinder 14 expands and contracts, the bucket 12 performs a tilting operation or a dumping operation.
The tilting operation of the bucket 12 is an operation in which the bucket 12 performs a rotating operation such that the distal end of the bucket 12 is separated from the ground 200 in a state in which the opening 12M of the bucket 12 faces upward. As the bucket cylinder 14 extends, the bell crank 15 performs a rotating operation such that the upper end of the bell crank 15 moves forward and the lower end of the bell crank 15 moves backward. When the lower end of the bell crank 15 moves backward, the bucket 12 is pulled backward by the bucket link 16 and performs the tilting operation. When the bucket 12 performs the tilting operation, the excavation target is scooped by the bucket 12, and the excavated object is held by the bucket 12.
The dumping operation of the bucket 12 is an operation in which the bucket 12 rotates so that the distal end of the bucket 12 approaches the ground 200 in a state in which the opening 12M of the bucket 12 faces downward. As the bucket cylinder 14 contracts, the bell crank 15 rotates such that the upper end of the bell crank 15 moves backward and the lower end of the bell crank 15 moves forward. When the lower end of the bell crank 15 moves forward, the bucket 12 is pushed forward by the bucket link 16 and performs the dumping operation. When the bucket 12 performs the dumping operation, the excavated object held by the bucket 12 is discharged from the bucket 12.

[Operation of Work Machine]

FIG. 8 is a diagram illustrating an operation of the work machine 1 according to the embodiment. The work machine 1 performs excavation work and loading work at a work site.
The excavation work is work for excavating an excavation target. Examples of the excavation target include a natural ground and a rocky mountain. The natural ground is a mountain formed of earth and sand placed on the ground 200. The rock mountain is a mountain formed of rocks or stones placed on the ground 200. In the embodiment, the excavation target is a natural ground 210. An excavated object is a part of the natural ground 210 excavated and held by the bucket 12.
The loading work is work for loading an excavated object excavated by the excavation work onto a loading target. Examples of the loading target include a haul vehicle, a hopper, and a belt conveyor. In the embodiment, the loading target is a dump body 230 of a dump truck 220 that is a type of haul vehicle.
The work machine 1 is operated by a driver so that cyclic work for repeating a series of work modes is performed. The cyclic work includes a plurality of work modes that are mutually related. The cyclic work includes traveling of the traveling apparatus 4 and an operation of the working equipment 10.
In the embodiment, the cyclic work includes six work modes. In the embodiment, a series of work modes of the work machine 1 includes an unloaded forward mode, an excavation mode, a loading reverse mode, a loading forward mode, a loading mode, and an unloaded reverse mode.
The order of the series of work modes is determined. After the unloaded forward mode is implemented, the excavation mode is implemented. After the excavation mode is performed, the loading reverse mode is performed. After the loading reverse mode is performed, the loading forward mode is performed. After the loading forward mode is performed, the loading mode is performed. After the loading mode is implemented, the unloaded reversing mode is implemented.
The unloaded forward mode is a work mode in which the work machine 1 moves forward to approach the excavation target in a state where the excavated object is not held by the bucket 12. In the unloaded forward mode, the work machine 1 moves forward so as to approach the natural ground 210 as indicated by an arrow M1 in FIG. 8 in a state where the excavated object is not held by the bucket 12.
The excavation mode is a work mode in which the bucket 12 of the working equipment 10 excavates the excavation target. In the excavation mode, an excavation operation is performed. When the bucket 12 performs a tilting operation in a state where at least a part of the bucket 12 is inserted into the natural ground 210, the natural ground 210 is excavated by the bucket 12, and the excavated object is held by the bucket 12.
The loading reverse mode is a work mode in which the work machine 1 moves backward to be separated from the excavation target in a state where the excavated object is held by the bucket 12. In the loading reverse mode, the work machine 1 moves backward to be separated from the natural ground 210 as indicated by an arrow M2 in FIG. 8 in a state where the excavated object is held by the bucket 12.
The loading forward mode is a work mode in which the work machine 1 moves forward to approach the loading target in a state where the excavated object is held by the bucket 12. In the loading forward mode, the work machine 1 moves forward while swinging toward the dump truck 220 as indicated by an arrow M3 in FIG. 8 in a state where the excavated object is held by the bucket 12. In a state where the work machine 1 moves forward toward the dump truck 220, the boom 11 performs a raising operation so that the bucket 12 is disposed above the dump body 230.
The loading mode is a work mode in which an excavated object held by the bucket 12 of the working equipment 10 is loaded onto a loading target. In the loading mode, the loading operation is performed. After the bucket 12 is disposed above the dump body 230, the bucket 12 performs a dumping operation, so that the excavated object held by the bucket 12 is discharged from the bucket 12 and loaded into the dump body 230.
The unloaded backward mode is a work mode in which the work machine 1 moves backward to be separated from the loading target in a state where the excavated object is not held by the bucket 12. The work machine 1 moves backward while swinging to be separated from the dump truck 220 as indicated by an arrow M4 in FIG. 8 in a state where the excavated object is not held by the bucket 12.
The work machine 1 repeats the cyclic work including the unloaded forward mode, the excavation mode, the loading reverse mode, the loading forward mode, the loading mode, and the unloaded backward mode until the excavated object is loaded onto the dump body 230 with the target loading amount.

[Measurement Range]

FIG. 9 is a side view illustrating the work machine 1 that performs the loading work according to the embodiment. FIG. 10 is a top view illustrating the work machine 1 that performs the loading work according to the embodiment.
As illustrated in FIGS. 9 and 10, when an excavated object is loaded onto the dump body 230 of the dump truck 220, the work machine 1 performs the raising operation of the boom 11 such that the bucket 12 is disposed above the dump body 230. After the boom 11 performs the raising operation and the bucket 12 is disposed above the dump body 230, the work machine 1 causes the bucket 12 to perform the dumping operation. The bucket 12 performs a dumping operation such that the excavated object held by the bucket 12 is discharged to the dump body 230 of the dump truck 220 in a state where the boom 11 performs the raising operation. The first imaging device 31 images at least the bucket 12 that performs the dumping operation in order to discharge the excavated object to the dump body 230. By the dumping operation of the bucket 12, the excavated object held by the bucket 12 is discharged from the bucket 12 and loaded into the dump body 230.
A measurement range Ra of the first imaging device 31 includes an imaging range of the first imaging device 31. The measurement range Ra is determined based on a first angle of view a indicating an angle of view of the optical system of the first imaging device 31. The first imaging device 31 images the bucket 12 that performs the dumping operation in order to discharge the excavated object to the dump body 230. The first imaging device 31 is fixed to the housing 9 such that the bucket 12 including the bucket teeth 18 is disposed in the measurement range Ra when the boom 11 performs the raising operation and the bucket 12 performs the dumping operation.
A measurement range Rb of the second imaging device 32 includes an imaging range of the second imaging device 32. The measurement range Rb is determined based on a second angle of view β indicating an angle of view of the optical system of the second imaging device 32. The second imaging device 32 images at least the ground 200 in a traveling direction of the tire 6. In the embodiment, when the work machine 1 moves forward, the second imaging device 32 images the ground 200 in front of the front tires 6F. The second imaging device 32 is fixed to the housing 9 such that at least the ground 200 in front of the front tires 6F is disposed in the measurement range Rb when the work machine 1 moves forward. In the embodiment, the second imaging device 32 is fixed to the housing 9 such that each of the front end of the front tire 6F, the ground 200 in front of the front tire 6F, and the ground 200 outside of the front tire 6F in the vehicle width direction is disposed in the measurement range Rb. The ground 200 in front of the front tire 6F includes the ground 200 between the front end of the front tire 6F and the rear end of the bucket 12 in the front-rear direction.
In the embodiment, the second angle of view β of the second imaging device 32 is wider than the first angle of view a of the first imaging device 31.
As illustrated in FIG. 10, the first imaging device 31L on the left side of the center CL images at least a left portion of the bucket 12 of the working equipment 10. The first imaging device 31R on the right side of the center CL images at least a right portion of the bucket 12 of the working equipment 10. The left portion of the bucket 12 is a portion of the bucket 12 between the left end 12E of the bucket 12 and the center CL. The right portion of the bucket 12 is a portion of the bucket 12 between the right end 12E of the bucket 12 and the center CL.
In the example illustrated in FIG. 10, the first imaging device 31L on the left side of the center CL is fixed to the left housing 9 such that the bucket teeth 181, 182, 183, 184, and 185 are disposed in the measurement range Ra when the boom 11 performs the raising operation and the bucket 12 performs the dumping operation. The first imaging device 31R on the right side of the center CL is fixed to the right housing 9 such that the bucket teeth 184, 185, 186, 187, and 188 are disposed in the measurement range Ra when the boom 11 performs the raising operation and the bucket 12 performs the dumping operation.
As illustrated in FIG. 10, the second imaging device 32L on the left side of the center CL images the ground 200 in front of the front tire 6F on the left side of the center CL. The second imaging device 32R on the right side of the center CL images the ground 200 in front of the front tire 6F on the right side of the center CL.
In the example illustrated in FIG. 10, the second imaging device 32L on the left side of the center CL is fixed to the left housing 9 such that the front end of the left front tire 6F, the ground 200 in front of the left front tire 6F, and the ground 200 on the left side of the left front tire 6F are disposed in the measurement range Rb. The second imaging device 32R on the right side of the center CL is fixed to the right housing 9 such that the front end of the right front tire 6F, the ground 200 in front of the right front tire 6F, and the ground 200 on the right side of the right front tire 6F are disposed in the measurement range Rb.
FIG. 11 is a diagram illustrating an example of image data captured by the first imaging device 31L on the left according to the embodiment. FIG. 12 is a diagram illustrating an example of image data captured by the second imaging device 32R on the left according to the embodiment.
As illustrated in FIG. 11, the first imaging device 31L images the bucket 12 performing the dumping operation so that the excavated object is discharged to the dump body 230 so as to look up from below. Accordingly, as illustrated in FIG. 11, the first imaging device 31L can simultaneously image a plurality of bucket teeth 18. In the example illustrated in FIG. 11, the bucket teeth 181, 182, 183, 184, 185, and 186 are disposed in the measurement range Ra of the first imaging device 31.
As illustrated in FIG. 12, the second imaging device 32L images the ground 200 in front of the front tires 6F and the ground 200 on the left side so as to look down from above. Accordingly, the second imaging device 32L can image the ground 200 around the front tire 6F. For example, when there is a rolling stone 240 on the ground 200 around the front tire 6F, the second imaging device 32L can image the rolling stone 240 on the ground 200.

[Monitoring System]

FIG. 13 is a functional block diagram illustrating a monitoring system 30 according to the embodiment. The work machine 1 includes the monitoring system 30. The monitoring system 30 monitors the working equipment 10 and the ground 200 on which the work machine 1 travels.
The monitoring system 30 includes the first imaging device 31, the second imaging device 32, the driving operation device 20, the angle sensor 90, the output device 24, the input device 25, and a monitoring controller 33. The first imaging device 31 includes a first imaging device 31L disposed on the left side of the center CL and a first imaging device 31R disposed on the right side of the center CL. The second imaging device 32 includes a second imaging device 32L disposed on the left side of the center CL and a second imaging device 32R disposed on the right side of the center CL.
Each of the first imaging device 31, the second imaging device 32, the driving operation device 20, the angle sensor 90, the output device 24, and the input device 25 is connected to the monitoring controller 33. Image data captured by the first imaging device 31 is input to the monitoring controller 33. Image data captured by the second imaging device 32 is input to the monitoring controller 33. Operation data of the driving operation device 20 is input to the monitoring controller 33. Detection data of the angle sensor 90 is input to the monitoring controller 33. The input data generated by the input device 25 is input to the monitoring controller 33.
In the embodiment, each of the first imaging device 31 and the second imaging device 32 acquires measurement data at a predetermined sampling rate (samples per second (SPS)). The measurement data includes image data. The sampling rate includes a frame rate (frames per second (FPS)). Each of the first imaging device 31 and the second imaging device 32 captures image data at a predetermined frame rate. When the first imaging device 31 captures image data at a predetermined frame rate, a moving image is captured. When the second imaging device 32 captures image data at a predetermined frame rate, a moving image is captured.
The monitoring controller 33 recognizes presence or absence of a defect of the working equipment 10 based on the image data captured by the first imaging device 31. When the defect of the working equipment 10 is recognized, the monitoring controller 33 causes the output device 24 to output a warning indicating that there is a defect in the working equipment 10. The defect of the working equipment 10 includes a defect of the bucket 12. The defect of the bucket 12 includes a defect of the bucket teeth 18 which are replacement members. The defect of the bucket teeth 18 includes a falling of the bucket teeth 18 from the bucket body 17.
The monitoring controller 33 recognizes presence or absence of the rolling stone 240 on the ground 200 on which the work machine 1 travels based on the image data captured by the second imaging device 32. When the rolling stone 240 on the ground 200 is recognized, the monitoring controller 33 causes the output device 24 to output a warning indicating that there is the rolling stone 240 on the ground 200. The rolling stone 240 of the ground 200 includes the rolling stone 240 of the ground 200 in the traveling direction of the front tires 6F. The rolling stone 240 on the ground 200 in the traveling direction of the front tire 6F includes the rolling stone 240 on the ground 200 in front of the front tire 6F when the work machine 1 moves forward.
In the cyclic work described with reference to FIG. 8, the monitoring controller 33 recognizes presence or absence of a defect of the working equipment 10 and recognizes presence or absence of the rolling stone 240. The monitoring controller 33 may recognize presence or absence of a defect of the working equipment 10 or presence or absence of the rolling stone 240 in work different from the cyclic work. The monitoring controller 33 may recognize the presence or absence of a defect of the working equipment 10 in starting inspection before the work machine 1 starts the work.
The monitoring controller 33 determines whether a reception surface of the first imaging device 31 is contaminated based on image data captured by the first imaging device 31. When it is determined that the first imaging device 31 is contaminated, the monitoring controller 33 causes the output device 24 to output a warning indicating that the reception surface of the first imaging device 31 is contaminated. When it is determined that the reception surface of the first imaging device 31 is contaminated, the monitoring controller 33 can cause the display device 24A to display data indicating a warning or cause the sounding device 24B to output a warning sound. The reception surface of the first imaging device 31 includes an incidence surface of the optical system of the first imaging device 31.
The monitoring controller 33 determines whether the first imaging device 31 is contaminated in the cycle operation described with reference to FIG. 8. The monitoring controller 33 may determine whether the incidence surface of the optical system of the first imaging device 31 is contaminated in work different from the cyclic work. The monitoring controller 33 may determine whether the incidence surface of the optical system of the first imaging device 31 is contaminated in the starting inspection before the work machine 1 starts work.
The monitoring controller 33 includes a measurement data acquisition unit 331, a vehicle body data acquisition unit 332, a state determination unit 333, an algorithm selection unit 334, a processing unit 335, a contamination determination unit 336, an output unit 337, and an algorithm storage unit 338.
The measurement data acquisition unit 331 acquires measurement data of the first imaging device 31 and measurement data of the second imaging device 32. The measurement data acquisition unit 331 acquires measurement data of the working equipment 10 measured by the first imaging device 31 from the first imaging device 31 at a predetermined sampling rate. The measurement data of the working equipment 10 includes image data of the working equipment 10 captured by the first imaging device 31. The sampling rate includes a frame rate. The measurement data acquisition unit 331 acquires image data of the working equipment 10 from the first imaging device 31 at a predetermined frame rate. The measurement data acquisition unit 331 acquires measurement data of the ground 200 measured by the second imaging device 32 from the second imaging device 32 at a predetermined sampling rate. The measurement data of the ground 200 includes image data of the ground 200 captured by the second imaging device 32. The sampling rate includes a frame rate. The measurement data acquisition unit 331 acquires the image data of the ground 200 from the second imaging device 32 at a predetermined frame rate. The measurement data acquisition unit 331 acquires image data of the ground 200 in front of the front tires 6F.
The vehicle body data acquisition unit 332 acquires vehicle body data indicating a state of the work machine 1. The state of the work machine 1 includes a working state of the work machine 1. The state of the work machine 1 includes a state of the working equipment 10 and a state of the traveling apparatus 4. The state of the working equipment 10 includes an attitude of the working equipment 10. The state of the traveling apparatus 4 includes forward movement and backward movement of the traveling apparatus 4. The vehicle body data includes attitude data indicating the attitude of the working equipment 10 and forward or backward data indicating forward or backward movement of the traveling apparatus 4.
The attitude data of the working equipment 10 includes an angle of the working equipment 10. The angle of the working equipment 10 is detected by the angle sensor 90. The attitude data of the working equipment 10 includes detection data of the angle sensor 90. The vehicle body data acquisition unit 332 acquires detection data of the angle sensor 90 as the attitude data of the working equipment 10.
The forward movement and the backward movement of the traveling apparatus 4 are switched by a forward or backward lever 20C of the driving operation device 20. The forward or backward data of the traveling apparatus 4 includes operation data of the forward or backward lever 20C. The vehicle body data acquisition unit 332 acquires the operation data of the forward or backward lever 20C as the forward or backward data of the traveling apparatus 4.
For example, when a rotation sensor that detects rotation of the wheel 5 is provided, the rotation sensor can detect forward movement or backward movement of the traveling apparatus 4 by detecting the rotation direction of the wheel 5. The vehicle body data acquisition unit 332 may acquire detection data of the rotation sensor as forward or backward data of the traveling apparatus 4.
The vehicle body data acquisition unit 332 may acquire state data indicating a state of the drive system 40 as forward or backward data of the traveling apparatus 4. The drive controller 48 inputs state data indicating the state of the drive system 40 to the monitoring controller 33. The vehicle body data acquisition unit 332 determines the state of the drive system 40 based on the state data from the drive controller 48. The state of the drive system 40 includes an operating state of the transmission 44. The transmission 44 switches a traveling direction of the work machine 1 between the forward direction and the backward direction. The transmission 44 includes a forward gear 44F that operates to advance the work machine 1 and a reverse gear 44R that operates to reverse the work machine 1. The vehicle body data acquisition unit 332 may acquire state data indicating the state of the transmission 44 as forward or backward data of the traveling apparatus 4.
The state determination unit 333 determines the state of the work machine 1 based on the vehicle body data acquired by the vehicle body data acquisition unit 332. The state determination unit 333 can determine whether the boom 11 is performing the raising operation or the lowering operation based on the attitude data of the working equipment 10 acquired by the vehicle body data acquisition unit 332. The state determination unit 333 can determine whether the bucket 12 is performing the tilting operation or the dumping operation based on the attitude data of the working equipment 10 acquired by the vehicle body data acquisition unit 332. The state determination unit 333 can determine whether the work machine 1 moves forward or backward based on the forward or backward data of the traveling apparatus 4 acquired by the vehicle body data acquisition unit 332.
The state determination unit 333 can determine whether the bucket 12 of the working equipment 10 is disposed in the measurement range Ra of the first imaging device 31 based on the attitude data of the working equipment 10. As described above, the first imaging device 31 is fixed to the housing 9 such that the bucket 12 is disposed in the measurement range Ra when the boom 11 performs the raising operation and the bucket 12 performs the dumping operation. The state determination unit 333 can determine whether the bucket teeth 18 are disposed in the measurement range Ra of the first imaging device 31 based on the attitude data of the working equipment 10. The state determination unit 333 can determine whether the inter-tooth protectors 19 are disposed in the measurement range Ra of the first imaging device 31 based on the attitude data of the working equipment 10.
The state of the work machine 1 may include a working mode of the work machine 1. When a weight sensor that detects whether an excavated object is held in the bucket 12 is provided, the state determination unit 333 can determine whether the work mode of the work machine 1 is any one of the unloaded forward mode, the excavation mode, the loading reverse mode, the loading forward mode, the loading mode, and the unloaded backward mode based on attitude data of the working equipment 10, forward or backward data of the traveling apparatus 4, and detection data of the weight sensor. As the weight sensor that detects whether an excavated object is held by the bucket 12, a pressure sensor that detects a pressure in a bottom chamber of the boom cylinder 13 is exemplified.
The algorithm selection unit 334 selects a specific algorithm for processing measurement data acquired by the measurement data acquisition unit 331 based on the state of the work machine 1 determined by the state determination unit 333. The algorithm storage unit 338 stores a plurality of algorithms different from each other in advance. The algorithm selection unit 334 selects a specific algorithm from a plurality of algorithms stored in the algorithm storage unit 338 based on the state of the work machine 1 determined by the state determination unit 333.
The algorithm stored in the algorithm storage unit 338 includes a first algorithm that processes image data captured by the first imaging device 31 and a second algorithm that processes image data captured by the second imaging device 32. The first algorithm includes an algorithm for identifying the working equipment 10 from the image data captured by the first imaging device 31. In the embodiment, the first algorithm includes an algorithm for identifying the bucket teeth 18 from the image data captured by the first imaging device 31. The second algorithm includes an algorithm for identifying the rolling stone 240 on the ground 200 from the image data captured by the second imaging device 32.
The algorithms include artificial intelligence (AI) algorithms. The first algorithm is different from the second algorithm. The first algorithm includes a first AI algorithm. The second algorithm includes a second AI algorithm. The first AI algorithm recognizes the bucket teeth 18 from the image data captured by the first imaging device 31 based on the first learning model generated by learning a feature of the bucket teeth 18. The first learning model is a learning model that has a feature of an object as an input and has the bucket teeth 18 as an output. The second AI algorithm is a learning model that specifies the rolling stone 240 from image data captured by the second imaging device 32 based on a second learning model generated by learning the feature of the rolling stone 240. The second learning model is a learning model that has the feature of the object as an input and has the rolling stone 240 as an output.
When the state determination unit 333 determines that the bucket 12 of the working equipment 10 is disposed in the measurement range Ra of the first imaging device 31 based on the attitude data of the working equipment 10, the algorithm selection unit 334 selects the first algorithm as a specific algorithm from the first algorithm and the second algorithm stored in the algorithm storage unit 338. When the state determination unit 333 determines that the bucket 12 of the working equipment 10 is not disposed in the measurement range Ra of the first imaging device 31 based on the attitude data of the working equipment 10, the algorithm selection unit 334 does not select the first algorithm.
When the state determination unit 333 determines that the traveling apparatus 4 is moving forward based on the forward or backward data of the traveling apparatus 4, the algorithm selection unit 334 selects the second algorithm as a specific algorithm between the first and second algorithms stored in the algorithm storage unit 338. When the state determination unit 333 determines that the traveling apparatus 4 is not moving forward based on the forward or backward data of the traveling apparatus 4, the algorithm selection unit 334 does not select the second algorithm.
The processing unit 335 processes the image data acquired by the measurement data acquisition unit 331 and recognizes the target. The processing unit 335 processes the image data based on the specific algorithm selected by the algorithm selection unit 334 and recognizes the target.
When the first algorithm is selected by the algorithm selection unit 334, the processing unit 335 processes the image data of the first imaging device 31 based on the first algorithm to recognize the working equipment 10. In the embodiment, the processing unit 335 processes the image data of the first imaging device 31 based on the first algorithm, and determines presence or absence of a defect of the working equipment 10. In the embodiment, the processing unit 335 determines presence or absence of a falling off of the bucket teeth 18 from the bucket body 17 in the image data of the first imaging device 31.
When the algorithm selection unit 334 selects the second algorithm, the processing unit 335 processes the image data of the second measurement device 32 based on the second algorithm to recognize the ground 200. In the embodiment, the processing unit 335 processes the image data of the second imaging device 32 based on the second algorithm and determines presence or absence of the rolling stone 240 on the ground 200. In the embodiment, the processing unit 335 determines the presence or absence of the rolling stone 240 on the ground 200 in front of the front tire 6F in the image data of the second imaging device 32.
Since a specific algorithm for processing the image data is selected based on the state of the work machine 1, the processing unit 335 can efficiently recognize a plurality of targets. In a situation where the bucket 12 is disposed in the measurement range Ra of the first imaging device 31, the processing unit 335 processes the image data captured by the first imaging device 31 based on the first algorithm to recognize the bucket teeth 18. In a situation where the bucket 12 is not disposed in the measurement range Ra of the first imaging device 31, the processing unit 335 does not perform a process of recognizing the bucket teeth 18. In a situation where there is a high possibility of the work machine 1 moving forward and the front tires 6F step on the rolling stone 240 on the ground 200 in front of the front tires 6F, the processing unit 335 processes the image data captured by the second imaging device 32 based on the second algorithm to recognize the rolling stone 240 on the ground 200 in front of the front tires 6F. In a situation where there is a low possibility of the work machine 1 reversing or stopping and the front tires 6F stepping on the rolling stone 240 on the ground 200 in front of the front tires 6F, the processing unit 335 does not perform the process of recognizing the rolling stone 240. As described above, the monitoring controller 33 may not always perform the process of recognizing both the bucket teeth 18 and the rolling stone 240. Therefore, a burden on the monitoring controller 33 required to monitor the bucket teeth 18 and the rolling stone 240 is reduced. Therefore, the monitoring controller 33 can efficiently monitor each of the bucket teeth 18 and the rolling stone 240.
The contamination determination unit 336 determines whether the incidence surface of the optical system of the first imaging device 31 is contaminated based on the state of the work machine 1 determined by the state determination unit 333 and the recognition result of the working equipment 10 by the processing unit 335. There is a high possibility of earth and sand or dust being scattered at the work site of the work machine 1. As a result, there is a possibility of foreign matters adhering to the incidence surface of the optical system of the first imaging device 31. There is a possibility of the incidence surface of the optical system of the first imaging device 31 being contaminated due to the adhesion of the foreign matters.
When the state determination unit 333 determines that the bucket 12 of the working equipment 10 is disposed in the measurement range Ra of the first imaging device 31 based on the attitude data of the working equipment 10 and the processing unit 335 recognizes the bucket 12 of the working equipment 10, the contamination determination unit 336 determines that the incidence surface of the optical system of the first imaging device 31 is not contaminated. In the embodiment, when the state determination unit 333 determines that the bucket teeth 18 are disposed in the measurement range Ra of the first imaging device 31 based on the attitude data of the working equipment 10 and the processing unit 335 recognizes the bucket teeth 18, the contamination determination unit 336 determines that the incidence surface of the optical system of the first imaging device 31 is not contaminated.
When the state determination unit 333 determines that the bucket 12 of the working equipment 10 is disposed in the measurement range Ra of the first imaging device 31 based on the attitude data of the working equipment 10 and the processing unit 335 does not recognize the bucket 12 of the working equipment 10, the contamination determination unit 336 determines that the incidence surface of the optical system of the first imaging device 31 is contaminated. In the embodiment, when the state determination unit 333 determines that the bucket teeth 18 are disposed in the measurement range Ra of the first imaging device 31 based on the attitude data of the working equipment 10 and the processing unit 335 does not recognize the bucket teeth 18, the contamination determination unit 336 determines that the incidence surface of the optical system of the first imaging device 31 is contaminated.
When the incidence surface of the optical system of the first imaging device 31 is not contaminated, the first imaging device 31 can correctly image the bucket teeth 18 by disposing the bucket teeth 18 in the measurement range Ra of the first imaging device 31. Therefore, the processing unit 335 can correctly identify the bucket teeth 18. Conversely, when the incidence surface of the optical system of the first imaging device 31 is contaminated, there is a high possibility that the first imaging device 31 cannot correctly image the bucket teeth 18 even when the bucket teeth 18 are disposed in the measurement range Ra of the first imaging device 31. Therefore, there is a high possibility that the processing unit 335 cannot identify the bucket teeth 18. As described above, in the embodiment, when the bucket teeth 18 that should be identified can be identified, the contamination determination unit 336 determines that the incidence surface of the optical system of the first imaging device 31 is not contaminated. When the bucket teeth 18 that should be identified cannot be identified, the contamination determination unit 336 determines that the incidence surface of the optical system of the first imaging device 31 is contaminated.
When the processing unit 335 processes the image data captured by the second imaging device 32 and recognizes the rolling stone 240, the contamination determination unit 336 may determine whether the incidence surface of the optical system of the second imaging device 32 is contaminated based on a recognition result of the rolling stone 240 by the processing unit 335. When the incidence surface of the optical system of the second imaging device 32 is not contaminated, the rolling stone 240 is disposed in the measurement range Rb of the second imaging device 32, and thus the second imaging device 32 can correctly image the rolling stone 240. Therefore, the processing unit 335 can correctly identify the rolling stone 240. Conversely, when the incidence surface of the optical system of the second imaging device 32 is contaminated, there is a high possibility that the second imaging device 32 cannot correctly image the rolling stone 240 even when the rolling stone 240 is disposed in the measurement range Rb of the second imaging device 32. Therefore, there is a high possibility that the processing unit 335 cannot identify the rolling stone 240. In this way, in the embodiment, when the rolling stone 240 that should be identified can be identified, the contamination determination unit 336 determines that the incidence surface of the optical system of the second imaging device 32 is not contaminated. When the rolling stone 240 that should be identified cannot be identified, the contamination determination unit 336 determines that the incidence surface of the optical system of the second imaging device 32 is contaminated.
The output unit 337 causes the output device 24 to output the recognition result of the object recognized by the processing unit 335. The output unit 337 causes the output device 24 to output the determination result of the presence or absence of contamination determined by the contamination determination unit 336.

[Determination of Defect of Working Equipment]

Each of FIGS. 14, 15, 16, 17, and 18 is a diagram illustrating a defect determination method of the working equipment 10 according to the embodiment. Hereinafter, as an example, a method of determining presence or absence of a defect of the bucket teeth 18 based on the image data captured by the first imaging device 31L on the left side of the center CL will be described.
As illustrated in FIG. 14, when the boom 11 performs the raising operation so that the bucket teeth 18 are disposed in the measurement range Ra of the first imaging device 31L and the bucket 12 performs the dumping operation, the algorithm selection unit 334 selects the first algorithm based on the detection data of the angle sensor 90 acquired by the vehicle body data acquisition unit 332.
The first imaging device 31L images the bucket 12 that performs the dumping operation in order to discharge the excavated object to the dump body 230. The measurement data acquisition unit 331 acquires the image data of the bucket 12 from the first imaging device 31L. The plurality of bucket teeth 18 (181, 182, 183, 184, and 185) are disposed in a recognition range 50 of the image data. Each of the plurality of bucket teeth 18 is attached to the bucket body 17 so that from bucket body 17 protrudes.
The processing unit 335 processes the image data acquired by the measurement data acquisition unit 331 based on the first algorithm and recognizes each of the plurality of bucket teeth 18 in the recognition range 50 of the image data.
After the plurality of bucket teeth 18 are recognized, as illustrated in FIG. 15, the processing unit 335 estimates positions of a root 51 and a blade edge 52 of each of the plurality of bucket teeth 18. The root 51 is a proximal end of the bucket teeth 18. The root 51 includes a boundary between the bucket teeth 18 and the bucket body 17. The blade edge 52 is a distal end of the bucket teeth 18. The processing unit 335 estimates the root 51 and the blade edge 52 using the first AI algorithm that analyzes input data and outputs output data. The processing unit 335 may estimate the positions of the root 51 and the blade edge 52 as the center where the probability is equal to or greater than a predetermined threshold using the first AI algorithm that outputs the probability of the certainty of the root 51 and the blade edge 52 based on the input data. In addition, the processing unit 335 may use a first AI algorithm that directly estimates the position of the root 51 and the position of the blade edge 52 based on the input data.
After the positions of the root 51 and the blade edge 52 of each of at least four bucket teeth 181, 182, 183, and 184 are estimated, as illustrated in FIG. 16, the processing unit 335 calculates a distance G1 between the root 51 of the bucket teeth 181 and the root 51 of the bucket teeth 182, a distance G2 between the root 51 of the bucket teeth 182 and the root 51 of the bucket teeth 183, and a distance G3 between the root 51 of the bucket teeth 183 and the root 51 of the bucket teeth 184. After the distances G1, G2, and G3 are calculated, the processing unit 335 calculates a reference inter-tooth distance G based on the distances G1, G2, and G3. In the embodiment, the reference inter-tooth distance G is a median value of the distances G1, G2, and G3.
After the reference inter-tooth distance G is calculated, as illustrated in FIG. 17, the processing unit 335 searches for the blade edge 52 corresponding to the root 51. The root 51 and the blade edge 52 corresponding to the root 51 include a proximal end and a distal end of one bucket tooth 18. In the embodiment, the processing unit 335 sets a search range 53 with the root 51 as a reference, and searches the blade edge 52 in the search range 53. The search range 53 is set in a rectangular shape in the image data. As illustrated in FIG. 17, a dimension of the search range 53 in the up-down direction in the image data is defined as [γ×G], and a dimension of the search range 53 in the horizontal direction in the image data is defined as [δ×G]. γ and δ are predetermined fixed values.
The bucket 12 appears largely in the image data captured by the first imaging device 31L when a distance between the first imaging device 31L and the bucket 12 is short. In the image data captured by the first imaging device 31L when the distance between the first imaging device 31L and the bucket 12 is long, the bucket 12 appears small. When the size of the bucket 12 is different in the image data, there is a possibility that the processing unit 335 cannot search for the blade edge 52 corresponding to the root 51. In the embodiment, the search range 53 is set based on the reference inter-tooth distance G. Accordingly, the processing unit 335 can search for the blade edge 52 corresponding to the root 51 regardless of the distance between the first imaging device 31L and the bucket 12.
After the processing unit 335 searches for the blade edge 52 corresponding to each of the plurality of roots 51, as illustrated in FIG. 18, the processing unit 335 calculates a distance L1 between the root 51 of the bucket tooth 181 and the blade edge 52, a distance L2 between the root 51 of the bucket tooth 182 and the blade edge 52, a distance L3 between the root 51 of the bucket tooth 183 and the blade edge 52, and a distance L4 between the root 51 of the bucket tooth 184 and the blade edge 52. The distance L1 corresponds to a protrusion amount of the bucket tooth 181 from the bucket body 17. The distance L2 corresponds to a protrusion amount of the bucket tooth 182 from the bucket body 17. The distance L3 corresponds to a protrusion amount of the bucket tooth 183 from the bucket body 17. The distance L4 corresponds to a protrusion amount of the bucket tooth 184 from the bucket body 17. After the processing unit 335 calculates the distances L1, L2, L3, and L4, the processing unit 335 calculates the reference tooth length L based on the distances L1, L2, L3, and L4. In the embodiment, the reference tooth length L is a median value of the distances L1, L2, L3, and L4. The processing unit 335 may calculate distances L5, L6, L7, and L8 corresponding to the four bucket teeth 185, 186, 187, and 188 by a similar method based on the image data captured by the first imaging device 31R on the right side of the center CL and may calculate the reference tooth length L based on the distances L1, L2, L3, L4, L5, L6, L7, and L8.
After the processing unit 335 calculates the reference tooth length L, the processing unit 335 determines a dimension threshold related to the protrusion amounts of the bucket teeth 18 from the bucket body 17 based on the reference tooth length L. In the embodiment, the dimension threshold is defined as [ε×L]. ε is a predetermined fixed value. In this way, in the embodiment, the processing unit 335 determines the dimension threshold based on the protrusion amounts (L1, L2, L3, and L4) of the plurality of bucket teeth 18 (181, 182, 183, and 184) from the bucket body 17.
The processing unit 335 determines whether there is a defect in the bucket teeth 18 (181, 182, 183, and 184) based on the dimension threshold. When the protrusion amounts of the bucket teeth 18 from the bucket body 17 are less than the dimension threshold, the processing unit 335 determines that there is a defect in the bucket teeth 18. When the protrusion amounts of the bucket teeth 18 from the bucket body 17 are equal to or greater than the dimension threshold, the processing unit 335 determines that there is no defect in the bucket teeth 18. In the embodiment, the processing unit 335 determines that the bucket teeth 18 are detached from the bucket body 17 when the protrusion amount of the bucket teeth 18 is less than the dimension threshold, and determines that the bucket teeth 18 are not detached from the bucket body 17 when the protrusion amounts of the bucket teeth 18 are equal to or greater than the dimension threshold.
The bucket 12 appears largely in the image data captured by the first imaging device 31L when a distance between the first imaging device 31L and the bucket 12 is short. In the image data captured by the first imaging device 31L when the distance between the first imaging device 31L and the bucket 12 is long, the bucket 12 appears small. When the size of the bucket 12 is different in the image data, there is a possibility that the processing unit 335 cannot correctly determine whether there is a defect in the bucket teeth 18. In the embodiment, the dimension threshold is determined based on the reference tooth length L. Accordingly, the processing unit 335 can determine whether there is a defect in the bucket teeth 18 regardless of the distance between the first imaging device 31L and the bucket 12.
The processing unit 335 classifies the measurement data into a plurality of samples based on a determination result indicating whether there is a defect in the bucket teeth 18. The processing unit 335 classifies the measurement data determined to have no defect into soundness determination samples. The processing unit 335 classifies the measurement data determined to have a defect into defect determination samples. The processing unit 335 classifies the measurement data in which the bucket teeth 18 cannot be recognized into undetermined samples. The processing unit 335 counts the number of soundness determination samples indicating the measurement data determined to have no defect. The processing unit 335 counts the number of defect determination samples indicating the measurement data determined to have a defect. The processing unit 335 counts the number of undetermined samples indicating the measurement data in which the bucket teeth 18 cannot be recognized. In the embodiment, the measurement data is image data, and the samples classified by the processing unit 335 are frames. In the following description, the soundness determination sample is appropriately referred to as a soundness determination frame, the defect determination sample is appropriately referred to as a defect determination frame, and the undetermined sample is appropriately referred to as an undetermined frame.
FIG. 19 is a diagram illustrating an example of a soundness determination frame according to an embodiment. As illustrated in FIG. 19, each of the distance L1 indicating the protrusion amount of the bucket tooth 181, the distance L2 indicating the protrusion amount of the bucket tooth 182, the distance L3 indicating the protrusion amount of the bucket tooth 183, and the distance L4 indicating the protrusion amount of the bucket tooth 184 is equal to or greater than the dimension threshold. In this case, the processing unit 335 determines that the bucket teeth 18 are not defective. The processing unit 335 classifies the image data determined to have no defect into a soundness determination frame.
FIG. 20 is a diagram illustrating an example of the defect determination frame according to the embodiment. In the example illustrated in FIG. 20, the distance L2 indicating the protrusion amount of the bucket tooth 182 is less than the dimension threshold. In this case, the processing unit 335 determines that the bucket tooth 182 is defective. The processing unit 335 classifies the image data determined to have a defect into a defect determination frame.
The method of determining whether the bucket teeth 18 falls off based on the image data captured by the first imaging device 31L on the left side of the center CL has been described above. The same applies to a method of determining whether the bucket teeth 18 falls off based on image data captured by the first imaging device 31R on the right side of the center CL. The processing unit 335 determines whether the four bucket teeth 185,186,187, and 188 fall off based on the image data acquired by the measurement data acquisition unit 331 from the first imaging device 31R.
The output unit 337 causes the display device 24A to display a symbol 60 indicating the bucket teeth 18. The output unit 337 changes a display form of the symbol 60 based on presence or absence of the defect of the bucket teeth 18.
FIG. 21 is a diagram illustrating an example of the symbol 60 indicating the bucket teeth 18 displayed on the display device 24A according to the embodiment. As illustrated in FIG. 21, when the processing unit 335 cannot recognize the bucket teeth 18 in the image data, the output unit 337 causes the display device 24A to display the symbol 60 indicating the bucket teeth 18 that cannot be recognized in the first display mode. When the processing unit 335 determines that there is no defect of the bucket teeth 18 in the image data, the output unit 337 causes the display device 24A to display the symbol 60 indicating the bucket teeth 18 determined to have no defect in the second display mode. When the processing unit 335 determines that there is the defect of the bucket teeth 18 in the image data, the output unit 337 causes the display device 24A to display the symbol 60 indicating the bucket teeth 18 determined to have the defect in the third display mode.
In the embodiment, the output unit 337 displays the symbol 60 indicating the bucket teeth 18 that cannot be recognized by the processing unit 335 in first color. The first color is, for example, gray. The output unit 337 displays the symbol 60 indicating the bucket teeth 18 determined to have no defect in second color. The second color is, for example, green. The output unit 337 displays a symbol indicating the bucket teeth 18 determined to have a defect in third color. The third color is, for example, red.
When there is a defect determination frame, the output unit 337 causes the output device 24 to output a warning indicating that there is the defect determination frame. The output unit 337 changes the form of the warning output from the output device 24 based on the number of defect determination frames counted by the processing unit 335.
The output unit 337 causes the output device 24 to output the warning in the first mode when the number of defect determination frames is less than a sample threshold, and causes the output device 24 to output a warning in the second mode when the number of defect determination frames is equal to or greater than the sample threshold. The sample threshold is a predetermined value.
The output device 24 includes the display device 24A that displays the symbol 60 indicating the bucket teeth 18 and the sounding device 24B that generates a warning sound. The output unit 337 may cause the display device 24A to display the display data in the first mode when the number of defect determination frames is less than the sample threshold, and may cause the display device 24A to display the display data in the second mode when the number of defect determination frames is equal to or greater than the sample threshold. The output unit 337 may cause the sounding device 24B to generate the warning sound in the first mode when the number of defect determination frames is less than the sample threshold, and may cause the sounding device 24B to generate the warning sound in the second mode when the number of defect determination frames is equal to or greater than the sample threshold.
FIG. 22 is a diagram illustrating a relationship between a defect determination frame and a form of a warning according to the embodiment. Presence or absence of the defect determination frame is determined at a predetermined frame rate. In the example illustrated in FIG. 22, presence or absence of a defect determination frame is determined at intervals of 0.2 [sec.]. In FIG. 22, a "normal" frame indicates a soundness determination frame or an undetermined frame. In FIG. 22, a "defect" frame includes a defect determination frame. In the example illustrated in FIG. 22, the sample threshold is 2.
As illustrated in FIG. 22, when a moving image captured by the first imaging device 31 includes one or both of the undetermined frame and the soundness determination frame, the warning sound is not output from the sounding device 24B. When the moving image captured by the first imaging device 31 includes only the undetermined frame, the symbol 60 indicating the unrecognizable bucket teeth 18 is displayed in the first display mode on the display device 24A. When the moving image captured by the first imaging device 31 includes only the soundness determination frame, the symbol 60 indicating the bucket teeth 18 determined to have no defect is displayed in the second display mode on the display device 24A.
The plurality of bucket teeth 18 are attached to the bucket body 17. In the embodiment, eight bucket teeth 18 are disposed at intervals in the vehicle width direction. The display device 24A displays the plurality of symbols 60 based on the positions of the plurality of bucket teeth 18. On the display device 24A, eight symbols 60 are displayed at intervals in accordance with the positions of the plurality of bucket teeth 181, 182, 183, 184, 185, 186, 187, and 188. When the moving image includes only undetermined frames, the eight symbols 60 are all displayed in the first display mode. When the moving image includes only the soundness determination frame, the eight symbols 60 are all displayed in the second display mode.
As illustrated in FIG. 22, when there is one defect determination frame in the moving image, the output unit 337 causes the sounding device 24B to output a first warning sound. When there are two defect determination frames in the moving image captured by the first imaging device 31, the output unit 337 causes the sounding device 24B to output a second warning sound.
In the embodiment, the output unit 337 causes the output device 24 to output a warning in the first mode when each of the number of pieces of continuously acquired image data less than the sample threshold is determined to be the defect determination frame by the measurement data acquisition unit 331. The output unit 337 causes the output device 24 to output a warning in the second mode when each of the number of pieces of continuously acquired image data equal to or greater than the sample threshold is determined to be the defect determination frame by the measurement data acquisition unit 331.
For example, when the sample threshold is 2 and it is determined that each of the number of pieces of image data continuously acquired by the measurement data acquisition unit 331 and less than 2 is a defect determination frame, the output unit 337 causes the sounding device 24B to output the first warning sound. That is, when the image data acquired before and after the defect determination frame is the soundness determination frame or the undetermined frame, the output unit 337 causes the sounding device 24B to output the first warning sound. When it is determined that each of the two or more pieces of image data continuously acquired by the measurement data acquisition unit 331 is a defect determination frame, the output unit 337 causes the sounding device 24B to output a second warning sound. That is, when the defect determination frames continue at least twice, the output unit 337 causes the sounding device 24B to output the second warning sound.
Even when the defect determination frames do not continue twice and there are at least two defect determination frames in the moving image per unit time captured by the first imaging device 31, the output unit 337 may cause the sounding device 24B to output the second warning sound.
Any sample threshold is used. The sample threshold may be any number of three or more, for example.
The form of the warning includes a volume of a warning sound. The output of the warning in the first mode includes generation of a warning sound at a first volume from the sounding device 24B. The output of the warning in the second mode includes generation of a warning sound at a second volume greater than the first volume from the sounding device 24B. In the example illustrated in FIG. 22, the volume of the second warning sound is greater than the volume of the first warning sound.
When there is a defect determination frame in the moving image captured by the first imaging device 31, the symbol 60 indicating the bucket teeth 18 determined to have a defect is displayed in the third display mode on the display device 24A. The output unit 337 displays the symbol 60 indicating the bucket teeth 18 determined to have a defect and the symbol 60 indicating the bucket teeth 18 determined to have no defect in different display forms. In the example illustrated in FIG. 22, when it is determined that there is a defect in the bucket tooth 184 and it is determined that there is no defect in the bucket teeth 181, 182, 183, 185, 186, 187, and 188, the output unit 337 displays the symbol indicating the bucket tooth 184 determined to have a defect in the third display mode and displays the symbol indicating the bucket teeth 181, 182, 183, 185, 186, 187, and 188 determined to have no defect in the second display mode.

[Determination of Rolling Stone]

Next, a method of determining presence or absence of the rolling stone 240 will be described. When the forward or backward lever 20C is operated so that the work machine 1 moves forward, the algorithm selection unit 334 selects the second algorithm based on the operation data of the forward or backward lever 20C acquired by the vehicle body data acquisition unit 332. In the embodiment, the processing unit 335 sets a determination reference for determining presence or absence of the rolling stone 240 on the ground 200 in the image data. The processing unit 335 sets a warning reference when a warning is output from the output device 24.
FIG. 23 is a diagram illustrating a determination reference according to the embodiment. In embodiments, the references include dimensions of the rolling stone 240 of the image data. The dimensions of the rolling stone 240 include the number of pixels of the rolling stone 240 in the image data. In the image data, thresholds related to the dimensions of the rolling stone 240 are set. The thresholds related to the dimension of the rolling stone 240 include a threshold Px related to the dimension of the rolling stone 240 in the horizontal direction and a threshold Py related to the dimension of the rolling stone 240 in the vertical direction. The algorithm storage unit 338 stores a standard value (initial value) related to the determination reference. The standard value is a recommended value related to a threshold related to the dimension of the rolling stone 240. The processing unit 335 sets the determination reference based on the standard value. In the embodiment, the processing unit 335 sets the threshold related to the dimension of the rolling stone 240 based on the standard value. The threshold is a standard value.
FIG. 24 is a diagram illustrating a warning reference according to the embodiment. In the embodiment, the warning reference includes a warning area 36 set on at least a part of the ground 200.
The setting of the warning reference includes setting of dimensions of the warning area 36. The dimensions of the warning area 36 include the number of pixels of the warning area in the image data. As illustrated in FIG. 24, a dimension Qx of the warning area 36 in the horizontal direction and a dimension Qy of the warning area 36 in the vertical direction are set in the image data.
The setting of the warning area 36 includes setting of the position of the warning area 36. The setting of the position of the warning area 36 includes setting of the position of the warning area 36 in the image data. The processing unit 335 sets the warning area 36 in a part of the ground 200 in front of the front tire 6F. The algorithm storage unit 338 stores a standard value (initial value) related to the warning reference. The standard value is a recommended value related to the dimensions and position of the warning area 36. The processing unit 335 sets a warning reference based on the standard value. In the embodiment, the processing unit 335 sets the dimensions and position of the warning area 36 in the image data based on the standard value.
The processing unit 335 acquires input data from the input device 25. The driver can change the setting of the determination reference or the setting of the warning reference by operating the input device 25. The processing unit 335 sets a determination reference based on the input data. The processing unit 335 sets a warning reference based on the input data.
FIG. 25 is a diagram illustrating an example of a setting screen of a determination reference and a warning reference according to the embodiment. As illustrated in FIG. 25, the processing unit 335 causes the display device 24A to display the setting screen of the determination reference. The processing unit 335 causes the display device 24A to display the setting screen of the warning reference. The driver can perform at least one of a change in the setting of the determination reference and a change in the setting of the warning reference by operating the input device 25 while viewing the setting screen.
The measurement data acquisition unit 331 acquires the measurement data of the ground 200 from the second imaging device 32 at a predetermined sampling rate. As described above, the measurement data of the ground 200 includes image data of the ground 200 captured by the second imaging device 32. The sampling rate includes a frame rate. The measurement data acquisition unit 331 acquires the image data of the ground 200 from the second imaging device 32 at a predetermined frame rate. The measurement data acquisition unit 331 acquires image data of the ground 200 in front of the front tires 6F.
The processing unit 335 determines presence or absence of the rolling stone 240 on the ground 200 based on the image data acquired by the measurement data acquisition unit 331. The processing unit 335 determines presence or absence of the rolling stone 240 on the ground 200 in the image data. In the embodiment, the processing unit 335 determines the presence or absence of the rolling stone 240 on the ground 200 in front of the front tire 6F in the image data.
The processing unit 335 uses a second AI algorithm that analyzes input data and outputs output data to determine the presence or absence of the rolling stone 240.
The processing unit 335 determines the presence or absence of the rolling stone 240 on the ground 200 in the image data based on the determination reference. The processing unit 335 determines that there is the rolling stone 240 when the dimensions of the rolling stone 240 are greater than thresholds (Px, Py), and determines that there is no rolling stone 240 when the dimensions of the rolling stone 240 are equal to or less than the thresholds (Px, Py). The processing unit 335 may determine that there is the rolling stone 240 when the dimension of the rolling stone 240 exceeds one of the thresholds Px and Py. The processing unit 335 may determine that there is the rolling stone 240 when the dimensions of the rolling stone 240 exceed both the threshold Px and the threshold Py.
The output unit 337 causes the output device 24 to output a warning based on the presence or absence of the rolling stone 240 on the ground 200 in front of the front tire 6F. The output unit 337 causes the output device 24 to output a warning based on the determination of the presence or absence of the rolling stone 240 by the processing unit 335 and the relationship between the rolling stone 240 and the warning standard. The warning reference includes the warning area 36 set on at least a part of the ground 200. The warning area 36 is set in a part of the ground 200 in front of the front tires 6F. When it is determined that there is the rolling stone 240, the output unit 337 causes the output device 24 to output a warning based on a positional relationship between the rolling stone 240 and the warning area 36. When it is determined that there is the rolling stone 240 in the warning area 36, the output unit 337 causes the output device 24 to output a warning indicating that there is the rolling stone 240 in the warning area 36.

[Monitoring Method]

FIG. 26 is a flowchart illustrating a monitoring method according to the embodiment.
The measurement data acquisition unit 331 acquires image data from each of the first imaging device 31 and the second imaging device 32 (Step S1).
The vehicle body data acquisition unit 332 acquires vehicle body data indicating a state of the work machine 1. In the embodiment, the vehicle body data acquisition unit 332 acquires detection data of the angle sensor 90 and operation data of the forward and backward lever 20C as the vehicle body data (Step S2).
The state determination unit 333 determines whether the bucket 12 of the working equipment 10 is disposed in the measurement range Ra of the first measurement device 31 based on the detection data of the angle sensor 90 acquired in Step S2 (Step S3).
In Step S3, when it is determined that the bucket 12 of the working equipment 10 is disposed in the measurement range Ra of the first measurement device 31 (Step S3: Yes), the state determination unit 333 determines whether the work machine 1 moves forward based on the operation data of the forward or backward lever 20C acquired in Step S2 (Step S4).
In Step S4, when it is determined that the work machine 1 has moved forward (Step S4: Yes), the algorithm selection unit 334 selects the first and second algorithms (Step S5).
When it is determined in Step S4 that the work machine 1 has not moved forward (Step S4: No), the algorithm selection unit 334 selects the first algorithm (Step S6) .
When it is determined in Step S3 that the bucket 12 of the working equipment 10 is not disposed in the measurement range Ra of the first measurement device 31 (Step S3: No), the state determination unit 333 determines whether the work machine 1 has moved forward based on the operation data of the forward or backward lever 20C acquired in Step S2 (Step S7).
When it is determined in Step S7 that the work machine 1 has moved forward (Step S7: Yes), the algorithm selection unit 334 selects the second algorithm (Step S8).
After the first algorithm is selected in Step S5 or Step S6, the processing unit 335 processes the image data captured by the first imaging device 31 based on the first algorithm and recognizes the bucket 12 of the working equipment 10 (Step S9).
The contamination determination unit 336 determines whether the bucket teeth 18 are recognized by the processing unit 335 in Step S9 (Step S10).
When it is determined in Step S10 that the bucket teeth 18 are recognized (Step S10: Yes), the processing unit 335 determines whether the bucket teeth 18 are defective (Step S11).
When it is determined in Step S11 that the bucket teeth 18 are defective (Step S11: Yes), the output unit 337 causes the output device 24 to output a warning indicating that the bucket teeth 18 are defective (Step S12).
When it is determined in Step S11 that the bucket teeth 18 are not defective (Step S11: No), no warning is output from the output device 24 (Step S13).
As described with reference to FIG. 22, when the moving image captured by the first imaging device 31 includes one or both of the undetermined frame and the soundness determination frame, the warning sound is not output from the sounding device 24B. When the moving image captured by the first imaging device 31 includes only the undetermined frame, the symbol 60 indicating the unrecognizable bucket teeth 18 is displayed in the first display mode on the display device 24A. When the moving image captured by the first imaging device 31 includes only the soundness determination frame, the symbol 60 indicating the bucket teeth 18 determined to have no defect is displayed in the second display mode on the display device 24A. When there is one defect determination frame in the moving image, the first warning sound is output from the sounding device 24B. When there are two defect determination frames in the moving image captured by the first imaging device 31, the second warning sound is output from the sounding device 24B.
When it is determined in Step S10 that the bucket teeth 18 are not recognized (Step S10: No), the contamination determination unit 336 determines that the incidence surface of the optical system of the first imaging device 31 is contaminated. When it is determined that the incidence surface of the optical system of the first imaging device 31 is contaminated, the output unit 337 causes the output device 24 to output a warning indicating that the incidence surface of the optical system of the first imaging device 31 is contaminated (Step S14).
After the second algorithm is selected in Step S5 or Step S8, the processing unit 335 processes the image data captured by the second imaging device 32 based on the second algorithm and recognizes the ground 200 (Step S15).
After the ground 200 is recognized, the processing unit 335 determines whether there is the rolling stone 240 on the ground 200 in front of the front tire 6F (Step S16).
When it is determined in Step S16 that there is the rolling stone 240 on the ground 200 in front of the front tire 6F (Step S16: Yes), the output unit 337 causes the output device 24 to output a warning indicating that there is the rolling stone 240 (Step S17).
When it is determined in Step S16 that there is no rolling stone 240 (Step S16: No), no warning is output from the output device 24 (Step S13).
After any one of Step S12, Step S13, Step S14, and Step S17 is processed or when it is determined that the work machine 1 has not moved forward in Step S7, the processing unit 335 determines whether the monitoring processing ends (Step S18).
When it is determined in Step S18 that the monitoring process continues (Step S18: No), the process returns to Step S1 and Step S2.
When it is determined in Step S18 that the monitoring process ends (Step S18: Yes), the monitoring process ends.

[Display Device]

FIG. 27 is a diagram illustrating an example of the display device 24A according to an embodiment. As illustrated in FIG. 27, the output unit 337 causes the display device 24A to display a target image 37 indicating the ground 200 generated based on the image data acquired by the measurement data acquisition unit 331. The target image 37 includes a target image 37L indicating image data captured by the second imaging device 32L on the left side and a target image 37R indicating image data captured by the second imaging device 32R on the right side. The target image 37L and the target image 37R are displayed side by side on a display screen of the display device 24A. The target image 37L and the target image 37R are simultaneously displayed on the display screen of the display device 24A. The target image 37L is displayed in the left region of the display screen of the display device 24A. The target image 37R is displayed in the right region of the display screen of the display device 24A.
The output unit 337 causes the display device 24A to display an area image 360 indicating the warning area 36. The output unit 337 superimposes the area image 360 on the target image 37 to display the area image on the display device 24A. The area image 360 indicating the warning area 36 set to include the ground 200 in front of the left front tire 6F is displayed on the display device 24A with being superimposed on the target image 37L. The area image 360 indicating the warning area 36 set to include the ground 200 in front of the right front tire 6F is displayed on the display device 24A with being superimposed on the target image 37R.
The output unit 337 causes the display device 24A to display a symbol 70 indicating front tire 6F. The symbol 70 includes a symbol 70L indicating the left front tire 6F and a symbol 70R indicating the right front tire 6F.
The output unit 337 causes the output device 24 to output a warning indicating that there is the rolling stone 240 on the ground 200 in front of the front tire 6F. The output of a warning to the output device 24 includes display of a warning image indicating that there is the rolling stone 240 on the ground 200 in front of the front tires 6F by the display device 24A. The output of the warning to the output device 24 includes generation of a warning sound indicating that there is the rolling stone 240 on the ground 200 in front of the front tire 6F by the sounding device 24B.
In the embodiment, the warning image includes a frame image 80 displayed to surround the rolling stone 240 in the target image 37, a frame image 81 displayed at an edge of the target image 37, and a background image 82 of the symbol 70.
The output unit 337 causes the display device 24A to display the frame image 80 such that the rolling stone 240 is emphasized.
In the example illustrated in FIG. 27, the rolling stone 240 of the target image 37L is disposed outside of the warning area 36. When the processing unit 335 determines that there is the rolling stone 240 outside of the warning area 36, the output unit 337 causes the display device 24A to display a frame image 80L of the first mode.
In the example illustrated in FIG. 27, at least a part of the rolling stone 240 of the target image 37R is disposed inside the warning area 36. When the processing unit 335 determines that there is the rolling stone 240 is inside the warning area 36, the output unit 337 causes the display device 24A to display a frame image 80R of the second mode different from the first mode.
The output unit 337 displays each of the frame image 80L and the frame image 80R on the display device 24A such that the rolling stone 240 disposed inside the warning area 36 is emphasized more than the rolling stone 240 disposed outside of the warning area 36. In the example illustrated in FIG. 27, the frame image 80R is displayed with a solid line, and the frame image 80L is displayed with a dotted line. The frame image 80R may be displayed in the first color, and the frame image 80L may be displayed in the second color. As the first color, red is exemplified. As the second color, green is exemplified.
The output unit 337 causes the display device 24A to display the frame image 81 such that the target image 37 in which there is at least a part of the rolling stone 240 inside the warning area 36 is emphasized.
In the example illustrated in FIG. 27, there is at least a part of the rolling stone 240 displayed in the target image 37R inside the warning area 36. There is the rolling stone 240 displayed in the target image 37L outside of the warning area 36. The output unit 337 causes the display device 24A to display the frame image 81 such that the target image 37R is emphasized more than the target image 37L.
The output unit 337 causes the display device 24A to display the background image 82 such that the front tire 6F having a high possibility of passing over the rolling stone 240 is emphasized.
In the example illustrated in FIG. 27, there is a high possibility of the right front tire 6F passing over the rolling stone 240. That is, there is a high possibility of the right front tire 6F stepping on the rolling stone 240 due to the forward movement of the work machine 1. The output unit 337 causes the display device 24A to display the background image 82 of the symbol 70R such that the right front tire 6F having a high possibility of passing over the rolling stone 240 is emphasized.
In the example illustrated in FIG. 27, when there is a high possibility of the left front tire 6F passing over the rolling stone 240, the output unit 337 can the display device 24A to display the background image 82 of the symbol 70L such that the left front tire 6F having a high possibility of passing over the rolling stone 240 is emphasized.
The output unit 337 causes the display device 24A to display a symbol 60 indicating the bucket teeth 18. The output unit 337 displays the plurality of symbols 60 based on the positions of the plurality of bucket teeth 18. The symbols 60 include a symbol 61 indicating the bucket tooth 181, a symbol 62 indicating the bucket tooth 182, a symbol 63 indicating the bucket tooth 183, a symbol 64 indicating the bucket tooth 184, a symbol 65 indicating the bucket tooth 185, a symbol 66 indicating the bucket tooth 186, a symbol 67 indicating the bucket tooth 187, and a symbol 68 indicating the bucket tooth 188.
The plurality of symbols 60 are displayed in an upper area of the display screen of the display device 24A. Of the plurality of symbols 60, the symbol 61 is displayed on the leftmost side, the symbol 62 is located on the left side next to the symbol 61, the symbol 63 is located on the left side next to the symbol 62, the symbol 64 is located on the left side next to the symbol 63, the symbol 65 is located on the left side next to the symbol 64, the symbol 66 is located on the left side next to the symbol 65, the symbol 67 is located on the left side next to the symbol 66, and the symbol 68 is located on the rightmost side.
FIG. 27 illustrates an example in which the processing unit 335 cannot recognize the eight bucket teeth 18 and the eight symbols 60 are displayed in the first display form. When the incidence surface of the optical system of the first imaging device 31 is contaminated and the eight bucket teeth 18 are not recognized, the output unit 337 displays the eight symbols 60 in the first display form as illustrated in FIG. 27.
FIG. 28 is a diagram illustrating an example of the display device 24A according to an embodiment. FIG. 28 is a diagram illustrating an extracted upper region of the display device 24A.
FIG. 28 illustrates examples of the symbols 60 when the processing unit 335 determines presence or absence of a defect of each of the eight bucket teeth 18. As illustrated in FIG. 28, the output unit 337 changes the display form of the symbols 60 based on the presence or absence of the defect of the bucket teeth 18. The output unit 337 displays the symbol 60 indicating the bucket teeth 18 determined to have a defect and the symbol 60 indicating the bucket teeth 18 determined to have no defect in different display forms. As illustrated in FIG. 28, when it is determined that there is a defect in the bucket tooth 184 and it is determined that there is no defect in the bucket teeth 181, 182, 183, 185, 186, 187, and 188, the output unit 337 displays the symbol 64 indicating the bucket tooth 184 determined to have a defect in the third display mode, and displays symbols 61, 62, 63, 65, 66, 67, and 68 indicating the bucket teeth 181, 182, 183, 185, 186, 187, and 188 determined to have no defect in the second display mode. The output unit 337 causes the display device 24A to display the symbol 64 in the third color and causes the display device 24A to display the symbols 61, 62, 63, 65, 66, 67, and 68 in the second color. The third color is, for example, red. The second color is, for example, green.

[Computer System]

FIG. 29 is a block diagram illustrating a computer system 1000 according to an embodiment. Each of the above-described monitoring controller 33 and drive controller 48 includes the computer system 1000. The computer system 1000 includes a processor 1001 such as a central processing unit (CPU), a main memory 1002 including a nonvolatile memory such as a read only memory (ROM) and a volatile memory such as a random access memory (RAM), a storage 1003, and an interface 1004 including an input/output circuit. The functions of the above-described monitoring controller 33 and drive controller 48 are stored in the storage 1003 as a computer program. The processor 1001 reads the computer program from the storage 1003, loads the computer program in the main memory 1002, and executes the above-described processes according to the program. The computer program may be distributed to the computer system 1000 via a network.
According to the above-described embodiment, the computer program or the computer system 1000 can execute: acquiring measurement data of the first measurement device 31 and the second measurement device 32 that are mounted on the work machine 1 and measure a target; acquiring vehicle body data indicating a state of the work machine 1; determining the state of the work machine 1 based on the vehicle body data; selecting a specific algorithm for processing the measurement data based on the determined state of the work machine 1; processing the measurement data based on the selected specific algorithm to recognize the target; and causing an output device to output a recognition result of the target.

[Effects]

As described above, the monitoring system 30 of the work machine 1 includes the measurement data acquisition unit 331 that is mounted on the work machine 1 and acquires the image data of the first imaging device 31 and the second imaging device 32 that measure the target, the vehicle body data acquisition unit 332 that acquires the vehicle body data indicating the state of the work machine 1, the state determination unit 333 that determines the state of the work machine 1 based on the vehicle body data, the algorithm selection unit 334 that selects a specific algorithm for processing the image data based on the state of the work machine 1 determined by the state determination unit 333, the processing unit 335 that processes the image data based on the specific algorithm selected by the algorithm selection unit 334 and recognizes the target, and the output unit 337 that causes the output device 24 to output the recognition result of the target recognized by the processing unit 335.
According to the embodiment, since a specific algorithm for processing the image data is selected based on the state of the work machine 1, the processing unit 335 can efficiently recognize a plurality of targets. In a situation where the bucket 12 is disposed in the measurement range Ra of the first imaging device 31, the processing unit 335 processes the image data captured by the first imaging device 31 based on the first algorithm to recognize the bucket teeth 18. In a situation where the bucket 12 is not disposed in the measurement range Ra of the first imaging device 31, the processing unit 335 does not perform a process of recognizing the bucket teeth 18. In a situation where there is a high possibility of the work machine 1 moving forward and the front tires 6F step on the rolling stone 240 on the ground 200 in front of the front tires 6F, the processing unit 335 processes the image data captured by the second imaging device 32 based on the second algorithm to recognize the rolling stone 240 on the ground 200 in front of the front tires 6F. In a situation where there is a low possibility of the work machine 1 reversing or stopping and the front tires 6F stepping on the rolling stone 240 on the ground 200 in front of the front tires 6F, the processing unit 335 does not perform the process of recognizing the rolling stone 240. As described above, the monitoring controller 33 may not always perform the process of recognizing both the bucket teeth 18 and the rolling stone 240. Therefore, a burden on the monitoring controller 33 required to monitor the bucket teeth 18 and the rolling stone 240 is reduced. Therefore, the monitoring controller 33 can efficiently monitor each of the bucket teeth 18 and the rolling stone 240.

[Another Embodiment]

In the above-described embodiment, each of the first imaging device 31 and the second imaging device 32 is disposed in the housing 9 holding the headlight 28. At least one of the first imaging device 31 and the second imaging device 32 may be disposed in a portion of the work machine 1 different from the housing 9.
In the above-described embodiment, the second imaging device 32 is disposed above the first imaging device 31. At least a part of the second imaging device 32 may be disposed at the same position as the first imaging device 31 in the up-down direction. The second imaging device 32 may be disposed below the first imaging device 31.
In the above-described embodiment, the defect of the bucket teeth 18 is falling off of the bucket teeth 18 from the bucket body 17. The defect of the bucket teeth 18 may be abrasion of the bucket teeth 18 or breakage of the bucket teeth 18.
In the above-described embodiment, the processing unit 335 monitors a defect of the bucket teeth 18 as the replacement member. The processing unit 335 may monitor a defect of the inter-tooth protector 19 as a replacement member. The defect of the inter-tooth protector 19 includes a falling of the inter-tooth protector 19 from the bucket body 17, abrasion of the inter-tooth protector 19, and breakage of the inter-tooth protector 19.
In the above-described embodiment, the setting of the warning reference includes the setting of the dimensions of the warning area 36 and the setting of the position of the warning area 36. The setting of the warning reference may be either the setting of the dimension of the warning area 36 or the setting of the position of the warning area 36. For example, the position of the warning area 36 may be a predetermined fixed value, and only the dimensions of the warning area 36 may be set. The dimensions of the warning area 36 may be predetermined fixed values, and only the position of the warning area 36 may be set.
In the above-described embodiment, the processing unit 335 determines presence or absence of the rolling stone 240 on the ground 200 based on the determination reference, and causes the output device 24 to output a warning based on the relationship between the rolling stone 240 and the warning reference. The processing unit 335 may determine that there is the rolling stone 240 when both the determination reference and the warning reference are satisfied. In this case, the processing unit 335 may use a warning reference as a determination reference. The processing unit 335 may determine presence or absence of the rolling stone 240 on the ground 200 regardless of the determination reference and output a warning when both the determination reference and the warning reference are satisfied. In this case, the processing unit 335 may use the determination reference as the warning reference.
In the above-described embodiment, the output unit 337 causes the display device 24A to display the area image 360 indicating the warning area 36. The output unit 337 may cause the display device 24A not to display the area image 360 indicating the warning area 36. The output unit 337 may display the area image 360 indicating the warning area 36 on the display device 24A when it is determined that there is the rolling stone 240. When it is determined that there is the rolling stone 240 in the warning area 36, the output unit 337 may cause the display device 24A to display an area image 360 indicating the warning area 36.
In the embodiment described above, the second algorithm is selected in a state where the work machine 1 can move forward or moves forward. Based on the second algorithm, it is determined whether there is the rolling stone 240 on the ground 200 in front of the front tires 6F. The second algorithm may be selected in a state where the work machine 1 can move backward or moves backward, and it may be determined whether there is the rolling stone 240 on the ground 200 behind the rear tire 6R.
FIG. 30 is a diagram illustrating an operation of the work machine 1 according to another embodiment. As illustrated in FIG. 30, the work machine 1 includes the second imaging device 32 that images the ground 200 behind the rear tires 6R. The measurement data acquisition unit 331 acquires image data of the ground 200 behind the rear tire 6R from the second imaging device 32. The algorithm selection unit 334 may select the second algorithm when the work machine 1 moves backward. The processing unit 335 may determine whether there is the rolling stone 240 on the ground 200 behind the rear tire 6R based on the second algorithm when the work machine 1 moves backward.
In the above-described embodiment, the output device 24 is disposed in the drive room of the work machine 1. The output device 24 may be disposed outside of the work machine 1.
FIG. 31 is a diagram illustrating a monitoring system 300 according to another embodiment. In the example illustrated in FIG. 31, the work machine 1 is remotely operated by a driving operation device 20F provided at a remote place of the work machine 1.
Similarly to the above-described embodiment, each of the first imaging device 31 and the second imaging device 32 is provided in the work machine 1. An imaging device for a remote operation (not illustrated) that images the front of the work machine 1 is provided in the work machine 1. The imaging device for the remote operation captures image data of a scene in front of the work machine 1 similar to a scene viewed through a windshield by the driver seated in the drive seat in the drive room as illustrated in FIG. 4.
Some or all of an output device 24F, an input device 25F, a monitoring controller 33F, and the driving operation device 20F are disposed in a remote operation facility provided at a remote place of the work machine 1. Each of the output device 24F, the input device 25F, the monitoring controller 33F, and the driving operation device 20F is provided separately from the work machine 1. A remote operation display device that displays image data captured by the imaging device for the remote operation is provided in the remote operation facility.
The work machine 1 and the monitoring controller 33F wirelessly communicate with each other via a wireless communication system. The work machine 1 includes a wireless communication device 301 of a wireless communication system, and a wireless communication device 302 of the wireless communication system is connected to the monitoring controller 33F. The wireless communication device 301 of the work machine 1 and the wireless communication device 302 of the monitoring controller 33F may communicate via another device.
The image data captured by the imaging device for the remote operation is transmitted to the display device for the remote operation via the wireless communication system. The driver of the remote operation facility can operate the driving operation device 20F while viewing the image data displayed on the remote operation display device. An operation signal generated by operating the driving operation device 20F is transmitted to the work machine 1 via the wireless communication system. Accordingly, the work machine 1 is remotely operated.
The monitoring controller 33F includes the measurement data acquisition unit 331, the vehicle body data acquisition unit 332, the state determination unit 333, the algorithm selection unit 334, the processing unit 335, the contamination determination unit 336, the output unit 337, and the algorithm storage unit 338 as described with reference to FIG. 13.
The image data captured by the first imaging device 31 and the second imaging device 32 is transmitted from the work machine 1 to the monitoring controller 33F via the wireless communication system. The processing unit 335 of the monitoring controller 33F can determine presence or absence of a defect of the bucket teeth 18 based on the image data captured by the first imaging device 31. The output unit 337 of the monitoring controller 33F can cause the output device 24F to output a warning indicating that there is a defect of the bucket teeth 18. The processing unit 335 of the monitoring controller 33F can determine the presence or absence of the rolling stone 240 on the ground 200 based on the image data captured by the second imaging device 32. The output unit 337 of the monitoring controller 33F can cause the output device 24F to output a warning indicating that there is the rolling stone 240 on the ground 200.
In the above-described embodiment, each of the first measurement device 31 and the second measurement device 32 is an imaging device. At least one of the first measurement device 31 and the second measurement device 32 may be a laser device. At least one of the first measurement device 31 and the second measurement device 32 may be a radar device. The laser device irradiates a measurement target with laser light to acquire measurement data of the measurement target. The radar device irradiates a measurement target with sound waves to acquire measurement data of the measurement target. The laser device can irradiate the working equipment 10 with laser light to acquire measurement data of the working equipment 10 at a predetermined sampling rate. The radar device can irradiate the working equipment 10 with sound waves to acquire measurement data of the working equipment 10 at a predetermined sampling rate. The output unit 337 can change the form of the warning to be output from the output device 24 based on the number of defect determination samples indicating the measurement data determined to have a defect in the working equipment 10.
In the above-described embodiment, the reception surfaces of the first measurement device 31 and the second measurement device 32 are the incidence surfaces of the optical system. When at least one of the first measurement device 31 and the second measurement device 32 is a laser device, the reception surface of the laser device includes a light reception surface of laser light. The contamination determination unit 336 can determine whether the light reception surface of the laser device is contaminated according to the above-described embodiment. When at least one of the first measurement device 31 and the second measurement device 32 is a radar device, the reception surface of the laser device includes a radio wave reception surface of the radar device. The contamination determination unit 336 can determine whether the radio wave reception surface of the radar device is contaminated according to the above-described embodiment.
In the above-described embodiment, the work machine 1 is a wheel loader. The work machine 1 may be another work machine such as a bulldozer or an excavator. Each of the bulldozer and the excavator has working equipment and a crawler belt that rotates with coming into contact with the ground 200. The crawler belt is a rotating member that rotates with coming into contact with the ground 200. When the crawler belt rotates, the work machine travels.

Reference Signs List

1 WORK MACHINE
2 VEHICLE BODY
2A JOINT MECHANISM
2F FRONT VEHICLE BODY
2R REAR VEHICLE BODY
3 DRIVE CAB
4 TRAVELING APPARATUS
5 WHEEL
5F FRONT WHEEL
5R REAR WHEEL
6 TIRE
6F FRONT TIRE
6R REAR TIRE
7 FRONT FENDER
7L FRONT FENDER
7R FRONT FENDER
8 SUPPORT MEMBER
8L SUPPORT MEMBER
8R SUPPORT MEMBER
9 HOUSING
9L HOUSING
9R HOUSING
10 WORKING EQUIPMENT
11 BOOM
12 BUCKET
12E END
12M OPENING
13 BOOM CYLINDER
14 BUCKET CYLINDER
15 BELL CRANK
16 BUCKET LINK
17 BUCKET BODY
17A BOTTOM PLATE PORTION
17B UPPER PLATE PORTION
17C LEFT PLATE PORTION
17D RIGHT PLATE PORTION
18 BUCKET TOOTH
19 INTER-TOOTH PROTECTOR
20 DRIVING OPERATION DEVICE
20A ACCELERATOR PEDAL
20B BRAKE PEDAL
20C FORWARD OR BACKWARD LEVER
20F DRIVING OPERATION DEVICE
21 OPERATION PANEL
22 MONITOR DEVICE
23 REAR VIEW MONITOR DEVICE
24 OUTPUT DEVICE
24A DISPLAY DEVICE
24B SOUNDING DEVICE
24F OUTPUT DEVICE
25 INPUT DEVICE
25F INPUT DEVICE
27 PILLAR
28 HEADLIGHT
28L HEADLIGHT
28R HEADLIGHT
29 BLINKER LAMP
29L BLINKER LAMP
29R BLINKER LAMP
30 MONITORING SYSTEM
31 FIRST IMAGING DEVICE (FIRST MEASUREMENT DEVICE)
31L FIRST IMAGING DEVICE
31R FIRST IMAGING DEVICE
32 SECOND IMAGING DEVICE (SECOND MEASUREMENT DEVICE)
32L SECOND IMAGING DEVICE
32R SECOND IMAGING DEVICE
33 MONITORING CONTROLLER
33F MONITORING CONTROLLER
36 WARNING AREA
37 TARGET IMAGE
37L TARGET IMAGE
37R TARGET IMAGE
40 DRIVE SYSTEM
41 ENGINE
42 FUEL INJECTION DEVICE
43 POWER TAKE-OFF
44 TRANSMISSION
44F FORWARD GEAR
44R REVERSE GEAR
45F FRONT AXLE
45R REAR AXLE
46 HYDRAULIC PUMP
47 CONTROL VALVE
48 DRIVE CONTROLLER
50 RECOGNITION RANGE
51 ROOT
52 BLADE EDGE
53 SEARCH RANGE
60 SYMBOL
61 SYMBOL
62 SYMBOL
63 SYMBOL
64 SYMBOL
65 SYMBOL
66 SYMBOL
67 SYMBOL
68 SYMBOL
70 SYMBOL
70L SYMBOL
70R SYMBOL
80 FRAME IMAGE
80L FRAME IMAGE
80R FRAME IMAGE
81 FRAME IMAGE
82 BACKGROUND IMAGE
90 ANGLE SENSOR
91 BOOM ANGLE SENSOR
92 BUCKET ANGLE SENSOR
181 BUCKET TOOTH
182 BUCKET TOOTH
183 BUCKET TOOTH
184 BUCKET TOOTH
185 BUCKET TOOTH
186 BUCKET TOOTH
187 BUCKET TOOTH
188 BUCKET TOOTH
191 INTER-TOOTH PROTECTOR
192 INTER-TOOTH PROTECTOR
193 INTER-TOOTH PROTECTOR
194 INTER-TOOTH PROTECTOR
195 INTER-TOOTH PROTECTOR
196 INTER-TOOTH PROTECTOR
197 INTER-TOOTH PROTECTOR
200 GROUND
210 NATURAL GROUND
220 DUMP TRUCK
230 DUMP BODY
240 ROLLING STONE
300 MONITORING SYSTEM
301 WIRELESS COMMUNICATION DEVICE
302 WIRELESS COMMUNICATION DEVICE
331 MEASUREMENT DATA ACQUISITION UNIT
332 VEHICLE BODY DATA ACQUISITION UNIT
333 STATE DETERMINATION UNIT
334 ALGORITHM SELECTION UNIT
335 PROCESSING UNIT
336 CONTAMINATION DETERMINATION UNIT
337 OUTPUT UNIT
338 ALGORITHM STORAGE UNIT
360 AREA IMAGE
1000 COMPUTER SYSTEM
1001 PROCESSOR
1002 MAIN MEMORY
1003 STORAGE
1004 INTERFACE
CL CENTER
FX ROTATION AXIS
G REFERENCE TOOTH DISTANCE
G1 DISTANCE
G2 DISTANCE
G3 DISTANCE
L REFERENCE TOOTH LENGTH
L1 DISTANCE
L2 DISTANCE
L3 DISTANCE
L4 DISTANCE
M1 ARROW
M2 ARROW
M3 ARROW
M4 ARROW
Px THRESHOLD
Py THRESHOLD
Qx DIMENSION
Qy DIMENSION
Ra MEASUREMENT RANGE
Rb MEASUREMENT RANGE
RX ROTATION AXIS
α FIRST ANGLE OF VIEW
β SECOND ANGLE OF VIEW

Claims

A monitoring system of a work machine comprising:
a measurement data acquisition unit mounted on a work machine and configured to acquire measurement data of a measurement device that measures a target;

a vehicle body data acquisition unit configured to acquire vehicle body data indicating a state of the work machine;

a state determination unit configured to determine the state of the work machine based on the vehicle body data;

an algorithm selection unit configured to select a specific algorithm for processing the measurement data based on the state of the work machine determined by the state determination unit; and

a processing unit configured to process the measurement data and recognize the target based on the specific algorithm selected by the algorithm selection unit.
The monitoring system of the work machine according to claim 1,
wherein the work machine includes working equipment and a traveling apparatus, and

wherein the vehicle body data includes attitude data indicating an attitude of the working equipment and forward or backward data indicating a forward or backward movement of the traveling apparatus.
The monitoring system of the work machine according to claim 1, comprising:
an algorithm storage unit configured to store a plurality of algorithms different from each other,

wherein the algorithm selection unit selects a specific algorithm from the plurality of algorithms stored in the algorithm storage unit based on the state of the work machine.
The monitoring system of the work machine according to claim 3,
wherein the measurement device includes a first measurement device and a second measurement device, and

wherein the algorithm stored in the algorithm storage unit includes a first algorithm that processes measurement data of the first measurement device and a second algorithm that processes measurement data of the second measurement device.
The monitoring system of the work machine according to claim 4,
wherein the work machine includes working equipment and a traveling apparatus,

wherein a target to be measured by the first measurement device includes the working equipment, and

wherein the target to be measured by the second measurement device includes a ground on which the traveling apparatus travels.
The monitoring system of the work machine according to claim 4,
wherein the work machine includes working equipment and a traveling apparatus,

wherein the vehicle body data includes attitude data indicating an attitude of the working equipment and forward or backward data indicating forward or backward movement of the traveling apparatus,

wherein, when the state determination unit determines that the working equipment is disposed in a measurement range of the first measurement device based on the attitude data, the algorithm selection unit selects the first algorithm,

wherein, when the state determination unit determines that the working equipment is not disposed in the measurement range of the first measurement device based on the attitude data, the algorithm selection unit does not select the first algorithm,

wherein, when the state determination unit determines that the traveling apparatus is moving forward based on the forward or backward data, the algorithm selection unit selects the second algorithm, and

wherein, when the state determination unit determines that the traveling apparatus is not moving forward based on the forward or backward data, the algorithm selection unit does not select the second algorithm.
The monitoring system of the work machine according to claim 5,
wherein the processing unit processes the measurement data of the first measurement device based on the first algorithm to determine presence or absence of a defect of the working equipment.
The monitoring system of the work machine according to claim 5,
wherein the processing unit processes the measurement data of the second measurement device based on the second algorithm to determine whether there is a rolling stone on the ground.
The monitoring system of the work machine according to any one of claims 1 to 8, comprising:
an output unit configured to cause an output device to output a recognition result of the target recognized by the processing unit.
A monitoring method for a work machine comprising:
acquiring measurement data of a measurement device that is mounted on a work machine and measures a target;

acquiring vehicle body data indicating a state of the work machine;

determining a state of the work machine based on the vehicle body data;

selecting a specific algorithm for processing the measurement data based on the determined state of the work machine; and

processing the measurement data based on the selected specific algorithm to recognize the target.