JP6419677B2 - Perimeter monitoring system for work machines - Google Patents

Description

  The present invention relates to a work machine periphery monitoring system for monitoring the surroundings of a work machine.

  An excavator provided with a sensor for detecting a person existing around the excavator is known (see Patent Document 1). This excavator generates a virtual viewpoint image for peripheral monitoring that simultaneously displays an image obtained by looking down the vicinity of the excavator from the sky and an image viewed from the excavator in the horizontal direction based on images taken by the three cameras. Display on the screen of the display device.

JP 2014-183500 A

However, shovel above does not change the display mode between when not detected a when detecting a human around sheet Yoberu. Therefore, the presence of the person who detected it may hardly be recognized.

In view of the above, it is desired to provide a work machine periphery monitoring system that enables an operator to quickly recognize the presence of a detected obstacle .

A work machine periphery monitoring system according to an embodiment of the present invention includes a detection unit that detects an obstacle existing around a work machine, and a control unit mounted on the work machine, wherein the control unit includes: A first image generated using a captured image of an imaging device attached to the work machine and a second image including a graphic representing a surrounding area of the work machine are displayed on a display device, and the second image is displayed on the display device. In the figure representing the surrounding area of the work machine included, the area corresponding to the direction of the obstacle detected by the detection unit is emphasized .

By the means described above, a work machine periphery monitoring system is provided that enables the operator to quickly recognize the presence of the detected obstacle .

It is a side view of the shovel in which the periphery monitoring system which concerns on the Example of this invention is mounted. It is a functional block diagram which shows the structural example of a periphery monitoring system. It is an example of the captured image of a rear camera. It is the schematic which shows an example of the geometric relationship used when cut out the identification process target image from a captured image. It is a top view of real space behind an excavator. It is a figure which shows the flow of the process which produces | generates the normalization image from a captured image. It is a figure which shows the relationship between a captured image, an identification process target image area, and a normalized image. It is a figure which shows the relationship between an identification process target image area | region and an identification process inappropriate area | region. It is a figure which shows the example of a normalized image. It is the schematic which shows another example of the geometric relationship used when cut out the identification process target image from a captured image. It is a figure which shows an example of the characteristic image in a captured image. It is a flowchart which shows the flow of an example of an image extraction process. It is a flowchart which shows the flow of an example of a periphery monitoring process. It is a flowchart which shows the flow of an example of a restriction | limiting cancellation | release process. It is a figure which shows the example of an output image. 6 is a correspondence table showing a correspondence relationship between detection states and display colors of frames and regions. It is an example of the viewpoint conversion image as an output image. It is an example of the output image containing a viewpoint conversion image. It is an example of the output image containing a viewpoint conversion image. It is an example of the output image containing a viewpoint conversion image.

  FIG. 1 is a side view of an excavator as a construction machine on which a periphery monitoring system 100 according to an embodiment of the present invention is mounted. An upper swing body 3 is mounted on the lower traveling body 1 of the excavator via a swing mechanism 2 so as to be rotatable. A boom 4 is attached to the upper swing body 3. An arm 5 is attached to the tip of the boom 4, and a bucket 6 is attached to the tip of the arm 5. The boom 4, the arm 5, and the bucket 6 constitute an excavation attachment, and are hydraulically driven by the boom cylinder 7, the arm cylinder 8, and the bucket cylinder 9, respectively. Further, the upper swing body 3 is provided with a cabin 10 and is mounted with a power source such as an engine. An imaging device 40 is attached to the upper part of the upper swing body 3. Specifically, the rear camera 40B, the left side camera 40L, and the right side camera 40R are attached to the rear end upper part, the left end upper part, and the right end upper part of the upper swing body 3. A controller 30 and an output device 50 are installed in the cabin 10.

  FIG. 2 is a functional block diagram illustrating a configuration example of the periphery monitoring system 100. The periphery monitoring system 100 mainly includes a controller 30, an imaging device 40, and an output device 50.

  The controller 30 is a control device that performs drive control of the shovel. In the present embodiment, the controller 30 is configured by an arithmetic processing unit including a CPU and an internal memory, and realizes various functions by causing the CPU to execute a drive control program stored in the internal memory.

  Further, the controller 30 determines whether there is a person around the shovel based on the outputs of the various devices, and controls the various devices according to the determination result. Specifically, the controller 30 receives the outputs of the imaging device 40 and the input device 41, and executes software programs corresponding to the extraction unit 31, the identification unit 32, the tracking unit 33, and the control unit 35, respectively. Then, according to the execution result, a control command is output to the machine control device 51 to execute drive control of the shovel, or various information is output from the output device 50. The controller 30 may be a control device dedicated to image processing.

  The imaging device 40 is a device that captures an image around the excavator, and outputs the captured image to the controller 30. In the present embodiment, the imaging device 40 is a wide camera that employs an imaging element such as a CCD, and is mounted on the upper part of the upper swing body 3 so that the optical axis is directed obliquely downward.

  The input device 41 is a device that receives input from an operator. In this embodiment, the input device 41 includes an operation device (operation lever, operation pedal, etc.), a gate lock lever, a button installed at the tip of the operation device, a button attached to the in-vehicle display, a touch panel, and the like.

  The output device 50 is a device that outputs various types of information, and includes, for example, a display unit such as an in-vehicle display that displays various types of image information, an in-vehicle speaker that outputs various types of audio information, an alarm buzzer, an alarm lamp, and the like. In the present embodiment, the output device 50 outputs various types of information in response to control commands from the controller 30.

  The machine control device 51 is a device that controls the movement of the excavator, and includes, for example, a control valve that controls the flow of hydraulic oil in the hydraulic system, a gate lock valve, an engine control device, and the like.

  The extraction unit 31 is a functional element that extracts an identification processing target image from a captured image captured by the imaging device 40. Specifically, the extraction unit 31 extracts simple features based on local luminance gradients or edges, geometric features based on Hough transform, features related to the area or aspect ratio of a region divided based on luminance, and the like. The image to be identified is extracted by image processing with a relatively small amount of computation (hereinafter referred to as “previous image recognition processing”). The identification processing target image is an image portion (a part of the captured image) that is a target of subsequent image processing, and includes a human candidate image. The human candidate image is an image portion (part of the captured image) that is considered to be a human image.

  The identification unit 32 is a functional element that identifies whether the human candidate image included in the identification processing target image extracted by the extraction unit 31 is a human image. Specifically, the identification unit 32 has a relatively large amount of calculation such as an image recognition process using an image feature description represented by HOG (Histograms of Oriented Gradients) feature and a classifier generated by machine learning. Image processing (hereinafter referred to as “rear-stage image recognition processing”) identifies whether the human candidate image is a human image. The rate at which the identification unit 32 identifies the human candidate image as a human image increases as the extraction processing target image is extracted by the extraction unit 31 with higher accuracy. Note that the identification unit 32 identifies and extracts all human candidate images as human images when, for example, a captured image having a desired quality cannot be obtained in an environment unsuitable for imaging at night or in bad weather. All of the human candidate images in the identification processing target image extracted by the unit 31 may be identified as people. This is to prevent human detection omissions.

  Next, with reference to FIG. 3, how the human image appears in the captured image behind the excavator captured by the rear camera 40B will be described. Note that the two captured images in FIG. 3 are examples of captured images of the rear camera 40B. 3 represents the presence of a human image, and is not displayed in an actual captured image.

  The rear camera 40B is a wide camera, and is attached at a height at which a person is looked down obliquely from above. For this reason, how the human image is seen in the captured image varies greatly depending on the direction in which the person is seen from the rear camera 40B. For example, the human image in the captured image is displayed so as to be inclined closer to the left and right ends of the captured image. This is due to image collapse caused by the wide-angle lens of the wide camera. Further, the closer to the rear camera 40B, the larger the head is displayed. In addition, the leg part enters the blind spot of the excavator's car body and disappears. These are caused by the installation position of the rear camera 40B. Therefore, it is difficult to identify a human image included in the captured image by image processing without performing any processing on the captured image.

  Therefore, the periphery monitoring system 100 according to the embodiment of the present invention promotes the identification of the human image included in the identification processing target image by normalizing the identification processing target image. Note that “normalization” means that the identification processing target image is converted into an image having a predetermined size and a predetermined shape. In this embodiment, an identification processing target image that can take various shapes in a captured image is converted into a rectangular image of a predetermined size by projective conversion. As the projective transformation, for example, an 8-variable projective transformation matrix is used.

  Here, an example of processing (hereinafter, referred to as “normalization processing”) in which the periphery monitoring system 100 normalizes the identification processing target image will be described with reference to FIGS. 4 to 6. FIG. 4 is a schematic diagram illustrating an example of a geometric relationship used when the extraction unit 31 cuts out an identification processing target image from a captured image.

  A box BX in FIG. 4 is a virtual three-dimensional object in real space, and is a virtual rectangular parallelepiped defined by eight vertices A to H in this embodiment. The point Pr is a reference point that is set in advance to refer to the identification processing target image. In the present embodiment, the reference point Pr is a point set in advance as an assumed standing position of a person, and is located at the center of a quadrilateral ABCD defined by four vertices A to D. The size of the box BX is set based on the direction of the person, the stride, the height, and the like. In this embodiment, the rectangle ABCD and the rectangle EFGH are squares, and the length of one side is, for example, 800 mm. Further, the height of the rectangular parallelepiped is, for example, 1800 mm. That is, the box BX is a rectangular parallelepiped having a width of 800 mm, a depth of 800 mm, and a height of 1800 mm.

  A quadrangle ABGH defined by the four vertices A, B, G, and H forms a virtual plane region TR corresponding to the region of the identification target image in the captured image. Further, the quadrangle ABGH as the virtual plane region TR is inclined with respect to the virtual ground that is a horizontal plane.

  In the present embodiment, a box BX as a virtual rectangular parallelepiped is employed to define the relationship between the reference point Pr and the virtual plane region TR. However, as long as the virtual plane region TR that faces the direction of the imaging device 40 and is inclined with respect to the virtual ground can be determined in association with an arbitrary reference point Pr, the relationship using other virtual three-dimensional objects, etc. May be adopted, and other mathematical relations such as a function and a conversion table may be adopted.

  FIG. 5 is a top view of the real space behind the shovel, and shows the positional relationship between the rear camera 40B and the virtual plane regions TR1 and TR2 when the virtual plane regions TR1 and TR2 are referenced using the reference points Pr1 and Pr2. Show. In this embodiment, the reference point Pr can be arranged at each of the lattice points of the virtual grid on the virtual ground. However, the reference points Pr may be irregularly arranged on the virtual ground, or may be arranged at equal intervals on a line segment radially extending from the projection point of the rear camera 40B on the virtual ground. For example, each line segment extends radially in increments of 1 degree, and the reference points Pr are arranged at intervals of 100 mm on each line segment.

  As shown in FIGS. 4 and 5, the first surface of the box BX defined by the quadrangle ABFE (see FIG. 4) is positive to the rear camera 40B when the virtual plane region TR1 is referenced using the reference point Pr1. It arranges so that it may be. That is, the line segment connecting the rear camera 40B and the reference point Pr1 is orthogonal to the first surface of the box BX arranged in association with the reference point Pr1 when viewed from above. Similarly, the first surface of the box BX is arranged to face the rear camera 40B even when the virtual plane region TR2 is referenced using the reference point Pr2. That is, a line segment connecting the rear camera 40B and the reference point Pr2 is orthogonal to the first surface of the box BX arranged in association with the reference point Pr2 when viewed from above. This relationship holds even when the reference point Pr is arranged on any lattice point. That is, the box BX is arranged so that the first surface thereof is always directly opposite the rear camera 40B.

  FIG. 6 is a diagram illustrating a flow of processing for generating a normalized image from a captured image. Specifically, FIG. 6A is an example of a captured image of the rear camera 40B, and displays a box BX arranged in association with the reference point Pr in real space. FIG. 6B is a diagram in which a region of the identification processing target image (hereinafter referred to as “identification processing target image region TRg”) in the captured image is cut out, and is displayed in the captured image of FIG. Corresponding to the virtual plane region TR. FIG. 6C shows a normalized image TRgt obtained by normalizing the identification processing target image having the identification processing target image region TRg.

  As shown in FIG. 6A, the box BX arranged in the real space in relation to the reference point Pr1 determines the position of the virtual plane region TR in the real space, and the imaging corresponding to the virtual plane region TR. An identification processing target image region TRg on the image is determined.

  Thus, if the position of the reference point Pr in the real space is determined, the position of the virtual plane region TR in the real space is uniquely determined, and the identification processing target image region TRg in the captured image is also uniquely determined. Then, the extraction unit 31 can generate a normalized image TRgt having a predetermined size by normalizing the identification processing target image having the identification processing target image region TRg. In the present embodiment, the size of the normalized image TRgt is, for example, 64 pixels long × 32 pixels wide.

  FIG. 7 is a diagram illustrating a relationship among a captured image, an identification processing target image region, and a normalized image. Specifically, FIG. 7A1 shows the identification processing target image region TRg3 in the captured image, and FIG. 7A2 shows the normalized image TRgt3 of the identification processing target image having the identification processing target image region TRg3. . FIG. 7B1 shows an identification processing target image region TRg4 in the captured image, and FIG. 7B2 shows a normalized image TRgt4 of the identification processing target image having the identification processing target image region TRg4. Similarly, FIG. 7C1 shows the identification processing target image region TRg5 in the captured image, and FIG. 7C2 shows the normalized image TRgt5 of the identification processing target image having the identification processing target image region TRg5.

  As shown in FIG. 7, the identification processing target image region TRg5 in the captured image is larger than the identification processing target image region TRg4 in the captured image. This is because the distance between the virtual plane area corresponding to the identification processing target image area TRg5 and the rear camera 40B is smaller than the distance between the virtual plane area corresponding to the identification processing target image area TRg4 and the rear camera 40B. Similarly, the identification processing target image region TRg4 in the captured image is larger than the identification processing target image region TRg3 in the captured image. This is because the distance between the virtual plane area corresponding to the identification processing target image area TRg4 and the rear camera 40B is smaller than the distance between the virtual plane area corresponding to the identification processing target image area TRg3 and the rear camera 40B. That is, the identification processing target image area in the captured image is smaller as the distance between the corresponding virtual plane area and the rear camera 40B is larger. On the other hand, the normalized images TRgt3, TRgt4, and TRgt5 are all rectangular images having the same size.

  In this manner, the extraction unit 31 can normalize the identification processing target image that can take various shapes and sizes in the captured image to a rectangular image of a predetermined size, and can normalize a human candidate image including a human image. Specifically, the extraction unit 31 arranges an image portion (hereinafter referred to as “head image portion”) that is estimated to be the head of the human candidate image in a predetermined region of the normalized image. In addition, an image portion (hereinafter, referred to as a “body portion image portion”) that is estimated to be the trunk portion of the human candidate image is arranged in another predetermined region of the normalized image, and yet another predetermined portion of the normalized image. An image portion (hereinafter referred to as a “leg image portion”) estimated to be a leg portion of the human candidate image is arranged in the region. Further, the extraction unit 31 can acquire the normalized image in a state where the inclination (image collapse) of the human candidate image with respect to the shape of the normalized image is suppressed.

  Next, referring to FIG. 8, normalization processing in a case where the identification processing target image area includes an image area that is not suitable for identification that adversely affects identification of human images (hereinafter referred to as “identification process inappropriate area”). Will be described. The identification processing inappropriate area is a known area where no human image can exist, for example, an area in which the excavator's vehicle body is reflected (hereinafter referred to as “vehicle body reflection area”), an area that protrudes from the captured image ( Hereinafter, it is referred to as “extrusion area”). FIG. 8 is a diagram illustrating the relationship between the identification processing target image area and the identification processing inappropriate area, and corresponds to FIG. 7 (C1) and FIG. 7 (C2). Further, the right-slanted hatched area in the left diagram of FIG. 8 corresponds to the protrusion area R1, and the left-slanted hatched area corresponds to the vehicle body reflection area R2.

  In the present embodiment, when the identification process target image region TRg5 includes a part of the protrusion region R1 and the vehicle body reflection region R2, the extraction unit 31 performs mask processing on these identification processing inappropriate regions, and then performs an identification processing target image. A normalized image TRgt5 of the identification processing target image having the region TRg5 is generated. Note that the extraction unit 31 may mask the portion corresponding to the identification processing inappropriate region in the normalized image TRgt5 after generating the normalized image TRgt5.

  The right figure of FIG. 8 shows normalized image TRgt5. Further, in the right diagram of FIG. 8, the diagonally hatched area that falls to the right represents the mask area M1 corresponding to the protruding area R1, and the diagonally hatched area that falls to the left represents the mask area M2 corresponding to a part of the vehicle body reflection area R2. Represents.

  In this way, the extraction unit 31 masks the image of the identification process inappropriate area, thereby preventing the image of the identification process inappropriate area from affecting the identification process performed by the identification unit 32. By this mask processing, the identification unit 32 can identify whether the image is a human image using an image of a region other than the mask region in the normalized image without being affected by the image of the identification processing inappropriate region. Note that the extraction unit 31 may use any known method other than the mask process so that the image in the identification process inappropriate region does not affect the identification process performed by the identification unit 32.

  Next, characteristics of the normalized image generated by the extraction unit 31 will be described with reference to FIG. FIG. 9 is a diagram illustrating an example of a normalized image. Further, the 14 normalized images shown in FIG. 9 include images of human candidates that are closer to the rear camera 40B as the normalized image is closer to the left end of the diagram, and the normalized image closer to the right end of the diagram is It includes images of human candidates that are located far from the rear camera 40B.

  As shown in FIG. 9, the extraction unit 31 performs any normalization regardless of the rear horizontal distance (the horizontal distance in the Y-axis direction shown in FIG. 5) between the virtual plane region TR and the rear camera 40B in the real space. In the image, the head image portion, the torso image portion, the leg image portion, and the like can be arranged at substantially the same ratio. Therefore, the extraction unit 31 can reduce the calculation load when the identification unit 32 executes the identification process, and can improve the reliability of the identification result. The rear horizontal distance described above is an example of information regarding the positional relationship between the virtual plane region TR and the rear camera 40B in the real space, and the extraction unit 31 adds the information to the extracted identification processing target image. . Further, the information on the positional relationship described above includes a top view angle with respect to the optical axis of the rear camera 40B of a line segment connecting the reference point Pr corresponding to the virtual plane region TR and the rear camera 40B.

  With the above configuration, the periphery monitoring system 100 generates the normalized image TRgt from the identification processing target image region TRg corresponding to the virtual plane region TR that faces the imaging device 40 and is inclined with respect to the virtual ground that is a horizontal plane. . Therefore, normalization in consideration of how the person looks in the height direction and depth direction can be realized. As a result, even when a captured image of the imaging device 40 attached to the construction machine so as to capture an image of a person from above is used, a person existing around the construction machine can be detected more reliably. In particular, even when a person approaches the imaging device 40, the normalized image can be generated from the identification processing target image that occupies a sufficiently large area on the captured image, so that the person can be reliably detected.

  In addition, the periphery monitoring system 100 defines the virtual plane region TR as a rectangular region formed by the four vertices A, B, G, and H of the box BX that is a virtual cuboid in real space. Therefore, the reference point Pr in the real space can be geometrically associated with the virtual plane region TR, and further, the virtual plane region TR in the real space and the identification processing target image region TRg in the captured image can be geometrically related. Can be associated.

  In addition, the extraction unit 31 performs mask processing on the image of the identification processing inappropriate area included in the identification processing target image area TRg. Therefore, the identification unit 32 can identify whether the image is a human image by using an image of a region other than the mask region in the normalized image without being affected by the image of the identification inappropriate region including the vehicle body reflection region R2.

  Further, the extraction unit 31 can extract an identification processing target image for each reference point Pr. In addition, each of the identification processing target image areas TRg is associated with one of the reference points Pr set in advance as an assumed standing position of the person via the corresponding virtual plane area TR. Therefore, the periphery monitoring system 100 can extract an identification processing target image that is likely to include a human candidate image by extracting the reference point Pr that is likely to be present by an arbitrary method. In this case, it is possible to prevent the identification processing target image having a low possibility of including the human candidate image from being subjected to the identification processing by the image processing having a relatively large calculation amount, and to realize the speedup of the human detection processing. .

  Next, an example of processing in which the extraction unit 31 extracts an identification processing target image that is highly likely to include a human candidate image will be described with reference to FIGS. 10 and 11. FIG. 10 is a schematic diagram illustrating an example of a geometric relationship used when the extraction unit 31 cuts out an identification processing target image from a captured image, and corresponds to FIG. FIG. 11 is a diagram illustrating an example of a feature image in a captured image. Note that the feature image is an image that represents a characteristic part of a person, and is preferably an image that represents a part in which the height from the ground in real space is difficult to change. Therefore, the feature image includes, for example, an image of a helmet, an image of a shoulder, an image of a head, an image of a reflector or a marker attached to a person, and the like.

  In particular, the helmet has a feature that its shape is approximately a sphere, and when the projected image is projected onto the captured image, it is always nearly circular regardless of the imaging direction. In addition, the helmet has a hard surface and is glossy or semi-glossy, and when the projected image is projected on the captured image, a local high-intensity region and a radial luminance gradient centered on the region are generated. It has the feature of being easy. Therefore, the helmet image is particularly suitable as a feature image. The feature that the projected image is close to a circle, the feature that it is easy to generate a radial brightness gradient centered on a local high brightness area, etc. are used for image processing to find the helmet image from the captured image. May be. The image processing for finding the helmet image from the captured image includes, for example, luminance smoothing processing, Gaussian smoothing processing, luminance maximum point searching processing, luminance minimum point searching processing, and the like.

  In the present embodiment, the extraction unit 31 finds out a helmet image (an image that can be estimated to be a helmet strictly) in the captured image by the pre-stage image recognition process. This is because a person working around the excavator is considered to be wearing a helmet. Then, the extraction unit 31 derives the most relevant reference point Pr from the position of the found helmet image. Then, the extraction unit 31 extracts an identification processing target image corresponding to the reference point Pr.

  Specifically, the extraction unit 31 derives a highly relevant reference point Pr from the position of the helmet image in the captured image using the geometric relationship shown in FIG. The geometric relationship in FIG. 10 is different from the geometric relationship in FIG. 4 in that the virtual head position HP in the real space is determined, but is common in other points.

  The virtual head position HP represents the head position of a person assumed to be present on the reference point Pr, and is disposed immediately above the reference point Pr. In this embodiment, it is arranged at a height of 1700 mm on the reference point Pr. Therefore, if the virtual head position HP in the real space is determined, the position of the reference point Pr in the real space is uniquely determined, and the position of the virtual plane region TR in the real space is also uniquely determined. Further, the identification processing target image region TRg in the captured image is also uniquely determined. Then, the extraction unit 31 can generate a normalized image TRgt having a predetermined size by normalizing the identification processing target image having the identification processing target image region TRg.

  Conversely, if the position of the reference point Pr in the real space is determined, the virtual head position HP in the real space is uniquely determined, and the head image position AP on the captured image corresponding to the virtual head position HP in the real space is also unique. It is decided. Therefore, the head image position AP can be set in advance in association with each of the preset reference points Pr. The head image position AP may be derived from the reference point Pr in real time.

  Therefore, the extraction unit 31 searches for the helmet image in the captured image of the rear camera 40B by the pre-stage image recognition process. The upper part of FIG. 11 shows a state where the extraction unit 31 has found the helmet image HRg. And the extraction part 31 determines the representative position RP, when the helmet image HRg is found. The representative position RP is a position derived from the size, shape, etc. of the helmet image HRg. In this embodiment, the representative position RP is the position of the central pixel in the helmet image area including the helmet image HRg. The lower diagram in FIG. 11 is an enlarged view of a helmet image region that is a rectangular image region partitioned by white lines in the upper diagram in FIG. 11, and shows that the position of the central pixel in the helmet image region is the representative position RP.

  Thereafter, the extraction unit 31 derives a head image position AP that is closest to the representative position RP using, for example, a nearest neighbor search algorithm. In the lower part of FIG. 11, six head image positions AP1 to AP6 are preset near the representative position RP, and the head image position AP5 is the head image position AP closest to the representative position RP. It shows that.

  Then, using the geometrical relationship shown in FIG. 10, the extraction unit 31 traces the virtual head position HP, the reference point Pr, and the virtual plane area TR from the nearest head image position AP derived. The identification processing target image region TRg to be extracted is extracted. Thereafter, the extraction unit 31 normalizes the identification processing target image having the extracted identification processing target image region TRg to generate a normalized image TRgt.

  In this way, the extraction unit 31 includes the representative position RP of the helmet image HRg that is the position of the human characteristic image in the captured image, and one of the preset head image positions AP (head image position AP5). Are associated with each other to extract an identification processing target image.

  Note that the extraction unit 31 directly associates the head image position AP with the reference point Pr, the virtual plane region TR, or the identification processing target image region TRg instead of using the geometrical relationship shown in FIG. An identification processing target image corresponding to the head image position AP may be extracted using a table.

  In addition, the extraction unit 31 may derive the reference point Pr from the representative position RP using a known algorithm other than the nearest neighbor search algorithm such as a hill-climbing method or a Mean-shift method. For example, when using the hill-climbing method, the extraction unit 31 derives a plurality of head image positions AP in the vicinity of the representative position RP, and the reference points Pr corresponding to the representative position RP and each of the plurality of head image positions AP. Is linked. At this time, the extraction unit 31 weights the reference point Pr so that the weight becomes larger as the representative position RP and the head image position AP are closer. Then, the weight distribution of the plurality of reference points Pr is climbed, and the identification processing target image region TRg is extracted from the reference point Pr having the weight closest to the maximum point of the weight.

  Next, an example of processing (hereinafter referred to as “image extraction processing”) in which the extraction unit 31 of the controller 30 extracts an identification processing target image will be described with reference to FIG. FIG. 12 is a flowchart showing an example of the image extraction process.

  First, the extraction unit 31 searches for a helmet image in the captured image (step ST1). In the present embodiment, the extraction unit 31 performs a raster scan on the image captured by the rear camera 40B by the preceding image recognition process to find a helmet image.

  When the helmet image HRg is found from the captured image (YES in step ST1), the extraction unit 31 acquires the representative position RP of the helmet image HRg (step ST2).

  Thereafter, the extraction unit 31 acquires a head image position AP that is closest to the acquired representative position RP (step ST3).

  Thereafter, the extraction unit 31 extracts an identification processing target image corresponding to the acquired head image position AP (step ST4). In the present embodiment, the extraction unit 31 uses the geometrical relationship shown in FIG. 10 and uses the head image position AP in the captured image, the virtual head position HP in the real space, and the assumed standing position of the person in the real space. The identification processing target image is extracted by following the correspondence between the reference point Pr and the virtual plane region TR in the real space.

  If the helmet image HRg is not found in the captured image (NO in step ST1), the extraction unit 31 shifts the process to step ST5 without extracting the identification processing target image.

  Then, the extraction part 31 determines whether the helmet image was searched over the whole captured image (step ST5).

  If it is determined that the entire captured image has not yet been searched (NO in step ST5), the extraction unit 31 performs the processing in steps ST1 to ST4 on another region of the captured image.

  On the other hand, when it is determined that the search for the helmet image over the entire captured image has been completed (YES in step ST5), the extraction unit 31 ends the current image extraction process.

  In this way, the extraction unit 31 first finds the helmet image HRg, and from the representative position RP of the found helmet image HRg, the head image position AP, the virtual head position HP, the reference point (assumed standing position) Pr, the virtual An identification processing target image region TRg is specified through the plane region TR. Then, by extracting and normalizing the identification processing target image having the identified identification processing target image region TRg, a normalized image TRgt of a predetermined size can be generated.

  With the above configuration, the extraction unit 31 of the periphery monitoring system 100 finds a helmet image as a feature image in the captured image, and associates the representative position RP of the helmet image with one of the head image positions AP as a predetermined image position. Thus, the identification processing target image is extracted. Therefore, it is possible to narrow down the image portion to be subjected to the subsequent image recognition processing with a simple system configuration.

  The extraction unit 31 first finds the helmet image HRg from the captured image, derives one of the head image positions AP corresponding to the representative position RP of the helmet image HRg, and sets it as one of the head image positions AP. A corresponding identification processing target image may be extracted. Alternatively, the extraction unit 31 first acquires one of the head image positions AP, and the helmet image is in a helmet image area that is a predetermined area including the position of the feature image corresponding to one of the head image positions AP. May exist, an identification processing target image corresponding to one of the head image positions AP may be extracted.

  Further, the extraction unit 31 may extract the identification processing target image from the representative position RP of the helmet image in the captured image using a predetermined geometrical relationship as illustrated in FIG. In this case, the predetermined geometric relationship is such that the identification processing target image region TRg in the captured image, the virtual plane region TR in the real space corresponding to the identification processing target image region TRg, and the real space corresponding to the virtual plane region TR. Reference point Pr (assumed standing position of a person), virtual head position HP corresponding to reference point Pr (virtual feature position that is a position in a real space of a characteristic portion of the person corresponding to the assumed standing position of the person), and The geometrical relationship with the head image position AP (predetermined image position in the captured image corresponding to the virtual feature position) in the captured image corresponding to the virtual head position HP is represented.

  Here, referring to FIG. 2 again, the description of the other functional elements of the controller 30 is continued.

  The tracking unit 33 is a functional element that tracks the identification result output by the identification unit 32 every predetermined time and outputs the final human detection result. In the present embodiment, the tracking unit 33 determines that the corresponding person candidate image is a human image when the identification results regarding the same person for a predetermined number of consecutive times satisfy a predetermined condition. That is, it is determined that a person exists at the corresponding three-dimensional position (actual position). Whether or not they are the same person is determined based on their actual positions. Specifically, the tracking unit 33 recognizes the person within a predetermined time based on the actual position (reference point PrI) of the person shown in the image identified as the person image in the first identification process by the identification unit 32. Derive the reachable range. The reachable range is set based on the maximum turning speed of the shovel, the maximum traveling speed of the shovel, the maximum movement speed of the person, and the like. Then, if the real position (reference point PrII) of the person shown in the image identified as the human image in the second identification process is within the range, it is determined that the person is the same person. The same applies to the third and subsequent identification processes. And the tracking part 33 determines with a person existing in a corresponding three-dimensional position, for example, when it identifies that it is a person's image of the same person in 4 times among 6 continuous identification results. In addition, even if a person image is identified in the first identification process, if a person image of the same person is not identified in the subsequent three identification processes, the corresponding three-dimensional It is determined that there is no person at the position.

  As described above, the combination of the extraction unit 31, the identification unit 32, and the tracking unit 33 constitutes a human detection unit 34 that detects whether or not there is a person around the shovel based on the captured image of the imaging device 40. .

  With this configuration, the person detection unit 34 makes false reports (determining that a person is present despite the absence of a person), misreporting (determining that a person is present despite the presence of a person), etc. Can be suppressed.

  Further, the person detection unit 34 can determine whether the person is approaching or moving away from the shovel based on the transition of the actual position of the person shown in the image identified as the person image. And the person detection part 34 may output a control command to the control part 35, and may output a warning, when the distance from the shovel of the person's real position is less than predetermined value. In this case, the person detection unit 34 may adjust the predetermined value according to the excavator operation information (for example, turning speed, turning direction, traveling speed, traveling direction, etc.).

  Further, the human detection unit 34 may discriminate and recognize at least two stages of the human detection state and the human non-detection state. For example, a state in which at least one of a condition regarding distance and a condition regarding reliability is satisfied is determined as a first person detection state (warning state), and a state where both are satisfied is determined as a second person detection state (alarm). State). The condition regarding the distance includes, for example, that the distance from the excavator at the actual position of the person shown in the image identified as the human image is less than a predetermined value. The condition relating to reliability includes, for example, identification of a person image of the same person in four out of six consecutive identification results. In the first person detection state (warning state), a first alarm is output as a preliminary alarm with low accuracy but quick response. The first alarm is, for example, a low-volume beep sound, and is automatically stopped when neither of the two conditions is satisfied. In the second person detection state (alarm state), a second alarm is output as a formal alarm with high accuracy but slow response. The second alarm is, for example, a loud melody sound, and is not automatically stopped even when at least one of the conditions is not satisfied, and an operator's operation is required for the stop.

  The control unit 35 is a functional element that controls various devices. In the present embodiment, the control unit 35 controls various devices in accordance with an operator input via the input device 41. For example, the display image displayed on the screen of the in-vehicle display is switched according to an image switching command input through the touch panel. The display image includes a through image of the rear camera 40B, a through image of the right side camera 40R, a through image of the left side camera 40L, a viewpoint conversion image, and the like. The viewpoint conversion image is, for example, a bird's-eye view image (image viewed from a virtual viewpoint directly above the shovel) synthesized from images captured by a plurality of cameras.

  Further, the control unit 35 controls various devices according to the final human detection result of the tracking unit 33 constituting the human detection unit 34. For example, a control command is output to the machine control device 51 according to the final human detection result of the tracking unit 33 to switch the shovel state between the first state and the second state. The first state includes a state where the restriction of the shovel movement is released, a state where the alarm output is stopped, and the like. The second state includes a state where movement of the excavator is limited or stopped, a state where an alarm is output, and the like. In this embodiment, the control unit 35 outputs a control command to the machine control device 51 when it is determined that a person is present within a predetermined range around the shovel based on the final human detection result of the tracking unit 33. The shovel state is switched from the first state to the second state. For example, the excavator stops moving. In this case, the operation by the operator is invalidated. Specifically, a control command is output to the gate lock valve to disconnect the operating device from the hydraulic system, thereby forcibly creating a no-operation state and stopping the excavator movement. Alternatively, the engine may be stopped by outputting a control command to the engine control device. Alternatively, the movement of the hydraulic actuator may be limited by outputting a control command to a control valve that controls the flow rate of the hydraulic oil flowing into the hydraulic actuator to change the opening area of the control valve, the opening area change rate, and the like. In this case, the maximum turning speed, the maximum traveling speed, etc. are reduced.

  In addition, the control unit 35 returns the shovel state to the first state when a predetermined release condition is satisfied after the shovel state is set to the second state. That is, when a predetermined release condition is satisfied after restricting or stopping the shovel movement, the restriction or stop is released. The predetermined release condition includes, for example, “determining that there is no person within a predetermined range around the excavator” (hereinafter referred to as “first release condition”). Further, the predetermined release condition additionally includes, for example, “a state in which the excavator does not start” is secured (hereinafter referred to as “second release condition”). Further, the predetermined release condition may include “the operator confirming that there is no person in the vicinity of the excavator” (hereinafter referred to as “third release condition”). In the present embodiment, whether or not the excavator's movement is restricted or stopped, and whether or not each of the first release condition, the second release condition, and the third release condition is satisfied is managed using a flag. The

  The first release condition is, for example, that “the control unit 35 determines that there is no person within a predetermined range around the shovel based on the final human detection result of the tracking unit 33 configuring the human detection unit 34”. Including.

  The second release condition is, for example, “all operation devices are in a neutral position for a predetermined time or more”, “the gate lock lever is lowered (the operation device is invalid)”, This includes “an operator's limbs being released from all operating devices”, “a predetermined button operation has been performed”, and the like. “All the operation devices are in the neutral position” is detected by the control unit 35 based on, for example, the presence / absence of a command from each operation device, the output value of a sensor that detects the operation amount of each operation device, and the like. . The condition “over a predetermined time or more” has an effect of preventing the second release condition from being satisfied only by instantaneously becoming the neutral position. “The operator's limbs are separated from the operating device” is based on, for example, a captured image of a camera that images the driver's cab, an output of an electrostatic sensor attached to the operating device (for example, a grip of the operating lever) The control unit 35 detects this. “The predetermined button operation has been performed” means, for example, that a confirmation button (for example, a horn button or the same) is displayed in a state where a message such as “Are the excavator not moving? When the software button (displayed on the screen) is pressed, the control unit 35 detects.

  The third release condition is satisfied, for example, when the confirmation button is pressed while a message such as “Did you confirm that there is no person around the shovel?” Is displayed on the screen of the in-vehicle display. Note that the third release condition may be omitted.

  In a case where the third release condition is included in the predetermined release condition, the excavator enters the restriction release enabled state when the first release condition and the second release condition are satisfied. The restriction release enabled state means a state in which the restriction can be released as long as the operator confirms that there is no person around the excavator.

  There is no restriction on the order in which each of the first release condition, the second release condition, and the third release condition is satisfied. For example, even when the conditions are satisfied in the order of the third release condition, the second release condition, and the first release condition, the control unit 35 releases the restriction or stop of the shovel movement.

  Moreover, the control part 35 may cancel the restriction | limiting or a stop, when a predetermined | prescribed waiting time passes after a predetermined cancellation | release condition is satisfied. This is to prevent the operator from being panicked by sudden release.

  Further, when the movement of the excavator is restricted or stopped, the control unit 35 outputs a control command to the in-vehicle display as the output device 50, and displays a captured image including the human image that caused the cause. Good. For example, when a human image is included only in the captured image of the left side camera 40L, the through image of the left side camera 40L may be displayed alone. Alternatively, when a human image is included in each of the captured image of the left camera 40L and the captured image of the rear camera 40B, the through images of the two cameras may be displayed side by side, and the captured images of the two cameras may be displayed simultaneously. One synthesized image (for example, a viewpoint conversion image) may be displayed. Further, an image indicating that the device is being restricted or stopped, guidance on a release method, and the like may be displayed. In addition, an image portion corresponding to a human candidate image identified as a human image may be highlighted. For example, the outline of the identification processing target image region TRg may be displayed in a predetermined color. In addition, when a waiting time after the predetermined release condition is satisfied is set, the wait time countdown may be displayed after displaying that fact when the predetermined release condition is satisfied. Good. Further, when an alarm is output during the waiting time, the volume of the alarm may be gradually reduced as the waiting time elapses.

  Further, when the movement of the excavator is restricted or stopped, the control unit 35 outputs a control command to the in-vehicle speaker as the output device 50, and outputs a warning on the side where the person who caused the output exists. Good. In this case, the in-vehicle speaker includes, for example, a right side speaker installed on the right wall in the cab, a left side speaker installed on the left wall, and a rear speaker installed on the rear wall. Then, when a human image is included only in the captured image of the left side camera 40L, the control unit 35 outputs an alarm from only the left side speaker. Alternatively, the control unit 35 may localize the sound using a surround system including a plurality of speakers.

  In addition, when the human detection unit 34 identifies the human candidate image as a human image, the control unit 35 may output only an alarm without limiting or stopping the excavator movement. Also in this case, the control unit 35 determines that a state in which at least one of the condition regarding distance and the condition regarding reliability is satisfied as described above is the first person detection state (warning state), and the state in which both are satisfied is determined. The second person detection state (alarm state) may be determined. And the control part 35 may stop the warning in a 2nd person detection state (alarm state), when predetermined cancellation | release conditions are satisfy | filled similarly to the case where the movement of an excavator is restrict | limited or stopped. . This is because, unlike the alarm in the first person detection state (warning state) that can be automatically stopped, the operation of the operator is required to stop the alarm in the second person detection state (warning state). .

  Next, an example of processing in which the control unit 35 of the controller 30 monitors the periphery of the excavator (hereinafter referred to as “periphery monitoring processing”) will be described with reference to FIG. FIG. 13 is a flowchart showing an example of the flow of the periphery monitoring process. The controller 30 repeatedly executes the periphery monitoring process at a predetermined control cycle.

  First, the control unit 35 determines whether or not there is a person around the excavator (step ST11). In the present embodiment, the control unit 35 determines whether or not there is a person around the shovel based on the final human detection result of the tracking unit 33.

  Thereafter, when it is determined that there is a person around the shovel (YES in step ST11), the control unit 35 restricts or stops the shovel movement (step ST12). In the present embodiment, for example, when it is determined that the current person detection state is the second person detection state (alarm state), the control unit 35 determines that there is a person around the shovel and stops the movement of the shovel. .

  At this time, the control unit 35 outputs a control command to the in-vehicle speaker as the output device 50 to output a second alarm. In addition, a control command is output to an in-vehicle display as the output device 50 to display a captured image including a human image that has caused a restriction or stop.

  When it is determined that there is no person around the shovel (NO in step ST11), the control unit 35 determines whether or not the shovel movement has already been restricted or stopped (step S13). In the present embodiment, the control unit 35 refers to the value of the corresponding flag to determine whether or not the shovel movement has already been restricted or stopped.

  When it is determined that the excavator's movement has already been restricted or stopped (YES in step ST13), the control unit 35 performs a process for releasing the restriction or stop (hereinafter referred to as “limit release process”). Execute (step ST14).

  When it is determined that the excavator movement has not been restricted or stopped (NO in step ST13), the control unit 35 ends the current excavator periphery monitoring process without executing the restriction release process.

  Next, with reference to FIG. 14, a process in which the control unit 35 of the controller 30 releases the restriction or stop of the shovel movement will be described. FIG. 14 is a flowchart illustrating an example of the flow of restriction release processing.

  First, the control unit 35 determines whether or not the first release condition is satisfied (step ST21). In the present embodiment, the control unit 35 determines whether or not there is a person within a predetermined range around the excavator. Specifically, it is determined whether or not the current person detection state has escaped from the second person detection state (alarm state). It may be determined whether the first person detection state (warning state) and the second person detection state (warning state) have been removed.

  When it is determined that the first release condition is satisfied (YES in step ST21), the control unit 35 determines whether or not the second release condition is satisfied (step ST22). In the present embodiment, the control unit 35 determines whether or not a state where the excavator does not start is secured. Specifically, it is determined whether or not the gate lock lever is lowered (whether or not the operating device is disabled).

  When it is determined that the second release condition is satisfied (YES in step ST22), the control unit 35 determines whether the third release condition is satisfied (step ST23). In the present embodiment, the control unit 35 determines whether or not the operator has confirmed that there is no person around the excavator. Specifically, it is determined whether or not the confirmation button has been pressed in a state where a message such as “Did you confirm that there are no people around the shovel?” Is displayed on the screen of the in-vehicle display.

  When it is determined that the third release condition is satisfied (YES in step ST23), the control unit 35 releases the restriction or stop of the shovel movement (step ST24).

  At this time, the control unit 35 outputs a control command to the in-vehicle speaker as the output device 50 to stop the output of the second alarm. Moreover, a control command is output to the vehicle-mounted display as the output device 50, and the display of the captured image including the human image that has caused the restriction or stop is stopped. For example, the through image displayed before the second alarm is output is displayed again.

When it is determined that the first release condition is not satisfied (NO in step ST21), the control unit 35 is determined that the second release condition is not satisfied (NO in step ST22).
If it is determined that the third release condition is not satisfied (NO in step ST23), the current restriction release process is terminated without releasing the restriction or stop of the shovel movement.

  With the above configuration, the controller 30 can limit or stop the movement of the shovel when it is determined that there is a person around the shovel.

  In addition, when the controller 30 determines that there is no person around the shovel after restricting or stopping the shovel movement, the controller 30 only determines that the state where the shovel does not start is secured. The restriction or suspension can be lifted. Further, the controller 30 can release the restriction or stop only when it is determined that the state where the shovel does not start is secured and it is determined by the operator that there is no person around the shovel. Therefore, the controller 30 can prevent the shovel from starting unintentionally when the restriction or stop is released.

  Next, an example of an output image displayed on the in-vehicle display during execution of the periphery monitoring process will be described with reference to FIG. FIG. 15 is an example of an output image generated based on a captured image of the rear camera 40B. FIG. 15A shows an example of an output image when there is no person within a predetermined range around the excavator, FIG. 15B shows an example of an output image in the first person detection state, and FIG. Shows an example of an output image in the second person detection state.

  Specifically, the output image of FIG. 15 includes a camera image portion G1 and an indicator portion G2. The camera image portion G1 is a portion that displays an image generated based on the captured images of one or a plurality of cameras. The indicator part G2 is a part for displaying a person detection state / person non-detection state of each of a plurality of areas around the excavator. In the camera image portion G1, a line segment L1 superimposed and displayed on the camera image indicates that the distance from the shovel is a predetermined first distance (for example, 5 meters). Further, a line segment L2 superimposed and displayed on the camera image indicates that the distance from the shovel is a predetermined second distance (for example, 2.5 meters). In the indicator portion G2, the outer peripheral line L1g of the partial circle drawn around the excavator icon CG1 indicates that the distance from the excavator is a predetermined first distance (for example, 5 meters), and the line segment L1 of the camera image portion G1 Correspond. A partial rectangular outer peripheral line L2g drawn around the excavator icon CG1 indicates that the distance from the excavator is a predetermined second distance (for example, 2.5 meters), and corresponds to the line segment L2 of the camera image part G1. To do.

  The partial circle is divided into six areas A1 to A6, and the partial rectangle is divided into three areas B1 to B3.

  In the state shown in FIG. 15A, the controller 30 detects a person present on the right rear side of the shovel. However, since the person's real position is beyond the first distance, the controller 30 does not highlight the person's image and does not output the first alarm. However, the controller 30 may highlight the person's image, such as displaying the outline of the corresponding identification processing target image region TRg as a white frame, and may output a first alarm. Further, a message such as “peripheral monitoring process in progress” may be displayed regardless of whether or not a person has already been detected. This is because the operator can recognize that the peripheral monitoring process is being executed.

  In the first person detection state shown in FIG. 15 (B), the controller 30 detects a person existing within the first distance on the right rear side of the excavator and beyond the second distance. Therefore, the controller 30 highlights the person's image and outputs the first alarm. Specifically, the controller 30 displays the outline of the corresponding identification processing target image region TRg as a yellow frame F1 in the camera image portion G1. In the indicator portion G2, a region A4 corresponding to the person's actual position is displayed in yellow. However, the display of the yellow frame may be omitted. Moreover, you may display the message which tells that it is a 1st person detection state (warning state).

  In the second person detection state shown in FIG. 15C, the controller 30 detects a person existing within the second distance on the right rear side of the shovel. Therefore, the controller 30 highlights the person's image and outputs a second alarm. Specifically, the controller 30 displays the outline of the corresponding identification process target image region TRg as a red frame F2 in the camera image portion G1. In the indicator portion G2, a region B2 corresponding to the person's actual position is displayed in red. In addition, the controller 30 restricts the movement of the shovel, and blinks a message “Excavator operation is being restricted” indicating that the second person detection state (alarm state) has occurred. However, the display of the message indicating that the second person detection state (alarm state) is present may be omitted.

  15A to 15C, the camera image portion G1 is displayed on the left side of the screen and the indicator portion G2 is displayed on the right side of the screen, but the camera image portion G1 is displayed on the right side of the screen. The indicator portion G2 may be displayed on the left side of the screen. Further, the camera image part G1 may be displayed on one side of the screen divided vertically, and the indicator part G2 may be displayed on the other side. Further, the display of the indicator part G2 may be omitted.

  In addition, the spread angle of each of the regions A1 to A6 is 45 degrees, and the spread angle of each of the regions B1 to B3 is 90 degrees. This difference in the spread angle is based on a difference in properties between the first person detection state (warning state) and the second person detection state (warning state). Specifically, the first person detection state (warning state) is a state in which a preliminary alarm is output with low accuracy but quick response, and a relatively wide space range that is relatively far from the excavator is monitored. It is a range. For this reason, when the expansion angle of the areas A1 to A6 is increased, the monitoring range corresponding to each area is increased together with the display range, and the actual position of the person who caused the first alarm becomes difficult to understand. This is because the same display result is obtained anywhere in the wide monitoring range. On the other hand, the second person detection state (alarm state) is a state in which a formal alarm with high accuracy but slow response is output, and a relatively narrow space range relatively close to the excavator is a monitoring range. Yes. Therefore, when the expansion angle of the areas B1 to B3 is reduced, the monitoring range corresponding to each area is reduced together with the display range, and it is difficult to understand in which direction the person who caused the second alarm is located. End up. This is because the display range is small and difficult to see. Therefore, desirably, as shown in FIGS. 15A to 15C, the spread angles of the regions A1 to A6 are set to be smaller than the spread angles of the regions B1 to B3.

  15A to 15C illustrate a case where a human image is detected in the captured image of the rear camera 40B when the through image of the rear camera 40B is displayed. However, the above description is similarly applied to a case where a human image is detected in at least one of the left camera 40L and the right camera 40R when a through image of the rear camera 40B is displayed. . In this case, the output image displayed on the camera image portion G1 may be automatically switched from a through image of the rear camera 40B to a viewpoint conversion image synthesized from a through image of another camera or captured images of a plurality of cameras. Good. For example, if a human image is detected in the captured image of the left camera 40L when the through image of the rear camera 40B is displayed, the controller 30 outputs the output image displayed on the camera image portion G1 to the left camera 40L. The through image may be switched to.

  Next, the relationship between the detection state and the display colors of the frame and the region will be described with reference to FIG. FIG. 16 is a correspondence table showing a correspondence relationship between detection states and display colors of frames and regions.

  The first line of the correspondence table does not display the outline of the identification processing target image area when the detection state is neither the first person detection state (warning state) nor the second person detection state (warning state). It represents that no area | region of the part G2 is colored.

  In the second line, when the detection state is the alert state, the outline of the identification processing target image region corresponding to the human image that causes the alert state is displayed as a yellow frame, and the regions A1 to A6 are displayed. Indicates that one of these is displayed in yellow.

  In the third line, when the detection state is the alarm state, the outline of the identification processing target image region corresponding to the human image that caused the alarm state is displayed as a red frame, and the regions B1 to B3 are displayed. Is displayed in red.

  The fourth line shows that when the detection state is a warning state and an alarm state, a contour line corresponding to the human image that causes the warning state is displayed in a yellow frame and causes the alarm state. It represents that the outline corresponding to the human image is displayed with a red frame. Moreover, the area | region corresponding to the person image which caused the alert state among areas A1-A6 is displayed in yellow, and the area | region corresponding to the person image which caused the alarm state among areas B1-B3 is red. Indicates that it is displayed.

  With the above configuration, the controller 30 outputs an alarm and highlights an image portion of the person when it is determined that there is a person around the excavator. Therefore, the operator can confirm the person who caused the alarm on the screen. In addition, when an error occurs, the operator can check on the screen what is the cause of the error.

  Further, the controller 30 displays a frame image as a human detection marker on the camera image portion G1 for the first time when the first person detection state (warning state) is entered, and changes the color of the corresponding region of the indicator portion G2. Let Therefore, even a frame image corresponding to a human candidate image that is identified as a human image but still has low reliability is displayed, and the display image can be prevented from becoming complicated. In the above-described embodiment, the frame image is displayed as the human detection marker, but another emphasized image such as a reverse display image may be adopted as the human detection marker.

  Further, the image of the person who causes the first person detection state (warning state) and the image of the person who causes the second person detection state (warning state) are highlighted in a distinguishable manner. Further, the color of the frame image in the camera image part G1 is made to correspond to the color of the area in the indicator part G2. Therefore, the operator can confirm the person who caused the second alarm on the screen. In the above-described embodiment, the controller 30 varies the color of the frame image in the camera image portion G1 and the color of the region in the indicator portion G2 according to the detection state. However, the controller 30 may change attributes other than colors such as blinking / lighting state and transmittance according to the detection state.

  Next, another example of an output image displayed on the in-vehicle display during execution of the periphery monitoring process will be described with reference to FIG. FIG. 17 is an example of a viewpoint conversion image as an output image generated based on the captured images of the rear camera 40B, the left camera 40L, and the right camera 40R. In the present embodiment, the viewpoint conversion image is an image (hereinafter referred to as “periphery monitoring image”) used when the operator monitors the periphery of the excavator. The periphery monitoring image is, for example, an image that combines a road surface image when the excavator periphery is viewed from directly above and a horizontal image that is disposed around the road surface image and is viewed around the excavator in the horizontal direction. . FIG. 17 shows an example of an output image when the first person detection state and the second person detection state coexist. The output image of FIG. 17 includes a viewpoint conversion image portion G3 corresponding to the camera image portion G1 of FIG. The part corresponding to the indicator part G2 in FIG. 15 is integrated into the viewpoint conversion image part G3. Specifically, the excavator icon CG1 in FIG. 15 corresponds to the excavator icon CG2 in FIG. 17, and the areas A1 to A6 in FIG. 15 correspond to the areas C1 to C6 in FIG. 15 corresponds to the combination of the regions C1 and C2 in FIG. 17, the region B2 in FIG. 15 corresponds to the combination of the regions C3 and C4 in FIG. 17, and the region B3 in FIG. 15 corresponds to the region in FIG. Corresponds to a combination of C5 and C6. A line segment L3 superimposed and displayed on the viewpoint conversion image indicates that the distance from the excavator is a predetermined third distance (for example, 2.5 meters).

  In the detection state shown in FIG. 17, the controller 30 detects a person (a person who caused the first person detection state) existing within a first distance (for example, 5 meters) on the left side of the excavator and beyond a third distance. Detected. Moreover, the person (person who caused the 2nd person detection state) which exists within the 3rd distance behind a shovel is detected. Therefore, the controller 30 highlights the images of those persons, outputs a second alarm, and restricts the movement of the shovel. Specifically, the controller 30 displays a yellow circle MA1 as a human detection marker at the position of the reference point Pr corresponding to the person who caused the first person detection state, and an area corresponding to the position C2 is displayed in yellow. Also, a red circle MA2 as a human detection marker is displayed at the position of the reference point Pr corresponding to the person who caused the second person detection state, and the combination of the regions C3 and C4 corresponding to the position is red. Is displayed. Further, the controller 30 may cause the excavator to limit the movement of the shovel, and blink the message “Excavator operation being restricted” indicating that the second person detection state (alarm state) is present. Further, the display of the yellow circle MA1 may be omitted. This is to make the screen easier to see.

  Next, still another example of the output image displayed on the vehicle-mounted display during the periphery monitoring process will be described with reference to FIG. FIG. 18 is an example of an output image including a viewpoint conversion image generated based on captured images of the rear camera 40B, the left camera 40L, and the right camera 40R. FIG. 18 shows an example of an output image when the first person detection state and the second person detection state coexist as in the case of FIG. The output image of FIG. 18 includes an indicator part G2 and a viewpoint conversion image part G3. A partial rectangular outer peripheral line L2g drawn around the excavator icon CG1 indicates that the distance from the excavator is a predetermined third distance (for example, 2.5 meters), and corresponds to the line segment L3 of the viewpoint conversion image portion G3. .

  In the detection state shown in FIG. 18, the controller 30 detects a person (a person who caused the first person detection state) existing within a first distance (for example, 5 meters) on the left side of the excavator and beyond a third distance. Detected. Moreover, the person (person who caused the 2nd person detection state) which exists within the 3rd distance behind a shovel is detected. Therefore, the controller 30 highlights the images of those persons, outputs a second alarm, and restricts the movement of the shovel. Specifically, the controller 30 displays a yellow circle MA1 as a human detection marker at the position of the reference point Pr corresponding to the person who caused the first person detection state, and an indicator corresponding to the position. A region A2 of the portion G2 is displayed in yellow. Further, a red circle MA2 as a human detection marker is displayed at the position of the reference point Pr corresponding to the person who caused the second person detection state, and the area B2 of the indicator portion G2 corresponding to the position is displayed in red. Is displayed. In addition, the controller 30 restricts the movement of the shovel, and blinks a message “Excavator operation is being restricted” indicating that the second person detection state (alarm state) has occurred. Note that the display of the yellow circle MA1 may be omitted. This is to make the screen easier to see.

With the above configuration, the controller 30 can achieve the same effect as when the output image of FIG. 15 is displayed. In Patent Document 1, when the excavator displays a virtual viewpoint image for periphery monitoring, the display mode does not change depending on whether a person is detected around the shovel or not. Therefore, there is a possibility that the area including the detected person image is difficult to be recognized as a highly important area. On the other hand, the periphery monitoring system 100 according to the embodiment of the present invention allows the operator to quickly recognize a highly important area in the screen of the display device.

  Next, still another example of an output image displayed on the in-vehicle display during execution of the periphery monitoring process will be described with reference to FIG. Each of FIGS. 19A to 19E is an example of an output image including a peripheral monitoring image as a viewpoint conversion image generated based on a plurality of captured images. A line segment L3 drawn around the excavator icon CG2 indicates that the distance from the excavator is a predetermined third distance (for example, 2.5 meters).

  In the present embodiment, the periphery monitoring image is obtained by projecting the captured images of the rear camera 40B, the left side camera 40L, and the right side camera 40R onto a space model, and then projecting the projection image projected onto the space model into another image. It is an image obtained by reprojecting on a two-dimensional plane. The “space model” is a projection target of the captured image in the virtual space, and is configured by one or a plurality of planes or curved surfaces including a plane or curved surface other than the plane on which the peripheral monitoring image is located.

  The control unit 35 outputs a control command to a display device serving as the output device 50, and relatively enlarges and displays a part of the periphery monitoring image displayed on the screen of the display device. In the present embodiment, the control unit 35 locally enlarges and displays a specific image portion of the periphery monitoring image displayed on the screen of the display device. For example, the control unit 35 locally enlarges and displays a specific image portion in the periphery monitoring image by generating and displaying an image when the periphery monitoring image in the virtual space is viewed from a different virtual viewpoint. The specific image portion is, for example, an image portion having high importance such as an image portion to be noticed by an operator of the excavator, and is hereinafter also referred to as “attention image portion”. In this embodiment, the direction of the target image portion viewed from the shovel is one, and image portions in two or more directions are not simultaneously adopted as the target image portion. However, the present invention does not exclude that image portions in two or more directions are simultaneously employed as the attention image portion. Further, the direction of the target image portion viewed from the excavator may be limited to the directions of the six areas A1 to A6 shown in FIG. 15A, or may be limited to the directions of the three areas B1 to B3. Good. Further, the size and shape of the attention image portion may be fixed or dynamically determined.

  “Locally enlarged display” means, for example, that the attention image portion of the peripheral monitoring image is partially enlarged and displayed while maintaining a smooth connection with other image portions.

  Specifically, the control unit 35 locally enlarges and displays the image portion of interest in the peripheral monitoring image while displaying substantially the entire peripheral monitoring image in depth by loupe processing, projective conversion, scale conversion, and the like. The attention image part is, for example, an image part corresponding to a position where it is determined that a person exists in real space, an image part corresponding to a space in the traveling direction of the lower traveling body 1, or a space behind the upper swing body 3. Corresponding image portion or the like. Hereinafter, the process of locally enlarging the target image portion while displaying substantially the entire peripheral monitoring image is referred to as “local enlargement process”. “Substantially the entire periphery monitoring image” means that the periphery of the periphery monitoring image may protrude from the screen. In addition, when an image portion corresponding to a position where it is determined that a person exists in the real space is the attention image portion, the image of the person included in the attention image portion displayed locally enlarged is larger than a predetermined size. Is displayed. For example, an image of a person within a predetermined distance (for example, 12 m) from the excavator is displayed on the screen of the display device so as to be larger than a predetermined size (for example, 7 mm × 7 mm). The distance from the excavator is, for example, the shortest distance between the person and the excavator (for example, the side surface or the rear surface of the upper swing body 3), the distance from the center position of the cabin 10 to the person, and from the pivot axis of the shovel to the person. Distance.

  FIG. 19A shows an output image including a surrounding monitoring image before the local enlargement process is executed. The output image in FIG. 19A shows a situation where the worker W1 exists within the third distance behind the excavator.

  FIG. 19B shows an example of an output image including a peripheral monitoring image after the local enlargement process is executed. The output image of FIG. 19B shows a situation where the worker W1 exists within the third distance behind the excavator as in the case of FIG. 19A. In this example, the control unit 35 executes a local enlargement process when the person detection unit 34 detects a person. Specifically, the control unit 35 employs an image portion corresponding to the position in the real space of the person detected by the human detection unit 34 as the attention image portion. Then, the peripheral monitoring image is rotated in the screen so that the attention image portion is located at the lower center of the screen, and the attention image portion is locally enlarged. The rotation of the periphery monitoring image is executed, for example, using the center of the periphery monitoring image corresponding to the turning center of the excavator as a rotation axis. The rotation of the peripheral monitoring image and the enlargement of the target image portion are in no particular order. Further, rotation of the periphery monitoring image may be omitted. Further, the image portion other than the attention image portion may be reduced. For example, the image portion that is farther from the target image portion may be reduced so as to be displayed smaller.

  When a plurality of people are detected around the shovel, the control unit 35 employs an image portion corresponding to the position of the person closest to the shovel as the attention image portion. Alternatively, the control unit 35 calculates the risk in each direction viewed from the shovel based on the detected distance between the person and the excavator, the direction of the lower traveling body 1, the direction of the upper swing body 3, and the like. You may employ | adopt the image part corresponding to the space in the direction where the degree is the highest as an attention image part.

  The control unit 35 may execute the local enlargement process when the first person detection state (warning state) caused by the worker W1 is brought about, and the second person detection caused by the worker W1 is caused. Local enlargement processing may be performed when a condition (alarm condition) occurs.

  FIG. 19C shows another example of the output image including the peripheral monitoring image after the local enlargement process is executed. The output image of FIG. 19C shows a situation where the worker W2 exists within the third distance on the left side of the excavator. In this example, as in the case of FIG. 19B, the control unit 35 employs an image portion corresponding to the position in the real space of the person detected by the human detection unit 34 as the attention image portion. Then, the peripheral monitoring image is rotated in the screen so that the attention image portion is located at the lower center of the screen, and the attention image portion is locally enlarged.

  Further, the control unit 35 may rotate the periphery monitoring image within the screen in accordance with the movement of the worker W2 so that the image of the moving worker W2 is always located at the lower center of the screen. That is, the periphery monitoring image may be rotated within the screen according to a change in the display position of the attention image portion. This is to cancel the change in the display position of the target image portion.

  FIG. 19D shows still another example of the output image including the peripheral monitoring image after the local enlargement process is executed. The output image in FIG. 19D shows a situation where the worker W2 exists within the third distance on the left side of the excavator, as in the case of FIG. 19C.

  The output image of FIG. 19D is different from the output image of FIG. 19C in that a frame F3 is displayed around the image of the worker W2, but is common in other points. As described above, the control unit 35 may display an image for emphasizing the actual position of the worker W2 as in the frame F3. This is because the operator who views the output image can more easily recognize information related to the worker W2 (the presence / absence of the worker, the presence direction seen from the shovel, etc.).

  The control unit 35 rotates the periphery monitoring image in the screen according to the movement of the worker W2 so that the image of the moving worker W2 is always located in the frame F3. In this case, the position of the frame F3 is fixed. However, the control unit 35 may move the frame F3 according to the movement of the image of the worker W2 within the screen without rotating the periphery monitoring image within the screen.

  FIG. 19E shows still another example of the output image including the peripheral monitoring image after the local enlargement process is executed. A line segment L4 drawn around the excavator icon CG2 indicates that the distance from the excavator is a predetermined fourth distance (for example, 5.0 meters). The output image of FIG. 19 (E) shows a situation where the worker W3 exists within the fourth distance on the left side of the excavator and the worker W4 exists within the fourth distance behind the shovel.

  In this example, the control unit 35 displays the boundary between the region A2x and the region A3x corresponding to the region A2 and the region A3 of the indicator part G2 illustrated in FIG. 15 with a solid line. The solid line color (yellow) represents that each of the workers W3 and W4 causes the first person detection state (warning state). Further, the control unit 35 displays a message image MS1 that conveys the current human detection state. The message image MS1 indicates that the current person detection state is the first person detection state (warning state).

  Moreover, the control part 35 employ | adopts the image part corresponding to the position of the worker W3 nearest to the shovel as an attention image part. Therefore, the control unit 35 locally enlarges the attention image portion including the image of the worker W3, and rotates the periphery monitoring image within the screen so that the attention image portion is located at the lower center of the screen. .

  For example, when the operator W3 enters the area B1x corresponding to the area B1 of the indicator portion G2 shown in FIG. 15, the control unit 35 displays the outline of the area B1x with a solid line. In this case, the solid line color (red) indicates that the worker W3 causes the second person detection state (warning state). FIG. 19E shows the boundary of the region B1x with a broken line for convenience of explaining the position of the region B1x, but this broken line is not actually displayed.

  As described above, the control unit 35 employs the image portion corresponding to the position in the real space of the person detected by the human detection unit 34 as the attention image portion. Then, the attention image portion can be made conspicuous by locally expanding and displaying the attention image portion. Therefore, the operator who sees the output image including the peripheral monitoring image after the local enlargement process is executed can recognize at a glance information related to the worker W1 (the presence / absence of the worker, the presence direction seen from the shovel, etc.). . Further, the peripheral monitoring image can be rotated in the screen so that the attention image portion is always located at a specific portion (for example, the lower center) in the screen. Therefore, the operator can recognize information related to the worker W1 at a glance only by looking at a specific portion in the screen when an alarm or the like is output.

  Next, still another example of the output image displayed on the vehicle-mounted display during the periphery monitoring process will be described with reference to FIG. Each of FIG. 20A and FIG. 20B is an example of an output image including a peripheral monitoring image generated based on three captured images, similar to each of FIG. 19A to FIG. It is.

  FIG. 20A and FIG. 20B each show an example of an output image including a peripheral monitoring image after the local enlargement process is executed. However, the output image of FIG. 20 is subjected to local enlargement processing when the human detection unit 34 detects a person in that local enlargement processing is performed when the travel lever as the operation device is operated. This is different from the output image of FIG. The output image of FIG. 20 corresponds to the position in the real space of the person detected by the human detection unit 34 in that an image part corresponding to the space in the traveling direction of the lower traveling body 1 is adopted as the attention image part. This is different from the output image shown in FIG.

  In the present embodiment, the control unit 35 executes the local enlargement process when no person is detected around the shovel and when the travel lever is operated. Further, when a person is detected after executing the local enlargement process according to the operation of the traveling lever, the control unit 35 adopts an image part corresponding to the detected position in the real space as the attention image part. The local enlargement process is executed after reworking. However, the control unit 35 may continue the local enlargement process being executed when a person is detected after executing the local enlargement process in accordance with the operation of the travel lever. On the contrary, the control unit 35 corresponds to the space in the traveling direction of the lower traveling body 1 when the travel lever is operated after performing the local enlargement process according to the detection of the person around the excavator. The local enlargement process may be executed after the image portion is adopted again as the attention image portion. In addition, the control unit 35 may continue the local enlargement process being executed when the operation of the traveling lever is performed after the local enlargement process is executed in response to detection of a person around the shovel.

  Moreover, the control part 35 may perform a local expansion process, when a person is not detected around the shovel and the traveling lever is not operated. In this case, the control unit 35 may employ, for example, an image portion corresponding to a space behind the upper swing body 3 as the attention image portion. Or you may employ | adopt the image part corresponding to the space in front of the lower traveling body 1 as an attention image part.

  FIG. 20A shows an example of an output image including a peripheral monitoring image after the local enlargement process is executed in accordance with the operation of the travel lever. The output image of FIG. 20 (A) shows a situation in which the lower traveling body 1 travels diagonally right rearward with respect to the upper swing body 3. In this example, the control unit 35 employs an image portion corresponding to a space in the traveling direction of the lower traveling body 1 as the attention image portion. The traveling direction of the lower traveling body 1 is derived from, for example, the turning angle of the upper swing body 3 with respect to the lower traveling body 1. This is because the traveling direction of the lower traveling body 1 can be uniquely determined based on the longitudinal direction of the upper swing body 3. Specifically, the traveling direction of the lower traveling body 1 includes a pair of azimuth sensors attached to the lower traveling body 1 and the upper revolving body 3, an angular velocity sensor such as a gyroscope attached to the upper revolving body 3, turning It is derived based on the output of a rotation angle sensor such as a resolver and a rotary encoder attached to the mechanism 2.

  And the control part 35 rotates a periphery monitoring image within a screen so that an attention image part may be located in the center upper part in a screen, and expands an attention image part locally. The rotation of the periphery monitoring image is executed, for example, using the center of the periphery monitoring image corresponding to the turning center of the excavator as a rotation axis. The rotation of the peripheral monitoring image and the enlargement of the target image portion are in no particular order. Further, rotation of the periphery monitoring image may be omitted. Further, the image portion other than the attention image portion may be reduced. For example, the image portion that is farther from the target image portion may be reduced so as to be displayed smaller. An arrow AR1 superimposed and displayed on the periphery monitoring image indicates the traveling direction of the lower traveling body 1. The crawler image LR1 is an image of the lower traveling body 1 captured by the imaging device 40.

  FIG. 20B shows another example of the output image including the peripheral monitoring image after the local enlargement process is executed according to the operation of the traveling lever. The output image of FIG. 20B shows a situation in which the lower traveling body 1 travels diagonally rightward with respect to the upper swing body 3. In this example, the control unit 35 employs an image portion corresponding to a space in the traveling direction of the lower traveling body 1 as the attention image portion. Then, the attention image portion is locally enlarged in a state where the excavation attachment of the excavator icon CG2 is directed upward. Image portions other than the target image portion may be reduced. For example, the image portion that is farther from the target image portion may be reduced so as to be displayed smaller. An arrow AR2 superimposed on the surrounding monitoring image indicates the traveling direction of the lower traveling body 1. The crawler image LR1 is an image of the lower traveling body 1 captured by the imaging device 40.

  As described above, the control unit 35 employs the image portion corresponding to the space in the traveling direction of the lower traveling body 1 as the attention image portion. Then, the attention image portion can be made conspicuous by locally expanding and displaying the attention image portion. Therefore, the operator who has viewed the output image including the peripheral monitoring image after the local enlargement process is executed can recognize at a glance information related to the traveling direction of the excavator (such as the presence or absence of an obstacle). Further, the peripheral monitoring image can be rotated in the screen so that the attention image portion is always located in a specific portion (for example, the upper center) in the screen. Therefore, when an alarm or the like is output, the operator can recognize at a glance information related to the traveling direction of the shovel only by looking at a specific portion in the screen.

  Moreover, the control part 35 may perform a local expansion process according to vehicle information other than the presence or absence of operation with respect to a travel lever. For example, the local enlargement process may be executed when the turning operation lever is operated. In this case, the control unit 35 may employ an image portion corresponding to a space in the turning direction of the excavation attachment as the attention image portion.

  In addition, when the detected position of the person is behind the upper swing body 3, the control unit 35 moves the peripheral monitoring image so that the attention image portion corresponding to the position of the person is located at the center of the screen. The image portion may be locally enlarged. Alternatively, when the detected position of the person is in the traveling direction of the lower traveling body 1, the control unit 35 moves the peripheral monitoring image so that the attention image portion corresponding to the position of the person is located at the center of the screen. The attention image portion may be locally enlarged.

  Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the scope of the present invention. Can be added.

  For example, in the above-described embodiment, it is assumed that a person is detected using a captured image of the imaging device 40 mounted on the upper swing body 3 of the excavator, but the present invention is not limited to this configuration. . The present invention can also be applied to a configuration that uses a captured image of an imaging device that is attached to the main body of another work machine such as a mobile crane, a fixed crane, a riff mag machine, or a forklift.

  In the above-described embodiment, the excavator blind spot area is imaged using three cameras, but the excavator blind spot area may be imaged using one, two, or four or more cameras.

  In the above-described embodiment, human detection is performed using an image captured by the imaging device 40, but human detection is performed using outputs from an ultrasonic sensor, a laser radar, a pyroelectric sensor, a millimeter wave radar, and the like. Also good.

  In the above-described embodiment, the human detection process is individually applied to each of the plurality of captured images. However, the human detection process is applied to one composite image generated from the plurality of captured images. Also good.

  DESCRIPTION OF SYMBOLS 1 ... Lower traveling body 2 ... Turning mechanism 3 ... Upper turning body 4 ... Boom 5 ... Arm 6 ... Bucket 7 ... Boom cylinder 8 ... Arm cylinder 9 ... Bucket cylinder 10: cabin 30 ... controller 31 ... extraction unit 32 ... identification unit 33 ... tracking unit 34 ... human detection unit 35 ... control unit 40 ... imaging device 40B: Rear camera 40L: Left side camera 40R ... Right side camera 41 ... Input device 50 ... Output device 51 ... Machine control device 100 ... Perimeter monitoring system AP, AP1 AP6 ... head image position BX ... box G1 ... camera image part G2 ... indicator part G3 ... viewpoint conversion image part HD ... head HP ... virtual head part HRg ... Helmet image M1, M2 ... Mask region Pr, Pr1, Pr2, Pr10-Pr12 ... Reference point R1 ... Projection region R2 ... Car body reflection region RP ... Representative position TR, TR1, TR2, TR10 to TR12 ... virtual plane region TRg, TRg3, TRg4, TRg5 ... identification processing target image region TRgt, TRgt3, TRgt4, TRgt5 ... normalized image

Claims (9)

A detector that detects obstacles around the work machine;
A control unit mounted on the work machine,
The controller is
Displaying a first image generated using a captured image of an imaging device attached to the work machine and a second image including a graphic representing a surrounding area of the work machine on a display device; and
In the graphic representing the surrounding area of the work machine included in the second image, the area corresponding to the direction of the obstacle detected by the detection unit is emphasized.
Perimeter monitoring system for work machines. The display area where the graphic representing the surrounding area of the work machine is displayed is displayed on the display device so as to include the rear of the graphic representing the work machine.
The work machine periphery monitoring system according to claim 1. The detection unit determines a non-detection state and a two-stage detection state based on a distance from the work machine,
The two-stage detection state includes a first detection state and a second detection state,
The distance of the obstacle causing the first detection state from the work machine is greater than the distance of the obstacle causing the second detection state from the work machine,
The work machine periphery monitoring system according to claim 1 or 2. The display area on which the graphic representing the surrounding area of the work machine is displayed is divided into a plurality of areas by boundary lines with respect to the distance and direction, and an area corresponding to a place where an obstacle is detected is emphasized.
The work machine periphery monitoring system according to any one of claims 1 to 3 . The control unit corresponds to the area corresponding to the first detection state, the area corresponding to the second detection state, and the non-detection state in a display area where a graphic representing the surrounding area of the work machine is displayed. Display different areas with different colors,
The work machine periphery monitoring system according to claim 3 . The display area where the graphic representing the surrounding area of the work machine is displayed has a shape of a partial circle in which a portion corresponding to the front of the graphic representing the work machine is cut out.
The work machine periphery monitoring system according to claim 1 . The figure representing the surrounding area of the work machine has a shape of a partial circle, and a plurality of fan shapes are formed so as to correspond to the direction by a plurality of lines extending from the figure representing the work machine to the outer circumference of the partial circle. Divided into shapes,
The work machine periphery monitoring system according to claim 1 . The control unit does not display a message notifying that the first detection state is in the first detection state, but displays a message notifying that the second detection state is in the second detection state. To
The work machine periphery monitoring system according to claim 3 . The display area where the first image is displayed is larger than the display area where the second image is displayed.
The work machine periphery monitoring system according to claim 1 .