JP6662622B2 - Perimeter monitoring system for work machine and construction machine equipped with perimeter monitoring system for work machine - Google Patents

Description

  The present invention relates to a work machine periphery monitoring system that monitors the periphery of a work machine.

  2. Description of the Related Art A shovel including a sensor for detecting an object (person) existing around the shovel is known (see Patent Document 1). When the shovel detects an object (person) on the right side of the shovel, an alarm is output from a speaker installed on the right wall in the cab, and a through image of a camera that captures an image of the right side of the shovel is displayed on a display. . Further, when an object (person) is detected on the left side of the shovel, an alarm is output from a speaker installed on the left wall in the driver's cab, and a through image of a camera that images the left side of the shovel is displayed on a display.

JP 2014-183500 A

  However, the above-mentioned shovel does not associate the object (person) detected by the sensor with the object (person) in the image displayed on the display. Therefore, the operator who looks at the display may not be able to recognize which object (person) in the image the object (person) detected by the sensor is.

  In view of the above, it is desired to provide a work machine periphery monitoring system that allows an operator to easily recognize in which region in a display image a person detected by the work machine is located.

A work machine periphery monitoring system according to an embodiment of the present invention includes a person detection unit that detects a person existing around the work machine, and a control unit that controls an output device mounted on the work machine. The control unit displays an output image generated using an image captured by an imaging device attached to the work machine on a display, and when the human detection unit detects a person present around the work machine, An alarm is output, and an image portion on the output image corresponding to the person detected by the human detection unit is displayed so as to be distinguishable. Is determined and recognized, the at least two-stage human detection state, the first person detection state that the condition regarding the reliability of the human detection result is satisfied, and the condition regarding the reliability of the human detection result is satisfied, And the working machine A second person detection state in which a condition relating to a distance from the first person detection state is satisfied; and the control unit includes: an image portion on the output image corresponding to a person who has brought the first person detection state; An image portion on the output image corresponding to the person who brought the state is displayed so as to be distinguishable .

  According to the above-described means, there is provided a work machine periphery monitoring system that allows an operator to easily recognize an area in a display image in which a person detected by the work machine is present.

1 is a side view of a shovel on which a periphery monitoring system according to an embodiment of the present invention is mounted. FIG. 2 is a functional block diagram illustrating a configuration example of a peripheral monitoring system. It is an example of the captured image of the rear camera. It is the schematic which shows an example of the geometric relationship used when cutting out the image for a classification process from a captured image. It is a top view of the real space behind the shovel. FIG. 7 is a diagram illustrating a flow of a process of generating a normalized image from a captured image. FIG. 3 is a diagram illustrating a relationship between a captured image, an image area to be identified, and a normalized image. FIG. 5 is a diagram illustrating a relationship between an image area to be identified and an inappropriate area to be identified; FIG. 4 is a diagram illustrating an example of a normalized image. It is the schematic which shows another example of the geometric relationship used when cutting out the image for a classification process from a captured image. FIG. 4 is a diagram illustrating an example of a feature image in a captured image. It is a flowchart which shows the flow of an example of an image extraction process. It is a flowchart which shows the flow of an example of a periphery monitoring process. It is a flowchart which shows the flow of an example of a restriction release process. FIG. 4 is a diagram illustrating an example of an output image. 6 is a correspondence table showing a correspondence between a detection state and a display color of a frame and an area. It is an example of a viewpoint conversion image as an output image. It is an example of the output image containing the viewpoint conversion image.

  FIG. 1 is a side view of a shovel as a construction machine on which a periphery monitoring system 100 according to an embodiment of the present invention is mounted. An upper swing body 3 is mounted on a lower traveling body 1 of the shovel via a swing mechanism 2. A boom 4 is attached to the upper swing body 3. An arm 5 is attached to the tip of the boom 4, and a bucket 6 is attached to the tip of the arm 5. The boom 4, the arm 5, and the bucket 6 constitute a digging attachment, and are hydraulically driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, respectively. A cabin 10 is provided on the upper swing body 3 and a power source such as an engine is mounted. An imaging device 40 is attached to the upper part of the upper swing body 3. Specifically, a rear camera 40B, a left camera 40L, and a right camera 40R are attached to the upper rear end, upper left end, and upper right end of the upper swing body 3. In the cabin 10, a controller 30 and an output device 50 are installed.

  FIG. 2 is a functional block diagram illustrating a configuration example of the periphery monitoring system 100. The peripheral monitoring system 100 mainly includes the controller 30, the imaging device 40, and the output device 50.

  The controller 30 is a control device that performs drive control of the shovel. In this embodiment, the controller 30 is configured by an arithmetic processing unit including a CPU and an internal memory, and implements various functions by causing the CPU to execute a drive control program stored in the internal memory.

  Further, the controller 30 determines whether a person is present around the shovel based on the output of the various devices, and controls the various devices according to the determination result. Specifically, the controller 30 receives the outputs of the imaging device 40 and the input device 41, and executes a software program corresponding to each of the extraction unit 31, the identification unit 32, the tracking unit 33, and the control unit 35. Then, in accordance with the execution result, a control command is output to the machine control device 51 to execute the drive control of the shovel, or the output device 50 outputs various information. Note that the controller 30 may be a control device dedicated to image processing.

  The imaging device 40 is a device that captures an image around the shovel, and outputs the captured image to the controller 30. In the present embodiment, the imaging device 40 is a wide camera employing an imaging device such as a CCD, and is attached so that the optical axis faces obliquely downward at the upper part of the upper swing body 3.

  The input device 41 is a device that receives an input from an operator. In the present embodiment, the input device 41 includes an operating device (an operating lever, an operating pedal, etc.), a gate lock lever, a button installed at the tip of the operating device, a button attached to a vehicle-mounted display, a touch panel, and the like.

  The output device 50 is a device that outputs various types of information, and includes, for example, an in-vehicle display that displays various types of image information, an in-vehicle speaker that outputs various types of audio information as sounds, an alarm buzzer, and an alarm lamp. In the present embodiment, the output device 50 outputs various information in response to a control command from the controller 30.

  The machine control device 51 is a device that controls the movement of the shovel, and includes, for example, a control valve that controls the flow of hydraulic oil in a hydraulic system, a gate lock valve, an engine control device, and the like.

  The extraction unit 31 is a functional element that extracts an image to be identified from an image captured by the imaging device 40. Specifically, the extraction unit 31 extracts a simple feature based on a local brightness gradient or edge, a geometric feature by Hough transform or the like, a feature related to an area or an aspect ratio of a region divided based on brightness, and the like. The image to be identified is extracted by image processing with a relatively small amount of calculation (hereinafter, referred to as “pre-stage image recognition processing”). The identification processing target image is an image portion (a part of a captured image) to be subjected to subsequent image processing, and includes a human candidate image. The human candidate image is an image portion (a part of a captured image) that is highly likely to be a human image.

  The identification unit 32 is a functional element that identifies whether the person candidate image included in the identification processing target image extracted by the extraction unit 31 is a human image. Specifically, the identification unit 32 has a relatively large amount of calculation such as an image recognition process using an image feature description described by a HOG (Histograms of Oriented Gradients) feature and a classifier generated by machine learning. Image processing (hereinafter, referred to as “post-stage image recognition processing”) identifies whether the human candidate image is a human image. The rate at which the identification unit 32 identifies a human candidate image as a human image increases as the extraction unit 31 extracts the identification processing target image with higher accuracy. Note that the identification unit 32 identifies and extracts all the human candidate images as human images in a case where a captured image of a desired quality cannot be obtained in an environment unsuitable for imaging at night, in bad weather, or the like. All of the person candidate images in the identification processing target image extracted by the unit 31 may be identified as a person. This is to prevent a person from being missed.

  Next, with reference to FIG. 3, how a human image looks in a captured image behind the shovel captured by the rear camera 40B will be described. Note that the two captured images in FIG. 3 are examples of captured images of the rear camera 40B. The dotted circle in FIG. 3 indicates the presence of a human image, and is not displayed in an actual captured image.

  The rear camera 40B is a wide camera, and is attached at a height at which a person can be looked down obliquely from above. Therefore, the appearance of the human image in the captured image greatly differs depending on the direction in which the human is viewed from the rear camera 40B. For example, a human image in a captured image is displayed to be tilted closer to the left and right ends of the captured image. This is due to image tilt caused by the wide-angle lens of the wide-angle camera. In addition, the head is displayed larger as it is closer to the rear camera 40B. In addition, the legs are invisible in the blind spot of the body of the shovel. These are caused by the installation position of the rear camera 40B. Therefore, it is difficult to identify a human image included in a captured image by image processing without performing any processing on the captured image.

  Therefore, the periphery monitoring system 100 according to the embodiment of the present invention promotes identification of a human image included in the identification processing target image by normalizing the identification processing target image. “Normalization” means that an image to be identified is converted into an image having a predetermined size and a predetermined shape. In the present embodiment, the identification processing target image that can take various shapes in the captured image is converted into a rectangular image of a predetermined size by projective transformation. As the projective transformation, for example, a projective transformation matrix of eight variables is used.

  Here, an example of a process in which the surroundings monitoring system 100 normalizes the image to be identified (hereinafter, referred to as “normalization process”) will be described with reference to FIGS. FIG. 4 is a schematic diagram illustrating an example of a geometric relationship used when the extraction unit 31 cuts out an image to be identified from a captured image.

  The box BX in FIG. 4 is a virtual three-dimensional object in the real space, and in this embodiment, is a virtual rectangular solid defined by eight vertices A to H. The point Pr is a reference point set in advance for referring to the image to be identified. In the present embodiment, the reference point Pr is a point preset as an assumed standing position of the person, and is located at the center of a square ABCD defined by four vertices A to D. The size of the box BX is set based on the direction of the person, the stride, the height, and the like. In this embodiment, the square ABCD and the square EFGH are square, and the length of one side is, for example, 800 mm. The height of the rectangular parallelepiped is, for example, 1800 mm. That is, the box BX is a rectangular parallelepiped having a width of 800 mm × a depth of 800 mm × a height of 1800 mm.

  A quadrangular ABGH defined by the four vertices A, B, G, and H forms a virtual plane region TR corresponding to the region of the identification processing target image in the captured image. Further, the square ABGH as the virtual plane area TR is inclined with respect to the virtual ground which is a horizontal plane.

  In this embodiment, a box BX as a virtual rectangular parallelepiped is employed to determine the relationship between the reference point Pr and the virtual plane area TR. However, if the virtual plane area TR facing the direction of the imaging device 40 and tilting with respect to the virtual ground can be determined in association with an arbitrary reference point Pr, other relations using other virtual three-dimensional objects, etc. May be adopted, and other mathematical relationships such as a function and a conversion table may be adopted.

  FIG. 5 is a top view of the real space behind the shovel, and shows the positional relationship between the rear camera 40B and the virtual plane regions TR1 and TR2 when the virtual plane regions TR1 and TR2 are referred to using the reference points Pr1 and Pr2. Show. In this embodiment, the reference point Pr can be arranged at each grid point of the virtual grid on the virtual ground. However, the reference points Pr may be irregularly arranged on the virtual ground, or may be arranged at equal intervals on a line segment extending radially from the projection point of the rear camera 40B on the virtual ground. For example, each line segment extends radially in increments of one degree, and the reference points Pr are arranged at 100 mm intervals on each line segment.

  As shown in FIGS. 4 and 5, the first surface of the box BX defined by the square ABFE (see FIG. 4) is positive for the rear camera 40B when the virtual plane region TR1 is referred to using the reference point Pr1. Are arranged to face each other. That is, a line segment connecting the rear camera 40B and the reference point Pr1 is orthogonal to the first surface of the box BX arranged in relation to the reference point Pr1 in a top view. Similarly, the first surface of the box BX is arranged to face the rear camera 40B even when the virtual plane region TR2 is referred to using the reference point Pr2. That is, a line segment connecting the rear camera 40B and the reference point Pr2 is orthogonal to the first surface of the box BX arranged in relation to the reference point Pr2 in a top view. This relationship holds even when the reference point Pr is located on any grid point. That is, the box BX is arranged such that the first surface always faces the rear camera 40B.

  FIG. 6 is a diagram illustrating a flow of a process of generating a normalized image from a captured image. Specifically, FIG. 6A is an example of a captured image of the rear camera 40B, and shows a box BX arranged in relation to the reference point Pr in the real space. FIG. 6B is a diagram in which a region of the identification processing target image in the captured image (hereinafter, referred to as “identification processing target image region TRg”) is cut out, and is shown in the captured image of FIG. Corresponding to the virtual plane region TR. FIG. 6C shows a normalized image TRgt obtained by normalizing the classification processing target image having the classification processing target image area TRg.

  As shown in FIG. 6A, a box BX arranged in relation to the reference point Pr1 in the real space determines the position of the virtual plane area TR in the real space, and captures an image corresponding to the virtual plane area TR. An identification processing target image region TRg on the image is determined.

  As described above, when the position of the reference point Pr in the real space is determined, the position of the virtual plane region TR in the real space is uniquely determined, and the identification processing target image region TRg in the captured image is also uniquely determined. Then, the extraction unit 31 can generate a normalized image TRgt of a predetermined size by normalizing the identification processing target image having the identification processing target image region TRg. In this embodiment, the size of the normalized image TRgt is, for example, 64 pixels vertically × 32 pixels horizontally.

  FIG. 7 is a diagram illustrating a relationship among a captured image, an image area to be identified, and a normalized image. Specifically, FIG. 7 (A1) shows an identification processing target image area TRg3 in the captured image, and FIG. 7 (A2) shows a normalized image TRgt3 of the identification processing target image having the identification processing target image area TRg3. . FIG. 7 (B1) shows an identification processing target image region TRg4 in the captured image, and FIG. 7 (B2) shows a normalized image TRgt4 of the identification processing target image having the identification processing target image region TRg4. Similarly, FIG. 7 (C1) shows an identification processing target image region TRg5 in the captured image, and FIG. 7 (C2) shows a normalized image TRgt5 of the identification processing target image having the identification processing target image region TRg5.

  As shown in FIG. 7, the identification processing target image region TRg5 in the captured image is larger than the identification processing target image region TRg4 in the captured image. This is because the distance between the virtual plane area corresponding to the identification processing target image area TRg5 and the rear camera 40B is smaller than the distance between the virtual plane area corresponding to the identification processing target image area TRg4 and the rear camera 40B. Similarly, the identification processing target image region TRg4 in the captured image is larger than the identification processing target image region TRg3 in the captured image. This is because the distance between the virtual plane area corresponding to the identification processing target image area TRg4 and the rear camera 40B is smaller than the distance between the virtual plane area corresponding to the identification processing target image area TRg3 and the rear camera 40B. That is, the image processing target image area in the captured image is smaller as the distance between the corresponding virtual plane area and the rear camera 40B is larger. On the other hand, the normalized images TRgt3, TRgt4, TRgt5 are all rectangular images of the same size.

  As described above, the extraction unit 31 can normalize an identification processing target image that can take various shapes and sizes in a captured image into a rectangular image of a predetermined size, and can normalize a human candidate image including a human image. Specifically, the extraction unit 31 arranges an image portion (hereinafter, referred to as a “head image portion”) estimated to be the head of the human candidate image in a predetermined area of the normalized image. Further, an image portion (hereinafter, referred to as a “body portion image portion”) estimated to be the body portion of the human candidate image is arranged in another predetermined region of the normalized image, and further another predetermined portion of the normalized image is provided. An image portion (hereinafter, referred to as a “leg image portion”) that is presumed to be a leg of the human candidate image is arranged in the region. In addition, the extraction unit 31 can acquire the normalized image in a state where the inclination (image collapse) of the human candidate image with respect to the shape of the normalized image is suppressed.

  Next, referring to FIG. 8, normalization processing in a case where the image processing target image area includes an image area that is unsuitable for identification that adversely affects the identification of a human image (hereinafter, referred to as an “inappropriate identification processing area”). Will be described. The unsuitable region for identification processing is a known region where a human image cannot exist. For example, a region where the body of the shovel is reflected (hereinafter, referred to as a “body reflection region”), a region which is out of the captured image ( Hereinafter, it will be referred to as “extended area”). FIG. 8 is a diagram illustrating a relationship between the identification processing target image area and the identification processing inappropriate area, and corresponds to FIGS. 7 (C1) and 7 (C2). In FIG. 8, the lower right hatched area corresponds to the protrusion area R1, and the lower left hatched area corresponds to the vehicle body reflection area R2.

  In the present embodiment, when the identification processing target image region TRg5 includes a part of the protruding region R1 and the body reflection region R2, the extraction unit 31 performs the masking process on the identification processing inappropriate region and then performs the identification processing target image. A normalized image TRgt5 of the identification processing target image having the region TRg5 is generated. Note that, after generating the normalized image TRgt5, the extraction unit 31 may perform a mask process on a portion of the normalized image TRgt5 corresponding to the inappropriate region for the identification process.

  The right diagram of FIG. 8 shows the normalized image TRgt5. In the right diagram of FIG. 8, the hatched hatched area falling to the right represents a mask area M1 corresponding to the protruding area R1, and the hatched hatched area falling left is a mask area M2 corresponding to a part of the vehicle body reflection area R2. Represents

  In this manner, the extraction unit 31 performs the masking process on the image of the unsuitable region for the identification process, thereby preventing the image of the unsuitable region for the identification process from affecting the identification process performed by the identification unit 32. By this mask processing, the identification unit 32 can identify whether the image is a human image by using an image of an area other than the mask area in the normalized image without being affected by the image of the inappropriate area for the identification processing. Note that the extraction unit 31 may use any known method other than the masking process so that the image of the area inappropriate for the identification processing does not affect the identification processing by the identification unit 32.

  Next, the features of the normalized image generated by the extraction unit 31 will be described with reference to FIG. FIG. 9 is a diagram illustrating an example of a normalized image. In addition, the 14 normalized images illustrated in FIG. 9 include images of human candidates located closer to the rear camera 40B as the normalized image is closer to the left end of the drawing, and the normalized images closer to the right end of the drawing are closer to the right end of the drawing. It includes an image of a candidate person located far from the rear camera 40B.

  As shown in FIG. 9, the extraction unit 31 performs any normalization regardless of the rear horizontal distance (horizontal distance in the Y-axis direction shown in FIG. 5) between the virtual plane area TR in the real space and the rear camera 40B. In the image, the head image portion, the body image portion, the leg image portion, and the like can be arranged at almost the same ratio. Therefore, the extraction unit 31 can reduce the calculation load when the identification unit 32 performs the identification processing, and can improve the reliability of the identification result. Note that the above-described rear horizontal distance is an example of information regarding the positional relationship between the virtual plane area TR and the rear camera 40B in the real space, and the extraction unit 31 adds the information to the extracted identification processing target image. . In addition, the information on the above-described positional relationship includes a top view angle with respect to the optical axis of the rear camera 40B of a line connecting the reference point Pr corresponding to the virtual plane area TR and the rear camera 40B.

  With the above configuration, the periphery monitoring system 100 generates the normalized image TRgt from the identification processing target image region TRg corresponding to the virtual plane region TR facing the direction of the imaging device 40 and inclined with respect to the virtual ground which is a horizontal plane. . Therefore, normalization can be realized in consideration of how the person looks in the height direction and the depth direction. As a result, even when a captured image of the imaging device 40 attached to the construction machine is used so as to capture a person obliquely from above, a person existing around the construction machine can be detected more reliably. In particular, even when a person approaches the imaging device 40, a normalized image can be generated from the identification processing target image that occupies a sufficiently large area on the captured image, so that the person can be reliably detected.

  Further, the periphery monitoring system 100 defines a virtual plane area TR as a rectangular area formed by four vertices A, B, G, and H of the box BX, which is a virtual cuboid in the real space. Therefore, the reference point Pr in the real space and the virtual plane region TR can be geometrically associated with each other, and further, the virtual plane region TR in the real space and the identification processing target image region TRg in the captured image are geometrically formed. Can be assigned.

  In addition, the extraction unit 31 performs a mask process on the image of the inappropriate identification processing area included in the identification processing target image area TRg. Therefore, the identification unit 32 can identify whether the image is a human image by using an image of an area other than the mask area in the normalized image without being affected by an image of the identification processing inappropriate area including the vehicle body reflection area R2.

  Further, the extraction unit 31 can extract an image to be identified for each reference point Pr. Further, each of the identification processing target image regions TRg is associated with one of the reference points Pr preset as the assumed standing position of the person via the corresponding virtual plane region TR. Therefore, the surroundings monitoring system 100 can extract the identification processing target image having a high possibility of including the person candidate image by extracting the reference point Pr having a high possibility that a person exists by an arbitrary method. In this case, it is possible to prevent the identification processing by the image processing with a relatively large amount of calculation from being performed on the identification processing target image having a low possibility of including the human candidate image, and to realize a high-speed human detection processing. .

  Next, with reference to FIGS. 10 and 11, an example of a process in which the extraction unit 31 extracts an image to be identified that is likely to include a human candidate image will be described. FIG. 10 is a schematic diagram illustrating an example of the geometric relationship used when the extraction unit 31 cuts out the image to be identified from the captured image, and corresponds to FIG. FIG. 11 is a diagram illustrating an example of a characteristic image in a captured image. The feature image is an image representing a characteristic portion of a person, and is preferably an image representing a portion of the real space where the height from the ground is hard to change. Therefore, the characteristic images include, for example, an image of a helmet, an image of a shoulder, an image of a head, an image of a reflector or a marker attached to a person, and the like.

  In particular, the helmet has a characteristic that its shape is approximately spherical, and when a projected image is projected on a captured image, it is always almost circular regardless of the imaging direction. In addition, the helmet has a hard surface and gloss or semi-gloss, and when a projected image is projected on a captured image, a local high-luminance region and a radial luminance gradient centered on the region are generated. It has the feature of being easy. Therefore, the image of the helmet is particularly suitable as the feature image. The feature that the projected image is close to a circle and the feature that a radial brightness gradient centering on a local high-brightness region is easily generated are used for image processing for finding an image of a helmet from a captured image. You may. The image processing for finding the helmet image from the captured image includes, for example, luminance smoothing processing, Gaussian smoothing processing, luminance maximum point search processing, luminance minimum point search processing, and the like.

  In the present embodiment, the extraction unit 31 finds a helmet image (strictly, an image that can be presumed to be a helmet) in the captured image by the preceding-stage image recognition processing. This is because people working around the shovel are considered to be wearing helmets. Then, the extraction unit 31 derives the most relevant reference point Pr from the position of the found helmet image. Then, the extraction unit 31 extracts an image to be identified that corresponds to the reference point Pr.

  Specifically, the extraction unit 31 derives a highly relevant reference point Pr from the position of the helmet image in the captured image using the geometric relationship shown in FIG. The geometric relationship in FIG. 10 is different from the geometric relationship in FIG. 4 in that the virtual head position HP in the real space is determined, but is common in other points.

  The virtual head position HP indicates the position of the head of a person that is assumed to exist on the reference point Pr, and is located directly above the reference point Pr. In this embodiment, it is arranged at a height of 1700 mm above the reference point Pr. Therefore, if the virtual head position HP in the real space is determined, the position of the reference point Pr in the real space is uniquely determined, and the position of the virtual plane region TR in the real space is also uniquely determined. Further, the identification processing target image region TRg in the captured image is also uniquely determined. Then, the extraction unit 31 can generate a normalized image TRgt of a predetermined size by normalizing the identification processing target image having the identification processing target image region TRg.

  Conversely, if the position of the reference point Pr in the real space is determined, the virtual head position HP in the real space is uniquely determined, and the head image position AP on the captured image corresponding to the virtual head position HP in the real space is also unique. Is decided. Therefore, the head image position AP can be set in advance in association with each of the preset reference points Pr. Note that the head image position AP may be derived in real time from the reference point Pr.

  Therefore, the extraction unit 31 searches for a helmet image in the captured image of the rear camera 40B by the preceding-stage image recognition processing. The upper part of FIG. 11 shows a state in which the extraction unit 31 has found a helmet image HRg. Then, when finding the helmet image HRg, the extraction unit 31 determines the representative position RP. Note that the representative position RP is a position derived from the size, shape, and the like of the helmet image HRg. In the present embodiment, the representative position RP is the position of the center pixel of the helmet image area including the helmet image HRg. The lower part of FIG. 11 is an enlarged view of the helmet image area which is a rectangular image area separated by a white line in the upper part of FIG. 11, and shows that the position of the center pixel of the helmet image area is the representative position RP.

  After that, the extraction unit 31 derives a head image position AP that is closest to the representative position RP by using, for example, a nearest neighbor search algorithm. In the lower part of FIG. 11, six head image positions AP1 to AP6 are set in advance near the representative position RP, and among them, the head image position AP5 is the head image position AP closest to the representative position RP. Indicates that

  Then, using the geometrical relationship shown in FIG. 10, the extraction unit 31 traces the virtual head position HP, the reference point Pr, and the virtual plane region TR from the derived nearest head image position AP, and The target image area TRg to be identified is extracted. After that, the extraction unit 31 normalizes the identification processing target image having the extracted identification processing target image region TRg to generate a normalized image TRgt.

  As described above, the extraction unit 31 determines the representative position RP of the helmet image HRg, which is the position of the human characteristic image in the captured image, and one of the preset head image positions AP (head image position AP5). Are associated with each other to extract an image to be identified.

  Note that, instead of using the geometrical relationship shown in FIG. 10, the extraction unit 31 refers to a reference that directly associates the head image position AP with the reference point Pr, the virtual plane area TR, or the identification processing target image area TRg. The identification processing target image corresponding to the head image position AP may be extracted using the table.

  Further, the extraction unit 31 may derive the reference point Pr from the representative position RP using a known algorithm other than the nearest neighbor search algorithm such as the hill-climbing method and the Mean-shift method. For example, when the hill-climbing method is used, the extraction unit 31 derives a plurality of head image positions AP near the representative position RP, and refers to the representative position RP and reference points Pr corresponding to each of the plurality of head image positions AP. And At this time, the extraction unit 31 weights the reference point Pr so that the closer the representative position RP and the head image position AP are, the larger the weight is. Then, the distribution of the weights of the plurality of reference points Pr is hill-climbed, and the identification processing target image region TRg is extracted from the reference point Pr having the weight closest to the maximum point of the weight.

  Next, an example of a process in which the extraction unit 31 of the controller 30 extracts an image to be identified (hereinafter, referred to as an “image extraction process”) will be described with reference to FIG. FIG. 12 is a flowchart illustrating a flow of an example of the image extraction process.

  First, the extraction unit 31 searches for a helmet image in a captured image (step ST1). In this embodiment, the extraction unit 31 finds a helmet image by raster-scanning the captured image of the rear camera 40B by the preceding-stage image recognition processing.

  When the helmet image HRg is found in the captured image (YES in step ST1), the extraction unit 31 acquires the representative position RP of the helmet image HRg (step ST2).

  After that, the extraction unit 31 acquires the head image position AP that is closest to the acquired representative position RP (step ST3).

  After that, the extraction unit 31 extracts an identification processing target image corresponding to the acquired head image position AP (step ST4). In the present embodiment, the extraction unit 31 uses the geometric relationship shown in FIG. 10 to determine the head image position AP in the captured image, the virtual head position HP in the real space, and the assumed standing position of the person in the real space. The identification processing target image is extracted by following the correspondence between the reference point Pr and the virtual plane region TR in the real space.

  When the helmet image HRg is not found in the captured image (NO in step ST1), the extraction unit 31 shifts the processing to step ST5 without extracting the identification processing target image.

  Thereafter, the extraction unit 31 determines whether a helmet image has been searched for over the entire captured image (step ST5).

  If it is determined that the entire captured image has not been searched yet (NO in step ST5), the extraction unit 31 performs the processing of steps ST1 to ST4 on another region of the captured image.

  On the other hand, when it is determined that the search for the helmet image over the entire captured image has been completed (YES in step ST5), the extraction unit 31 ends the current image extraction processing.

  As described above, the extraction unit 31 first finds the helmet image HRg, and from the representative position RP of the found helmet image HRg, the head image position AP, the virtual head position HP, the reference point (estimated standing position) Pr, and the virtual The identification processing target image area TRg is specified via the plane area TR. Then, by extracting and normalizing the identification processing target image having the specified identification processing target image region TRg, a normalized image TRgt of a predetermined size can be generated.

  With the above configuration, the extraction unit 31 of the periphery monitoring system 100 finds a helmet image as a characteristic image in a captured image, and associates a representative position RP of the helmet image with one of the head image positions AP as a predetermined image position. Thus, an image to be identified is extracted. Therefore, it is possible to narrow down the image portion to be subjected to the subsequent image recognition processing with a simple system configuration.

  Note that the extraction unit 31 first finds the helmet image HRg from the captured image, derives one of the head image positions AP corresponding to the representative position RP of the helmet image HRg, and extracts the one of the head image positions AP. The corresponding identification processing target image may be extracted. Alternatively, the extraction unit 31 first obtains one of the head image positions AP and places the helmet image in a helmet image area that is a predetermined area including the position of the feature image corresponding to the one of the head image positions AP. May exist, the identification processing target image corresponding to one of the head image positions AP may be extracted.

  In addition, the extraction unit 31 may extract the identification processing target image from the representative position RP of the helmet image in the captured image using a predetermined geometric relationship as illustrated in FIG. In this case, the predetermined geometric relationship is such that the identification processing target image region TRg in the captured image, the virtual plane region TR in the real space corresponding to the identification processing target image region TRg, and the real space corresponding to the virtual plane region TR A reference point Pr (an assumed standing position of a person), a virtual head position HP corresponding to the reference point Pr (a virtual feature position which is a position in a real space of a characteristic portion of the person corresponding to the assumed standing position of the person), and Represents a geometric relationship with a head image position AP (a predetermined image position in the captured image corresponding to the virtual feature position) in the captured image corresponding to the virtual head position HP.

  Here, with reference to FIG. 2 again, the description of the other functional elements of the controller 30 will be continued.

  The tracking unit 33 is a functional element that tracks the identification result output by the identification unit 32 every predetermined time and outputs a final human detection result. In this embodiment, the tracking unit 33 determines that the corresponding person candidate image is a human image when the identification results regarding the same person for the predetermined number of consecutive times satisfy the predetermined condition. That is, it is determined that a person exists at the corresponding three-dimensional position (real position). Whether they are the same person is determined based on the actual position. Specifically, the tracking unit 33 determines that the person within a predetermined time period based on the actual position (reference point PrI) of the person appearing in the image identified as the human image in the first identification processing by the identification unit 32. Deriving reachable range. The reachable range is set based on the maximum turning speed of the shovel, the maximum running speed of the shovel, the maximum moving speed of a person, and the like. If the actual position (reference point PrII) of the person appearing in the image identified as the human image in the second identification processing is within the range, it is determined that the person is the same person. The same applies to the third and subsequent identification processes. The tracking unit 33 determines that a person is present at the corresponding three-dimensional position when, for example, four times out of six consecutive identification results identify the same person image. Even if the image is identified as a human image in the first identification processing, if the same person image is not identified in the subsequent three consecutive identification processing, the corresponding three-dimensional image is obtained. It is determined that no person exists at the position.

  As described above, the combination of the extraction unit 31, the identification unit 32, and the tracking unit 33 configures a person detection unit 34 that detects whether or not a person is present around the shovel based on the image captured by the imaging device 40. .

  With this configuration, the human detection unit 34 can provide a false report (determining that a person exists despite the absence of a person), an unreporting (determining that no person exists despite the presence of a person), and the like. Can be suppressed.

  In addition, the human detection unit 34 can determine whether the person is approaching the shovel or moving away from the shovel based on a change in the actual position of the person appearing in the image identified as the human image. Then, the human detection unit 34 may output a control command to the control unit 35 to output an alarm when the distance of the real position of the person from the shovel is smaller than a predetermined value. In this case, the human detection unit 34 may adjust the predetermined value according to operation information of the shovel (for example, turning speed, turning direction, running speed, running direction, and the like).

  Further, the human detection unit 34 may determine and recognize at least two stages of the human detection state and the human non-detection state. For example, a state where at least one of the condition related to the distance and the condition related to the reliability is satisfied is determined as a first person detection state (alert state), and a state where both are satisfied is a second person detection state (alarm state). State). The condition regarding the distance includes, for example, that the distance from the shovel of the actual position of the person appearing in the image identified as the human image is less than a predetermined value. The condition relating to reliability includes, for example, that the same person image is identified by four times out of six consecutive identification results. In the first person detection state (alert state), the first alarm is output as a preliminary alarm with low accuracy but quick response. The first alarm is, for example, a low volume beep sound, and is automatically stopped when neither of the two conditions is satisfied. In the second person detection state (alarm state), the second alarm is output as a formal alarm having a high accuracy but a slow response. The second alarm is, for example, a loud melody sound. Even if at least one of the conditions is not satisfied, the second alarm is not automatically stopped, and the stop requires an operation of the operator.

  The control unit 35 is a functional element that controls various devices. In the present embodiment, the control unit 35 controls various devices according to the input of the operator via the input device 41. For example, a display image displayed on a screen of an in-vehicle display is switched according to an image switching command input through a touch panel. The display image includes a through image of the rear camera 40B, a through image of the right camera 40R, a through image of the left camera 40L, a viewpoint conversion image, and the like. The viewpoint conversion image is, for example, a bird's-eye view image (image viewed from a virtual viewpoint directly above the shovel) synthesized from images captured by a plurality of cameras.

  The control unit 35 controls various devices in accordance with the final human detection result of the tracking unit 33 included in the human detection unit 34. For example, a control command is output to the machine control device 51 according to the final human detection result of the tracking unit 33 to switch the state of the shovel between the first state and the second state. The first state includes a state in which the restriction on the movement of the shovel is released, a state in which the output of the alarm is stopped, and the like. The second state includes a state in which the movement of the shovel is restricted or stopped, a state in which an alarm is output, and the like. In the present embodiment, the control unit 35 outputs a control command to the machine control device 51 when determining that a person exists within a predetermined range around the shovel based on the final human detection result of the tracking unit 33. The state of the shovel is switched from the first state to the second state. For example, the movement of the shovel is stopped. In this case, the operation by the operator is invalidated. The invalidation of the operation by the operator is realized by, for example, setting the operation device to a non-responsive state. Specifically, a control command is output to the gate lock valve to disconnect the operating device from the hydraulic system, thereby forcibly creating a no-operation state and stopping the movement of the shovel. Alternatively, a control command may be output to the engine control device to stop the engine. Alternatively, the movement of the hydraulic actuator may be limited by outputting a control command to a control valve that controls the flow rate of hydraulic oil flowing into the hydraulic actuator to change the opening area of the control valve, the opening area change speed, and the like. In this case, the maximum turning speed, the maximum traveling speed, and the like are reduced. Further, the movement of the hydraulic actuator may be stopped by closing the control valve.

  Further, the control unit 35 returns the state of the shovel to the first state when a predetermined release condition is satisfied after the state of the shovel is changed to the second state. That is, when a predetermined release condition is satisfied after restricting or stopping the movement of the shovel, the restriction or stop is released. The predetermined release condition includes, for example, “determining that no person exists within a predetermined range around the shovel” (hereinafter, referred to as “first release condition”). Further, the predetermined release condition additionally includes, for example, “a state in which the shovel does not start moving” (hereinafter, referred to as “second release condition”). Further, the predetermined release condition may include “the operator has confirmed that there is no person around the shovel” (hereinafter, referred to as “third release condition”). In the present embodiment, whether the movement of the shovel is restricted or stopped, and whether each of the first release condition, the second release condition, and the third release condition are satisfied is managed using a flag. You.

  The first cancellation condition is, for example, “the control unit 35 determines that no person exists within a predetermined range around the shovel based on the final human detection result of the tracking unit 33 included in the human detection unit 34”. Including.

  The second release condition includes, for example, “all operating devices have been in the neutral position for a predetermined time or more”, “gate lock lever has been lowered (the operating device has been disabled)”, This includes "the operator's limbs are separated from all the operation devices", "the predetermined release operation is performed", and the like. The control unit 35 detects that “all the operation devices are in the neutral position” based on, for example, the presence or absence of a command from each operation device, the output value of a sensor that detects the operation amount of each operation device, and the like. . The condition "over a predetermined time" has an effect of preventing the second cancellation condition from being satisfied only by momentarily reaching the neutral position. “The limb of the operator is separated from the operating device” is, for example, based on an image captured by a camera that captures an image of the driver's cab, an output of an electrostatic sensor attached to the operating device (eg, a grip of the operating lever), The control unit 35 detects this. The "predetermined release operation has been performed" means that, for example, a confirmation button (for example, a horn button or the same The control unit 35 detects when a software button (displayed on the screen) is pressed. For example, the control unit 35 may determine that “the state in which the shovel does not start moving is secured” when an operator performs a release operation such as an operation input to a lever, button, panel, or the like in the driver's seat. .

  The third release condition is satisfied, for example, when the confirmation button is pressed while a message such as “Did you confirm that there is no person around the shovel?” Is displayed on the screen of the vehicle-mounted display. Note that the third cancellation condition may be omitted.

  In a case where the third release condition is included in the predetermined release condition, when the first release condition and the second release condition are satisfied, the shovel is in a state where restriction can be released. The state in which the restriction can be released means a state in which the restriction can be released only by the operator confirming that there is no person around the shovel.

  There is no limitation on the order in which the first release condition, the second release condition, and the third release condition are satisfied. For example, even when the condition is satisfied in the order of the third release condition, the second release condition, and the first release condition, the control unit 35 releases the restriction or stop of the movement of the shovel.

  Further, the control unit 35 may release the restriction or stop when a predetermined waiting time has elapsed after the predetermined release condition is satisfied. This is to prevent the operator from panicing due to sudden release.

  In addition, when the movement of the shovel is restricted or stopped, the control unit 35 outputs a control command to an in-vehicle display as the output device 50, and may display a captured image including a human image that caused the control instruction. Good. For example, when a human image is included only in the captured image of the left camera 40L, the through image of the left camera 40L may be displayed alone. Alternatively, when a human image is included in each of the captured image of the left camera 40L and the captured image of the rear camera 40B, the through images of the two cameras may be displayed side by side at the same time, and the captured images of the two cameras may be displayed. One combined image (for example, a viewpoint converted image) may be displayed. Further, an image indicating that the operation is being restricted or stopped, guidance for a release method, or the like may be displayed. Further, an image portion corresponding to a human candidate image identified as a human image may be highlighted. For example, the outline of the identification processing target image region TRg may be displayed in a predetermined color. Further, when a waiting time after the predetermined release condition is satisfied is set, the operator may be notified that the waiting time exists when the predetermined release condition is satisfied. For example, after displaying that the waiting time exists, the countdown of the waiting time may be displayed. If the alarm is output during the waiting time, the volume of the alarm may be gradually reduced as the waiting time elapses.

  Further, when the movement of the shovel is restricted or stopped, the control unit 35 outputs a control command to an in-vehicle speaker as the output device 50, and may output an alarm on the side where the person who caused the problem exists. Good. In this case, the in-vehicle speakers include, for example, a right speaker installed on a right wall in a driver's cab, a left speaker installed on a left wall, and a rear speaker installed on a rear wall. Then, when only the captured image of the left camera 40L includes a human image, the control unit 35 outputs an alarm only from the left speaker. Alternatively, the control unit 35 may localize the sound using a surround system including a plurality of speakers.

  Further, when the human detection unit 34 identifies the human candidate image as a human image, the control unit 35 may output only an alarm without restricting or stopping the movement of the shovel. Also in this case, the control unit 35 determines a state in which at least one of the condition related to the distance and the condition related to the reliability is satisfied as described above as a first person detection state (alert state), and determines a state in which both are satisfied. The second person detection state (alarm state) may be determined. Then, similarly to the case where the movement of the shovel is restricted or stopped, the control unit 35 may stop the alarm in the second person detection state (alarm state) when a predetermined release condition is satisfied. . This is because, unlike the alarm in the first-person detection state (alert state), which can be automatically stopped, the operation of the operator is required to stop the alarm in the second-person detection state (alarm state). .

  Next, an example of a process in which the control unit 35 of the controller 30 monitors the periphery of the shovel (hereinafter, referred to as “periphery monitoring process”) will be described with reference to FIG. FIG. 13 is a flowchart showing a flow of an example of the periphery monitoring process. The controller 30 repeatedly executes the periphery monitoring process at a predetermined control cycle.

  First, the control unit 35 determines whether or not a person exists around the shovel (step ST11). In the present embodiment, the control unit 35 determines whether or not a person is present around the shovel based on the final result of the detection of the person by the tracking unit 33.

  Thereafter, when it is determined that there is a person around the shovel (YES in step ST11), the control unit 35 restricts or stops the movement of the shovel (step ST12). In the present embodiment, for example, when determining that the current person detection state is the second person detection state (alarm state), the control unit 35 determines that there is a person around the shovel and stops the movement of the shovel. .

  At this time, the control unit 35 outputs a control command to an in-vehicle speaker as the output device 50 to output a second alarm. In addition, a control command is output to an in-vehicle display as the output device 50 to display a captured image including a human image that has caused a restriction or a stop.

  When it is determined that there is no person around the shovel (NO in step ST11), the control unit 35 determines whether the movement of the shovel is already restricted or stopped (step S13). In the present embodiment, the control unit 35 refers to the value of the corresponding flag to determine whether the movement of the shovel is already restricted or stopped.

  When it is determined that the movement of the shovel is already restricted or stopped (YES in step ST13), the control unit 35 performs a process for releasing the restriction or stop (hereinafter, referred to as “restriction release process”). Execute (step ST14).

  When it is determined that the movement of the shovel has not yet been restricted or stopped (NO in step ST13), the control unit 35 terminates the current excavator periphery monitoring processing without executing the restriction release processing.

  Next, a process in which the control unit 35 of the controller 30 releases the restriction or stop of the movement of the shovel will be described with reference to FIG. FIG. 14 is a flowchart illustrating a flow of an example of the restriction release processing.

  First, the control unit 35 determines whether the first release condition has been satisfied (step ST21). In the present embodiment, the control unit 35 determines whether or not a person exists within a predetermined range around the shovel. Specifically, it is determined whether or not the current person detection state has escaped from the second person detection state (alarm state). It may be determined whether the first person detection state (alert state) and the second person detection state (alarm state) have been escaped.

  When determining that the first release condition is satisfied (YES in step ST21), control unit 35 determines whether the second release condition is satisfied (step ST22). In the present embodiment, the control unit 35 determines whether or not a state in which the shovel does not start moving is secured. Specifically, it is determined whether or not the gate lock lever is lowered (whether or not the operating device is disabled).

  When it is determined that the second release condition is satisfied (YES in step ST22), control unit 35 determines whether the third release condition is satisfied (step ST23). In the present embodiment, the control unit 35 determines whether or not the operator has confirmed that there is no person around the shovel. Specifically, it is determined whether or not the confirmation button has been pressed while a message such as “Did you confirm that there is no person around the shovel?” Is displayed on the screen of the vehicle-mounted display.

  When it is determined that the third release condition is satisfied (YES in step ST23), the control unit 35 releases the restriction or stop of the movement of the shovel (step ST24).

  At this time, the control unit 35 outputs a control command to the in-vehicle speaker as the output device 50 to stop outputting the second alarm. In addition, a control command is output to an in-vehicle display as the output device 50 to stop displaying a captured image including a human image that has caused a restriction or a stop. For example, the live view image that was displayed before the second alarm was output is displayed again. Further, the control unit 35 may display a message indicating that the restriction or stop of the movement of the shovel has been released.

Note that when the control unit 35 determines that the first release condition is not satisfied (NO in step ST21), and when determines that the second release condition is not satisfied (NO in step ST22),
If it is determined that the third release condition is not satisfied (NO in step ST23), the current restriction release processing is ended without releasing restriction or stop of the movement of the shovel.

  With the above configuration, the controller 30 can limit or stop the movement of the shovel when it is determined that a person is present around the shovel.

  In addition, when the controller 30 determines that no person is present around the shovel after restricting or stopping the movement of the shovel, the controller 30 performs the control only when determining that the state in which the shovel does not start is secured. Restrictions or suspensions can be lifted. Further, the controller 30 can release the restriction or stop only when it is determined that the state in which the shovel does not start moving is secured, and when it is determined that the operator has confirmed that there is no person around the shovel. Therefore, the controller 30 can prevent the shovel from unintentionally starting to move when the restriction or the stop is released.

  Next, an example of an output image displayed on the in-vehicle display during execution of the periphery monitoring process will be described with reference to FIG. FIG. 15 is an example of an output image generated based on the captured image of the rear camera 40B. FIG. 15A shows an example of an output image when no person is within a predetermined range around the shovel, FIG. 15B shows an example of an output image in the first person detection state, and FIG. Shows an example of an output image in the second person detection state.

  Specifically, the output image of FIG. 15 includes a camera image portion G1 and an indicator portion G2. The camera image portion G1 is a portion that displays an image generated based on images captured by one or more cameras. The indicator portion G2 is a portion for displaying the human detection state / non-human detection state of each of a plurality of areas around the shovel. In the camera image portion G1, the line segment L1 superimposed and displayed on the camera image indicates that the distance from the shovel is a predetermined first distance (for example, 5 meters). A line segment L2 superimposed on the camera image indicates that the distance from the shovel is a predetermined second distance (for example, 2.5 meters). In the indicator portion G2, the outer peripheral line L1g of the partial circle drawn around the shovel icon CG1 indicates that the distance from the shovel is a predetermined first distance (for example, 5 meters). Corresponding. A partial rectangular outer line L2g drawn around the shovel icon CG1 indicates that the distance from the shovel is a predetermined second distance (for example, 2.5 meters), and corresponds to the line segment L2 of the camera image portion G1. I do.

  The partial circle is divided into six areas A1 to A6, and the partial rectangle is divided into three areas B1 to B3.

  In the state shown in FIG. 15A, the controller 30 has detected a person present at the right rear of the shovel. However, since the actual position of the person is longer than the first distance, the controller 30 does not highlight the image of the person and does not output the first alarm. However, the controller 30 may highlight the image of the person, such as displaying the outline of the corresponding image processing area TRg as a white frame, or may output a first alarm. Further, a message such as “periphery monitoring process is being performed” may be displayed regardless of whether a person has already been detected. This is to enable the operator to recognize that the peripheral monitoring process is being performed.

  In the first person detection state illustrated in FIG. 15B, the controller 30 detects a person existing within a first distance behind the shovel and within a second distance or more. Therefore, the controller 30 highlights the image of the person and outputs the first alarm. Specifically, in the camera image portion G1, the controller 30 displays the outline of the corresponding identification processing target image region TRg as a yellow frame F1. In the indicator portion G2, the area A4 corresponding to the actual position of the person is displayed in yellow. However, the display of the yellow frame may be omitted. Further, a message indicating that the first person is being detected (alert state) may be displayed.

  In the second person detection state illustrated in FIG. 15C, the controller 30 detects a person existing within a second distance to the right rear of the shovel. Therefore, the controller 30 highlights the image of the person and outputs the second alarm. Specifically, in the camera image portion G1, the controller 30 displays the outline of the corresponding identification processing target image region TRg as a red frame F2. In the indicator part G2, the area B2 corresponding to the actual position of the person is displayed in red. In addition, the controller 30 restricts the movement of the shovel, and blinks the message “Limiting shovel operation” indicating that the second person is being detected (alarm state). However, the display of the message indicating the second person detection state (alarm state) may be omitted.

  In FIGS. 15A to 15C, a camera image portion G1 is displayed on the left side of the screen and an indicator portion G2 is displayed on the right side of the screen, but the camera image portion G1 is displayed on the right side of the screen. Then, the indicator portion G2 may be displayed on the left side of the screen. Further, the camera image portion G1 may be displayed on one of the vertically divided screens, and the indicator portion G2 may be displayed on the other. The display of the indicator portion G2 may be omitted.

  Further, each of the regions A1 to A6 has a spread angle of 45 degrees, and each of the regions B1 to B3 has a spread angle of 90 degrees. This difference in the spread angle is based on the difference in properties between the first person detection state (alert state) and the second person detection state (alarm state). Specifically, the first-person detection state (alert state) is a state in which a preliminary alarm with a low accuracy but a quick response is output, and a relatively wide space range relatively distant from the shovel is monitored. Range. Therefore, when the spread angles of the regions A1 to A6 are increased, the monitoring ranges corresponding to the respective regions are increased together with the display ranges, and it becomes difficult to recognize the actual position of the person who caused the first alarm. This is because the same display result is obtained anywhere in a wide monitoring range. On the other hand, the second person detection state (alarm state) is a state in which a formal alarm with a high accuracy but a slow response is output, and a relatively narrow space area relatively close to the shovel is a monitoring range. I have. Therefore, when the spread angles of the areas B1 to B3 are reduced, the monitoring range corresponding to each area is reduced together with its display range, making it difficult to know in which direction the person who caused the second alarm is located. I will. This is because the display range is small and it is difficult to see. Therefore, desirably, as shown in FIGS. 15A to 15C, the spread angles of the regions A1 to A6 are set to be smaller than the spread angles of the regions B1 to B3.

  FIGS. 15A to 15C illustrate a case where a human image is detected in a captured image of the rear camera 40B while a through image of the rear camera 40B is displayed. However, the above description is similarly applied to a case where a human image is detected in at least one of the left camera 40L and the right camera 40R when a through image of the rear camera 40B is displayed. . In this case, the output image displayed in the camera image portion G1 may be automatically switched from the through image of the rear camera 40B to the through image of another camera or the viewpoint conversion image synthesized from the captured images of a plurality of cameras. Good. For example, when the through image of the rear camera 40B is displayed and the human image is detected in the captured image of the left camera 40L, the controller 30 outputs the output image displayed in the camera image portion G1 to the left camera 40L. May be switched to the through image.

  Next, the relationship between the detection state and the display colors of the frame and the area will be described with reference to FIG. FIG. 16 is a correspondence table showing the correspondence between the detection state and the display colors of the frame and the area.

  When the detection state is neither the first person detection state (alert state) nor the second person detection state (alarm state), the first line of the correspondence table does not display the outline of the image area to be identified and displays an indicator. This means that no area of the portion G2 is colored.

  In the second line, when the detection state is the alert state, the outline of the image processing target image area corresponding to the human image that caused the alert state is displayed as a yellow frame, and the areas A1 to A6 are displayed. Is displayed in yellow.

  In the third row, when the detection state is the alarm state, the outline of the image processing target image area corresponding to the human image that caused the alarm state is displayed as a red frame, and the areas B1 to B3 are displayed. Is displayed in red.

  In the fourth line, when the detection state is the alert state and the alarm state, the outline corresponding to the human image that caused the alert state is displayed in a yellow frame, and the cause of the alarm state is indicated. The outline corresponding to the changed human image is displayed in a red frame. Further, an area corresponding to the person image that caused the alert state among the areas A1 to A6 is displayed in yellow, and an area corresponding to the person image that caused the alarm state among the areas B1 to B3 is displayed in red. It is displayed with.

  With the above configuration, the controller 30 outputs an alarm when it is determined that a person is present around the shovel and highlights the image portion of the person. Therefore, the operator can confirm the person who caused the alarm on the screen. Further, even when a false report occurs, the operator can check on the screen what is the cause of the false report.

  Further, the controller 30 displays a frame image as a human detection marker on the camera image portion G1 only when the first person detection state (alert state) is entered, and changes the color of the area corresponding to the indicator portion G2. Let it. For this reason, even a frame image corresponding to a human candidate image that has been identified as a human image but whose reliability is still low can be displayed, and the display image can be prevented from becoming complicated. In the above-described embodiment, the frame image is displayed as the human detection marker, but another emphasized image such as an inverted display image may be used as the human detection marker.

  Further, the image of the person who caused the first person detection state (alert state) and the image of the person who caused the second person detection state (alarm state) are highlighted so as to be distinguishable. Further, the color of the frame image in the camera image portion G1 is made to correspond to the color of the region in the indicator portion G2. Therefore, the operator can confirm the person who caused the second alarm on the screen. In the above-described embodiment, the controller 30 changes the color of the frame image in the camera image portion G1 and the color of the region in the indicator portion G2 according to the detection state. However, the controller 30 may make the attributes other than the color such as the blinking / lighting state and the transmittance different according to the detection state.

  Next, another example of the output image displayed on the in-vehicle display during the execution of the periphery monitoring process will be described with reference to FIG. FIG. 17 is an example of a viewpoint conversion image as an output image generated based on the captured images of the rear camera 40B, the left camera 40L, and the right camera 40R. FIG. 17 shows an example of an output image when the first person detection state and the second person detection state coexist. The output image of FIG. 17 includes a viewpoint conversion image portion G3 corresponding to the camera image portion G1 of FIG. The part corresponding to the indicator part G2 in FIG. 15 is integrated into the viewpoint conversion image part G3. Specifically, the shovel icon CG1 in FIG. 15 corresponds to the shovel icon CG2 in FIG. 17, and the areas A1 to A6 in FIG. 15 correspond to the areas C1 to C6 in FIG. The area B1 in FIG. 15 corresponds to the combination of the areas C1 and C2 in FIG. 17, the area B2 in FIG. 15 corresponds to the combination of the areas C3 and C4 in FIG. 17, and the area B3 in FIG. This corresponds to the combination of C5 and C6. The line segment L3 superimposed and displayed on the viewpoint conversion image indicates that the distance from the shovel is a predetermined third distance (for example, 2.5 meters).

  In the detection state shown in FIG. 17, the controller 30 detects a person (a person who caused the first person detection state) within a first distance (for example, 5 meters) on the left side of the shovel and beyond the third distance. Detected. In addition, a person existing within a third distance behind the shovel (a person who caused the second person detection state) is detected. Therefore, the controller 30 highlights the images of those persons, outputs the second alarm, and limits the movement of the shovel. Specifically, the controller 30 displays a yellow circle MA1 as a human detection marker at the position of the reference point Pr corresponding to the person who caused the first person detection state, and displays an area corresponding to the position. C2 is displayed in yellow. Further, a red circle MA2 as a person detection marker is displayed at the position of the reference point Pr corresponding to the person who caused the second person detection state, and the combination of the areas C3 and C4 corresponding to the position is displayed in red. To display. Further, the controller 30 may limit the movement of the shovel, and blink a message indicating that the shovel is in the second-person detection state (alarm state), “the shovel operation is being limited”. The display of the yellow circle MA1 may be omitted. This is to make the screen easier to see.

  Next, another example of the output image displayed on the in-vehicle display during the execution of the periphery monitoring process will be described with reference to FIG. FIG. 18 is an example of an output image including a viewpoint conversion image generated based on the captured images of the rear camera 40B, the left camera 40L, and the right camera 40R. FIG. 18 shows an example of an output image when the first person detection state and the second person detection state coexist, as in the case of FIG. The output image of FIG. 18 includes an indicator portion G2 and a viewpoint conversion image portion G3. An outer peripheral line L2g of a partial rectangle drawn around the shovel icon CG1 indicates that the distance from the shovel is a predetermined third distance (for example, 2.5 meters), and corresponds to the line segment L3 of the viewpoint conversion image portion G3. .

  In the detection state illustrated in FIG. 18, the controller 30 detects a person (a person who caused the first person detection state) within a first distance (for example, 5 meters) on the left side of the shovel and beyond the third distance. Detected. In addition, a person existing within a third distance behind the shovel (a person who caused the second person detection state) is detected. Therefore, the controller 30 highlights the images of those persons, outputs a second alarm, and limits the movement of the shovel. Specifically, the controller 30 displays a yellow circle MA1 as a human detection marker at the position of the reference point Pr corresponding to the person who caused the first person detection state, and displays an indicator corresponding to the position. The area A2 of the portion G2 is displayed in yellow. Further, a red circle MA2 as a person detection marker is displayed at the position of the reference point Pr corresponding to the person who caused the second person detection state, and the area B2 of the indicator portion G2 corresponding to that position is displayed in red. To display. In addition, the controller 30 restricts the movement of the shovel, and blinks the message “Limiting shovel operation” indicating that the second person is being detected (alarm state). The display of the yellow circle MA1 may be omitted. This is to make the screen easier to see.

  With the configuration described above, the controller 30 can achieve the same effect as the case where the output image of FIG. 15 is displayed.

  As described above, when it is determined that a person is present around the shovel, the controller 30 restricts or stops the movement of the shovel and displays an image of the person. Then, when it is determined that there is no person around the shovel after restricting or stopping the movement of the shovel, the restriction or stop is performed only when it is determined that the state in which the shovel does not start moving is secured. It is determined that it can be released. Then, when the predetermined waiting time has elapsed, the restriction or the stop is actually released. Therefore, the operation restriction of the shovel executed in response to the detection of a person can be more appropriately released.

  Further, when the movement of the shovel is restricted or stopped, the controller 30 can output an alarm from the side where the person who caused the movement is present to the operator. Therefore, before the operator looks at the screen of the in-vehicle display, the operator can recognize the direction in which the person exists. The operator can visually confirm that the person is present in the direction as recognized by visually recognizing the direction in which the person is present based on the direction in which the alarm was heard, and then looking at the screen of the in-vehicle display. . As described above, the controller 30 notifies the operator of the direction in which the person is present in cooperation with the alarm and the display, so that the operator can recognize the situation around the shovel in a short time.

  Even if it is recognized that a person has been detected by the alarm, if the direction of the person is not known, the operator must first look at the entire screen to find out in which direction the person exists. It is. On the other hand, if the direction of the person is known before looking at the screen, the operator can visually confirm the presence of the person only by looking at a part of the screen (the part corresponding to the direction of the person). is there.

  Although the preferred embodiment of the present invention has been described in detail, the present invention is not limited to the above-described embodiment, and various modifications and substitutions can be made to the above-described embodiment without departing from the scope of the present invention. Can be added.

  For example, in the above-described embodiment, a case is assumed where a person is detected using an image captured by the imaging device 40 mounted on the upper swing body 3 of the shovel, but the present invention is not limited to this configuration. . The present invention can also be applied to a configuration using an image captured by an imaging device attached to a main body of another work machine such as a mobile crane, a fixed crane, a lift mag machine, a forklift, or the like.

  Further, in the above-described embodiment, the blind spot area of the shovel is imaged using three cameras, but the blind spot area of the shovel may be imaged using one, two, or four or more cameras.

  In the above-described embodiment, the human detection is performed using the image captured by the imaging device 40. However, the human detection is performed using the output of an ultrasonic sensor, a laser radar, a pyroelectric sensor, a millimeter-wave radar, and the like. Is also good.

  In the above-described embodiment, the human detection process is individually applied to each of the plurality of captured images. However, the human detection process is applied to one composite image generated from the plurality of captured images. Is also good.

  DESCRIPTION OF SYMBOLS 1 ... Lower traveling body 2 ... Revolving mechanism 3 ... Upper revolving body 4 ... Boom 5 ... Arm 6 ... Bucket 7 ... Boom cylinder 8 ... Arm cylinder 9 ... Bucket cylinder 10 Cabin 30 Controller 31 Extraction unit 32 Identification unit 33 Tracking unit 34 Person detection unit 35 Control unit 40 Imaging device 40B: Rear camera 40L: Left camera 40R: Right camera 41: Input device 50: Output device 51: Machine control device 100: Surrounding monitoring system AP, AP1 AP6 ... head image position BX ... box G1 ... camera image part G2 ... indicator part G3 ... viewpoint conversion image part HD ... head HP ... virtual head part HRg: Helmet image M1, M2: Mask area Pr, Pr1, Pr2, Pr10 to Pr12: Reference point R1: Projection area R2: Body reflection area RP: Representative position TR, TR1, TR2, TR10 to TR12 ... virtual plane area TRg, TRg3, TRg4, TRg5 ... identification processing target image area TRgt, TRgt3, TRgt4, TRgt5 ... normalized image

Claims (5)

A human detection unit that detects a person present around the work machine;
A control unit that controls an output device mounted on the work machine,
The control unit includes:
Displaying an output image generated using a captured image of an imaging device attached to the work machine on a display,
Outputs an alarm when the human detection unit detects a person present around the work machine, and
An image portion on the output image corresponding to the person detected by the human detection unit is displayed so as to be distinguishable ,
The human detection unit determines and recognizes a human non-detection state and a human detection state of at least two stages,
The at least two-stage human detection state includes a first human detection state in which a condition relating to reliability of the human detection result is satisfied, and a condition in which a condition relating to reliability of the human detection result is satisfied, and a distance from the work machine. A second person detection state in which a condition regarding the second person is satisfied,
The controller distinguishes an image portion on the output image corresponding to a person who has brought the first person detection state from an image portion on the output image corresponding to a person who has brought the second person detection state. Display as possible ,
Perimeter monitoring system for work machines. The first person detection state includes a state in which a condition regarding reliability of the human detection result is not satisfied, but a condition regarding a distance from the work machine is satisfied.
The peripheral monitoring system for a work machine according to claim 1. The control unit is configured to output a content of an alarm that is output when the first person detection state is recognized by the human detection unit and an alarm that is output when the second person detection state is recognized by the human detection unit. Different from the contents,
A work machine peripheral monitoring system according to claim 1 or 2. The output image includes an indicator portion indicating a human detection state / non-human detection state of each of a plurality of regions around the work machine,
The control unit displays the area of the first person detection state, the area of the second person detection state, and the area of the non-human detection state in different colors so as to be distinguishable,
Working machine surroundings monitoring system according to any one of claims 1 to 3. 請 Motomeko construction machine having a working machine surroundings monitoring system according to any one of 1 to 4.