JP2017158033A - Periphery monitoring system for work machine - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a periphery monitoring system capable of detecting a person present in the periphery of a shovel.SOLUTION: A periphery monitoring system 100 is provided that detects a person present in the periphery of a shovel using a captured image of an imaging apparatus 40 fitted to the shovel. The system includes: an extraction part 31 that extracts a plurality of predetermined image portions from a captured image as a discerning processing target image; and a discerning part 32 that discerns whether an image included in the discerning processing target image is a person image by an image recognition processing. When the number of discerning processing target images extracted by the extraction part 31 is larger than a predetermined criterion, it is determined that the captured image is not proper for a person detection.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a work machine periphery monitoring system for monitoring the periphery of a work machine.

  An image sensor and a thermopile array that senses heat, overlaps the imaging range and heat detection range, and only the human-like range indicated by the thermopile array output is limited to the face extraction range, which is unnecessary for image identification processing A human body detection device that reduces the amount of computation processing is known (see Patent Document 1).

JP 2006-059015 A

  On the other hand, it is desired to provide a work machine periphery monitoring system that can reliably detect people around the work machine.

  A work machine periphery monitoring system according to an embodiment of the present invention is a work machine periphery monitoring system that detects a person existing around the work machine using a captured image of an imaging device attached to the work machine, An extraction unit that extracts a plurality of predetermined image portions from the captured image as identification processing target images; and an identification unit that identifies whether an image included in the identification processing target image is a human image by image recognition processing. When the number of identification processing target images extracted by the extraction unit is larger than a predetermined reference, it is determined that the captured image is in an unsuitable state for human detection.

  By the above-mentioned means, a work machine periphery monitoring system capable of reliably detecting a person around the work machine is provided.

It is a side view of the shovel in which the periphery monitoring system which concerns on the Example of this invention is mounted. It is a functional block diagram which shows the structural example of a periphery monitoring system. It is an example of the captured image of a rear camera. It is the schematic which shows an example of the geometric relationship used when cut out the identification process target image from a captured image. It is a top view of real space behind an excavator. It is a figure which shows the flow of the process which produces | generates the normalization image from a captured image. It is a figure which shows the relationship between a captured image, an identification process target image area, and a normalized image. It is a figure which shows the relationship between an identification process target image area | region and an identification process inappropriate area. It is a figure which shows the example of a normalized image. It is a figure explaining the relationship between the back horizontal distance between the virtual plane area | region in real space and a back camera, and the magnitude | size of the head image part in a normalized image. It is the schematic which shows another example of the geometric relationship used when cut out the identification process target image from a captured image. It is a figure which shows an example of the characteristic image in a captured image. It is a flowchart which shows the flow of an example of an image extraction process. It is a flowchart which shows the flow of another example of an image extraction process. It is a flowchart which shows the flow of another example of an image extraction process. It is a flowchart which shows the flow of another example of an image extraction process. It is a flowchart which shows the flow of an example of a person detection suitability determination process. It is a frequency distribution map of a helmet degree.

  FIG. 1 is a side view of an excavator (excavator) as a work machine (construction machine) on which a periphery monitoring system 100 according to an embodiment of the present invention is mounted. An upper swing body 3 is mounted on the lower traveling body 1 of the excavator via a swing mechanism 2. A boom 4 is attached to the upper swing body 3. An arm 5 is attached to the tip of the boom 4, and a bucket 6 is attached to the tip of the arm 5. The boom 4, the arm 5, and the bucket 6 constitute an excavation attachment, and are hydraulically driven by the boom cylinder 7, the arm cylinder 8, and the bucket cylinder 9, respectively. Further, the upper swing body 3 is provided with a cabin 10 and is mounted with a power source such as an engine. An imaging device 40 is attached to the upper part of the upper swing body 3. Specifically, the rear camera 40B, the left side camera 40L, and the right side camera 40R are attached to the rear end upper part, the left end upper part, and the right end upper part of the upper swing body 3. A controller 30 and an output device 50 are installed in the cabin 10.

  FIG. 2 is a functional block diagram illustrating a configuration example of the periphery monitoring system 100. The periphery monitoring system 100 mainly includes a controller 30, an imaging device 40, an input device 42, an output device 50, and a machine control device 51. In the present embodiment, the imaging device 40, the input device 42, the output device 50, and the machine control device 51 are connected to the controller 30 via the CAN.

  The controller 30 is a control device that performs drive control of the shovel. In the present embodiment, the controller 30 is configured by an arithmetic processing unit including a CPU and an internal memory, and realizes various functions by causing the CPU to execute a drive control program stored in the internal memory.

  Further, the controller 30 determines whether there is a person around the shovel based on the outputs of the various devices, and controls the various devices according to the determination result. Specifically, the controller 30 receives the outputs of the imaging device 40 and the input device 42 and executes software programs corresponding to the extraction unit 31, the identification unit 32, the tracking unit 33, and the control unit 35, respectively. Then, according to the execution result, a control command is output to the machine control device 51 to execute drive control of the shovel, or various information is output from the output device 50. The controller 30 may be a control device dedicated to image processing.

  The imaging device 40 is a device that captures an image around the excavator, and outputs the captured image to the controller 30. In the present embodiment, the imaging device 40 is a wide camera that employs an imaging element such as a CCD, and is mounted on the upper part of the upper swing body 3 so that the optical axis is directed obliquely downward.

  The input device 42 is a device that receives input from the operator. In the present embodiment, the input device 42 includes an operation device (operation lever, operation pedal, etc.), a gate lock lever, a button installed at the tip of the operation device, a button attached to the in-vehicle display, a touch panel, and the like.

  The output device 50 is a device that outputs various types of information, and includes, for example, an in-vehicle display that displays various types of image information, an in-vehicle speaker that outputs various types of audio information as audio, an alarm buzzer, an alarm lamp, and the like. In the present embodiment, the output device 50 outputs various types of information in response to control commands from the controller 30.

  The machine control device 51 is a device that controls the movement of the excavator, and includes, for example, a control valve that controls the flow of hydraulic oil in the hydraulic system, a gate lock valve, an engine control device, and the like.

  The extraction unit 31 is a functional element that extracts an identification processing target image from a captured image captured by the imaging device 40. Specifically, the extraction unit 31 extracts simple features based on local luminance gradients or edges, geometric features based on Hough transform, features related to the area or aspect ratio of a region divided based on luminance, and the like. The image to be identified is extracted by image processing with a relatively small amount of computation (hereinafter referred to as “previous image recognition processing”). The identification processing target image is an image portion (a part of the captured image) that is a target of subsequent image processing, and includes a human candidate image. The human candidate image is an image portion (part of the captured image) that is considered to be a human image. The captured image may be a color image or a gray scale image. The extraction unit 31 may have a plurality of types of functions for converting a color image into a gray scale. Each person candidate image may be considered to have a large or small difference in the degree of humanity or the level indicating the degree. The degree or a level indicating the degree can also be regarded as an evaluation value. Moreover, the extraction part 31 may be comprised by the multistage which each performs several narrowing down. For example, it may be configured as a first extraction unit at the front stage and a second extraction unit at the rear stage connected in series.

  The identification unit 32 is a functional element that identifies whether the human candidate image included in the identification processing target image extracted by the extraction unit 31 is a human image. Specifically, the identification unit 32 has a relatively large amount of calculation such as an image recognition process using an image feature description represented by HOG (Histograms of Oriented Gradients) feature and a classifier generated by machine learning. Image processing (hereinafter referred to as “rear-stage image recognition processing”) identifies whether the human candidate image is a human image. The rate at which the identification unit 32 identifies the human candidate image as a human image increases as the extraction processing target image is extracted by the extraction unit 31 with higher accuracy. Note that the identification unit 32 identifies and extracts all human candidate images as human images when, for example, a captured image having a desired quality cannot be obtained in an environment unsuitable for imaging at night or in bad weather. All of the human candidate images in the identification processing target image extracted by the unit 31 may be identified as people. This is to prevent human detection omissions.

  Next, with reference to FIG. 3, how the human image appears in the captured image behind the excavator captured by the rear camera 40B will be described. Note that the two captured images in FIG. 3 are examples of captured images of the rear camera 40B. 3 represents the presence of a human image, and is not displayed in an actual captured image.

  The rear camera 40B is a wide camera, and is attached at a height at which a person is looked down obliquely from above. For this reason, how the human image is seen in the captured image varies greatly depending on the direction in which the person is seen from the rear camera 40B. For example, the human image in the captured image is displayed so as to be inclined closer to the left and right ends of the captured image. This is due to image collapse caused by the wide-angle lens of the wide camera. Further, the closer to the rear camera 40B, the larger the head is displayed. In addition, the leg part enters the blind spot of the excavator's car body and disappears. These are caused by the installation position of the rear camera 40B. Therefore, it is difficult to identify a human image included in the captured image by image processing without performing any processing on the captured image.

  Therefore, the periphery monitoring system 100 according to the embodiment of the present invention promotes the identification of the human image included in the identification processing target image by normalizing the identification processing target image. Note that “normalization” means that the identification processing target image is converted into an image having a predetermined size and a predetermined shape. In this embodiment, an identification processing target image that can take various shapes in a captured image is converted into a rectangular image of a predetermined size by projective conversion. As the projective transformation, for example, an 8-variable projective transformation matrix is used.

  Here, an example of processing (hereinafter, referred to as “normalization processing”) in which the periphery monitoring system 100 normalizes the identification processing target image will be described with reference to FIGS. 4 to 6. FIG. 4 is a schematic diagram illustrating an example of a geometric relationship used when the extraction unit 31 cuts out an identification processing target image from a captured image.

  A box BX in FIG. 4 is a virtual three-dimensional object in real space, and is a virtual rectangular parallelepiped defined by eight vertices A to H in this embodiment. The point Pr is a reference point that is set in advance to refer to the identification processing target image. In the present embodiment, the reference point Pr is a point set in advance as an assumed standing position of a person, and is located at the center of a quadrilateral ABCD defined by four vertices A to D. The size of the box BX is set based on the direction of the person, the stride, the height, and the like. In this embodiment, the rectangle ABCD and the rectangle EFGH are squares, and the length of one side is, for example, 800 mm. Further, the height of the rectangular parallelepiped is, for example, 1800 mm. That is, the box BX is a rectangular parallelepiped having a width of 800 mm, a depth of 800 mm, and a height of 1800 mm.

  A quadrangle ABGH defined by the four vertices A, B, G, and H forms a virtual plane region TR corresponding to the region of the identification target image in the captured image. Further, the quadrangle ABGH as the virtual plane region TR is inclined with respect to the virtual ground that is a horizontal plane.

  In the present embodiment, a box BX as a virtual rectangular parallelepiped is employed to define the relationship between the reference point Pr and the virtual plane region TR. However, as long as the virtual plane region TR that faces the direction of the imaging device 40 and is inclined with respect to the virtual ground can be determined in association with an arbitrary reference point Pr, the relationship using other virtual three-dimensional objects, etc. May be adopted, and other mathematical relations such as a function and a conversion table may be adopted.

  FIG. 5 is a top view of the real space behind the shovel, and shows the positional relationship between the rear camera 40B and the virtual plane regions TR1 and TR2 when the virtual plane regions TR1 and TR2 are referenced using the reference points Pr1 and Pr2. Show. In this embodiment, the reference point Pr can be arranged at each of the lattice points of the virtual grid on the virtual ground. However, the reference points Pr may be irregularly arranged on the virtual ground, or may be arranged at equal intervals on a line segment radially extending from the projection point of the rear camera 40B on the virtual ground. For example, each line segment extends radially in increments of 1 degree, and the reference points Pr are arranged at intervals of 100 mm on each line segment.

  As shown in FIGS. 4 and 5, the first surface of the box BX defined by the quadrangle ABFE (see FIG. 4) is positive to the rear camera 40B when the virtual plane region TR1 is referenced using the reference point Pr1. It arranges so that it may be. That is, the line segment connecting the rear camera 40B and the reference point Pr1 is orthogonal to the first surface of the box BX arranged in association with the reference point Pr1 when viewed from above. Similarly, the first surface of the box BX is arranged to face the rear camera 40B even when the virtual plane region TR2 is referenced using the reference point Pr2. That is, a line segment connecting the rear camera 40B and the reference point Pr2 is orthogonal to the first surface of the box BX arranged in association with the reference point Pr2 when viewed from above. This relationship holds even when the reference point Pr is arranged on any lattice point. That is, the box BX is arranged so that the first surface thereof is always directly opposite the rear camera 40B.

  FIG. 6 is a diagram illustrating a flow of processing for generating a normalized image from a captured image. Specifically, FIG. 6A is an example of a captured image of the rear camera 40B, and displays a box BX arranged in association with the reference point Pr in real space. FIG. 6B is a diagram in which a region of the identification processing target image (hereinafter referred to as “identification processing target image region TRg”) in the captured image is cut out, and is displayed in the captured image of FIG. Corresponding to the virtual plane region TR. FIG. 6C shows a normalized image TRgt obtained by normalizing the identification processing target image having the identification processing target image region TRg.

  As shown in FIG. 6A, the box BX arranged in the real space in relation to the reference point Pr1 determines the position of the virtual plane region TR in the real space, and the imaging corresponding to the virtual plane region TR. An identification processing target image region TRg on the image is determined.

  Thus, if the position of the reference point Pr in the real space is determined, the position of the virtual plane region TR in the real space is uniquely determined, and the identification processing target image region TRg in the captured image is also uniquely determined. Then, the extraction unit 31 can generate a normalized image TRgt having a predetermined size by normalizing the identification processing target image having the identification processing target image region TRg. In the present embodiment, the size of the normalized image TRgt is, for example, 64 pixels long × 32 pixels wide.

  FIG. 7 is a diagram illustrating a relationship among a captured image, an identification processing target image region, and a normalized image. Specifically, FIG. 7A1 shows the identification processing target image region TRg3 in the captured image, and FIG. 7A2 shows the normalized image TRgt3 of the identification processing target image having the identification processing target image region TRg3. . FIG. 7B1 shows an identification processing target image region TRg4 in the captured image, and FIG. 7B2 shows a normalized image TRgt4 of the identification processing target image having the identification processing target image region TRg4. Similarly, FIG. 7C1 shows the identification processing target image region TRg5 in the captured image, and FIG. 7C2 shows the normalized image TRgt5 of the identification processing target image having the identification processing target image region TRg5.

  As shown in FIG. 7, the identification processing target image region TRg5 in the captured image is larger than the identification processing target image region TRg4 in the captured image. This is because the distance between the virtual plane area corresponding to the identification processing target image area TRg5 and the rear camera 40B is smaller than the distance between the virtual plane area corresponding to the identification processing target image area TRg4 and the rear camera 40B. Similarly, the identification processing target image region TRg4 in the captured image is larger than the identification processing target image region TRg3 in the captured image. This is because the distance between the virtual plane area corresponding to the identification processing target image area TRg4 and the rear camera 40B is smaller than the distance between the virtual plane area corresponding to the identification processing target image area TRg3 and the rear camera 40B. That is, the identification processing target image area in the captured image is smaller as the distance between the corresponding virtual plane area and the rear camera 40B is larger. On the other hand, the normalized images TRgt3, TRgt4, and TRgt5 are all rectangular images having the same size.

  In this manner, the extraction unit 31 can normalize the identification processing target image that can take various shapes and sizes in the captured image to a rectangular image of a predetermined size, and can normalize a human candidate image including a human image. Specifically, the extraction unit 31 arranges an image portion (hereinafter referred to as “head image portion”) that is estimated to be the head of the human candidate image in a predetermined region of the normalized image. In addition, an image portion (hereinafter, referred to as a “body portion image portion”) that is estimated to be the trunk portion of the human candidate image is arranged in another predetermined region of the normalized image, and yet another predetermined portion of the normalized image. An image portion (hereinafter referred to as a “leg image portion”) estimated to be a leg portion of the human candidate image is arranged in the region. Further, the extraction unit 31 can acquire the normalized image in a state where the inclination (image collapse) of the human candidate image with respect to the shape of the normalized image is suppressed.

  Next, referring to FIG. 8, normalization processing in a case where the identification processing target image area includes an image area that is not suitable for identification that adversely affects identification of human images (hereinafter referred to as “identification process inappropriate area”). Will be described. The identification processing inappropriate area is a known area where no human image can exist, for example, an area in which the excavator's vehicle body is reflected (hereinafter referred to as “vehicle body reflection area”), an area that protrudes from the captured image ( Hereinafter, it is referred to as “extrusion area”). FIG. 8 is a diagram illustrating the relationship between the identification processing target image area and the identification processing inappropriate area, and corresponds to FIG. 7 (C1) and FIG. 7 (C2). Further, the right-slanted hatched area in the left diagram of FIG. 8 corresponds to the protrusion area R1, and the left-slanted hatched area corresponds to the vehicle body reflection area R2.

  In the present embodiment, when the identification process target image region TRg5 includes a part of the protrusion region R1 and the vehicle body reflection region R2, the extraction unit 31 performs mask processing on these identification processing inappropriate regions, and then performs an identification processing target image. A normalized image TRgt5 of the identification processing target image having the region TRg5 is generated. Note that the extraction unit 31 may mask the portion corresponding to the identification processing inappropriate region in the normalized image TRgt5 after generating the normalized image TRgt5.

  The right figure of FIG. 8 shows normalized image TRgt5. Further, in the right diagram of FIG. 8, the diagonally hatched area that falls to the right represents the mask area M1 corresponding to the protruding area R1, and the diagonally hatched area that falls to the left represents the mask area M2 corresponding to a part of the vehicle body reflection area R2. Represents.

  In this way, the extraction unit 31 masks the image of the identification process inappropriate area, thereby preventing the image of the identification process inappropriate area from affecting the identification process performed by the identification unit 32. By this mask processing, the identification unit 32 can identify whether the image is a human image using an image of a region other than the mask region in the normalized image without being affected by the image of the identification processing inappropriate region. Note that the extraction unit 31 may use any known method other than the mask process so that the image in the identification process inappropriate region does not affect the identification process performed by the identification unit 32.

  Next, characteristics of the normalized image generated by the extraction unit 31 will be described with reference to FIG. FIG. 9 is a diagram illustrating an example of a normalized image. Further, the 14 normalized images shown in FIG. 9 include images of human candidates that are closer to the rear camera 40B as the normalized image is closer to the left end of the diagram, and the normalized image closer to the right end of the diagram is It includes images of human candidates that are located far from the rear camera 40B.

  As shown in FIG. 9, the extraction unit 31 performs any normalization regardless of the rear horizontal distance (the horizontal distance in the Y-axis direction shown in FIG. 5) between the virtual plane region TR and the rear camera 40B in the real space. In the image, the head image portion, the torso image portion, the leg image portion, and the like can be arranged at substantially the same ratio. Therefore, the extraction unit 31 can reduce the calculation load when the identification unit 32 executes the identification process, and can improve the reliability of the identification result. The rear horizontal distance described above is an example of information regarding the positional relationship between the virtual plane region TR and the rear camera 40B in the real space, and the extraction unit 31 adds the information to the extracted identification processing target image. . Further, the information on the positional relationship described above includes a top view angle with respect to the optical axis of the rear camera 40B of a line segment connecting the reference point Pr corresponding to the virtual plane region TR and the rear camera 40B.

  Next, the relationship between the rear horizontal distance between the virtual plane region TR and the rear camera 40B in the real space and the size of the head image portion in the normalized image will be described with reference to FIG. 10 shows the sizes L10, L11, and L12 of the head image portion when a person is present at three reference points Pr10, Pr11, and P12 that have different rear horizontal distances from the rear camera 40B. The horizontal axis corresponds to the rear horizontal distance. 10 is a graph showing the relationship between the rear horizontal distance and the size of the head image portion. The vertical axis corresponds to the size of the head image portion, and the horizontal axis corresponds to the rear horizontal distance. In addition, the horizontal axis of the upper figure of FIG. 10 and the lower figure of FIG. 10 is common. In this embodiment, the camera height is 2100 mm, the height of the center of the head HD from the ground is 1600 mm, and the head diameter is 250 mm.

  As shown in the upper diagram of FIG. 10, when a person is present at the position indicated by the reference point Pr10, the size L10 of the head image portion is the projected image of the head HD on the virtual plane region TR10 viewed from the rear camera 40B. Corresponds to the size of. Similarly, when a person is present at the positions indicated by the reference points Pr11 and Pr12, the sizes L11 and L12 of the head image portion are the projection images of the head HD on the virtual plane regions TR11 and TR12 viewed from the rear camera 40B. Corresponds to the size of. Note that the size of the head image portion in the normalized image varies with the size of the projection image.

  As shown in the lower diagram of FIG. 10, the size of the head image portion in the normalized image remains substantially the same when the rear horizontal distance is greater than or equal to D1 (eg, 700 mm), but the rear horizontal distance is less than D1. It suddenly increases.

  Therefore, the identification unit 32 changes the content of the identification process according to the rear horizontal distance. For example, when using a supervised learning (machine learning) technique, the identification unit 32 groups learning samples used in the identification process at a predetermined rear horizontal distance (for example, 650 mm). Specifically, the learning samples are divided into a short distance group and a long distance group. With this configuration, the identification unit 32 can identify the human image with higher accuracy.

  With the above configuration, the periphery monitoring system 100 generates the normalized image TRgt from the identification processing target image region TRg corresponding to the virtual plane region TR that faces the imaging device 40 and is inclined with respect to the virtual ground that is a horizontal plane. . Therefore, normalization in consideration of how the person looks in the height direction and depth direction can be realized. As a result, even when a captured image of the imaging device 40 attached to the construction machine so as to capture an image of a person from above is used, a person existing around the construction machine can be detected more reliably. In particular, even when a person approaches the imaging device 40, the normalized image can be generated from the identification processing target image that occupies a sufficiently large area on the captured image, so that the person can be reliably detected.

  In addition, the periphery monitoring system 100 defines the virtual plane region TR as a rectangular region formed by the four vertices A, B, G, and H of the box BX that is a virtual cuboid in real space. Therefore, the reference point Pr in the real space can be geometrically associated with the virtual plane region TR, and further, the virtual plane region TR in the real space and the identification processing target image region TRg in the captured image can be geometrically related. Can be associated.

  In addition, the extraction unit 31 performs mask processing on the image of the identification processing inappropriate area included in the identification processing target image area TRg. Therefore, the identification unit 32 can identify whether the image is a human image by using an image of a region other than the mask region in the normalized image without being affected by the image of the identification inappropriate region including the vehicle body reflection region R2.

  In addition, when the identification processing target image is extracted, the extraction unit 31 adds the rear horizontal distance between the two to the identification processing target image as information regarding the positional relationship between the virtual plane region TR and the imaging device 40. And the identification part 32 changes the content of the identification process according to the back horizontal distance. Specifically, the identification unit 32 groups learning samples used in the identification process at a predetermined rear horizontal distance (for example, 650 mm). With this configuration, the identification unit 32 can identify the human image with higher accuracy.

  Further, the extraction unit 31 can extract an identification processing target image for each reference point Pr. In addition, each of the identification processing target image areas TRg is associated with one of the reference points Pr set in advance as an assumed standing position of the person via the corresponding virtual plane area TR. Therefore, the periphery monitoring system 100 can extract an identification processing target image that is likely to include a human candidate image by extracting the reference point Pr that is likely to be present by an arbitrary method. In this case, it is possible to prevent the identification processing target image having a low possibility of including the human candidate image from being subjected to the identification processing by the image processing having a relatively large calculation amount, and to realize the speedup of the human detection processing. .

  Next, an example of processing in which the extraction unit 31 extracts an identification processing target image that is highly likely to include a human candidate image will be described with reference to FIGS. 11 and 12. FIG. 11 is a schematic diagram illustrating an example of a geometric relationship used when the extraction unit 31 cuts out an identification processing target image from a captured image, and corresponds to FIG. FIG. 12 is a diagram illustrating an example of a feature image in a captured image. Note that the feature image is an image that represents a characteristic part of a person, and is preferably an image that represents a part in which the height from the ground in real space is difficult to change. Therefore, the feature image includes, for example, an image of a helmet, an image of a shoulder, an image of a head, an image of a reflector or a marker attached to a person, and the like.

  In the present embodiment, the extraction unit 31 finds out a helmet image (an image that can be estimated to be a helmet strictly) in the captured image by the pre-stage image recognition process. This is because a person working around the excavator is considered to be wearing a helmet. Then, the extraction unit 31 derives the most relevant reference point Pr from the position of the found helmet image. Then, the extraction unit 31 extracts an identification processing target image corresponding to the reference point Pr.

  Specifically, the extraction unit 31 derives a highly relevant reference point Pr from the position of the helmet image in the captured image using the geometric relationship shown in FIG. The geometric relationship in FIG. 11 is different from the geometric relationship in FIG. 4 in that the virtual head position HP in the real space is determined, but is common in other points.

  The virtual head position HP represents the head position of a person assumed to be present on the reference point Pr, and is disposed immediately above the reference point Pr. In this embodiment, it is arranged at a height of 1700 mm on the reference point Pr. Therefore, if the virtual head position HP in the real space is determined, the position of the reference point Pr in the real space is uniquely determined, and the position of the virtual plane region TR in the real space is also uniquely determined. Further, the identification processing target image region TRg in the captured image is also uniquely determined. Then, the extraction unit 31 can generate a normalized image TRgt having a predetermined size by normalizing the identification processing target image having the identification processing target image region TRg.

  Conversely, if the position of the reference point Pr in the real space is determined, the virtual head position HP in the real space is uniquely determined, and the head image position AP on the captured image corresponding to the virtual head position HP in the real space is also unique. It is decided. Therefore, the head image position AP can be set in advance in association with each of the preset reference points Pr. The head image position AP may be derived from the reference point Pr in real time.

  Therefore, the extraction unit 31 searches for the helmet image in the captured image of the rear camera 40B by the pre-stage image recognition process. The upper part of FIG. 12 shows a state where the extraction unit 31 has found the helmet image HRg. And the extraction part 31 determines the representative position RP, when the helmet image HRg is found. The representative position RP is a position derived from the size, shape, etc. of the helmet image HRg. In this embodiment, the representative position RP is the position of the central pixel in the helmet image area including the helmet image HRg. The lower diagram in FIG. 12 is an enlarged view of the helmet image region, which is a rectangular image region separated by white lines in the upper diagram in FIG. 12, and shows that the position of the central pixel in the helmet image region is the representative position RP.

  Thereafter, the extraction unit 31 derives a head image position AP that is closest to the representative position RP using, for example, a nearest neighbor search algorithm. In the lower part of FIG. 12, six head image positions AP1 to AP6 are preset near the representative position RP, and the head image position AP5 is the head image position AP closest to the representative position RP. It shows that.

  Then, the extraction unit 31 uses the geometrical relationship shown in FIG. 11 to follow the virtual head position HP, the reference point Pr, and the virtual plane region TR from the nearest head image position AP derived. The identification processing target image region TRg to be extracted is extracted. Thereafter, the extraction unit 31 normalizes the identification processing target image having the extracted identification processing target image region TRg to generate a normalized image TRgt.

  In this way, the extraction unit 31 includes the representative position RP of the helmet image HRg that is the position of the human characteristic image in the captured image, and one of the preset head image positions AP (head image position AP5). Are associated with each other to extract an identification processing target image.

  Note that the extraction unit 31 directly associates the head image position AP with the reference point Pr, the virtual plane region TR, or the identification processing target image region TRg instead of using the geometrical relationship shown in FIG. An identification processing target image corresponding to the head image position AP may be extracted using a table.

  In addition, the extraction unit 31 may derive the reference point Pr from the representative position RP using a known algorithm other than the nearest neighbor search algorithm such as a hill-climbing method or a Mean-shift method. For example, when using the hill-climbing method, the extraction unit 31 derives a plurality of head image positions AP in the vicinity of the representative position RP, and the reference points Pr corresponding to the representative position RP and each of the plurality of head image positions AP. Is linked. At this time, the extraction unit 31 weights the reference point Pr so that the weight becomes larger as the representative position RP and the head image position AP are closer. Then, the weight distribution of the plurality of reference points Pr is climbed, and the identification processing target image region TRg is extracted from the reference point Pr having the weight closest to the maximum point of the weight.

  Next, an example of processing (hereinafter referred to as “image extraction processing”) in which the extraction unit 31 of the controller 30 extracts an identification processing target image will be described with reference to FIG. FIG. 13 is a flowchart showing an exemplary flow of image extraction processing. For example, the extraction unit 31 performs this image extraction process every time a captured image is acquired.

  First, the extraction unit 31 searches for a helmet image in the captured image (step ST1). In the present embodiment, the extraction unit 31 performs a raster scan on the image captured by the rear camera 40B by the preceding image recognition process to find a helmet image.

  When the helmet image HRg is found from the captured image (YES in step ST1), the extraction unit 31 acquires the representative position RP of the helmet image HRg (step ST2).

  Thereafter, the extraction unit 31 acquires a head image position AP that is closest to the acquired representative position RP (step ST3).

  Thereafter, the extraction unit 31 extracts an identification processing target image corresponding to the acquired head image position AP (step ST4). In the present embodiment, the extraction unit 31 uses the geometrical relationship shown in FIG. 11 and uses the head image position AP in the captured image, the virtual head position HP in the real space, and the assumed standing position of the person in the real space. The identification processing target image is extracted by following the correspondence between the reference point Pr and the virtual plane region TR in the real space.

  If the helmet image HRg is not found in the captured image (NO in step ST1), the extraction unit 31 shifts the process to step ST5 without extracting the identification processing target image.

  Then, the extraction part 31 determines whether the helmet image was searched over the whole captured image (step ST5).

  If it is determined that the entire captured image has not yet been searched (NO in step ST5), the extraction unit 31 performs the processing in steps ST1 to ST4 on another region of the captured image.

  On the other hand, when it is determined that the search for the helmet image over the entire captured image has been completed (YES in step ST5), the extraction unit 31 ends the current image extraction process.

  In this way, the extraction unit 31 first finds the helmet image HRg, and from the representative position RP of the found helmet image HRg, the head image position AP, the virtual head position HP, the reference point (assumed standing position) Pr, the virtual An identification processing target image region TRg is specified through the plane region TR. Then, by extracting and normalizing the identification processing target image having the identified identification processing target image region TRg, a normalized image TRgt of a predetermined size can be generated.

  Next, another example of image extraction processing will be described with reference to FIG. FIG. 14 is a flowchart showing the flow of another example of image extraction processing.

  First, the extraction unit 31 acquires one of the head image positions AP (step ST11). Thereafter, the extraction unit 31 acquires a helmet image region corresponding to the head image position AP (step ST12). In the present embodiment, the helmet image area is an image area of a predetermined size that is preset for each of the head image positions AP.

  Thereafter, the extraction unit 31 searches for a helmet image within the helmet image region (step ST13). In the present embodiment, the extraction unit 31 raster scans the inside of the helmet image area by the preceding image recognition process and finds out the helmet image.

  When the helmet image HRg is found in the helmet image area (YES in step ST13), the extraction unit 31 extracts an identification processing target image corresponding to the head image position AP at that time (step ST14). In the present embodiment, the extraction unit 31 uses the geometrical relationship shown in FIG. 11 and uses the head image position AP in the captured image, the virtual head position HP in the real space, and the assumed standing position of the person in the real space. The identification processing target image is extracted by following the correspondence between the reference point Pr and the virtual plane region TR in the real space.

  If the helmet image HRg is not found in the helmet image area (NO in step ST13), the extraction unit 31 shifts the process to step ST15 without extracting the identification process target image.

  Thereafter, the extraction unit 31 determines whether all the head image positions AP have been acquired (step ST15). And when it determines with not having acquired all the head image positions AP yet (NO of step ST15), the extraction part 31 acquires another head image position AP which has not been acquired, and step ST11-step ST14 Execute the process. On the other hand, when it is determined that all the head image positions AP have been acquired (YES in step ST15), the extraction unit 31 ends the current image extraction process.

  Thus, when the extraction unit 31 first acquires one of the head image positions AP and finds the helmet image HRg in the helmet image area corresponding to the acquired head image position AP, the head at that time From the partial image position AP, the identification processing target image area TRg is specified through the virtual head position HP, the reference point (assumed standing position) Pr, and the virtual plane area TR. Then, by extracting and normalizing the identification processing target image having the identified identification processing target image region TRg, a normalized image TRgt of a predetermined size can be generated.

  With the above configuration, the extraction unit 31 of the periphery monitoring system 100 finds a helmet image as a feature image in the captured image, and associates the representative position RP of the helmet image with one of the head image positions AP as a predetermined image position. Thus, the identification processing target image is extracted. Therefore, it is possible to narrow down the image portion to be subjected to the subsequent image recognition processing with a simple system configuration.

  The extraction unit 31 first finds the helmet image HRg from the captured image, derives one of the head image positions AP corresponding to the representative position RP of the helmet image HRg, and sets it as one of the head image positions AP. A corresponding identification processing target image may be extracted. Alternatively, the extraction unit 31 first acquires one of the head image positions AP, and the helmet image is in a helmet image area that is a predetermined area including the position of the feature image corresponding to one of the head image positions AP. May exist, an identification processing target image corresponding to one of the head image positions AP may be extracted.

  Further, the extraction unit 31 may extract the identification processing target image from the representative position RP of the helmet image in the captured image using a predetermined geometrical relationship as illustrated in FIG. In this case, the predetermined geometric relationship is such that the identification processing target image region TRg in the captured image, the virtual plane region TR in the real space corresponding to the identification processing target image region TRg, and the real space corresponding to the virtual plane region TR. Reference point Pr (assumed standing position of a person), virtual head position HP corresponding to reference point Pr (virtual feature position that is a position in a real space of a characteristic portion of the person corresponding to the assumed standing position of the person), and The geometrical relationship with the head image position AP (predetermined image position in the captured image corresponding to the virtual feature position) in the captured image corresponding to the virtual head position HP is represented.

  Next, another example of the image extraction process will be described with reference to FIG. FIG. 15 is a flowchart showing the flow of still another example of the image extraction process. The image extraction process of FIG. 15 is the same as the image extraction process of FIG. 13 and FIG. 14 in that it determines whether a helmet image as a feature image is included in the normalized image after generating the normalized image. Different. The image extraction process in each of FIGS. 13 and 14 is for normalizing an image portion including the helmet image after finding the helmet image.

  In the present embodiment, the extraction unit 31 normalizes each of the plurality of predetermined image portions in the captured image to generate a plurality of normalized images, and among the normalized images, the normalized image including the helmet image is identified. Extract as an image. The plurality of predetermined image portions are, for example, a plurality of identification processing target image regions TRg predetermined on the captured image. The identification processing target image region TRg (see FIG. 6) corresponds to the virtual plane region TR in the real space, and the virtual plane region TR corresponds to the reference point Pr in the real space. Then, the identification unit 32 identifies whether the human candidate image included in the identification processing target image extracted by the extraction unit 31 is a human image.

  First, the extraction unit 31 generates a normalized image from one of the predetermined image portions (step ST21). In the present embodiment, one of the identification processing target image regions TRg corresponding to one of the preset reference points Pr in the captured image of the rear camera 40B is set as a predetermined image portion, and the predetermined image portion is converted into a predetermined size by projective transformation. Convert to a rectangular image.

  Thereafter, the extraction unit 31 determines whether or not a helmet image is included in the generated normalized image (step ST22). In the present embodiment, the extraction unit 31 raster scans the normalized image by the previous image recognition process to find a helmet image.

  If it is determined that the helmet image is included in the normalized image (YES in step ST22), the extraction unit 31 extracts the normalized image as an identification processing target image (step ST23). At this time, the identification unit 32 identifies whether the human candidate image included in the identification processing target image extracted by the extraction unit 31 is a human image. That is, the identification unit 32 determines whether the image included in the normalized image is a human image with respect to the normalized image determined to include the helmet image before the next normalization by the extraction unit 31 is performed. Identify. Therefore, if the periphery monitoring system 100 has a memory capacity capable of storing one normalized image, the periphery monitoring system 100 can execute image extraction processing and identification processing on the entire captured image. However, after the extraction unit 31 extracts a plurality of normalized images as identification processing target images, the identification unit 32 identifies whether the human candidate image included in each of the plurality of identification processing target images is a human image. May be.

  When it is determined that the helmet image is not included in the normalized image (NO in step ST22), the extraction unit 31 proceeds with the process without extracting the normalized image as the identification processing target image.

  Thereafter, the extraction unit 31 determines whether or not a normalized image has been generated from all the predetermined image portions (step ST24). In the present embodiment, it is determined whether or not a normalized image has been generated from all of the predetermined image portions that are the identification processing target image regions TRg corresponding to the respective reference points Pr in the captured image of the rear camera 40B.

  When it is determined that the normalized image has not been generated from all the predetermined image portions (NO in step ST24), the extraction unit 31 performs the processes in steps ST21 to ST23 on another predetermined image portion.

  On the other hand, when it is determined that the normalized image has been generated from all the predetermined image portions (YES in step ST24), the extraction unit 31 ends the current image extraction process.

  In the example of FIG. 15, the periphery monitoring system 100 determines whether or not a helmet image is included in the normalized image when one normalized image is generated. However, at the stage where a plurality of normalized images are generated, whether or not a helmet image is included in each of the plurality of normalized images may be collectively determined. Further, it may be determined collectively whether or not a helmet image is included in each of all the normalized images at a stage where all the normalized images are generated.

  By such an image extraction process, the periphery monitoring system 100 can reduce the number of memory accesses during the human detection process. This is because the process of finding out the helmet image and the process of extracting the identification process target image can be executed simultaneously.

  Further, the periphery monitoring system 100 that employs the image extraction process of FIG. 15 can reduce the memory capacity compared to the case of employing the image extraction process of FIG. 13 or FIG. The image extraction process of FIG. 13 or FIG. 14 requires at least a memory capacity that can store an image part before normalization (an image larger than the normalized image), but the image extraction process of FIG. This is because the image can be executed if there is enough memory capacity to store the normalized image in order to generate the image first.

  Further, the periphery monitoring system 100 that employs the image extraction process of FIG. 15 determines whether or not a helmet image is included in the normalized image. Therefore, compared with the configuration for determining whether or not a helmet image is included in an image before normalization as in the image extraction process of FIG. 13 or FIG. 14, processing required for determining whether or not a helmet image is included. You can save time. This is because variations in the size and distortion of the helmet image to be searched can be reduced. As a result, the time required for extracting the identification processing target image can be shortened. Moreover, since the variation in the size and distortion of the helmet image to be searched can be reduced, the accuracy of the determination as to whether or not the helmet image is included can be increased.

  Further, the periphery monitoring system 100 that employs the image extraction process of FIG. 15 is a case where the variation in the head position with respect to the human foot position is large (for example, when the image of the crooked person is included in the captured image). , More reliable human detection can be realized. This is because a configuration for determining whether or not the helmet image is included in the normalized image is employed. That is, it is not a configuration in which the identification processing target image region is specified using the head image position related to the position of the helmet image as in the image extraction processing of FIG. 13 or FIG. 14, but as a predetermined image portion to be normalized This is because the image processing target image area is set regardless of the head image position.

  Next, another example of the image extraction process will be described with reference to FIG. FIG. 16 is a flowchart showing the flow of still another example of the image extraction process. In the image extraction process of FIG. 16, a helmet image as a feature image is added to a partially normalized image (hereinafter referred to as “partial normalized image”) at a stage where a part of the predetermined image portion is normalized. It differs from the image extraction process of FIG. 15 in that it is determined whether or not it is included. The image extraction process in FIG. 15 is for determining whether or not a helmet image is included in the normalized image when all of one predetermined image portion is normalized.

  First, the extraction unit 31 generates a partially normalized image from a part of the predetermined image portion (step ST31). And the extraction part 31 determines whether a helmet image is contained in the partial normalization image in the step which produced | generated the partial normalization image (step ST32). In the present embodiment, the extraction unit 31 rasterizes the partially normalized image by the pre-stage image recognition process at the stage where the partially normalized image is generated by normalizing a part of the predetermined image portion corresponding to the upper half of the normalized image. Scan to find the helmet image.

  If it is determined that the helmet image is included in the partially normalized image (YES in step ST32), the extraction unit 31 normalizes the remaining portion of the predetermined image portion to generate a normalized image (step ST33). In the present embodiment, the extraction unit 31 normalizes the remaining portion of the predetermined image portion corresponding to the lower half of the normalized image to generate a normalized image. Then, the extraction unit 31 extracts the normalized image as an identification processing target image (step ST34). At this time, the identification unit 32 identifies whether the human candidate image included in the identification processing target image extracted by the extraction unit 31 is a human image. However, after the extraction unit 31 extracts a plurality of normalized images as identification processing target images, the identification unit 32 identifies whether the human candidate image included in each of the plurality of identification processing target images is a human image. May be.

  When it is determined that the helmet image is not included in the partially normalized image (NO in step ST32), the extraction unit 31 proceeds with the process without generating a normalized image by normalizing the remaining portion of the predetermined image portion.

  Thereafter, the extraction unit 31 determines whether or not a partially normalized image has been generated from all the predetermined image portions (step ST35). In the present embodiment, it is determined whether or not a partial normalized image has been generated from all of the predetermined image portions that are the identification processing target image regions TRg corresponding to the respective reference points Pr in the captured image of the rear camera 40B.

  When it is determined that the partially normalized image has not been generated from all the predetermined image portions (NO in step ST35), the extraction unit 31 performs the processes in steps ST31 to ST34 on another predetermined image portion.

  On the other hand, when it is determined that a partially normalized image has been generated from all the predetermined image portions (YES in step ST35), the extraction unit 31 ends the current image extraction process.

  In the example of FIG. 16, the periphery monitoring system 100 normalizes a part of a predetermined image portion corresponding to the upper half of the normalized image to generate a partially normalized image. This is because the helmet image has a high probability of being included in the upper half of the normalized image (the portion corresponding to the upper body of a person). However, a partially normalized image may be generated by normalizing a part of a predetermined image portion corresponding to a portion smaller than the upper half of the normalized image. For example, a partially normalized image may be generated by normalizing a part of a predetermined image portion corresponding to a portion corresponding to 1/3 of the normalized image. Alternatively, the periphery monitoring system 100 may generate a partially normalized image by normalizing another part of the normalized image. For example, when searching for a shoulder image as a feature image, a part of a predetermined image portion corresponding to the right half of the normalized image may be normalized to generate a partially normalized image. When searching for an image, a partially normalized image may be generated by normalizing a part of a predetermined image portion corresponding to the central half of the normalized image.

  With such an image extraction process, the periphery monitoring system 100 that employs the image extraction process of FIG. 16 can achieve the same effects as when the image extraction process of FIG. 15 is employed.

  In addition, the periphery monitoring system 100 that employs the image extraction process of FIG. 16 can further reduce the processing time required for the human detection process as compared to the case of employing the image extraction process of FIG. This is because, when it is determined that the helmet image is not included in the partial normalized image, normalization of the next predetermined image portion can be started without normalizing the remaining portion of the predetermined image portion.

  Further, in the image extraction process in each of FIGS. 13 to 16, the extraction unit 31 finds out a helmet image (an image that can be estimated to be a helmet strictly) in a predetermined image portion of the captured image by the preceding image recognition process. And the extraction part 31 extracts the identification process target image corresponding to the found helmet image.

  However, when the background image is complicated, the extraction unit 31 may erroneously recognize an image that is not a helmet image as a helmet image. Also, the helmet image may be missed without being recognized as a helmet image. That is, the extraction unit 31 may reduce the reliability of the extraction result of the identification processing target image depending on the content of the captured image. The helmet image here can be considered as a human candidate image. That is, it is a scene that reduces the reliability of the human detection result. The captured images that reduce the reliability of the extraction results include, for example, captured images captured in a night environment, backlight environment, shaded environment, and the like, captured images captured with the camera lens dirty, flare, and subject blur. And the like.

  Therefore, the controller 30 is configured to detect a state where the extraction result of the identification processing target image extracted by the extraction unit 31 is low in reliability as a state unsuitable for human detection (hereinafter referred to as “human detection unsuitable state”). May be. This is because various measures can be taken in response to detection of an unsuitable state of human detection. For example, it is possible to notify the operator of the excavator that a human detection inappropriate state has occurred. Alternatively, it is possible to selectively collect data effective for improvement of the peripheral monitoring system by recording image characteristics and the like when a human detection inappropriate state occurs. Alternatively, the operation content of the periphery monitoring system 100 can be changed in accordance with detection of a person detection inappropriate state. The change in the operation content of the periphery monitoring system 100 includes, for example, a change in various parameters used in the preceding image recognition process, a change in various parameters used in the subsequent image recognition process, and the like.

  For example, the controller 30 may calculate an evaluation value for each predetermined image portion using a predetermined evaluation formula, and determine whether the captured image is in a human detection inappropriate state based on the distribution of the evaluation values. . Specifically, the controller 30 may determine whether or not a human detection inappropriate state has occurred based on the helmet degree distribution. The evaluation value here can also be regarded as a degree of humanity in a helmet image, that is, a person candidate image, or a level indicating the degree. It may be considered that there is a difference between the evaluation values.

  “Helmet degree” is an example of an evaluation value, and is a value representing the helmet-likeness of the determination target image in the predetermined image portion of the captured image. When the search target is not a helmet but a head, the degree of headness representing headiness is used as another example of the evaluation value.

  The “determination target image” means an image that is a determination target of whether or not it is a feature image (helmet image). The “helmet image” in the image extraction process in each of FIGS. 13 to 16 means a determination target image having a helmet degree equal to or higher than a predetermined acceptance / rejection determination value (for example, zero). Alternatively, it may be a predetermined upper number when the determination target images having a helmet degree equal to or higher than a predetermined acceptance / rejection determination value are arranged in descending order of the helmet degree. The determination target image includes an image (an image other than the helmet image) having a helmet degree less than the acceptance determination value. A predetermined helmet degree selectable as a specification is set as the acceptance / rejection determination value. The acceptance / rejection determination value is compared with the helmet degree of the determination target image acquired by the extraction unit 31.

  The acceptance / rejection determination value is a value for determining whether or not to adopt the determination target image as a helmet image. For example, if the helmet degree as the evaluation value is equal to or higher than the acceptance / rejection determination value, the determination target image is adopted as the helmet image. The acceptance / rejection determination value is set in advance by machine learning, statistical analysis, or the like, for example. The acceptance / rejection determination value may be set separately for each work environment of the excavator such as a night environment, a backlight environment, and a shade environment. The work environment of the excavator may be input using, for example, the input device 42 in the cabin 10.

  The degree of helmet is derived based on, for example, an image feature amount calculated at a calculation cost lower than that of the HOG feature amount used in the subsequent image recognition process. For example, the degree of helmet is derived by performing multiple regression analysis by inputting an image feature quantity having a dimension number lower than that of the HOG feature quantity into a machine learning algorithm such as a random forest. The degree of helmet is derived, for example, as a real number between −1 and +1. The closer to +1, the stronger the helmet, and the closer to −1, the weaker the helmet. The same applies to the evaluation value representing the feature image likelihood of other feature images such as “head degree” representing the head likelihood related to the “head image”.

  Next, referring to FIG. 17, a process in which the controller 30 determines whether the captured image is in a person detection inappropriate state based on the helmet degree distribution (hereinafter, referred to as “person detection suitability determination process”) will be described. To do. FIG. 17 is a flowchart showing an exemplary flow of the human detection suitability determination process.

  The controller 30 calculates the helmet degree of the determination target image in each predetermined image portion at the time of image extraction processing, and stores it in an internal memory or the like. The helmet degree may be stored so that it can be rearranged in a predetermined order (descending order), for example. Then, for example, the controller 30 starts executing the person detection suitability determination process at the time when the degree of helmet of the determination target image in a predetermined number of predetermined image portions is stored, and then the person detection is performed every time the degree of helmet is newly calculated. The execution of the suitability determination process is repeated.

  Further, the controller 30 executes the image extraction process and the human detection suitability determination process in parallel. This is because the degree of helmet calculated in the image extraction process is used in the human detection suitability determination process. However, the person detection suitability determination process may be executed after the image extraction process is completed. As the image extraction process, for example, any of the image extraction processes shown in FIGS.

  As shown in FIG. 17, the controller 30 first acquires the nth highest helmet degree (step ST41). For example, the controller 30 refers to the helmet degree stored in the internal memory, and obtains the nth highest helmet degree that is the helmet degree of a predetermined rank. “N” is an integer of 1 or more set as appropriate, and for example, a value corresponding to 2% of the total number of predetermined image portions (determination target images) is adopted. For example, the helmet degrees of the determination target images are stored in the internal memory in the descending order.

  Then, it is determined whether or not the nth highest helmet degree is equal to or greater than a predetermined suitability determination value (step ST42). A predetermined helmet degree selectable as a specification is set as a suitability determination value. It is determined whether the nth highest helmet degree is equal to or higher than the suitability determination value.

  The suitability determination value is a value for determining whether the captured image is in a human detection inappropriate state or a human detection compatible state. Similar to the acceptance / rejection determination value, the suitability determination value is set in advance by machine learning, statistical analysis, or the like, for example. The suitability determination value may be set separately for each work environment of the excavator such as a night environment, a backlight environment, and a shade environment.

  When it is determined that the n-th highest helmet degree is equal to or higher than the suitability determination value (YES in step ST42), the controller 30 determines that the captured image is in a human detection inappropriate state (step ST43). If the suitability determination value is greater than or equal to the acceptance determination value, the condition that “the nth highest helmet degree is greater than or equal to the suitability determination value” is satisfied. Means an existing state. In the example of FIG. 17, a state of a captured image in which n or more helmet images exist is defined as a person detection inappropriate state. “N” can be considered as a reference for the number of human candidate images sent from the extraction unit 31 to the identification unit 32. That is, this is a criterion for determining whether there are many or few candidate images to be processed by the identification unit 32.

  When it is determined that the captured image is in the human detection inappropriate state, the controller 30 may notify the shovel operator that the captured image is in the human detection inappropriate state. For example, the controller 30 may output a control command to an in-vehicle display as the output device 50 and display a text message informing that. Alternatively, the controller 30 may output a control command to a vehicle-mounted speaker serving as the output device 50 and output a voice message to that effect.

  In addition, when it is determined that the captured image is in a person detection inappropriate state, the controller 30 may stop the person detection by the periphery monitoring system. This is to prevent the person detection result based on the extraction result with low reliability from being presented to the operator of the excavator.

  In addition, when it is determined that the captured image is in a human detection inappropriate state, the controller 30 may record the captured image in NVRAM or the like. This is so that it can be used to improve the peripheral monitoring system.

  When it is determined that the nth highest helmet degree is less than the suitability determination value (NO in step ST42), the controller 30 is in a state where the captured image is suitable for human detection (hereinafter referred to as “human detection conforming state”). Is determined (step ST44). If the suitability determination value is greater than or equal to the acceptance determination value, the condition that “the nth highest helmet degree is less than the suitability determination value” is satisfied. Means a state. Considering the example of FIG. 17, in this case, the human detection inappropriate state is not achieved. This is because the number of person candidate images to be processed by the classifier is less than the reference “n”.

  Next, with reference to FIG. 18, the characteristics of the human detection conformity state and the human detection inappropriate state will be described. FIG. 18 is a frequency distribution diagram (histogram) of helmet degrees having real values between −1 and +1 derived from each of all predetermined image portions in a captured image captured in a certain environment. The frequency on the vertical axis can be considered as the detection number corresponding to the helmet degree. FIG. 18A shows a frequency distribution diagram when the human detection suitability state, and FIG. 18B shows a frequency distribution diagram when the human detection inappropriate state. In FIG. 18A and FIG. 18B, in order to maintain clarity, the illustration of each bin is omitted and only the frequency curve is shown. A range surrounded by the frequency curve and the horizontal axis represents the total number of predetermined image portions in the captured image.

  18A and 18B, the alternate long and short dash line drawn when the helmet degree is +0.5 (first reference) indicates the position of the suitability determination value, and the helmet degree is +0.2 (second). A two-dot chain line drawn at (reference) indicates the position of the acceptance / rejection determination value. Therefore, a determination target image with a helmet degree of +0.2 or more is adopted as a helmet image, and a determination target image with a helmet degree of less than +0.2 is not adopted as a helmet image. Below, what was not employ | adopted as a helmet image among determination object images is called a non-helmet image. The first standard and the second standard can be changed. In the histogram of FIG. 18A, a determination target image having a helmet degree of less than +0.2 is not extracted as a person candidate image. It can be said that the determination target image having a helmet degree of +0.2 or more was extracted as a human candidate image by the extraction unit 31. The extracted person candidate image is handled as an identification processing target image processed by the identification unit. Note that the extraction processing target image may be extracted by the extraction unit 31 in a plurality of stages. Then, the identification unit 32 performs image recognition processing on all of the identification processing target images extracted by the extraction unit 31.

  In FIG. 18A, the nth highest helmet degree is −0.3, which is smaller than the suitability determination value (+0.5). That is, the condition that “the nth highest helmet degree is less than the suitability determination value” is satisfied. This means that there are less than n helmet images having a helmet degree larger than the suitability determination value. From the viewpoint of the criterion “n”, it can be said that the human detection suitability state has a small number of identification processing target images to be processed by the identification unit 32. In addition, it can also be said that it is a person detection conformity state when the determination target image having the nth highest helmet degree is a non-helmet image. Therefore, the controller 30 determines that the actually acquired captured image is in the human detection compatible state.

  As described above, the captured image that provides the frequency distribution of the helmet degree as shown in FIG. 18A has a feature that the number of found helmet images is small and the number of non-helmet images is large. Therefore, there are relatively few processing target images to be processed by the identification unit 32.

  In FIG. 18B, the nth highest helmet degree is +0.6, which is larger than the suitability determination value (+0.5). That is, the condition that “the nth highest helmet degree is equal to or higher than the suitability determination value” is satisfied. It means a state where there are n or more helmet images having a helmet degree larger than the suitability determination value. Considering from the viewpoint of the criterion “n”, it can be said that the number of identification processing target images to be processed by the identification unit 32 is large and the human detection inappropriate state. In addition, it can also be said that it is a person detection unsuitable state when the determination target image which has the nth highest helmet degree is a helmet image. Therefore, the controller 30 determines that the actually acquired captured image is in a human detection inappropriate state.

  As described above, the captured image that provides the frequency distribution of the helmet degree as shown in FIG. 18B has a feature that the number of found helmet images is large and the number of non-helmet images is small. In other words, the number of erroneously recognized helmet images is large. Therefore, there are relatively many processing target images to be processed by the identification unit 32.

  As described above, the controller 30 can determine whether the captured image is in the human detection inappropriate state or the human detection compatible state based only on the captured image of the imaging device 40. Specifically, it can be determined whether or not the captured image is unsuitable for human detection based on the number of identification processing target images extracted by the extraction unit 31. According to this feature, the human detection unit includes a front stage (extraction unit 31) and a rear stage (identification unit 32), and the fact that the calculation cost of the rear stage is higher is an opportunity to reduce the burden on the rear stage. In addition, it is possible to determine whether it is a scene that takes time for subsequent processing.

  In addition, when the periphery monitoring system 100 determines that the captured image is in a human detection inappropriate state, the periphery monitoring system 100 can alert the operator of the excavator to that effect so as not to excessively depend on the current human detection result. .

  Or the periphery monitoring system 100 can prevent the presentation of the human detection result based on the extraction result with low reliability by stopping the human detection when it is determined that the captured image is in the human detection inappropriate state.

  Alternatively, the periphery monitoring system 100 can use the captured image for improvement of the periphery monitoring system by recording the captured image in NVRAM or the like when it is determined that the captured image is in a human detection inappropriate state.

  Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the scope of the present invention. Can be added.

  For example, in the above-described embodiment, the periphery monitoring system 100 extracts the identification processing target image while using the image of the helmet representing a part of the person as the feature image, but uses the whole body image of the person as the feature image. However, the identification processing target image may be extracted.

  The evaluation value may be widely used as an index as to whether a part of the captured image is a human image. It can be said that the helmet degree as the evaluation value is a value representing humanity.

  In the above-described embodiment, it is assumed that a person is detected using an image captured by the imaging device 40 mounted on the upper swing body 3 of the excavator. However, the present invention is not limited to this configuration. . The present invention can also be applied to a configuration that uses a captured image of an imaging device that is attached to the main body of another work machine such as a mobile crane, a fixed crane, a riff mag machine, or a forklift.

  In the above-described embodiment, the excavator blind spot area is imaged using three cameras, but the excavator blind spot area may be imaged using one, two, or four or more cameras.

  In the above-described embodiment, the human detection process is individually applied to each of the plurality of captured images. However, the human detection process is applied to one composite image generated from the plurality of captured images. Also good.

  DESCRIPTION OF SYMBOLS 1 ... Lower traveling body 2 ... Turning mechanism 3 ... Upper turning body 4 ... Boom 5 ... Arm 6 ... Bucket 7 ... Boom cylinder 8 ... Arm cylinder 9 ... Bucket cylinder 10: cabin 30 ... controller 31 ... extraction unit 32 ... identification unit 33 ... tracking unit 34 ... human detection unit 35 ... control unit 40 ... imaging device 40B: Rear camera 40L: Left camera 40R: Right camera 42 ... Input device 50 ... Output device 51 ... Machine control device 100 ... Perimeter monitoring system AP, AP1 AP6 ... head image position BX ... box HD ... head HP ... virtual head position HRg ... helmet image M1, M2 ... mask regions Pr, Pr1, Pr2, r10 to Pr12 ... Reference point R1 ... Projection area R2 ... Car body reflection area RP ... Representative position TR, TR1, TR2, TR10 to TR12 ... Virtual plane area TRg, TRg3, TRg4, TRg5 ... Identification processing target image area TRgt, TRgt3, TRgt4, TRgt5 ... Normalized image

Claims (7)

A work machine periphery monitoring system that detects a person existing around the work machine using a captured image of an image pickup device attached to the work machine, wherein a plurality of predetermined image portions from the captured image are used as identification processing target images. An extraction unit that extracts, and an identification unit that identifies whether an image included in the identification processing target image is a human image by image recognition processing;
When the number of the identification processing target images extracted by the extraction unit is greater than a predetermined reference, it is determined that the captured image is in an unsuitable state for human detection.
Perimeter monitoring system for work machines. Arranging the evaluation values calculated for each of the plurality of predetermined image portions in descending order,
When the evaluation value in a predetermined order is equal to or higher than a predetermined suitability determination value, it is determined that the captured image is unsuitable for human detection.
The work machine periphery monitoring system according to claim 1. The evaluation value is calculated based on the image feature amount of each predetermined image portion, and represents the humanity of each predetermined image portion,
The extraction unit extracts a predetermined image portion having the evaluation value equal to or higher than a predetermined acceptance / rejection determination value as the identification processing target image.
The work machine periphery monitoring system according to claim 2. The suitability determination value is not less than the acceptance determination value.
The work machine periphery monitoring system according to claim 3. The calculation cost required for calculating the image feature amount of each predetermined image portion is lower than the calculation cost required for calculating the image feature amount used in the image recognition processing executed by the identification unit.
The work machine periphery monitoring system according to claim 3 or 4. When it is determined that the captured image is in an unsuitable state for human detection, a notification to that effect is given.
The work machine periphery monitoring system according to any one of claims 1 to 5. Recording the captured image when it is determined that the captured image is unsuitable for human detection;
The work machine periphery monitoring system according to any one of claims 1 to 6.