JP2017102604A - Periphery monitoring system for work machine - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a periphery monitoring system capable of detecting a person in the periphery of a shovel car with a simpler system configuration.SOLUTION: A periphery monitoring system 100 for a work machine that detects a person being present in the periphery of a shovel car by using a captured image of an imaging apparatus 40 mounted on a shovel car includes: an extraction part 31 for extracting normalized images including a feature image representing a feature point of the person, from among a plurality of the normalized images generated by individually normalizing a plurality of predetermined image portions in the captured image; and an identification part 32 for identifying, by image recognition processing, whether an image included in identification processing target images extracted by the extraction part 31 is an image of the person or not.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a work machine periphery monitoring system for monitoring the periphery of a work machine.

  An image sensor and a thermopile array that senses heat, overlaps the imaging range and heat detection range, and only the human-like range indicated by the thermopile array output is limited to the face extraction range, which is unnecessary for image identification processing A human body detection device that reduces the amount of computation processing is known (see Patent Document 1).

JP 2006-059015 A

  However, the above-described apparatus is provided with an image sensor and a thermopile array, and it is necessary to accurately overlap the imaging range and the heat detection range, so that the system configuration becomes complicated.

  In view of the above, it is desired to provide a work machine periphery monitoring system that can detect a person around the work machine with a simpler system configuration.

  A work machine periphery monitoring system according to an embodiment of the present invention is a work machine periphery monitoring system that detects a person existing around the work machine using a captured image of an imaging device attached to the work machine, Of the plurality of normalized images generated by normalizing each of the plurality of predetermined image portions in the captured image, the normalized image including a feature image representing a human characteristic portion is extracted as an identification processing target image. An extraction unit; and an identification unit that identifies whether an image included in the identification processing target image extracted by the extraction unit is a human image by image recognition processing.

  By the above-mentioned means, a work machine periphery monitoring system capable of detecting a person around the work machine with a simpler system configuration is provided.

It is a side view of the shovel in which the periphery monitoring system which concerns on the Example of this invention is mounted. It is a functional block diagram which shows the structural example of a periphery monitoring system. It is an example of the captured image of a rear camera. It is the schematic which shows an example of the geometric relationship used when cut out the identification process target image from a captured image. It is a top view of real space behind an excavator. It is a figure which shows the flow of the process which produces | generates the normalization image from a captured image. It is a figure which shows the relationship between a captured image, an identification process target image area, and a normalized image. It is a figure which shows the relationship between an identification process target image area | region and an identification process inappropriate area. It is a figure which shows the example of a normalized image. It is a figure explaining the relationship between the back horizontal distance between the virtual plane area | region in real space and a back camera, and the magnitude | size of the head image part in a normalized image. It is the schematic which shows another example of the geometric relationship used when cut out the identification process target image from a captured image. It is a figure which shows an example of the characteristic image in a captured image. It is a side view of the shovel equipped with the illuminating device. It is a flowchart which shows the flow of an example of an image extraction process. It is a flowchart which shows the flow of another example of an image extraction process. It is a flowchart which shows the flow of another example of an image extraction process. It is a flowchart which shows the flow of another example of an image extraction process.

  FIG. 1 is a side view of an excavator as a construction machine on which a periphery monitoring system 100 according to an embodiment of the present invention is mounted. An upper swing body 3 is mounted on the lower traveling body 1 of the excavator via a swing mechanism 2. A boom 4 is attached to the upper swing body 3. An arm 5 is attached to the tip of the boom 4, and a bucket 6 is attached to the tip of the arm 5. The boom 4, the arm 5, and the bucket 6 constitute an excavation attachment, and are hydraulically driven by the boom cylinder 7, the arm cylinder 8, and the bucket cylinder 9, respectively. Further, the upper swing body 3 is provided with a cabin 10 and is mounted with a power source such as an engine. An imaging device 40 is attached to the upper part of the upper swing body 3. Specifically, the rear camera 40B, the left side camera 40L, and the right side camera 40R are attached to the rear end upper part, the left end upper part, and the right end upper part of the upper swing body 3. A controller 30 and an output device 50 are installed in the cabin 10.

  FIG. 2 is a functional block diagram illustrating a configuration example of the periphery monitoring system 100. The periphery monitoring system 100 mainly includes a controller 30, an imaging device 40, and an output device 50.

  The controller 30 is a control device that performs drive control of the shovel. In the present embodiment, the controller 30 is configured by an arithmetic processing unit including a CPU and an internal memory, and realizes various functions by causing the CPU to execute a drive control program stored in the internal memory.

  Further, the controller 30 determines whether there is a person around the shovel based on the outputs of the various devices, and controls the various devices according to the determination result. Specifically, the controller 30 receives the outputs of the imaging device 40 and the input device 42 and executes software programs corresponding to the extraction unit 31, the identification unit 32, the tracking unit 33, and the control unit 35, respectively. Then, according to the execution result, a control command is output to the machine control device 51 to execute drive control of the shovel, or various information is output from the output device 50. The controller 30 may be a control device dedicated to image processing.

  The imaging device 40 is a device that captures an image around the excavator, and outputs the captured image to the controller 30. In the present embodiment, the imaging device 40 is a wide camera that employs an imaging element such as a CCD, and is mounted on the upper part of the upper swing body 3 so that the optical axis is directed obliquely downward.

  The input device 42 is a device that receives input from the operator. In the present embodiment, the input device 42 includes an operation device (operation lever, operation pedal, etc.), a gate lock lever, a button installed at the tip of the operation device, a button attached to the in-vehicle display, a touch panel, and the like.

  The output device 50 is a device that outputs various types of information, and includes, for example, an in-vehicle display that displays various types of image information, an in-vehicle speaker that outputs various types of audio information as audio, an alarm buzzer, an alarm lamp, and the like. In the present embodiment, the output device 50 outputs various types of information in response to control commands from the controller 30.

  The machine control device 51 is a device that controls the movement of the excavator, and includes, for example, a control valve that controls the flow of hydraulic oil in the hydraulic system, a gate lock valve, an engine control device, and the like.

  The extraction unit 31 is a functional element that extracts an identification processing target image from a captured image captured by the imaging device 40. Specifically, the extraction unit 31 extracts simple features based on local luminance gradients or edges, geometric features based on Hough transform, features related to the area or aspect ratio of a region divided based on luminance, and the like. The image to be identified is extracted by image processing with a relatively small amount of computation (hereinafter referred to as “previous image recognition processing”). The identification processing target image is an image portion (a part of the captured image) that is a target of subsequent image processing, and includes a human candidate image. The human candidate image is an image portion (part of the captured image) that is considered to be a human image. The captured image may be a color image or a gray scale image. The extraction unit 31 may have a plurality of types of functions for converting a color image into a clay scale.

  The identification unit 32 is a functional element that identifies whether the human candidate image included in the identification processing target image extracted by the extraction unit 31 is a human image. Specifically, the identification unit 32 has a relatively large amount of calculation such as an image recognition process using an image feature description represented by HOG (Histograms of Oriented Gradients) feature and a classifier generated by machine learning. Image processing (hereinafter referred to as “rear-stage image recognition processing”) identifies whether the human candidate image is a human image. The rate at which the identification unit 32 identifies the human candidate image as a human image increases as the extraction processing target image is extracted by the extraction unit 31 with higher accuracy. Note that the identification unit 32 identifies and extracts all human candidate images as human images when, for example, a captured image having a desired quality cannot be obtained in an environment unsuitable for imaging at night or in bad weather. All of the human candidate images in the identification processing target image extracted by the unit 31 may be identified as people. This is to prevent human detection omissions.

  Next, with reference to FIG. 3, how the human image appears in the captured image behind the excavator captured by the rear camera 40B will be described. Note that the two captured images in FIG. 3 are examples of captured images of the rear camera 40B. 3 represents the presence of a human image, and is not displayed in an actual captured image.

  The rear camera 40B is a wide camera, and is attached at a height at which a person is looked down obliquely from above. For this reason, how the human image is seen in the captured image varies greatly depending on the direction in which the person is seen from the rear camera 40B. For example, the human image in the captured image is displayed so as to be inclined closer to the left and right ends of the captured image. This is due to image collapse caused by the wide-angle lens of the wide camera. Further, the closer to the rear camera 40B, the larger the head is displayed. In addition, the leg part enters the blind spot of the excavator's car body and disappears. These are caused by the installation position of the rear camera 40B. Therefore, it is difficult to identify a human image included in the captured image by image processing without performing any processing on the captured image.

  Therefore, the periphery monitoring system 100 according to the embodiment of the present invention promotes the identification of the human image included in the identification processing target image by normalizing the identification processing target image. Note that “normalization” means that the identification processing target image is converted into an image having a predetermined size and a predetermined shape. In this embodiment, an identification processing target image that can take various shapes in a captured image is converted into a rectangular image of a predetermined size by projective conversion. As the projective transformation, for example, an 8-variable projective transformation matrix is used.

  Here, an example of processing (hereinafter, referred to as “normalization processing”) in which the periphery monitoring system 100 normalizes the identification processing target image will be described with reference to FIGS. 4 to 6. FIG. 4 is a schematic diagram illustrating an example of a geometric relationship used when the extraction unit 31 cuts out an identification processing target image from a captured image.

  A box BX in FIG. 4 is a virtual three-dimensional object in real space, and is a virtual rectangular parallelepiped defined by eight vertices A to H in this embodiment. The point Pr is a reference point that is set in advance to refer to the identification processing target image. In the present embodiment, the reference point Pr is a point set in advance as an assumed standing position of a person, and is located at the center of a quadrilateral ABCD defined by four vertices A to D. The size of the box BX is set based on the direction of the person, the stride, the height, and the like. In this embodiment, the rectangle ABCD and the rectangle EFGH are squares, and the length of one side is, for example, 800 mm. Further, the height of the rectangular parallelepiped is, for example, 1800 mm. That is, the box BX is a rectangular parallelepiped having a width of 800 mm, a depth of 800 mm, and a height of 1800 mm.

  A quadrangle ABGH defined by the four vertices A, B, G, and H forms a virtual plane region TR corresponding to the region of the identification target image in the captured image. Further, the quadrangle ABGH as the virtual plane region TR is inclined with respect to the virtual ground that is a horizontal plane.

  In the present embodiment, a box BX as a virtual rectangular parallelepiped is employed to define the relationship between the reference point Pr and the virtual plane region TR. However, as long as the virtual plane region TR that faces the direction of the imaging device 40 and is inclined with respect to the virtual ground can be determined in association with an arbitrary reference point Pr, the relationship using other virtual three-dimensional objects, etc. May be adopted, and other mathematical relations such as a function and a conversion table may be adopted.

  FIG. 5 is a top view of the real space behind the shovel, and shows the positional relationship between the rear camera 40B and the virtual plane regions TR1 and TR2 when the virtual plane regions TR1 and TR2 are referenced using the reference points Pr1 and Pr2. Show. In this embodiment, the reference point Pr can be arranged at each of the lattice points of the virtual grid on the virtual ground. However, the reference points Pr may be irregularly arranged on the virtual ground, or may be arranged at equal intervals on a line segment radially extending from the projection point of the rear camera 40B on the virtual ground. For example, each line segment extends radially in increments of 1 degree, and the reference points Pr are arranged at intervals of 100 mm on each line segment.

  As shown in FIGS. 4 and 5, the first surface of the box BX defined by the quadrangle ABFE (see FIG. 4) is positive to the rear camera 40B when the virtual plane region TR1 is referenced using the reference point Pr1. It arranges so that it may be. That is, the line segment connecting the rear camera 40B and the reference point Pr1 is orthogonal to the first surface of the box BX arranged in association with the reference point Pr1 when viewed from above. Similarly, the first surface of the box BX is arranged to face the rear camera 40B even when the virtual plane region TR2 is referenced using the reference point Pr2. That is, a line segment connecting the rear camera 40B and the reference point Pr2 is orthogonal to the first surface of the box BX arranged in association with the reference point Pr2 when viewed from above. This relationship holds even when the reference point Pr is arranged on any lattice point. That is, the box BX is arranged so that the first surface thereof is always directly opposite the rear camera 40B.

  FIG. 6 is a diagram illustrating a flow of processing for generating a normalized image from a captured image. Specifically, FIG. 6A is an example of a captured image of the rear camera 40B, and displays a box BX arranged in association with the reference point Pr in real space. FIG. 6B is a diagram in which a region of the identification processing target image (hereinafter referred to as “identification processing target image region TRg”) in the captured image is cut out, and is displayed in the captured image of FIG. Corresponding to the virtual plane region TR. FIG. 6C shows a normalized image TRgt obtained by normalizing the identification processing target image having the identification processing target image region TRg.

  As shown in FIG. 6A, the box BX arranged in the real space in relation to the reference point Pr1 determines the position of the virtual plane region TR in the real space, and the imaging corresponding to the virtual plane region TR. An identification processing target image region TRg on the image is determined.

  Thus, if the position of the reference point Pr in the real space is determined, the position of the virtual plane region TR in the real space is uniquely determined, and the identification processing target image region TRg in the captured image is also uniquely determined. Then, the extraction unit 31 can generate a normalized image TRgt having a predetermined size by normalizing the identification processing target image having the identification processing target image region TRg. In the present embodiment, the size of the normalized image TRgt is, for example, 64 pixels long × 32 pixels wide.

  FIG. 7 is a diagram illustrating a relationship among a captured image, an identification processing target image region, and a normalized image. Specifically, FIG. 7A1 shows the identification processing target image region TRg3 in the captured image, and FIG. 7A2 shows the normalized image TRgt3 of the identification processing target image having the identification processing target image region TRg3. . FIG. 7B1 shows an identification processing target image region TRg4 in the captured image, and FIG. 7B2 shows a normalized image TRgt4 of the identification processing target image having the identification processing target image region TRg4. Similarly, FIG. 7C1 shows the identification processing target image region TRg5 in the captured image, and FIG. 7C2 shows the normalized image TRgt5 of the identification processing target image having the identification processing target image region TRg5.

  As shown in FIG. 7, the identification processing target image region TRg5 in the captured image is larger than the identification processing target image region TRg4 in the captured image. This is because the distance between the virtual plane area corresponding to the identification processing target image area TRg5 and the rear camera 40B is smaller than the distance between the virtual plane area corresponding to the identification processing target image area TRg4 and the rear camera 40B. Similarly, the identification processing target image region TRg4 in the captured image is larger than the identification processing target image region TRg3 in the captured image. This is because the distance between the virtual plane area corresponding to the identification processing target image area TRg4 and the rear camera 40B is smaller than the distance between the virtual plane area corresponding to the identification processing target image area TRg3 and the rear camera 40B. That is, the identification processing target image area in the captured image is smaller as the distance between the corresponding virtual plane area and the rear camera 40B is larger. On the other hand, the normalized images TRgt3, TRgt4, and TRgt5 are all rectangular images having the same size.

  In this manner, the extraction unit 31 can normalize the identification processing target image that can take various shapes and sizes in the captured image to a rectangular image of a predetermined size, and can normalize a human candidate image including a human image. Specifically, the extraction unit 31 arranges an image portion (hereinafter referred to as “head image portion”) that is estimated to be the head of the human candidate image in a predetermined region of the normalized image. In addition, an image portion (hereinafter, referred to as a “body portion image portion”) that is estimated to be the trunk portion of the human candidate image is arranged in another predetermined region of the normalized image, and yet another predetermined portion of the normalized image. An image portion (hereinafter referred to as a “leg image portion”) estimated to be a leg portion of the human candidate image is arranged in the region. Further, the extraction unit 31 can acquire the normalized image in a state where the inclination (image collapse) of the human candidate image with respect to the shape of the normalized image is suppressed.

  Next, referring to FIG. 8, normalization processing in a case where the identification processing target image area includes an image area that is not suitable for identification that adversely affects identification of human images (hereinafter referred to as “identification process inappropriate area”). Will be described. The identification processing inappropriate area is a known area where no human image can exist, for example, an area in which the excavator's vehicle body is reflected (hereinafter referred to as “vehicle body reflection area”), an area that protrudes from the captured image ( Hereinafter, it is referred to as “extrusion area”). FIG. 8 is a diagram illustrating the relationship between the identification processing target image area and the identification processing inappropriate area, and corresponds to FIG. 7 (C1) and FIG. 7 (C2). Further, the right-slanted hatched area in the left diagram of FIG. 8 corresponds to the protrusion area R1, and the left-slanted hatched area corresponds to the vehicle body reflection area R2.

  In the present embodiment, when the identification process target image region TRg5 includes a part of the protrusion region R1 and the vehicle body reflection region R2, the extraction unit 31 performs mask processing on these identification processing inappropriate regions, and then performs an identification processing target image. A normalized image TRgt5 of the identification processing target image having the region TRg5 is generated. Note that the extraction unit 31 may mask the portion corresponding to the identification processing inappropriate region in the normalized image TRgt5 after generating the normalized image TRgt5.

  The right figure of FIG. 8 shows normalized image TRgt5. Further, in the right diagram of FIG. 8, the diagonally hatched area that falls to the right represents the mask area M1 corresponding to the protruding area R1, and the diagonally hatched area that falls to the left represents the mask area M2 corresponding to a part of the vehicle body reflection area R2. Represents.

  In this way, the extraction unit 31 masks the image of the identification process inappropriate area, thereby preventing the image of the identification process inappropriate area from affecting the identification process performed by the identification unit 32. By this mask processing, the identification unit 32 can identify whether the image is a human image using an image of a region other than the mask region in the normalized image without being affected by the image of the identification processing inappropriate region. Note that the extraction unit 31 may use any known method other than the mask process so that the image in the identification process inappropriate region does not affect the identification process performed by the identification unit 32.

  Next, characteristics of the normalized image generated by the extraction unit 31 will be described with reference to FIG. FIG. 9 is a diagram illustrating an example of a normalized image. Further, the 14 normalized images shown in FIG. 9 include images of human candidates that are closer to the rear camera 40B as the normalized image is closer to the left end of the diagram, and the normalized image closer to the right end of the diagram is It includes images of human candidates that are located far from the rear camera 40B.

  As shown in FIG. 9, the extraction unit 31 performs any normalization regardless of the rear horizontal distance (the horizontal distance in the Y-axis direction shown in FIG. 5) between the virtual plane region TR and the rear camera 40B in the real space. In the image, the head image portion, the torso image portion, the leg image portion, and the like can be arranged at substantially the same ratio. Therefore, the extraction unit 31 can reduce the calculation load when the identification unit 32 executes the identification process, and can improve the reliability of the identification result. The rear horizontal distance described above is an example of information regarding the positional relationship between the virtual plane region TR and the rear camera 40B in the real space, and the extraction unit 31 adds the information to the extracted identification processing target image. . Further, the information on the positional relationship described above includes a top view angle with respect to the optical axis of the rear camera 40B of a line segment connecting the reference point Pr corresponding to the virtual plane region TR and the rear camera 40B.

  Next, the relationship between the rear horizontal distance between the virtual plane region TR and the rear camera 40B in the real space and the size of the head image portion in the normalized image will be described with reference to FIG. 10 shows the sizes L10, L11, and L12 of the head image portion when a person is present at three reference points Pr10, Pr11, and P12 that have different rear horizontal distances from the rear camera 40B. The horizontal axis corresponds to the rear horizontal distance. 10 is a graph showing the relationship between the rear horizontal distance and the size of the head image portion. The vertical axis corresponds to the size of the head image portion, and the horizontal axis corresponds to the rear horizontal distance. In addition, the horizontal axis of the upper figure of FIG. 10 and the lower figure of FIG. 10 is common. In this embodiment, the camera height is 2100 mm, the height of the center of the head HD from the ground is 1600 mm, and the head diameter is 250 mm.

  As shown in the upper diagram of FIG. 10, when a person is present at the position indicated by the reference point Pr10, the size L10 of the head image portion is the projected image of the head HD on the virtual plane region TR10 viewed from the rear camera 40B. Corresponds to the size of. Similarly, when a person is present at the positions indicated by the reference points Pr11 and Pr12, the sizes L11 and L12 of the head image portion are the projection images of the head HD on the virtual plane regions TR11 and TR12 viewed from the rear camera 40B. Corresponds to the size of. Note that the size of the head image portion in the normalized image varies with the size of the projection image.

  As shown in the lower diagram of FIG. 10, the size of the head image portion in the normalized image remains substantially the same when the rear horizontal distance is greater than or equal to D1 (eg, 700 mm), but the rear horizontal distance is less than D1. It suddenly increases.

  Therefore, the identification unit 32 changes the content of the identification process according to the rear horizontal distance. For example, when using a supervised learning (machine learning) technique, the identification unit 32 groups learning samples used in the identification process at a predetermined rear horizontal distance (for example, 650 mm). Specifically, the learning samples are divided into a short distance group and a long distance group. With this configuration, the identification unit 32 can identify the human image with higher accuracy.

  With the above configuration, the periphery monitoring system 100 generates the normalized image TRgt from the identification processing target image region TRg corresponding to the virtual plane region TR that faces the imaging device 40 and is inclined with respect to the virtual ground that is a horizontal plane. . Therefore, normalization in consideration of how the person looks in the height direction and depth direction can be realized. As a result, even when a captured image of the imaging device 40 attached to the construction machine so as to capture an image of a person from above is used, a person existing around the construction machine can be detected more reliably. In particular, even when a person approaches the imaging device 40, the normalized image can be generated from the identification processing target image that occupies a sufficiently large area on the captured image, so that the person can be reliably detected.

  In addition, the periphery monitoring system 100 defines the virtual plane region TR as a rectangular region formed by the four vertices A, B, G, and H of the box BX that is a virtual cuboid in real space. Therefore, the reference point Pr in the real space can be geometrically associated with the virtual plane region TR, and further, the virtual plane region TR in the real space and the identification processing target image region TRg in the captured image can be geometrically related. Can be associated.

  In addition, the extraction unit 31 performs mask processing on the image of the identification processing inappropriate area included in the identification processing target image area TRg. Therefore, the identification unit 32 can identify whether the image is a human image by using an image of a region other than the mask region in the normalized image without being affected by the image of the identification inappropriate region including the vehicle body reflection region R2.

  In addition, when the identification processing target image is extracted, the extraction unit 31 adds the rear horizontal distance between the two to the identification processing target image as information regarding the positional relationship between the virtual plane region TR and the imaging device 40. And the identification part 32 changes the content of the identification process according to the back horizontal distance. Specifically, the identification unit 32 groups learning samples used in the identification process at a predetermined rear horizontal distance (for example, 650 mm). With this configuration, the identification unit 32 can identify the human image with higher accuracy.

  Further, the extraction unit 31 can extract an identification processing target image for each reference point Pr. In addition, each of the identification processing target image areas TRg is associated with one of the reference points Pr set in advance as an assumed standing position of the person via the corresponding virtual plane area TR. Therefore, the periphery monitoring system 100 can extract an identification processing target image that is likely to include a human candidate image by extracting the reference point Pr that is likely to be present by an arbitrary method. In this case, it is possible to prevent the identification processing target image having a low possibility of including the human candidate image from being subjected to the identification processing by the image processing having a relatively large calculation amount, and to realize the speedup of the human detection processing. .

  Next, an example of processing in which the extraction unit 31 extracts an identification processing target image that is highly likely to include a human candidate image will be described with reference to FIGS. 11 and 12. FIG. 11 is a schematic diagram illustrating an example of a geometric relationship used when the extraction unit 31 cuts out an identification processing target image from a captured image, and corresponds to FIG. FIG. 12 is a diagram illustrating an example of a feature image in a captured image. Note that the feature image is an image that represents a characteristic part of a person, and is preferably an image that represents a part in which the height from the ground in real space is difficult to change. Therefore, the feature image includes, for example, an image of a helmet, an image of a shoulder, an image of a head, an image of a reflector or a marker attached to a person, and the like.

  The reflector or marker has, for example, a specific color, shape, pattern, and the like, and is configured to be attached to a specific position on the body such as the chest, shoulder, or back. When searching for an image of a reflector as a feature image, the extraction unit 31 uses, for example, the color, shape, pattern, and the like of a pixel group having a luminance equal to or higher than a predetermined value in the captured image as the color and shape of the reflector registered in advance. If the pattern matches the pattern, the pixel group is recognized as an image of the reflector. Then, the most relevant reference point Pr is derived based on the position and size of the image of the reflecting plate and information on the specific position of the body to which the reflecting plate is attached. The same applies when searching for a marker image as a feature image.

  The helmet has a feature that its shape is approximately a sphere, and when the projected image is projected onto the captured image, it is always nearly circular regardless of the imaging direction. In addition, the helmet has a hard surface and is glossy or semi-glossy, and when the projected image is projected on the captured image, a local high-intensity region and a radial luminance gradient centered on the region are generated. It has the feature of being easy. Therefore, the helmet image is particularly suitable as a feature image. The feature that the projected image is close to a circle, the feature that it is easy to generate a radial brightness gradient centered on a local high brightness area, etc. are used for image processing to find the helmet image from the captured image. May be. The image processing for finding the helmet image from the captured image includes, for example, luminance smoothing processing, Gaussian smoothing processing, luminance maximum point searching processing, luminance minimum point searching processing, and the like.

  Further, in order to stably bring out a feature that a radial gradation (luminance gradient) centered on a local high-luminance region is likely to occur in the captured image, even if an illumination device is installed near the imaging device 40 Good. In this case, the lighting device is attached on the lighting device 41 so as to illuminate the imaging region of the imaging device 40, for example.

  FIG. 13 is a side view of an excavator on which the illumination device 41 is mounted. Specifically, the rear light 41B, the left side light 41L, and the right side light 41R are mounted on the rear camera 40B, the left side camera 40L, and the right side camera 40R.

  With this configuration, the lighting device can further emphasize a radial luminance gradient centered on a local high-luminance region in a captured image formed by reflected light from the helmet. In addition, the lighting device can make a helmet with a color that is difficult to distinguish from the background color stand out, and can make the helmet stand out even in places where there is little ambient light, such as indoors and at night.

  In the present embodiment, the extraction unit 31 finds out a helmet image (an image that can be estimated to be a helmet strictly) in the captured image by the pre-stage image recognition process. This is because a person working around the excavator is considered to be wearing a helmet. Then, the extraction unit 31 derives the most relevant reference point Pr from the position of the found helmet image. Then, the extraction unit 31 extracts an identification processing target image corresponding to the reference point Pr.

  Specifically, the extraction unit 31 derives a highly relevant reference point Pr from the position of the helmet image in the captured image using the geometric relationship shown in FIG. The geometric relationship in FIG. 11 is different from the geometric relationship in FIG. 4 in that the virtual head position HP in the real space is determined, but is common in other points.

  The virtual head position HP represents the head position of a person assumed to be present on the reference point Pr, and is disposed immediately above the reference point Pr. In this embodiment, it is arranged at a height of 1700 mm on the reference point Pr. Therefore, if the virtual head position HP in the real space is determined, the position of the reference point Pr in the real space is uniquely determined, and the position of the virtual plane region TR in the real space is also uniquely determined. Further, the identification processing target image region TRg in the captured image is also uniquely determined. Then, the extraction unit 31 can generate a normalized image TRgt having a predetermined size by normalizing the identification processing target image having the identification processing target image region TRg.

  Conversely, if the position of the reference point Pr in the real space is determined, the virtual head position HP in the real space is uniquely determined, and the head image position AP on the captured image corresponding to the virtual head position HP in the real space is also unique. It is decided. Therefore, the head image position AP can be set in advance in association with each of the preset reference points Pr. The head image position AP may be derived from the reference point Pr in real time.

  Therefore, the extraction unit 31 searches for the helmet image in the captured image of the rear camera 40B by the pre-stage image recognition process. The upper part of FIG. 12 shows a state where the extraction unit 31 has found the helmet image HRg. And the extraction part 31 determines the representative position RP, when the helmet image HRg is found. The representative position RP is a position derived from the size, shape, etc. of the helmet image HRg. In this embodiment, the representative position RP is the position of the central pixel in the helmet image area including the helmet image HRg. The lower diagram in FIG. 12 is an enlarged view of the helmet image region, which is a rectangular image region separated by white lines in the upper diagram in FIG. 12, and shows that the position of the central pixel in the helmet image region is the representative position RP.

  Thereafter, the extraction unit 31 derives a head image position AP that is closest to the representative position RP using, for example, a nearest neighbor search algorithm. In the lower part of FIG. 12, six head image positions AP1 to AP6 are preset near the representative position RP, and the head image position AP5 is the head image position AP closest to the representative position RP. It shows that.

  Then, the extraction unit 31 uses the geometrical relationship shown in FIG. 11 to follow the virtual head position HP, the reference point Pr, and the virtual plane region TR from the nearest head image position AP derived. The identification processing target image region TRg to be extracted is extracted. Thereafter, the extraction unit 31 normalizes the identification processing target image having the extracted identification processing target image region TRg to generate a normalized image TRgt.

  In this way, the extraction unit 31 includes the representative position RP of the helmet image HRg that is the position of the human characteristic image in the captured image, and one of the preset head image positions AP (head image position AP5). Are associated with each other to extract an identification processing target image.

  Note that the extraction unit 31 directly associates the head image position AP with the reference point Pr, the virtual plane region TR, or the identification processing target image region TRg instead of using the geometrical relationship shown in FIG. An identification processing target image corresponding to the head image position AP may be extracted using a table.

  In addition, the extraction unit 31 may derive the reference point Pr from the representative position RP using a known algorithm other than the nearest neighbor search algorithm such as a hill-climbing method or a Mean-shift method. For example, when using the hill-climbing method, the extraction unit 31 derives a plurality of head image positions AP in the vicinity of the representative position RP, and the reference points Pr corresponding to the representative position RP and each of the plurality of head image positions AP. Is linked. At this time, the extraction unit 31 weights the reference point Pr so that the weight becomes larger as the representative position RP and the head image position AP are closer. Then, the weight distribution of the plurality of reference points Pr is climbed, and the identification processing target image region TRg is extracted from the reference point Pr having the weight closest to the maximum point of the weight.

  Next, an example of processing (hereinafter referred to as “image extraction processing”) in which the extraction unit 31 of the controller 30 extracts an identification processing target image will be described with reference to FIG. FIG. 14 is a flowchart showing an exemplary flow of image extraction processing.

  First, the extraction unit 31 searches for a helmet image in the captured image (step ST1). In the present embodiment, the extraction unit 31 performs a raster scan on the image captured by the rear camera 40B by the preceding image recognition process to find a helmet image.

  When the helmet image HRg is found from the captured image (YES in step ST1), the extraction unit 31 acquires the representative position RP of the helmet image HRg (step ST2).

  Thereafter, the extraction unit 31 acquires a head image position AP that is closest to the acquired representative position RP (step ST3).

  Thereafter, the extraction unit 31 extracts an identification processing target image corresponding to the acquired head image position AP (step ST4). In the present embodiment, the extraction unit 31 uses the geometrical relationship shown in FIG. 11 and uses the head image position AP in the captured image, the virtual head position HP in the real space, and the assumed standing position of the person in the real space. The identification processing target image is extracted by following the correspondence between the reference point Pr and the virtual plane region TR in the real space.

  If the helmet image HRg is not found in the captured image (NO in step ST1), the extraction unit 31 shifts the process to step ST5 without extracting the identification processing target image.

  Then, the extraction part 31 determines whether the helmet image was searched over the whole captured image (step ST5).

  If it is determined that the entire captured image has not yet been searched (NO in step ST5), the extraction unit 31 performs the processing in steps ST1 to ST4 on another region of the captured image.

  On the other hand, when it is determined that the search for the helmet image over the entire captured image has been completed (YES in step ST5), the extraction unit 31 ends the current image extraction process.

  In this way, the extraction unit 31 first finds the helmet image HRg, and from the representative position RP of the found helmet image HRg, the head image position AP, the virtual head position HP, the reference point (assumed standing position) Pr, the virtual An identification processing target image region TRg is specified through the plane region TR. Then, by extracting and normalizing the identification processing target image having the identified identification processing target image region TRg, a normalized image TRgt of a predetermined size can be generated.

  Next, another example of image extraction processing will be described with reference to FIG. FIG. 15 is a flowchart showing the flow of another example of image extraction processing.

  First, the extraction unit 31 acquires one of the head image positions AP (step ST11). Thereafter, the extraction unit 31 acquires a helmet image region corresponding to the head image position AP (step ST12). In the present embodiment, the helmet image area is an image area of a predetermined size that is preset for each of the head image positions AP.

  Thereafter, the extraction unit 31 searches for a helmet image within the helmet image region (step ST13). In the present embodiment, the extraction unit 31 raster scans the inside of the helmet image area by the preceding image recognition process and finds out the helmet image.

  When the helmet image HRg is found in the helmet image area (YES in step ST13), the extraction unit 31 extracts an identification processing target image corresponding to the head image position AP at that time (step ST14). In the present embodiment, the extraction unit 31 uses the geometrical relationship shown in FIG. 11 and uses the head image position AP in the captured image, the virtual head position HP in the real space, and the assumed standing position of the person in the real space. The identification processing target image is extracted by following the correspondence between the reference point Pr and the virtual plane region TR in the real space.

  If the helmet image HRg is not found in the helmet image area (NO in step ST13), the extraction unit 31 shifts the process to step ST15 without extracting the identification process target image.

  Thereafter, the extraction unit 31 determines whether all the head image positions AP have been acquired (step ST15). And when it determines with not having acquired all the head image positions AP yet (NO of step ST15), the extraction part 31 acquires another head image position AP which has not been acquired, and step ST11-step ST14 Execute the process. On the other hand, when it is determined that all the head image positions AP have been acquired (YES in step ST15), the extraction unit 31 ends the current image extraction process.

  Thus, when the extraction unit 31 first acquires one of the head image positions AP and finds the helmet image HRg in the helmet image area corresponding to the acquired head image position AP, the head at that time From the partial image position AP, the identification processing target image area TRg is specified through the virtual head position HP, the reference point (assumed standing position) Pr, and the virtual plane area TR. Then, by extracting and normalizing the identification processing target image having the identified identification processing target image region TRg, a normalized image TRgt of a predetermined size can be generated.

  With the above configuration, the extraction unit 31 of the periphery monitoring system 100 finds a helmet image as a feature image in the captured image, and associates the representative position RP of the helmet image with one of the head image positions AP as a predetermined image position. Thus, the identification processing target image is extracted. Therefore, it is possible to narrow down the image portion to be subjected to the subsequent image recognition processing with a simple system configuration.

  The extraction unit 31 first finds the helmet image HRg from the captured image, derives one of the head image positions AP corresponding to the representative position RP of the helmet image HRg, and sets it as one of the head image positions AP. A corresponding identification processing target image may be extracted. Alternatively, the extraction unit 31 first acquires one of the head image positions AP, and the helmet image is in a helmet image area that is a predetermined area including the position of the feature image corresponding to one of the head image positions AP. May exist, an identification processing target image corresponding to one of the head image positions AP may be extracted.

  Further, the extraction unit 31 may extract the identification processing target image from the representative position RP of the helmet image in the captured image using a predetermined geometrical relationship as illustrated in FIG. In this case, the predetermined geometric relationship is such that the identification processing target image region TRg in the captured image, the virtual plane region TR in the real space corresponding to the identification processing target image region TRg, and the real space corresponding to the virtual plane region TR. Reference point Pr (assumed standing position of a person), virtual head position HP corresponding to reference point Pr (virtual feature position that is a position in a real space of a characteristic portion of the person corresponding to the assumed standing position of the person), and The geometrical relationship with the head image position AP (predetermined image position in the captured image corresponding to the virtual feature position) in the captured image corresponding to the virtual head position HP is represented.

  Next, another example of the image extraction process will be described with reference to FIG. FIG. 16 is a flowchart showing the flow of still another example of the image extraction process. The image extraction process in FIG. 16 is the same as the image extraction process in FIG. 14 and FIG. 15 in that after the normalized image is generated, it is determined whether the normalized image includes a helmet image as a feature image. Different. The image extraction process in each of FIGS. 14 and 15 is for normalizing the image portion including the helmet image after finding the helmet image.

  In the present embodiment, the extraction unit 31 normalizes each of the plurality of predetermined image portions in the captured image to generate a plurality of normalized images, and among the normalized images, the normalized image including the helmet image is identified. Extract as an image. The plurality of predetermined image portions are, for example, a plurality of identification processing target image regions TRg predetermined on the captured image. The identification processing target image region TRg (see FIG. 6) corresponds to the virtual plane region TR in the real space, and the virtual plane region TR corresponds to the reference point Pr in the real space. Then, the identification unit 32 identifies whether the human candidate image included in the identification processing target image extracted by the extraction unit 31 is a human image.

  First, the extraction unit 31 generates a normalized image from one of the predetermined image portions (step ST21). In the present embodiment, one of the identification processing target image regions TRg corresponding to one of the preset reference points Pr in the captured image of the rear camera 40B is set as a predetermined image portion, and the predetermined image portion is converted into a predetermined size by projective transformation. Convert to a rectangular image.

  Thereafter, the extraction unit 31 determines whether or not a helmet image is included in the generated normalized image (step ST22). In the present embodiment, the extraction unit 31 raster scans the normalized image by the previous image recognition process to find a helmet image.

  If it is determined that the helmet image is included in the normalized image (YES in step ST22), the extraction unit 31 extracts the normalized image as an identification processing target image (step ST23). At this time, the identification unit 32 identifies whether the human candidate image included in the identification processing target image extracted by the extraction unit 31 is a human image. That is, the identification unit 32 determines whether the image included in the normalized image is a human image with respect to the normalized image determined to include the helmet image before the next normalization by the extraction unit 31 is performed. Identify. Therefore, if the periphery monitoring system 100 has a memory capacity capable of storing one normalized image, the periphery monitoring system 100 can execute image extraction processing and identification processing on the entire captured image. However, after the extraction unit 31 extracts a plurality of normalized images as identification processing target images, the identification unit 32 identifies whether the human candidate image included in each of the plurality of identification processing target images is a human image. May be.

  When it is determined that the helmet image is not included in the normalized image (NO in step ST22), the extraction unit 31 proceeds with the process without extracting the normalized image as the identification processing target image.

  Thereafter, the extraction unit 31 determines whether or not a normalized image has been generated from all the predetermined image portions (step ST24). In the present embodiment, it is determined whether or not a normalized image has been generated from all of the predetermined image portions that are the identification processing target image regions TRg corresponding to the respective reference points Pr in the captured image of the rear camera 40B.

  When it is determined that the normalized image has not been generated from all the predetermined image portions (NO in step ST24), the extraction unit 31 performs the processes in steps ST21 to ST23 on another predetermined image portion.

  On the other hand, when it is determined that the normalized image has been generated from all the predetermined image portions (YES in step ST24), the extraction unit 31 ends the current image extraction process.

  In the example of FIG. 16, the periphery monitoring system 100 determines whether or not a helmet image is included in the normalized image when one normalized image is generated. However, at the stage where a plurality of normalized images are generated, whether or not a helmet image is included in each of the plurality of normalized images may be collectively determined. Further, it may be determined collectively whether or not a helmet image is included in each of all the normalized images at a stage where all the normalized images are generated.

  By such an image extraction process, the periphery monitoring system 100 can reduce the number of memory accesses during the human detection process. This is because the process of finding out the helmet image and the process of extracting the identification process target image can be executed simultaneously.

  Further, the periphery monitoring system 100 that employs the image extraction process of FIG. 16 can reduce the memory capacity compared to the case of employing the image extraction process of FIG. 14 or FIG. The image extraction process of FIG. 14 or 15 requires at least a memory capacity that can store an image part before normalization (an image larger than the normalized image), but the image extraction process of FIG. 16 is normalized. This is because the image can be executed if there is enough memory capacity to store the normalized image in order to generate the image first.

  Further, the periphery monitoring system 100 that employs the image extraction process of FIG. 16 determines whether or not the helmet image is included in the normalized image. Therefore, compared with the configuration for determining whether or not a helmet image is included in an image before normalization as in the image extraction process of FIG. 14 or FIG. 15, processing required for determining whether or not a helmet image is included. You can save time. This is because variations in the size and distortion of the helmet image to be searched can be reduced. As a result, the time required for extracting the identification processing target image can be shortened. Moreover, since the variation in the size and distortion of the helmet image to be searched can be reduced, the accuracy of the determination as to whether or not the helmet image is included can be increased.

  In addition, the periphery monitoring system 100 that employs the image extraction process of FIG. 16 has a large variation in the head position with respect to the position of the person's foot (for example, when the image of a crooked person is included in the captured image). , More reliable human detection can be realized. This is because a configuration for determining whether or not the helmet image is included in the normalized image is employed. That is, it is not a configuration for specifying the identification processing target image area using the head image position related to the position of the helmet image as in the image extraction processing of FIG. 14 or FIG. 15, but as a predetermined image portion to be normalized. This is because the image processing target image area is set regardless of the head image position.

  Next, still another example of the image extraction process will be described with reference to FIG. FIG. 17 is a flowchart showing the flow of still another example of the image extraction process. In the image extraction process of FIG. 17, a helmet image as a feature image is added to a partially normalized image (hereinafter referred to as “partial normalized image”) at a stage where a part of a predetermined image portion is normalized. It differs from the image extraction process of FIG. 16 in that it is determined whether or not it is included. The image extraction process in FIG. 16 is for determining whether or not a helmet image is included in the normalized image when all of one predetermined image portion is normalized.

  First, the extraction unit 31 generates a partially normalized image from a part of the predetermined image portion (step ST31). And the extraction part 31 determines whether a helmet image is contained in the partial normalization image in the step which produced | generated the partial normalization image (step ST32). In the present embodiment, the extraction unit 31 rasterizes the partially normalized image by the pre-stage image recognition process at the stage where the partially normalized image is generated by normalizing a part of the predetermined image portion corresponding to the upper half of the normalized image. Scan to find the helmet image.

  If it is determined that the helmet image is included in the partially normalized image (YES in step ST32), the extraction unit 31 normalizes the remaining portion of the predetermined image portion to generate a normalized image (step ST33). In the present embodiment, the extraction unit 31 normalizes the remaining portion of the predetermined image portion corresponding to the lower half of the normalized image to generate a normalized image. Then, the extraction unit 31 extracts the normalized image as an identification processing target image (step ST34). At this time, the identification unit 32 identifies whether the human candidate image included in the identification processing target image extracted by the extraction unit 31 is a human image. However, after the extraction unit 31 extracts a plurality of normalized images as identification processing target images, the identification unit 32 identifies whether the human candidate image included in each of the plurality of identification processing target images is a human image. May be.

  When it is determined that the helmet image is not included in the partially normalized image (NO in step ST32), the extraction unit 31 proceeds with the process without generating a normalized image by normalizing the remaining portion of the predetermined image portion.

  Thereafter, the extraction unit 31 determines whether or not a partially normalized image has been generated from all the predetermined image portions (step ST35). In the present embodiment, it is determined whether or not a partial normalized image has been generated from all of the predetermined image portions that are the identification processing target image regions TRg corresponding to the respective reference points Pr in the captured image of the rear camera 40B.

  When it is determined that the partially normalized image has not been generated from all the predetermined image portions (NO in step ST35), the extraction unit 31 performs the processes in steps ST31 to ST34 on another predetermined image portion.

  On the other hand, when it is determined that a partially normalized image has been generated from all the predetermined image portions (YES in step ST35), the extraction unit 31 ends the current image extraction process.

  In the example of FIG. 17, the periphery monitoring system 100 generates a partially normalized image by normalizing a part of a predetermined image portion corresponding to the upper half of the normalized image. This is because the helmet image has a high probability of being included in the upper half of the normalized image (the portion corresponding to the upper body of a person). However, a partially normalized image may be generated by normalizing a part of a predetermined image portion corresponding to a portion smaller than the upper half of the normalized image. For example, a partially normalized image may be generated by normalizing a part of a predetermined image portion corresponding to a portion corresponding to 1/3 of the normalized image. Alternatively, the periphery monitoring system 100 may generate a partially normalized image by normalizing another part of the normalized image. For example, when searching for a shoulder image as a feature image, a part of a predetermined image portion corresponding to the right half of the normalized image may be normalized to generate a partially normalized image. When searching for an image, a partially normalized image may be generated by normalizing a part of a predetermined image portion corresponding to the central half of the normalized image.

  With such an image extraction process, the periphery monitoring system 100 that employs the image extraction process of FIG. 17 can achieve the same effects as when the image extraction process of FIG. 16 is employed.

  In addition, the periphery monitoring system 100 that employs the image extraction process of FIG. 17 can further reduce the processing time required for the human detection process as compared to the case of employing the image extraction process of FIG. This is because, when it is determined that the helmet image is not included in the partial normalized image, normalization of the next predetermined image portion can be started without normalizing the remaining portion of the predetermined image portion.

  Here, referring to FIG. 2 again, the description of the other functional elements of the controller 30 is continued.

  The tracking unit 33 is a functional element that tracks the identification result output by the identification unit 32 every predetermined time and outputs the final human detection result. In the present embodiment, the tracking unit 33 determines that the corresponding person candidate image is a human image when the identification results regarding the same person for a predetermined number of consecutive times satisfy a predetermined condition. That is, it is determined that a person exists at the corresponding three-dimensional position (actual position). Whether or not they are the same person is determined based on their actual positions. Specifically, the tracking unit 33 recognizes the person within a predetermined time based on the actual position (reference point PrI) of the person shown in the image identified as the person image in the first identification process by the identification unit 32. Derive the reachable range. The reachable range is set based on the maximum turning speed of the shovel, the maximum traveling speed of the shovel, the maximum movement speed of the person, and the like. Then, if the real position (reference point PrII) of the person shown in the image identified as the human image in the second identification process is within the range, it is determined that the person is the same person. The same applies to the third and subsequent identification processes. And the tracking part 33 determines with a person existing in a corresponding three-dimensional position, for example, when it identifies that it is a person's image of the same person in 4 times among 6 continuous identification results. In addition, even if a person image is identified in the first identification process, if a person image of the same person is not identified in the subsequent three identification processes, the corresponding three-dimensional It is determined that there is no person at the position.

  As described above, the combination of the extraction unit 31, the identification unit 32, and the tracking unit 33 constitutes a human detection unit 34 that detects whether or not there is a person around the shovel based on the captured image of the imaging device 40. .

  With this configuration, the person detection unit 34 makes false reports (determining that a person is present despite the absence of a person), misreporting (determining that a person is present despite the presence of a person), etc. Can be suppressed.

  Further, the person detection unit 34 can determine whether the person is approaching or moving away from the shovel based on the transition of the actual position of the person shown in the image identified as the person image. And the person detection part 34 may output a control command to the control part 35, and may output a warning, when the distance from the shovel of the person's real position is less than predetermined value. In this case, the person detection unit 34 may adjust the predetermined value according to the excavator operation information (for example, turning speed, turning direction, traveling speed, traveling direction, etc.).

  Further, the human detection unit 34 may discriminate and recognize at least two stages of the human detection state and the human non-detection state. For example, a state in which at least one of a condition regarding distance and a condition regarding reliability is satisfied is determined as a first person detection state (warning state), and a state where both are satisfied is determined as a second person detection state (alarm). State). The condition regarding the distance includes, for example, that the distance from the excavator at the actual position of the person shown in the image identified as the human image is less than a predetermined value. The condition relating to reliability includes, for example, identification of a person image of the same person in four out of six consecutive identification results. In the first person detection state (warning state), a first alarm is output as a preliminary alarm with low accuracy but quick response. The first alarm is, for example, a low-volume beep sound, and is automatically stopped when neither of the two conditions is satisfied. In the second person detection state (alarm state), a second alarm is output as a formal alarm with high accuracy but slow response. The second alarm is, for example, a loud melody sound, and is not automatically stopped even when at least one of the conditions is not satisfied, and an operator's operation is required for the stop.

  The control unit 35 is a functional element that controls various devices. In the present embodiment, the control unit 35 controls various devices in accordance with an operator input via the input device 42. For example, the display image displayed on the screen of the in-vehicle display is switched according to an image switching command input through the touch panel. The display image includes a through image of the rear camera 40B, a through image of the right side camera 40R, a through image of the left side camera 40L, a viewpoint conversion image, and the like. The viewpoint conversion image is, for example, a bird's-eye view image (image viewed from a virtual viewpoint directly above the shovel) synthesized from images captured by a plurality of cameras.

  Further, the control unit 35 controls various devices according to the final human detection result of the tracking unit 33 constituting the human detection unit 34. For example, a control command is output to the machine control device 51 according to the final human detection result of the tracking unit 33 to switch the shovel state between the first state and the second state. The first state includes a state where the restriction of the shovel movement is released, a state where the alarm output is stopped, and the like. The second state includes a state where movement of the excavator is limited or stopped, a state where an alarm is output, and the like. In this embodiment, the control unit 35 outputs a control command to the machine control device 51 when it is determined that a person is present within a predetermined range around the shovel based on the final human detection result of the tracking unit 33. The shovel state is switched from the first state to the second state. For example, the excavator stops moving. In this case, the operation by the operator is invalidated. Specifically, a control command is output to the gate lock valve to disconnect the operating device from the hydraulic system, thereby forcibly creating a no-operation state and stopping the excavator movement. Alternatively, the engine may be stopped by outputting a control command to the engine control device. Alternatively, the movement of the hydraulic actuator may be limited by outputting a control command to a control valve that controls the flow rate of the hydraulic oil flowing into the hydraulic actuator to change the opening area of the control valve, the opening area change rate, and the like. In this case, the maximum turning speed, the maximum traveling speed, etc. are reduced.

  In addition, the control unit 35 returns the shovel state to the first state when a predetermined release condition is satisfied after the shovel state is set to the second state. That is, when a predetermined release condition is satisfied after restricting or stopping the shovel movement, the restriction or stop is released. The predetermined release condition includes, for example, “determining that there is no person within a predetermined range around the excavator” (hereinafter referred to as “first release condition”). Further, the predetermined release condition additionally includes, for example, “a state in which the excavator does not start” is secured (hereinafter referred to as “second release condition”). Further, the predetermined release condition may include “the operator confirming that there is no person in the vicinity of the excavator” (hereinafter referred to as “third release condition”). In the present embodiment, whether or not the excavator's movement is restricted or stopped, and whether or not each of the first release condition, the second release condition, and the third release condition is satisfied is managed using a flag. The

  The first release condition is, for example, that “the control unit 35 determines that there is no person within a predetermined range around the shovel based on the final human detection result of the tracking unit 33 configuring the human detection unit 34”. Including.

  The second release condition is, for example, “all operation devices are in a neutral position for a predetermined time or more”, “the gate lock lever is lowered (the operation device is invalid)”, This includes “an operator's limbs being released from all operating devices”, “a predetermined button operation has been performed”, and the like. “All the operation devices are in the neutral position” is detected by the control unit 35 based on, for example, the presence / absence of a command from each operation device, the output value of a sensor that detects the operation amount of each operation device, and the like. . The condition “over a predetermined time or more” has an effect of preventing the second release condition from being satisfied only by instantaneously becoming the neutral position. “The operator's limbs are separated from the operating device” is based on, for example, a captured image of a camera that images the driver's cab, an output of an electrostatic sensor attached to the operating device (for example, a grip of the operating lever), or the like. The control unit 35 detects this. “The predetermined button operation has been performed” means, for example, that a confirmation button (for example, a horn button or the same) is displayed in a state where a message such as “Are the excavator not moving? When the software button (displayed on the screen) is pressed, the control unit 35 detects.

  The third release condition is satisfied, for example, when the confirmation button is pressed while a message such as “Did you confirm that there is no person around the shovel?” Is displayed on the screen of the in-vehicle display. Note that the third release condition may be omitted.

  In a case where the third release condition is included in the predetermined release condition, the excavator enters the restriction release enabled state when the first release condition and the second release condition are satisfied. The restriction release enabled state means a state in which the restriction can be released as long as the operator confirms that there is no person around the excavator.

  There is no restriction on the order in which each of the first release condition, the second release condition, and the third release condition is satisfied. For example, even when the conditions are satisfied in the order of the third release condition, the second release condition, and the first release condition, the control unit 35 releases the restriction or stop of the shovel movement.

  Moreover, the control part 35 may cancel the restriction | limiting or a stop, when a predetermined | prescribed waiting time passes after a predetermined cancellation | release condition is satisfied. This is to prevent the operator from being panicked by sudden release.

  Further, when the movement of the excavator is restricted or stopped, the control unit 35 outputs a control command to the in-vehicle display as the output device 50, and displays a captured image including the human image that caused the cause. Good. For example, when a human image is included only in the captured image of the left side camera 40L, the through image of the left side camera 40L may be displayed alone. Alternatively, when a human image is included in each of the captured image of the left camera 40L and the captured image of the rear camera 40B, the through images of the two cameras may be displayed side by side, and the captured images of the two cameras may be displayed simultaneously. One synthesized image (for example, a viewpoint conversion image) may be displayed. Further, an image indicating that the device is being restricted or stopped, guidance on a release method, and the like may be displayed. In addition, an image portion corresponding to a human candidate image identified as a human image may be highlighted. For example, the outline of the identification processing target image region TRg may be displayed in a predetermined color. In addition, when a waiting time after the predetermined release condition is satisfied is set, the wait time countdown may be displayed after displaying that fact when the predetermined release condition is satisfied. Good. Further, when an alarm is output during the waiting time, the volume of the alarm may be gradually reduced as the waiting time elapses.

  Further, when the movement of the excavator is restricted or stopped, the control unit 35 outputs a control command to the in-vehicle speaker as the output device 50, and outputs a warning on the side where the person who caused the output exists. Good. In this case, the in-vehicle speaker includes, for example, a right side speaker installed on the right wall in the cab, a left side speaker installed on the left wall, and a rear speaker installed on the rear wall. Then, when a human image is included only in the captured image of the left side camera 40L, the control unit 35 outputs an alarm from only the left side speaker. Alternatively, the control unit 35 may localize the sound using a surround system including a plurality of speakers.

  In addition, when the human detection unit 34 identifies the human candidate image as a human image, the control unit 35 may output only an alarm without limiting or stopping the excavator movement. Also in this case, the control unit 35 determines that a state in which at least one of the condition regarding distance and the condition regarding reliability is satisfied as described above is the first person detection state (warning state), and the state in which both are satisfied is determined. The second person detection state (alarm state) may be determined. And the control part 35 may stop the warning in a 2nd person detection state (alarm state), when predetermined cancellation | release conditions are satisfy | filled similarly to the case where the movement of an excavator is restrict | limited or stopped. . This is because, unlike the alarm in the first person detection state (warning state) that can be automatically stopped, the operation of the operator is required to stop the alarm in the second person detection state (warning state). .

  Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the scope of the present invention. Can be added.

  For example, in the above-described embodiment, it is assumed that a person is detected using a captured image of the imaging device 40 mounted on the upper swing body 3 of the excavator, but the present invention is not limited to this configuration. . The present invention can also be applied to a configuration that uses a captured image of an imaging device that is attached to the main body of another work machine such as a mobile crane, a fixed crane, a riff mag machine, or a forklift.

  In the above-described embodiment, the excavator blind spot area is imaged using three cameras, but the excavator blind spot area may be imaged using one, two, or four or more cameras.

  In the above-described embodiment, the human detection process is individually applied to each of the plurality of captured images. However, the human detection process is applied to one composite image generated from the plurality of captured images. Also good.

  DESCRIPTION OF SYMBOLS 1 ... Lower traveling body 2 ... Turning mechanism 3 ... Upper turning body 4 ... Boom 5 ... Arm 6 ... Bucket 7 ... Boom cylinder 8 ... Arm cylinder 9 ... Bucket cylinder 10: cabin 30 ... controller 31 ... extraction unit 32 ... identification unit 33 ... tracking unit 34 ... human detection unit 35 ... control unit 40 ... imaging device 40B: Rear camera 40L ... Left side camera 40R ... Right side camera 41 ... Illumination device 41B ... Rear light 41L ... Left side light 41R ... Right side light 42 ... Input device 50 ... Output device 51 ... Machine control device 100 ... Ambient monitoring system AP, AP1-AP6 ... Head image position BX ... Box HD ... Head HP -Virtual head position HRg ... Helmet image M1, M2 ... Mask region Pr, Pr1, Pr2, Pr10-Pr12 ... Reference point R1 ... Projection region R2 ... Vehicle body reflection region RP Representative positions TR, TR1, TR2, TR10 to TR12 ... virtual plane region TRg, TRg3, TRg4, TRg5 ... identification processing target image region TRgt, TRgt3, TRgt4, TRgt5 ... normalized image

Claims (6)

A work machine periphery monitoring system that detects a person existing around the work machine using a captured image of an image pickup device attached to the work machine,
Of the plurality of normalized images generated by normalizing each of the plurality of predetermined image portions in the captured image, the normalized image including a feature image representing a human characteristic portion is extracted as an identification processing target image. An extractor;
An identification unit that identifies whether an image included in the identification target image extracted by the extraction unit is a human image by image recognition processing,
Perimeter monitoring system for work machines. The extraction unit determines whether or not the feature image is included at a stage where a predetermined part of the predetermined image portion is normalized,
The work machine periphery monitoring system according to claim 1. The extraction unit determines whether or not the feature image is included in each normalized image at the stage of normalizing one or a plurality of the predetermined image portions,
Whether the image included in the normalized image is a human image with respect to the normalized image determined to include the feature image before the next normalization by the extraction unit is performed. Identify the
The work machine periphery monitoring system according to claim 1. The feature image is an image of a helmet, an image of a shoulder, an image of a head, an image of a reflector, or an image of a marker.
The work machine periphery monitoring system according to any one of claims 1 to 3. An illumination device for illuminating the imaging region of the imaging device;
The feature image is an image of a helmet illuminated by the lighting device.
The work machine periphery monitoring system according to any one of claims 1 to 3. The predetermined image portion is an area set in advance on the captured image, and the area corresponds to a virtual plane area associated with a reference point set in advance as an assumed standing position of a person in real space.
The work machine periphery monitoring system according to any one of claims 1 to 5.