JP2021095839A - Shovel - Google Patents

Abstract

To provide a shovel capable of appropriately changing a state according to detection of a monitoring object.SOLUTION: A shovel includes: a person detection part 34 for detecting a monitoring object present in a peripheral prescribed range; and a controller 30 having a control part 35 for switching a state of the shovel to a first state or a second state based on a detection result of the person detection part 34. The first state includes a state where restriction of movement of the shovel is released, and the second state includes a state where the movement of the shovel is restricted. When the state of the shovel is switched from the first state to the second state and then a prescribed condition is satisfied, the controller 30 returns the state of the shovel to the first state. The predetermined condition includes non-detection of the monitoring object in the prescribed range, and securing of a state where the shovel is not moved.SELECTED DRAWING: Figure 1

Description

The present invention relates to excavators.

A shovel equipped with a sensor for detecting an object (person) existing around the shovel is known (see Patent Document 1). This excavator outputs an alarm from a speaker installed on the right wall of the driver's cab when an object (person) is detected on the right side of the excavator, and displays a through image of the camera that captures the right side of the excavator on the display. .. Further, when an object (person) is detected on the left side of the excavator, an alarm is output from a speaker installed on the left wall of the driver's cab, and a through image of a camera that captures the left side of the excavator is displayed on the display.

Japanese Unexamined Patent Publication No. 2014-183500

However, Patent Document 1 does not mention a change in the state of the excavator when the sensor detects a monitored object such as an object (person).

In view of the above, it is desired to provide a shovel that can appropriately change the state according to the detection of the monitored object.

The excavator according to the embodiment of the present invention is an excavator including a traveling body, a swivel body rotatably mounted on the traveling body, and a work attachment provided on the swivel body, and the swivel body includes the traveling body. A detection unit that is provided and detects a monitoring target existing in a predetermined range around the excavator, and a control unit that switches the state of the excavator to a first state or a second state based on the detection result of the detection unit. The first state includes a state in which the movement of the excavator is restricted, the second state includes a state in which the movement of the excavator is restricted, and the control unit controls the excavator. When a predetermined condition is satisfied after switching the state of the above from the first state to the second state, the state of the excavator is returned to the first state, and the predetermined condition is the said within the predetermined range. This includes non-detection of the monitoring target and ensuring that the excavator does not start moving.

By the above-mentioned means, a shovel capable of appropriately changing the state according to the detection of the monitored object is provided.

It is a side view of the excavator which mounts the peripheral monitoring system which concerns on embodiment of this invention. It is a functional block diagram which shows the configuration example of the peripheral monitoring system. This is an example of an image captured by a rear camera. It is a schematic diagram which shows an example of the geometric relation used when cutting out the identification processing target image from the captured image. It is a top view of the real space behind the excavator. It is a figure which shows the flow of the process which generates the normalized image from the captured image. It is a figure which shows the relationship between the captured image, the image area subject to identification processing, and a normalized image. It is a figure which shows the relationship between the image area which is the object of identification processing, and the area which is unsuitable for identification processing. It is a figure which shows the example of the normalized image. It is a schematic diagram which shows another example of the geometric relation used when cutting out the identification processing target image from the captured image. It is a figure which shows an example of the feature image in the captured image. It is a flowchart which shows the flow of an example of an image extraction process. It is a flowchart which shows the flow of an example of peripheral monitoring processing. It is a flowchart which shows the flow of an example of the restriction release process. It is a figure which shows the example of the output image. It is a correspondence table which shows the correspondence relation between the detection state and the display color of a frame and an area. This is an example of a viewpoint conversion image as an output image. This is an example of an output image including a viewpoint conversion image.

FIG. 1 is a side view of an excavator as a construction machine on which the peripheral monitoring system 100 according to the embodiment of the present invention is mounted. The upper swivel body 3 is mounted on the lower traveling body 1 of the excavator via the swivel mechanism 2. A boom 4 is attached to the upper swing body 3. An arm 5 is attached to the tip of the boom 4, and a bucket 6 is attached to the tip of the arm 5. The boom 4, arm 5, and bucket 6 form an excavation attachment, and are hydraulically driven by the boom cylinder 7, arm cylinder 8, and bucket cylinder 9, respectively. Further, the upper swing body 3 is provided with a cabin 10 and is equipped with a power source such as an engine. Further, an image pickup device 40 is attached to the upper part of the upper swing body 3. Specifically, the rear camera 40B, the left side camera 40L, and the right side camera 40R are attached to the upper rear end, the upper left end, and the upper right end of the upper swing body 3. Further, a controller 30 and an output device 50 are installed in the cabin 10.

FIG. 2 is a functional block diagram showing a configuration example of the peripheral monitoring system 100. The peripheral monitoring system 100 mainly includes a controller 30, an image pickup device 40, and an output device 50.

The controller 30 is a control device that controls the drive of the excavator. In this embodiment, the controller 30 is composed of a CPU and an arithmetic processing device including an internal memory, and causes the CPU to execute a drive control program stored in the internal memory to realize various functions.

Further, the controller 30 determines whether or not a person exists in the vicinity of the excavator based on the outputs of the various devices, and controls the various devices according to the determination result. Specifically, the controller 30 receives the outputs of the image pickup device 40 and the input device 41, and executes software programs corresponding to each of the extraction unit 31, the identification unit 32, the tracking unit 33, and the control unit 35. Then, a control command is output to the machine control device 51 to execute the excavator drive control according to the execution result, or various information is output from the output device 50. The controller 30 may be a control device dedicated to image processing.

The image pickup device 40 is a device that captures an image of the surroundings of the excavator, and outputs the captured image to the controller 30. In this embodiment, the image pickup device 40 is a wide camera that employs an image pickup device such as a CCD, and is attached so that the optical axis faces diagonally downward at the upper part of the upper swing body 3.

The input device 41 is a device that receives input from the operator. In this embodiment, the input device 41 includes an operation device (operation lever, operation pedal, etc.), a gate lock lever, a button installed at the tip of the operation device, a button attached to an in-vehicle display, a touch panel, and the like.

The output device 50 is a device that outputs various information, and includes, for example, an in-vehicle display that displays various image information, an in-vehicle speaker that outputs various audio information by voice, an alarm buzzer, an alarm lamp, and the like. In this embodiment, the output device 50 outputs various information in response to a control command from the controller 30.

The mechanical control device 51 is a device that controls the movement of the excavator, and includes, for example, a control valve, a gate lock valve, an engine control device, and the like that control the flow of hydraulic oil in a hydraulic system.

The extraction unit 31 is a functional element that extracts an image to be identified for identification processing from an image captured by the image pickup device 40. Specifically, the extraction unit 31 extracts simple features based on a local luminance gradient or edge, geometric features by Hough transform, etc., features related to the area or aspect ratio of a region divided based on luminance, and the like. The image to be identified is extracted by image processing with a relatively small amount of calculation (hereinafter referred to as "pre-stage image recognition processing"). The identification processing target image is an image portion (a part of the captured image) that is the target of subsequent image processing, and includes a person candidate image. The human candidate image is an image portion (a part of the captured image) that is likely to be a human image.

The identification unit 32 is a functional element that identifies whether the person candidate image included in the identification processing target image extracted by the extraction unit 31 is a human image. Specifically, the identification unit 32 has a relatively large amount of calculation such as an image recognition process using an image feature description represented by a HOG (Histograms of Oriented Gradients) feature and a classifier generated by machine learning. It is identified whether the person candidate image is a person image by image processing (hereinafter, referred to as "post-stage image recognition processing"). The rate at which the identification unit 32 identifies the human candidate image as a human image increases as the extraction unit 31 extracts the identification processing target image with higher accuracy. The identification unit 32 identifies and extracts all of the human candidate images as human images when an image of desired quality cannot be obtained in an environment unsuitable for imaging such as at night or in bad weather. All of the person candidate images in the identification processing target image extracted by the unit 31 may be identified as a person. This is to prevent human detection omission.

Next, with reference to FIG. 3, how the human image looks in the captured image behind the excavator captured by the rear camera 40B will be described. The two captured images in FIG. 3 are examples of captured images of the rear camera 40B. Further, the dotted line circle in FIG. 3 represents the existence of a human image and is not displayed in the actual captured image.

The rear camera 40B is a wide camera and is mounted at a height at which a person is viewed from diagonally above. Therefore, the appearance of the human image in the captured image greatly differs depending on the direction in which the person exists as seen from the rear camera 40B. For example, a human image in a captured image is displayed so as to be closer to the left and right edges of the captured image. This is due to image collapse caused by the wide-angle lens of the wide-angle camera. Further, the closer to the rear camera 40B, the larger the head is displayed. In addition, the legs enter the blind spot of the excavator's body and become invisible. These are due to the installation position of the rear camera 40B. Therefore, it is difficult to identify the human image included in the captured image by image processing without performing any processing on the captured image.

Therefore, the peripheral monitoring system 100 according to the embodiment of the present invention promotes the identification of the human image included in the identification processing target image by normalizing the identification processing target image. In addition, "normalization" means converting the image to be identified for identification processing into an image having a predetermined size and a predetermined shape. In this embodiment, the identification processing target image that can take various shapes in the captured image is converted into a rectangular image of a predetermined size by the projective transformation. As the projective transformation, for example, an 8-variable projective transformation matrix is used.

Here, with reference to FIGS. 4 to 6, an example of a process in which the peripheral monitoring system 100 normalizes the image to be identified (hereinafter referred to as “normalization process”) will be described. Note that FIG. 4 is a schematic view showing an example of the geometric relationship used when the extraction unit 31 cuts out the identification processing target image from the captured image.

The box BX in FIG. 4 is a virtual three-dimensional object in real space, and in this embodiment, it is a virtual rectangular parallelepiped defined by eight vertices A to H. Further, the point Pr is a reference point set in advance for referring to the image to be identified for identification processing. In this embodiment, the reference point Pr is a point preset as an assumed standing position of a person, and is located at the center of the quadrangle ABCD defined by the four vertices A to D. The size of the box BX is set based on the direction of the person, the stride length, the height, and the like. In this embodiment, the quadrangle ABCD and the quadrangle EFGH are square, and the length of one side is, for example, 800 mm. The height of the rectangular parallelepiped is, for example, 1800 mm. That is, the box BX is a rectangular parallelepiped having a width of 800 mm, a depth of 800 mm, and a height of 1800 mm.

The quadrangle ABGH defined by the four vertices A, B, G, and H forms a virtual plane region TR corresponding to the region of the identification processing target image in the captured image. Further, the quadrangle ABGH as the virtual plane region TR is inclined with respect to the virtual ground which is a horizontal plane.

In this embodiment, the box BX as a virtual rectangular parallelepiped is adopted to determine the relationship between the reference point Pr and the virtual plane region TR. However, if the virtual plane region TR that faces the direction of the image pickup apparatus 40 and is inclined with respect to the virtual ground can be defined in association with an arbitrary reference point Pr, other relationships such as using other virtual three-dimensional objects, etc. Geometric relations may be adopted, and other mathematical relations such as functions and conversion tables may be adopted.

FIG. 5 is a top view of the real space behind the excavator, and shows the positional relationship between the rear camera 40B and the virtual plane regions TR1 and TR2 when the virtual plane regions TR1 and TR2 are referred to using the reference points Pr1 and Pr2. Shown. In this embodiment, the reference point Pr can be arranged at each of the grid points of the virtual grid on the virtual ground. However, the reference points Pr may be irregularly arranged on the virtual ground, or may be arranged at equal intervals on a line segment extending radially from the projection point of the rear camera 40B on the virtual ground. For example, each line segment extends radially in 1 degree increments, and reference points Pr are arranged on each line segment at 100 mm intervals.

As shown in FIGS. 4 and 5, the first surface of the box BX defined by the quadrangle ABFE (see FIG. 4) is positive to the rear camera 40B when the virtual plane region TR1 is referenced using the reference point Pr1. Arranged to face each other. That is, the line segment connecting the rear camera 40B and the reference point Pr1 is orthogonal to the first surface of the box BX arranged in relation to the reference point Pr1 in the top view. Similarly, the first surface of the box BX is arranged so as to face the rear camera 40B even when the virtual plane region TR2 is referred to by using the reference point Pr2. That is, the line segment connecting the rear camera 40B and the reference point Pr2 is orthogonal to the first surface of the box BX arranged in relation to the reference point Pr2 in the top view. This relationship holds regardless of which grid point the reference point Pr is placed on. That is, the box BX is arranged so that its first surface always faces the rear camera 40B.

FIG. 6 is a diagram showing a flow of processing for generating a normalized image from a captured image. Specifically, FIG. 6A is an example of the captured image of the rear camera 40B, and projects the box BX arranged in relation to the reference point Pr in the real space. Further, FIG. 6B is a diagram obtained by cutting out a region of the identification processing target image in the captured image (hereinafter, referred to as “identification processing target image region TRg”), and is projected on the captured image of FIG. 6A. Corresponds to the virtual plane area TR. Further, FIG. 6C shows a normalized image TRgt in which the identification processing target image having the identification processing target image region TRg is normalized.

As shown in FIG. 6 (A), the box BX arranged in relation to the reference point Pr1 in the real space determines the position of the virtual plane region TR in the real space, and the imaging corresponding to the virtual plane region TR is performed. The image area TRg to be identified on the image is determined.

In this way, if the position of the reference point Pr in the real space is determined, the position of the virtual plane region TR in the real space is uniquely determined, and the identification processing target image region TRg in the captured image is also uniquely determined. Then, the extraction unit 31 can normalize the identification processing target image having the identification processing target image region TRg to generate a normalized image TRgt of a predetermined size. In this embodiment, the size of the normalized image TRgt is, for example, 64 pixels in length × 32 pixels in width.

FIG. 7 is a diagram showing the relationship between the captured image, the image area subject to identification processing, and the normalized image. Specifically, FIG. 7 (A1) shows the identification processing target image region TRg3 in the captured image, and FIG. 7 (A2) shows the normalized image TRgt3 of the identification processing target image having the identification processing target image region TRg3. .. Further, FIG. 7 (B1) shows the identification processing target image region TRg4 in the captured image, and FIG. 7 (B2) shows the normalized image TRgt4 of the identification processing target image having the identification processing target image region TRg4. Similarly, FIG. 7 (C1) shows the identification processing target image region TRg5 in the captured image, and FIG. 7 (C2) shows the normalized image TRgt5 of the identification processing target image having the identification processing target image region TRg5.

As shown in FIG. 7, the identification processing target image region TRg5 in the captured image is larger than the identification processing target image region TRg4 in the captured image. This is because the distance between the virtual plane area corresponding to the identification processing target image area TRg5 and the rear camera 40B is smaller than the distance between the virtual plane area corresponding to the identification processing target image area TRg4 and the rear camera 40B. Similarly, the identification processing target image region TRg4 in the captured image is larger than the identification processing target image region TRg3 in the captured image. This is because the distance between the virtual plane area corresponding to the identification processing target image area TRg4 and the rear camera 40B is smaller than the distance between the virtual plane area corresponding to the identification processing target image area TRg3 and the rear camera 40B. That is, the identification processing target image area in the captured image is smaller as the distance between the corresponding virtual plane area and the rear camera 40B is larger. On the other hand, the normalized images TRgt3, TRgt4, and TRgt5 are all rectangular images of the same size.

In this way, the extraction unit 31 can normalize the identification processing target image that can take various shapes and sizes in the captured image to a rectangular image of a predetermined size, and normalize the human candidate image including the human image. Specifically, the extraction unit 31 arranges an image portion (hereinafter, referred to as “head image portion”) presumed to be the head of the human candidate image in a predetermined region of the normalized image. Further, an image portion presumed to be the body portion of the human candidate image (hereinafter referred to as “body portion image portion”) is arranged in another predetermined region of the normalized image, and another predetermined region of the normalized image is provided. An image portion (hereinafter, referred to as “leg image portion”) that is presumed to be the leg portion of the person candidate image is arranged in the region. Further, the extraction unit 31 can acquire the normalized image in a state where the inclination (image collapse) of the person candidate image with respect to the shape of the normalized image is suppressed.

Next, with reference to FIG. 8, normalization processing is performed when the image area to be identified includes an image area unsuitable for identification that adversely affects the identification of human images (hereinafter, referred to as “identification processing unsuitable area”). Will be described. The area unsuitable for the identification process is a known area in which a human image cannot exist. For example, an area in which the vehicle body of the excavator is reflected (hereinafter, referred to as a “vehicle body reflection area”) and an area protruding from the captured image (the area beyond the captured image). Hereinafter, it is referred to as a “protruding area”) and the like. Note that FIG. 8 is a diagram showing the relationship between the image region subject to identification processing and the region unsuitable for identification processing, and corresponds to FIGS. 7 (C1) and 7 (C2). Further, the downward-sloping diagonal hatching area in FIG. 8 left corresponds to the protruding area R1, and the downward-sloping diagonal hatching area corresponds to the vehicle body reflection area R2.

In this embodiment, when the identification processing target image area TRg5 includes a part of the protruding area R1 and the vehicle body reflection area R2, the extraction unit 31 masks the areas unsuitable for the identification processing and then the identification processing target image. A normalized image TRgt5 of the identification processing target image having the region TRg5 is generated. After generating the normalized image TRgt5, the extraction unit 31 may mask the portion corresponding to the region unsuitable for the identification process in the normalized image TRgt5.

The right figure of FIG. 8 shows the normalized image TRgt5. Further, in the right figure of FIG. 8, the downward-sloping diagonal hatching area represents the mask area M1 corresponding to the protruding area R1, and the downward-sloping diagonal hatching area is the mask area M2 corresponding to a part of the vehicle body reflection area R2. Represents.

In this way, the extraction unit 31 masks the image of the region unsuitable for the identification process to prevent the image of the region unsuitable for the identification process from affecting the identification process by the identification unit 32. By this mask processing, the identification unit 32 can identify whether the image is a human image by using the image of the region other than the mask region in the normalized image without being affected by the image of the region unsuitable for the identification processing. The extraction unit 31 may use any known method other than the mask processing so that the image of the region unsuitable for the identification process does not affect the identification process by the identification unit 32.

Next, with reference to FIG. 9, the features of the normalized image generated by the extraction unit 31 will be described. Note that FIG. 9 is a diagram showing an example of a normalized image. Further, in the 14 normalized images shown in FIG. 9, the normalized image closer to the left end of the figure includes the image of the person candidate existing at a position closer to the rear camera 40B, and the normalized image closer to the right end of the figure. Includes an image of a candidate person existing at a position far from the rear camera 40B.

As shown in FIG. 9, the extraction unit 31 performs any normalization regardless of the rear horizontal distance (horizontal distance in the Y-axis direction shown in FIG. 5) between the virtual plane region TR and the rear camera 40B in the real space. In the image, the head image portion, the body portion image portion, the leg portion image portion, and the like can be arranged at almost the same ratio. Therefore, the extraction unit 31 can reduce the calculation load when the identification unit 32 executes the identification process, and can improve the reliability of the identification result. The above-mentioned rear horizontal distance is an example of information regarding the positional relationship between the virtual plane region TR and the rear camera 40B in the real space, and the extraction unit 31 adds the information to the extracted identification processing target image. .. Further, the above-mentioned information on the positional relationship includes a top view angle with respect to the optical axis of the rear camera 40B of the line segment connecting the reference point Pr corresponding to the virtual plane region TR and the rear camera 40B.

With the above configuration, the peripheral monitoring system 100 generates a normalized image TRgt from the identification processing target image region TRg corresponding to the virtual plane region TR that faces the direction of the image pickup apparatus 40 and is inclined with respect to the virtual ground that is a horizontal plane. .. Therefore, it is possible to realize normalization in consideration of how a person looks in the height direction and the depth direction. As a result, even when the captured image of the image pickup device 40 attached to the construction machine so as to image the person from diagonally above is used, the person existing around the construction machine can be detected more reliably. In particular, even when a person approaches the image pickup device 40, the normalized image can be generated from the identification processing target image that occupies a sufficiently large area on the captured image, so that the person can be reliably detected.

Further, the peripheral monitoring system 100 defines the virtual plane area TR as a rectangular area formed by the four vertices A, B, G, and H of the box BX, which is a virtual rectangular parallelepiped in the real space. Therefore, the reference point Pr in the real space and the virtual plane region TR can be geometrically associated with each other, and further, the virtual plane region TR in the real space and the identification processing target image region TRg in the captured image can be geometrically associated with each other. Can be associated.

In addition, the extraction unit 31 masks the image of the identification processing unsuitable region included in the identification processing target image region TRg. Therefore, the identification unit 32 can identify whether the image is a human image by using an image of a region other than the mask region in the normalized image without being affected by the image of the region unsuitable for the identification process including the vehicle body reflection region R2.

In addition, the extraction unit 31 can extract the identification processing target image for each reference point Pr. Further, each of the identification processing target image regions TRg is associated with one of the reference points Pr preset as the assumed standing position of the person via the corresponding virtual plane region TR. Therefore, the peripheral monitoring system 100 can extract the identification processing target image that is likely to include the person candidate image by extracting the reference point Pr that is likely to have a person by an arbitrary method. In this case, it is possible to prevent the identification processing target image, which is unlikely to include the person candidate image, from being subjected to the identification processing by the image processing having a relatively large amount of calculation, and it is possible to realize the speeding up of the person detection processing. ..

Next, with reference to FIGS. 10 and 11, an example of a process in which the extraction unit 31 extracts an image to be identified, which is likely to include a human candidate image, will be described. Note that FIG. 10 is a schematic view showing an example of the geometric relationship used when the extraction unit 31 cuts out the identification processing target image from the captured image, and corresponds to FIG. 4. Further, FIG. 11 is a diagram showing an example of a feature image in the captured image. The feature image is an image showing a characteristic part of a person, and preferably an image showing a part where the height from the ground in the real space is hard to change. Therefore, the feature image includes, for example, an image of a helmet, an image of a shoulder, an image of a head, an image of a reflector or a marker attached to a person, and the like.

In particular, the helmet has a characteristic that its shape is approximately spherical, and when the projected image is projected on the captured image, it is always close to a circle regardless of the imaging direction. In addition, the helmet has a hard surface and is glossy or semi-glossy, and when the projected image is projected on the captured image, a local high-luminance region and a radial brightness gradient centered on the region are generated. It has the feature of being easy. Therefore, the helmet image is particularly suitable as a feature image. The feature that the projected image is close to a circle, the feature that a radial luminance gradient centered on a local high-luminance region is likely to be generated, and the like are used for image processing to find an image of a helmet from an captured image. You may. Further, the image processing for finding the image of the helmet from the captured image includes, for example, a luminance smoothing process, a Gaussian smoothing process, a luminance maximum point search process, a luminance minimum point search process, and the like.

In this embodiment, the extraction unit 31 finds a helmet image (strictly speaking, an image that can be presumed to be a helmet) in the captured image by the pre-stage image recognition process. This is because people working around the excavator are considered to be wearing helmets. Then, the extraction unit 31 derives the most relevant reference point Pr from the position of the found helmet image. Then, the extraction unit 31 extracts the identification processing target image corresponding to the reference point Pr.

Specifically, the extraction unit 31 uses the geometric relationship shown in FIG. 10 to derive a highly relevant reference point Pr from the position of the helmet image in the captured image. The geometrical relationship of FIG. 10 is different from the geometrical relationship of FIG. 4 in that it determines the virtual head position HP in the real space, but is common in other respects.

The virtual head position HP represents the head position of a person who is assumed to exist on the reference point Pr, and is arranged directly above the reference point Pr. In this embodiment, it is arranged at a height of 1700 mm on the reference point Pr. Therefore, if the virtual head position HP in the real space is determined, the position of the reference point Pr in the real space is uniquely determined, and the position of the virtual plane region TR in the real space is also uniquely determined. Further, the identification processing target image area TRg in the captured image is also uniquely determined. Then, the extraction unit 31 can normalize the identification processing target image having the identification processing target image region TRg to generate a normalized image TRgt of a predetermined size.

On the contrary, if the position of the reference point Pr in the real space is determined, the virtual head position HP in the real space is uniquely determined, and the head image position AP on the captured image corresponding to the virtual head position HP in the real space is also unique. It is decided to. Therefore, the head image position AP can be preset in association with each of the preset reference points Pr. The head image position AP may be derived in real time from the reference point Pr.

Therefore, the extraction unit 31 searches for a helmet image in the captured image of the rear camera 40B by the pre-stage image recognition process. The upper figure of FIG. 11 shows a state in which the extraction unit 31 has found the helmet image HRg. Then, when the extraction unit 31 finds the helmet image HRg, the extraction unit 31 determines the representative position RP. The representative position RP is a position derived from the size, shape, etc. of the helmet image HRg. In this embodiment, the representative position RP is the position of the central pixel of the helmet image region including the helmet image HRg. The lower figure of FIG. 11 is an enlarged view of the helmet image area which is a rectangular image area separated by a white line in the upper figure of FIG. 11, and shows that the position of the center pixel of the helmet image area is the representative position RP.

After that, the extraction unit 31 derives the head image position AP closest to the representative position RP by using, for example, the nearest neighbor search algorithm. In the lower figure of FIG. 11, six head image positions AP1 to AP6 are preset near the representative position RP, and the head image position AP5 is the head image position AP closest to the representative position RP. Show that.

Then, the extraction unit 31 traces the virtual head position HP, the reference point Pr, and the virtual plane region TR from the derived nearest head image position AP by using the geometric relationship shown in FIG. The image area TRg to be identified for identification processing is extracted. After that, the extraction unit 31 normalizes the identification processing target image having the extracted identification processing target image region TRg to generate a normalized image TRgt.

In this way, the extraction unit 31 includes the representative position RP of the helmet image HRg, which is the position of the characteristic image of the person in the captured image, and one of the preset head image position APs (head image position AP5). The image to be identified is extracted by associating with.

Instead of using the geometric relationship shown in FIG. 10, the extraction unit 31 directly associates the head image position AP with the reference point Pr, the virtual plane region TR, or the identification processing target image region TRg. The identification processing target image corresponding to the head image position AP may be extracted by using the table.

Further, the extraction unit 31 may derive the reference point Pr from the representative position RP by using a known algorithm other than the nearest neighbor search algorithm such as the mountain climbing method and the Mean-shift method. For example, when the mountain climbing method is used, the extraction unit 31 derives a plurality of head image position APs in the vicinity of the representative position RP, and the reference point Pr corresponding to each of the representative position RP and the plurality of head image position APs. And link. At this time, the extraction unit 31 weights the reference point Pr so that the closer the representative position RP and the head image position AP are, the larger the weight is. Then, the weight distribution of the plurality of reference points Pr is climbed, and the identification processing target image region TRg is extracted from the reference points Pr having the weight closest to the maximum weight point.

Next, with reference to FIG. 12, an example of a process in which the extraction unit 31 of the controller 30 extracts the image to be identified (hereinafter referred to as “image extraction process”) will be described. Note that FIG. 12 is a flowchart showing a flow of an example of the image extraction process.

First, the extraction unit 31 searches for a helmet image in the captured image (step ST1). In this embodiment, the extraction unit 31 raster-scans the captured image of the rear camera 40B by the pre-stage image recognition process to find the helmet image.

When the helmet image HRg is found in the captured image (YES in step ST1), the extraction unit 31 acquires the representative position RP of the helmet image HRg (step ST2).

After that, the extraction unit 31 acquires the head image position AP closest to the acquired representative position RP (step ST3).

After that, the extraction unit 31 extracts the identification processing target image corresponding to the acquired head image position AP (step ST4). In this embodiment, the extraction unit 31 utilizes the geometric relationship shown in FIG. 10 as a head image position AP in the captured image, a virtual head position HP in the real space, and an assumed standing position of a person in the real space. The identification processing target image is extracted by tracing the correspondence between the reference point Pr and the virtual plane region TR in the real space.

If the extraction unit 31 does not find the helmet image HRg in the captured image (NO in step ST1), the extraction unit 31 shifts the processing to step ST5 without extracting the identification processing target image.

After that, the extraction unit 31 determines whether the helmet image has been searched for the entire captured image (step ST5).

When it is determined that the entire captured image has not been searched yet (NO in step ST5), the extraction unit 31 executes the processes of steps ST1 to ST4 for another region of the captured image.

On the other hand, when it is determined that the search for the helmet image over the entire captured image is completed (YES in step ST5), the extraction unit 31 ends the image extraction process this time.

In this way, the extraction unit 31 first finds the helmet image HRg, and from the representative position RP of the found helmet image HRg, the head image position AP, the virtual head position HP, the reference point (assumed standing position) Pr, and the virtual The image region TRg to be identified is specified via the plane region TR. Then, by extracting and normalizing the identification processing target image having the specified identification processing target image region TRg, a normalized image TRgt of a predetermined size can be generated.

With the above configuration, the extraction unit 31 of the peripheral monitoring system 100 finds a helmet image as a feature image in the captured image, and associates the representative position RP of the helmet image with one of the head image position APs as a predetermined image position. By doing so, the image to be identified is extracted. Therefore, it is possible to narrow down the image portion to be the target of the subsequent image recognition processing with a simple system configuration.

The extraction unit 31 first finds the helmet image HRg from the captured image, derives one of the head image position APs corresponding to the representative position RP of the helmet image HRg, and sets it as one of the head image position APs. The corresponding identification processing target image may be extracted. Alternatively, the extraction unit 31 first acquires one of the head image position APs, and the helmet image is within the helmet image area which is a predetermined area including the position of the feature image corresponding to one of the head image position APs. If is present, the identification processing target image corresponding to one of the head image position APs may be extracted.

Further, the extraction unit 31 may extract the identification processing target image from the representative position RP of the helmet image in the captured image by utilizing a predetermined geometric relationship as shown in FIG. In this case, the predetermined geometric relationship is the identification processing target image region TRg in the captured image, the virtual plane region TR in the real space corresponding to the identification processing target image region TRg, and the real space corresponding to the virtual plane region TR. Reference point Pr (assumed standing position of a person) and virtual head position HP corresponding to reference point Pr (virtual feature position which is a position in real space of a characteristic part of a person corresponding to a person's assumed standing position) , Represents the geometric relationship with the head image position AP (predetermined image position in the captured image corresponding to the virtual feature position) in the captured image corresponding to the virtual head position HP.

Here, with reference to FIG. 2 again, the description of the other functional elements of the controller 30 will be continued.

The tracking unit 33 is a functional element that tracks the identification result output by the identification unit 32 at predetermined time intervals and outputs the final human detection result. In this embodiment, the tracking unit 33 determines that the corresponding person candidate image is a person image when the identification result for the same person for a predetermined number of consecutive times satisfies the predetermined condition. That is, it is determined that a person exists at the corresponding three-dimensional position (existing position). Whether or not they are the same person is determined based on their actual position. Specifically, the tracking unit 33 allows the person to appear within a predetermined time based on the actual position (reference point PrI) of the person appearing in the image identified as a human image in the first identification process by the identification unit 32. Derive the reachable range. The reachable range is set based on the maximum turning speed of the excavator, the maximum traveling speed of the excavator, the maximum moving speed of a person, and the like. Then, if the actual position (reference point PrII) of the person appearing in the image identified as the human image in the second identification process is within that range, it is determined that the person is the same person. The same applies to the third and subsequent identification processes. Then, the tracking unit 33 determines that a person exists at the corresponding three-dimensional position when, for example, it is identified as a human image of the same person four times out of the six consecutive identification results. Further, even if a human image is identified in the first identification process, if the same person's image is not identified in the subsequent three consecutive identification processes, the corresponding three-dimensional image is used. It is determined that there is no person at the position.

In this way, the combination of the extraction unit 31, the identification unit 32, and the tracking unit 33 constitutes a person detection unit 34 that detects whether or not a person exists in the vicinity of the excavator based on the image captured by the image pickup device 40. ..

With this configuration, the person detection unit 34 misreports (determines that a person exists even though there is no person), misinformation (determines that a person does not exist even though there is a person), and the like. Can be suppressed.

In addition, the human detection unit 34 can determine whether the person is approaching the shovel or away from the shovel based on the transition of the actual position of the person appearing in the image identified as the human image. Then, the person detection unit 34 may output a control command to the control unit 35 to output an alarm when the distance from the shovel at the actual position of the person is less than a predetermined value. In this case, the person detection unit 34 may adjust a predetermined value according to the operation information of the excavator (for example, turning speed, turning direction, running speed, running direction, etc.).

Further, the human detection unit 34 may discriminate and recognize at least two stages of the human detection state and the non-human detection state. For example, a state in which at least one of the conditions related to distance and the condition related to reliability is satisfied is determined as a first person detection state (warning state), and a state in which both are satisfied is a second person detection state (alarm). State) may be judged. The condition regarding the distance includes, for example, that the distance from the excavator of the actual position of the person appearing in the image identified as the human image is less than a predetermined value. The reliability condition includes, for example, being identified as a human image of the same person in 4 out of 6 consecutive identification results. In the first person detection state (warning state), the first alarm is output as a preliminary alarm with low accuracy but quick response. The first alarm is, for example, a low-volume beep, which is automatically stopped when neither of the two conditions is satisfied. In the second person detection state (alarm state), the second alarm is output as a formal alarm with high accuracy but slow response. The second alarm is, for example, a loud melody sound, and is not automatically stopped even if at least one of the conditions is not satisfied, and an operator's operation is required to stop the second alarm.

The control unit 35 is a functional element that controls various devices. In this embodiment, the control unit 35 controls various devices according to the input of the operator via the input device 41. For example, the display image displayed on the screen of the in-vehicle display is switched according to the image switching command input through the touch panel. The display image includes a through image of the rear camera 40B, a through image of the right side camera 40R, a through image of the left side camera 40L, a viewpoint conversion image, and the like. The viewpoint conversion image is, for example, a bird's-eye view image (an image viewed from a virtual viewpoint directly above the excavator) synthesized from images captured by a plurality of cameras.

Further, the control unit 35 controls various devices according to the final human detection result of the tracking unit 33 constituting the human detection unit 34. For example, a control command is output to the machine control device 51 according to the final human detection result of the tracking unit 33, and the state of the excavator is switched between the first state and the second state. The first state includes a state in which the restriction on the movement of the excavator is released, a state in which the alarm output is stopped, and the like. The second state includes a state in which the movement of the excavator is restricted or stopped, a state in which an alarm is output, and the like. In this embodiment, the control unit 35 outputs a control command to the machine control device 51 when it is determined that a person exists within a predetermined range around the excavator based on the final human detection result of the tracking unit 33. The state of the excavator is switched from the first state to the second state. For example, stop the movement of the excavator. In this case, the operation by the operator is invalidated. The invalidation of the operation by the operator is realized, for example, by making the operating device unresponsive. Specifically, a control command is output to the gate lock valve to disconnect the operating device from the hydraulic system to forcibly create a non-operating state and stop the movement of the excavator. Alternatively, a control command may be output to the engine control device to stop the engine. Alternatively, the movement of the hydraulic actuator may be restricted by outputting a control command to the control valve that controls the flow rate of the hydraulic oil flowing into the hydraulic actuator to change the opening area of the control valve, the opening area change speed, and the like. In this case, the maximum turning speed, the maximum traveling speed, and the like are reduced. Further, the movement of the hydraulic actuator may be stopped by closing the control valve.

Further, the control unit 35 returns the shovel state to the first state when a predetermined release condition is satisfied after the shovel state is set to the second state. That is, when the predetermined release condition is satisfied after restricting or stopping the movement of the excavator, the restriction or stop is released. The predetermined release condition includes, for example, "determining that no person exists within a predetermined range around the excavator" (hereinafter, referred to as "first release condition"). In addition, the predetermined release condition additionally includes, for example, "a state in which the excavator does not start moving is secured" (hereinafter, referred to as "second release condition"). Further, the predetermined release condition may include "the operator has confirmed that there is no person around the excavator" (hereinafter, referred to as "third release condition"). In this embodiment, whether or not the movement of the excavator is restricted or stopped, and whether or not each of the first release condition, the second release condition, and the third release condition are satisfied are managed by using a flag. To.

The first release condition is, for example, "determining that the control unit 35 determines that no person exists within a predetermined range around the excavator based on the final person detection result of the tracking unit 33 constituting the person detection unit 34". Including.

The second release condition is, for example, "all the operating devices are in the neutral position for a predetermined time or longer", "the gate lock lever is lowered (the operating devices are invalid)", and so on. This includes "the operator's limbs are separated from all the operating devices", "a predetermined release operation has been performed", and the like. "All operating devices are in the neutral position" is detected by the control unit 35 based on, for example, the presence / absence of a command from each operating device, the output value of the sensor that detects the operating amount of each operating device, and the like. .. The condition "over a predetermined time" has an effect of preventing the second release condition from being satisfied only by momentarily entering the neutral position. "The operator's limbs are separated from the operating device" is based on, for example, an image captured by a camera that images the driver's cab, an output of an electrostatic sensor attached to the operating device (for example, a grip of an operating lever), and the like. The control unit 35 detects it. "The predetermined release operation has been performed" is confirmed by a confirmation button (for example, a horn button or the same) with a message such as "Is the state where the excavator does not start moving?" Is displayed on the screen of the in-vehicle display. The control unit 35 detects when the software button (software button displayed on the screen) is pressed. The control unit 35 may determine that "a state in which the excavator does not start is secured" when a release operation such as an operation input for a lever, a button, a panel, etc. in the driver's seat is performed by the operator. ..

The third release condition is satisfied, for example, when the confirmation button is pressed while a message such as "Did you confirm that there are no people around the excavator?" Is displayed on the screen of the vehicle-mounted display. The third release condition may be omitted.

When the predetermined release condition includes the third release condition, the excavator is in the restricted release state when the first release condition and the second release condition are satisfied. The state in which the restriction can be released means a state in which the restriction can be released as long as the operator confirms that there are no people around the excavator.

There is no limit to the order in which each of the first cancellation condition, the second cancellation condition, and the third cancellation condition is satisfied. For example, even when the conditions are satisfied in the order of the third release condition, the second release condition, and the first release condition, the control unit 35 releases the restriction or stop of the movement of the excavator.

Further, the control unit 35 may release the restriction or stop when a predetermined waiting time elapses after the predetermined release condition is satisfied. This is to prevent the operator from being confused by a sudden release.

Further, when the movement of the excavator is restricted or stopped, the control unit 35 may output a control command to the in-vehicle display as the output device 50 and display an captured image including the human image that caused the control command. Good. For example, when a human image is included only in the captured image of the left side camera 40L, the through image of the left side camera 40L may be displayed independently. Alternatively, when the captured image of the left side camera 40L and the captured image of the rear camera 40B each include a human image, the through images of the two cameras may be displayed side by side at the same time. One composite image including the image (for example, a viewpoint conversion image) may be displayed. In addition, an image indicating that the restriction or stoppage is in progress, guidance on how to release the image, or the like may be displayed. Further, the image portion corresponding to the person candidate image identified as the person image may be highlighted. For example, the outline of the image area TRg to be identified may be displayed in a predetermined color. Further, when the waiting time after the predetermined cancellation condition is satisfied is set, the operator may be notified that the waiting time exists when the predetermined cancellation condition is satisfied. For example, after displaying that there is a waiting time, the waiting time countdown may be displayed. If an alarm is output during the waiting time, the volume of the alarm may be gradually reduced as the waiting time elapses.

Further, when the movement of the excavator is restricted or stopped, the control unit 35 may output a control command to the vehicle-mounted speaker as the output device 50 and output an alarm on the side where the person who caused the excavator exists. Good. In this case, the vehicle-mounted speaker is composed of, for example, a right-side speaker installed on the right wall of the driver's cab, a left-side speaker installed on the left wall, and a rear speaker installed on the rear wall. Then, when the human image is included only in the captured image of the left side camera 40L, the control unit 35 outputs an alarm only from the left side speaker. Alternatively, the control unit 35 may localize the sound using a surround system including a plurality of speakers.

Further, when the person detection unit 34 identifies the person candidate image as a person image, the control unit 35 may output only an alarm without limiting or stopping the movement of the excavator. In this case as well, the control unit 35 determines that at least one of the distance-related conditions and the reliability-related conditions is satisfied as the first person detection state (warning state), and determines that both are satisfied. It may be determined that the second person is in the detection state (alarm state). Then, the control unit 35 may stop the alarm in the second person detection state (alarm state) when the predetermined release condition is satisfied, as in the case where the movement of the excavator is restricted or stopped. .. This is because, unlike the alarm in the first person detection state (alert state) that can be automatically stopped, the operation of the operator is required to stop the alarm in the second person detection state (alarm state). ..

Next, with reference to FIG. 13, an example of a process in which the control unit 35 of the controller 30 monitors the periphery of the excavator (hereinafter, referred to as “peripheral monitoring process”) will be described. FIG. 13 is a flowchart showing an example flow of the peripheral monitoring process, and the controller 30 repeatedly executes the peripheral monitoring process at a predetermined control cycle.

First, the control unit 35 determines whether or not there is a person around the excavator (step ST11). In this embodiment, the control unit 35 determines whether or not a person exists in the vicinity of the excavator based on the final human detection result of the tracking unit 33.

After that, when it is determined that a person exists around the excavator (YES in step ST11), the control unit 35 limits or stops the movement of the excavator (step ST12). In this embodiment, for example, when the control unit 35 determines that the current person detection state is the second person detection state (alarm state), the control unit 35 determines that there is a person around the excavator and stops the movement of the excavator. ..

At this time, the control unit 35 outputs a control command to the vehicle-mounted speaker as the output device 50 to output the second alarm. Further, a control command is output to the in-vehicle display as the output device 50 to display an captured image including a human image that has caused the restriction or stop.

When it is determined that there is no person around the excavator (NO in step ST11), the control unit 35 determines whether or not the movement of the excavator has already been restricted or stopped (step S13). In this embodiment, the control unit 35 determines whether or not the movement of the excavator has already been restricted or stopped by referring to the value of the corresponding flag.

When it is determined that the movement of the excavator has already been restricted or stopped (YES in step ST13), the control unit 35 performs a process for releasing the restriction or stop (hereinafter, referred to as "restriction release process"). Execute (step ST14).

When it is determined that the movement of the excavator has not been restricted or stopped (NO in step ST13), the control unit 35 ends the excavator peripheral monitoring process this time without executing the restriction release process.

Next, with reference to FIG. 14, a process in which the control unit 35 of the controller 30 releases the restriction or stop of the movement of the excavator will be described. FIG. 14 is a flowchart showing a flow of an example of the restriction release process.

First, the control unit 35 determines whether or not the first release condition is satisfied (step ST21). In this embodiment, the control unit 35 determines whether or not a person exists within a predetermined range around the excavator. Specifically, it is determined whether or not the current person detection state has exited the second person detection state (alarm state). It may be determined whether or not the first person detection state (warning state) and the second person detection state (alarm state) have been exited.

When it is determined that the first release condition is satisfied (YES in step ST21), the control unit 35 determines whether or not the second release condition is satisfied (step ST22). In this embodiment, the control unit 35 determines whether or not the state in which the excavator does not start is secured. Specifically, it is determined whether or not the gate lock lever is lowered (whether or not the operating device is disabled).

When it is determined that the second release condition is satisfied (YES in step ST22), the control unit 35 determines whether or not the third release condition is satisfied (step ST23). In this embodiment, the control unit 35 determines whether or not the operator has confirmed that there is no person around the excavator. Specifically, it is determined whether or not the confirmation button is pressed while a message such as "Did you confirm that there are no people around the excavator?" Is displayed on the screen of the in-vehicle display.

When it is determined that the third release condition is satisfied (YES in step ST23), the control unit 35 releases the restriction or stop of the movement of the excavator (step ST24).

At this time, the control unit 35 outputs a control command to the vehicle-mounted speaker as the output device 50 to stop the output of the second alarm. Further, a control command is output to the in-vehicle display as the output device 50 to stop the display of the captured image including the human image that caused the limitation or stop. For example, the through image that was displayed before the second alarm was output is redisplayed. Further, the control unit 35 may display a message informing that the restriction or stop of the movement of the excavator has been released.

When the control unit 35 determines that the first release condition is not satisfied (NO in step ST21) and determines that the second release condition is not satisfied (NO in step ST22),
If it is determined that the third release condition is not satisfied (NO in step ST23), the current restriction release process is terminated without releasing the restriction or stop of the excavator movement.

With the above configuration, the controller 30 can limit or stop the movement of the excavator when it is determined that a person exists around the excavator.

Further, when the controller 30 determines that there is no person around the excavator after limiting or stopping the movement of the excavator, the controller 30 determines that the state in which the excavator does not start is secured. The restriction or suspension can be lifted. Further, the controller 30 can release the restriction or stop only when it is determined that the state in which the excavator does not start moving is secured and it is determined by the operator that there is no person around the excavator. Therefore, the controller 30 can prevent the excavator from unintentionally starting to move when the restriction or the stop is released.

Next, with reference to FIG. 15, an example of an output image displayed on the vehicle-mounted display during execution of the peripheral monitoring process will be described. FIG. 15 is an example of an output image generated based on the captured image of the rear camera 40B. FIG. 15 (A) shows an example of an output image when there is no person within a predetermined range around the excavator, and FIG. 15 (B) shows an example of an output image in the first person detection state, FIG. 15 (C). Shows an example of an output image in the second person detection state.

Specifically, the output image of FIG. 15 includes a camera image portion G1 and an indicator portion G2. The camera image portion G1 is a portion that displays an image generated based on images captured by one or a plurality of cameras. The indicator portion G2 is a portion that displays the person detection state / person non-detection state of each of the plurality of areas around the excavator. In the camera image portion G1, the line segment L1 superimposed and displayed on the camera image indicates that the distance from the shovel is a predetermined first distance (for example, 5 meters). Further, the line segment L2 superimposed and displayed on the camera image indicates that the distance from the shovel is a predetermined second distance (for example, 2.5 meters). In the indicator portion G2, the outer peripheral line L1g of the partial circle drawn around the excavator icon CG1 indicates that the distance from the excavator is a predetermined first distance (for example, 5 meters), and the line segment L1 of the camera image portion G1 Correspond. Further, the outer peripheral line L2g of the partial rectangle drawn around the excavator icon CG1 indicates that the distance from the excavator is a predetermined second distance (for example, 2.5 meters), and corresponds to the line segment L2 of the camera image portion G1. To do.

The partial circle is divided into six regions A1 to A6, and the partial rectangle is divided into three regions B1 to B3.

In the state shown in FIG. 15 (A), the controller 30 detects a person existing on the right rear side of the excavator. However, since the actual position of the person is beyond the first distance, the controller 30 does not highlight the image of the person and does not output the first alarm. However, the controller 30 may highlight the image of the person, such as displaying the outline of the corresponding identification processing target image area TRg as a white frame, or may output the first alarm. Further, a message such as "peripheral monitoring process is being executed" may be displayed regardless of whether or not a person has already been detected. This is so that the operator can recognize that the peripheral monitoring process is being executed.

In the first person detection state shown in FIG. 15B, the controller 30 detects a person who exists within the first distance to the right rear of the excavator and beyond the second distance. Therefore, the controller 30 highlights the image of the person and outputs the first alarm. Specifically, the controller 30 displays the contour line of the corresponding identification processing target image region TRg as a yellow frame F1 in the camera image portion G1. Further, in the indicator portion G2, the area A4 corresponding to the actual position of the person is displayed in yellow. However, the display of the yellow frame may be omitted. In addition, a message indicating that the first person is in the detection state (warning state) may be displayed.

In the second person detection state shown in FIG. 15C, the controller 30 detects a person who exists within the second distance to the right rear of the excavator. Therefore, the controller 30 highlights the image of the person and outputs the second alarm. Specifically, the controller 30 displays the contour line of the corresponding identification processing target image region TRg as the red frame F2 in the camera image portion G1. Further, in the indicator portion G2, the area B2 corresponding to the actual position of the person is displayed in red. In addition, the controller 30 limits the movement of the excavator, and then blinks and displays the message "excavator operation restricted" informing that the second person is in the detection state (alarm state). However, the display of the message indicating that the second person is in the detection state (alarm state) may be omitted.

Further, in FIGS. 15A to 15C, the camera image portion G1 is displayed on the left side of the screen, the indicator portion G2 is displayed on the right side of the screen, but the camera image portion G1 is displayed on the right side of the screen. The indicator portion G2 may be displayed on the left side of the screen. Further, the camera image portion G1 may be displayed on one of the vertically divided screens, and the indicator portion G2 may be displayed on the other side. Further, the display of the indicator portion G2 may be omitted.

Further, each of the regions A1 to A6 has an expansion angle of 45 degrees, and each of the regions B1 to B3 has an expansion angle of 90 degrees. This difference in the spread angle is based on the difference in the properties of the first person detection state (warning state) and the second person detection state (alarm state). Specifically, the first person detection state (warning state) is a state in which a preliminary alarm with low accuracy but quick response is output, and a relatively wide space range relatively far from the excavator is monitored. It is in the range. Therefore, if the expansion angle of the areas A1 to A6 is increased, the monitoring range corresponding to each area increases together with the display range, and it becomes difficult to understand the actual position of the person who caused the first alarm. This is because the same display result will be obtained anywhere in a wide monitoring range. On the other hand, the second person detection state (alarm state) is a state in which a formal alarm with high accuracy but slow response is output, and a relatively narrow space range relatively close to the excavator becomes the monitoring range. There is. Therefore, if the expansion angle of the areas B1 to B3 is reduced, the monitoring range corresponding to each area becomes smaller together with the display range, and it becomes difficult to know in which direction the person who caused the second alarm is. It ends up. This is because the display range is small and difficult to see. Therefore, preferably, as shown in FIGS. 15A to 15C, the expansion angle of each of the regions A1 to A6 is set to be smaller than the expansion angle of each of the regions B1 to B3.

Further, in FIGS. 15A to 15C, a case where a human image is detected in the captured image of the rear camera 40B while the through image of the rear camera 40B is displayed will be described. However, the above description is similarly applied when a human image is detected in at least one captured image of the left side camera 40L and the right side camera 40R when the through image of the rear camera 40B is displayed. .. In that case, even if the output image displayed on the camera image portion G1 is automatically switched from the through image of the rear camera 40B to the through image of another camera or the viewpoint conversion image synthesized from the images captured by a plurality of cameras. Good. For example, when a human image is detected in the captured image of the left camera 40L while the through image of the rear camera 40B is displayed, the controller 30 uses the output image displayed in the camera image portion G1 as the left camera 40L. You may switch to the through image of.

Next, with reference to FIG. 16, the relationship between the detection state and the display colors of the frame and the area will be described. FIG. 16 is a correspondence table showing the correspondence between the detection state and the display colors of the frame and the area.

The first row of the correspondence table does not display the outline of the image area to be identified and is an indicator when the detection state is neither the first person detection state (warning state) nor the second person detection state (alarm state). Indicates that none of the regions of the portion G2 is colored.

In the second line, when the detection state is the alert state, the outline of the identification processing target image area corresponding to the human image that caused the alert state is displayed as a yellow frame, and the areas A1 to A6 Indicates that any of the above is displayed in yellow.

In the third line, when the detection state is the alarm state, the outline of the identification processing target image area corresponding to the human image that caused the alarm state is displayed as a red frame, and the areas B1 to B3. Indicates that any of the above is displayed in red.

In the fourth line, when the detection state is the alert state and the alarm state, the outline corresponding to the human image that caused the alert state is displayed in a yellow frame, and the cause that causes the alarm state. Indicates that the contour line corresponding to the human image is displayed in a red frame. Further, the areas A1 to A6 corresponding to the human image causing the alert state are displayed in yellow, and the areas B1 to B3 corresponding to the human image causing the alarm state are displayed in red. Indicates that it is displayed with.

With the above configuration, the controller 30 outputs an alarm when it is determined that a person exists around the excavator, and highlights the image portion of the person. Therefore, the operator can confirm the person who caused the alarm on the screen. In addition, even if a false alarm occurs, the operator can confirm on the screen what caused the false alarm.

Further, the controller 30 displays a frame image as a person detection marker on the camera image portion G1 for the first time when the first person detection state (warning state) is reached, and changes the color of the corresponding area of the indicator portion G2. Let me. Therefore, even a frame image corresponding to a person candidate image that has been identified as a human image but whose reliability is still low is displayed, and it is possible to prevent the displayed image from becoming complicated. In the above embodiment, the frame image is displayed as the person detection marker, but another emphasized image such as the inverted display image may be adopted as the person detection marker.

In addition, the image of the person who caused the first person detection state (warning state) and the image of the person who caused the second person detection state (alarm state) are highlighted so as to be distinguishable. Further, the color of the frame image in the camera image portion G1 and the color of the region in the indicator portion G2 are made to correspond to each other. Therefore, the operator can confirm the person who caused the second alarm on the screen. Further, in the above-described embodiment, the controller 30 makes the color of the frame image in the camera image portion G1 and the color of the region in the indicator portion G2 different depending on the detection state. However, the controller 30 may have different attributes other than color, such as blinking / lighting state and transmittance, depending on the detection state.

Next, another example of the output image displayed on the in-vehicle display during the execution of the peripheral monitoring process will be described with reference to FIG. FIG. 17 is an example of a viewpoint conversion image as an output image generated based on each captured image of the rear camera 40B, the left side camera 40L, and the right side camera 40R. FIG. 17 shows an example of an output image when the first person detection state and the second person detection state coexist. The output image of FIG. 17 includes a viewpoint conversion image portion G3 corresponding to the camera image portion G1 of FIG. The portion corresponding to the indicator portion G2 in FIG. 15 is integrated with the viewpoint conversion image portion G3. Specifically, the excavator icon CG1 of FIG. 15 corresponds to the excavator icon CG2 of FIG. 17, and the regions A1 to A6 of FIG. 15 correspond to the regions C1 to C6 of FIG. Further, the area B1 of FIG. 15 corresponds to the combination of the areas C1 and C2 of FIG. 17, the area B2 of FIG. 15 corresponds to the combination of the areas C3 and C4 of FIG. 17, and the area B3 of FIG. 15 corresponds to the area of FIG. Corresponds to the combination of C5 and C6. The line segment L3 superimposed and displayed on the viewpoint conversion image indicates that the distance from the shovel is a predetermined third distance (for example, 2.5 meters).

In the detection state shown in FIG. 17, the controller 30 detects a person who is within the first distance (for example, 5 meters) on the left side of the excavator and beyond the third distance (the person who caused the first person detection state). It is being detected. In addition, a person who exists within a third distance behind the excavator (a person who caused the second person detection state) is detected. Therefore, the controller 30 highlights the images of those people, outputs a second alarm, and limits the movement of the excavator. Specifically, the controller 30 displays the yellow circle MA1 as a person detection marker at the position of the reference point Pr corresponding to the person who caused the first person detection state, and the area corresponding to that position. C2 is displayed in yellow. Further, a red circle MA2 as a person detection marker is displayed at the position of the reference point Pr corresponding to the person who caused the second person detection state, and the combination of the areas C3 and C4 corresponding to the position is red. Display with. Further, the controller 30 may limit the movement of the excavator and then blink the message "excavator operation restriction in progress" indicating that the second person is in the detection state (alarm state). Further, the display of the yellow circle MA1 may be omitted. This is to make the screen easier to see.

Next, with reference to FIG. 18, another example of the output image displayed on the in-vehicle display during the execution of the peripheral monitoring process will be described. FIG. 18 is an example of an output image including a viewpoint conversion image generated based on each captured image of the rear camera 40B, the left side camera 40L, and the right side camera 40R. FIG. 18 shows an example of an output image when the first person detection state and the second person detection state coexist as in the case of FIG. The output image of FIG. 18 includes an indicator portion G2 and a viewpoint conversion image portion G3. The outer peripheral line L2g of the partial rectangle drawn around the excavator icon CG1 indicates that the distance from the excavator is a predetermined third distance (for example, 2.5 meters), and corresponds to the line segment L3 of the viewpoint conversion image portion G3. ..

In the detection state shown in FIG. 18, the controller 30 detects a person who is within the first distance (for example, 5 meters) on the left side of the excavator and beyond the third distance (the person who caused the first person detection state). It is being detected. In addition, a person who exists within a third distance behind the excavator (a person who caused the second person detection state) is detected. Therefore, the controller 30 highlights the images of those people, outputs a second alarm, and limits the movement of the excavator. Specifically, the controller 30 displays a yellow circle MA1 as a person detection marker at the position of the reference point Pr corresponding to the person who caused the first person detection state, and an indicator corresponding to that position. The area A2 of the portion G2 is displayed in yellow. Further, a red circle MA2 as a person detection marker is displayed at the position of the reference point Pr corresponding to the person who caused the second person detection state, and the area B2 of the indicator portion G2 corresponding to the position is red. Display with. In addition, the controller 30 limits the movement of the excavator, and then blinks and displays the message "excavator operation restricted" informing that the second person is in the detection state (alarm state). The display of the yellow circle MA1 may be omitted. This is to make the screen easier to see.

With the above configuration, the controller 30 can realize the same effect as when the output image of FIG. 15 is displayed.

In this way, when the controller 30 determines that a person exists in the vicinity of the excavator, the controller 30 limits or stops the movement of the excavator and displays an image of the person. Then, when it is determined that there is no person around the excavator after restricting or stopping the movement of the excavator, the restriction or stop is performed only when it is determined that the state in which the excavator does not start is secured. Judge that it can be released. Then, when the predetermined waiting time elapses, the restriction or suspension is actually released. Therefore, it is possible to more appropriately release the operation restriction of the excavator executed in response to the detection of a person.

Further, when the movement of the excavator is restricted or stopped, the controller 30 can output an alarm to the operator from the side where the person who caused the excavator exists. Therefore, the operator can recognize the direction in which the person exists before the operator looks at the screen of the in-vehicle display. The operator can visually confirm that the person exists in the recognized direction by audibly recognizing the direction in which the person is present according to the direction in which the alarm is heard and then looking at the screen of the in-vehicle display. .. In this way, the controller 30 informs the operator of the direction in which the person exists by coordinating the alarm and the display, so that the operator can recognize the situation around the excavator in a short time.

Even if it recognizes that a person has been detected by an alarm, when the direction of the person's existence is unknown, the operator must first look at the entire screen to find out in which direction the person is present. Is. On the other hand, when the direction of existence of the person is known before looking at the screen, the operator can visually confirm the existence of the person only by looking at a part of the screen (the part corresponding to the direction of existence). is there.

Although the preferred examples of the present invention have been described in detail above, the present invention is not limited to the above-mentioned examples, and various modifications and substitutions are made to the above-mentioned examples without departing from the scope of the present invention. Can be added.

For example, in the above-described embodiment, it is assumed that a person is detected by using an image captured by an image pickup device 40 mounted on the upper swing body 3 of the shovel, but the present invention is not limited to this configuration. .. It can also be applied to a configuration using an image taken by an image pickup device attached to the main body of another work machine such as a mobile crane, a fixed crane, a riff mag machine, and a forklift.

Further, in the above-described embodiment, the blind spot area of the excavator is imaged by using three cameras, but the blind spot area of the excavator may be imaged by using one, two, or four or more cameras.

Further, in the above-described embodiment, the person detection is performed using the image captured by the image pickup device 40, but the person detection is performed using the output of an ultrasonic sensor, a laser radar, a pyroelectric sensor, a millimeter wave radar, or the like. May be good.

Further, in the above-described embodiment, the human detection process is individually applied to each of the plurality of captured images, but the human detection process is applied to one composite image generated from the plurality of captured images. May be good.

1 ... Lower traveling body 2 ... Swivel mechanism 3 ... Upper swivel body 4 ... Boom 5 ... Arm 6 ... Bucket 7 ... Boom cylinder 8 ... Arm cylinder 9 ...・ Bucket cylinder 10 ・ ・ ・ Cabin 30 ・ ・ ・ Controller 31 ・ ・ ・ Extraction unit 32 ・ ・ ・ Identification unit 33 ・ ・ ・ Tracking unit 34 ・ ・ ・ Human detection unit 35 ・ ・ ・ Control unit 40 ・ ・ ・ Imaging device 40B ・ ・ ・ Rear camera 40L ・ ・ ・ Left side camera 40R ・ ・ ・ Right side camera 41 ・ ・ ・ Input device 50 ・ ・ ・ Output device 51 ・ ・ ・ Machine control device 100 ・ ・ ・ Peripheral monitoring system AP, AP1 AP6 ・ ・ ・ Head image position BX ・ ・ ・ Box G1 ・ ・ ・ Camera image part G2 ・ ・ ・ Indicator part G3 ・ ・ ・ Viewpoint conversion image part HD ・ ・ ・ Head HP ・ ・ ・ Virtual head position HRg ・・ ・ Helmet image M1, M2 ・ ・ ・ Mask area Pr, Pr1, Pr2, Pr10 ・ Pr12 ・ ・ ・ Reference point R1 ・ ・ ・ Overhang area R2 ・ ・ ・ Vehicle body reflection area RP ・ ・ ・ Representative position TR, TR1, TR2, TR10 to TR12 ... Virtual plane area TRg, TRg3, TRg4, TRg5 ... Identification processing target image area TRgt, TRgt3, TRgt4, TRgt5 ... Normalized image

Claims (8)

With the running body
A swivel body mounted on the traveling body so as to be swivel
The work attachment provided on the swivel body and
It is a shovel equipped with
A detection unit provided on the swivel body and detecting a monitoring target existing in a predetermined range around the excavator, and a detection unit.
A control unit that switches the state of the excavator to the first state or the second state based on the detection result of the detection unit is provided.
The first state includes a state in which the restriction on the movement of the excavator is released.
The second state includes a state in which the movement of the excavator is restricted.
After switching the state of the excavator from the first state to the second state, the control unit returns the state of the excavator to the first state when a predetermined condition is satisfied.
The predetermined condition includes non-detection of the monitoring target within the predetermined range and ensuring that the excavator does not start moving.
Excavator. The state in which the movement of the excavator is restricted is a state in which the operating speed of the excavator is reduced.
The excavator according to claim 1. The state in which the movement of the excavator is restricted is at least one of a state in which the maximum turning speed is reduced, a state in which the maximum traveling speed is reduced, and a state in which the operating speed of the actuator is reduced. There are two states,
The excavator according to claim 2. Ensuring that the excavator does not start moving includes that the operating device is in the neutral position for a predetermined time or longer.
The condition "for a predetermined time or longer" prevents the condition that "the state in which the excavator does not start is ensured" is satisfied only when the operating device is momentarily in the neutral position. ,
The excavator according to any one of claims 1 to 3. The fact that the excavator does not start moving includes the fact that the operator's limbs are separated from the operating device.
The fact that the operator's limbs are separated from the operating device is detected based on the captured image of the imaging device that images the driver's cab or the output of the electrostatic sensor attached to the imaging device.
The excavator according to any one of claims 1 to 3. The fact that the excavator does not start moving includes the fact that the operating device is disabled by the gate lock lever.
The invalidation of the operating device is realized by making the operating device unresponsive.
The excavator according to any one of claims 1 to 3. The state in which the excavator does not start is ensured includes the state in which the excavator does not start when the state of the excavator is returned to the first state.
Whether or not the state in which the excavator does not start is secured is determined when the state shifts to the first state.
The excavator according to any one of claims 1 to 6. When the movement of the excavator is restricted, the method of releasing the restriction on the movement of the excavator is guided.
The excavator according to any one of claims 1 to 7.

Images