JP6755672B2 - Peripheral monitoring system for work machines - Google Patents

Description

The present invention relates to a peripheral monitoring system for a working machine that monitors the periphery of the working machine.

It has an image sensor and a thermopile array that senses heat, overlaps the imaging range and heat detection range, and limits only the human body-like range indicated by the output of the thermopile array to the face extraction range, which is unnecessary for image identification processing. A human body detection device that reduces the amount of arithmetic processing is known (see Patent Document 1).

Japanese Unexamined Patent Publication No. 2006-059015

On the other hand, it is desired to provide a peripheral monitoring system for work machines that can detect people around the work machine at an early stage.

The peripheral monitoring system for a work machine according to an embodiment of the present invention is a peripheral monitoring system for a work machine that detects a person existing in the vicinity of the work machine by using an image captured by an image pickup device attached to the work machine. It has an extraction unit that extracts an image to be identified and an identification unit that identifies whether the image included in the identification processing target image extracted by the extraction unit is a human image by image recognition processing. The image is divided into a plurality of regions, and the extraction unit selects regions from the plurality of regions in a predetermined order, and among the plurality of image portions in the selected region, a feature image representing a characteristic portion of a person. The image portion including the above is extracted as the identification processing target image, and the predetermined order is dynamically set according to the vehicle state of the work machine .

By the above-mentioned means, a peripheral monitoring system for a work machine that can detect a person around the work machine at an early stage is provided.

It is a side view of the excavator equipped with the peripheral monitoring system which concerns on embodiment of this invention. It is a functional block diagram which shows the configuration example of the peripheral monitoring system. This is an example of an image captured by a rear camera. It is the schematic which shows an example of the geometric relation used when cutting out the identification processing target image from the captured image. It is a top view of the real space behind the excavator. It is a figure which shows the flow of the process which generates the normalized image from the captured image. It is a figure which shows the relationship between the captured image, the image area subject to identification processing, and a normalized image. It is a figure which shows the relationship between the image area which is the object of identification processing, and the area which is unsuitable for identification processing. It is a figure which shows the example of the normalized image. It is a figure explaining the relationship between the rear horizontal distance between a virtual plane area and a rear camera in a real space, and the size of the head image part in a normalized image. It is a schematic diagram which shows another example of the geometric relation used when cutting out the identification processing target image from the captured image. It is a figure which shows an example of the feature image in the captured image. It is a flowchart which shows the flow of an example of an image extraction process. It is a flowchart which shows the flow of another example of an image extraction process. It is a flowchart which shows the flow of still another example of an image extraction process. It is a flowchart which shows the flow of still another example of an image extraction process. It is a flowchart which shows the flow of an example of a selective image extraction process. It is a figure which shows an example of the captured image divided into a plurality of regions. It is a figure which shows another example of the captured image divided into a plurality of regions. It is a figure which shows still another example of the captured image divided into a plurality of regions.

FIG. 1 is a side view of an excavator (excavator) as a work machine (construction machine) on which the peripheral monitoring system 100 according to the embodiment of the present invention is mounted. The upper swivel body 3 is mounted on the lower traveling body 1 of the excavator via the swivel mechanism 2. A boom 4 is attached to the upper swing body 3. An arm 5 is attached to the tip of the boom 4, and a bucket 6 is attached to the tip of the arm 5. The boom 4, arm 5, and bucket 6 form an excavation attachment, and are hydraulically driven by the boom cylinder 7, arm cylinder 8, and bucket cylinder 9, respectively. Further, the upper swing body 3 is provided with a cabin 10 and is equipped with a power source such as an engine. Further, an image pickup device 40 is attached to the upper part of the upper swing body 3. Specifically, the rear camera 40B, the left side camera 40L, and the right side camera 40R are attached to the upper rear end, the upper left end, and the upper right end of the upper swing body 3. Further, a controller 30 and an output device 50 are installed in the cabin 10.

FIG. 2 is a functional block diagram showing a configuration example of the peripheral monitoring system 100. The peripheral monitoring system 100 mainly includes a controller 30, an image pickup device 40, an input device 42, a vehicle state detection device 44, an output device 50, and a machine control device 51. In this embodiment, the image pickup device 40, the input device 42, the vehicle state detection device 44, the output device 50, and the machine control device 51 are connected to the controller 30 via CAN.

The controller 30 is a control device that controls the drive of the excavator. In this embodiment, the controller 30 is composed of a CPU and an arithmetic processing device including an internal memory, and causes the CPU to execute a drive control program stored in the internal memory to realize various functions.

Further, the controller 30 determines whether or not a person exists in the vicinity of the excavator based on the outputs of the various devices, and controls the various devices according to the determination result. Specifically, the controller 30 receives the outputs of the image pickup device 40 and the input device 42, and executes software programs corresponding to each of the extraction unit 31, the identification unit 32, the tracking unit 33, and the control unit 35. Then, a control command is output to the machine control device 51 to execute the excavator drive control according to the execution result, or various information is output from the output device 50. The controller 30 may be a control device dedicated to image processing.

The image pickup device 40 is a device for capturing an image around the excavator, and outputs the captured image to the controller 30. In this embodiment, the image pickup device 40 is a wide camera that employs an image pickup device such as a CCD, and is attached so that the optical axis faces diagonally downward at the upper part of the upper swing body 3.

The input device 42 is a device that receives input from the operator. In this embodiment, the input device 42 includes an operation device (operation lever, operation pedal, etc.), a gate lock lever, a button installed at the tip of the operation device, a button attached to an in-vehicle display, a touch panel, and the like.

The vehicle state detection device 44 is a device that detects the vehicle state of the excavator, and includes, for example, a boom angle sensor, an arm angle sensor, a bucket angle sensor, a resolver, a rotary encoder, a pilot pressure sensor, and the like. The vehicle state includes, for example, the posture of the excavation attachment, the turning direction, the turning speed, the traveling direction, the traveling speed, the operation content of the operating device, and the like. The posture of the drilling attachment is detected, for example, based on the outputs of the boom angle sensor, arm angle sensor, and bucket angle sensor. The turning direction and turning speed are detected based on, for example, the output of a resolver, a rotary encoder, or the like attached to the turning shaft of the upper turning body 3. The turning direction and turning speed may be detected based on the operation content of the turning operation lever. The operation content of the swivel operation lever is detected based on, for example, the output of the pilot pressure sensor. The traveling direction and traveling speed are detected based on, for example, the output of a rotary encoder or the like attached to the rotation shaft of the traveling hydraulic motor that drives the lower traveling body 1. The traveling direction and traveling speed may be detected based on the operation contents of the traveling lever or the traveling pedal. The operation content of the travel lever or the travel pedal is detected based on, for example, the output of the pilot pressure sensor.

The output device 50 is a device that outputs various information, and includes, for example, an in-vehicle display that displays various image information, an in-vehicle speaker that outputs various audio information, an alarm buzzer, an alarm lamp, and the like. In this embodiment, the output device 50 outputs various information in response to a control command from the controller 30.

The mechanical control device 51 is a device that controls the movement of the excavator, and includes, for example, a control valve, a gate lock valve, an engine control device, and the like that control the flow of hydraulic oil in the hydraulic system.

The extraction unit 31 is a functional element that extracts an image to be identified for identification processing from the image captured by the image pickup device 40. Specifically, the extraction unit 31 extracts simple features based on a local luminance gradient or edge, geometric features by Hough transform, etc., features related to the area or aspect ratio of a region divided based on luminance, and the like. The image to be identified is extracted by image processing with a relatively small amount of calculation (hereinafter, referred to as "pre-stage image recognition processing"). The identification processing target image is an image portion (a part of the captured image) that is the target of subsequent image processing, and includes a person candidate image. The human candidate image is an image portion (a part of the captured image) that is highly likely to be a human image. The captured image may be a color image or a grayscale image. The extraction unit 31 may have a plurality of types of functions for grayscale a color image.

The identification unit 32 is a functional element that identifies whether the person candidate image included in the identification processing target image extracted by the extraction unit 31 is a human image. Specifically, the identification unit 32 has a relatively large amount of calculation such as image recognition processing using an image feature amount description represented by a HOG (Histograms of Oriented Gradients) feature amount and a classifier generated by machine learning. It is identified whether the person candidate image is a person image by image processing (hereinafter, referred to as "post-stage image recognition processing"). The rate at which the identification unit 32 identifies the human candidate image as a human image increases as the extraction unit 31 extracts the identification processing target image with higher accuracy.

Next, with reference to FIG. 3, how the human image looks in the captured image behind the excavator captured by the rear camera 40B will be described. The two captured images in FIG. 3 are examples of captured images of the rear camera 40B. Further, the dotted line circle in FIG. 3 represents the existence of a human image and is not displayed in the actual captured image.

The rear camera 40B is a wide camera and is mounted at a height at which a person is viewed from diagonally above. Therefore, the appearance of the human image in the captured image greatly differs depending on the direction in which the person exists as seen from the rear camera 40B. For example, a human image in a captured image is displayed so as to be closer to the left and right edges of the captured image. This is due to image collapse caused by the wide-angle lens of the wide-angle camera. Further, the closer to the rear camera 40B, the larger the head is displayed. In addition, the legs enter the blind spot of the excavator's body and disappear. These are due to the installation position of the rear camera 40B. Therefore, it is difficult to identify the human image included in the captured image by image processing without performing any processing on the captured image.

Therefore, the peripheral monitoring system 100 according to the embodiment of the present invention promotes the identification of the human image included in the identification processing target image by normalizing the identification processing target image. In addition, "normalization" means converting the image to be identified for identification processing into an image having a predetermined size and a predetermined shape. In this embodiment, the identification processing target image that can take various shapes in the captured image is converted into a rectangular image of a predetermined size by projective transformation. As the projective transformation, for example, an 8-variable projective transformation matrix is used.

Here, with reference to FIGS. 4 to 6, an example of a process in which the peripheral monitoring system 100 normalizes the image to be identified (hereinafter, referred to as “normalization process”) will be described. Note that FIG. 4 is a schematic view showing an example of the geometric relationship used when the extraction unit 31 cuts out the identification processing target image from the captured image.

The box BX in FIG. 4 is a virtual three-dimensional object in real space, and in this embodiment, it is a virtual rectangular parallelepiped defined by eight vertices A to H. Further, the point Pr is a reference point set in advance for referring to the image to be identified. In this embodiment, the reference point Pr is a point preset as an assumed standing position of a person, and is located at the center of the quadrangle ABCD defined by the four vertices A to D. The size of the box BX is set based on the orientation of the person, the stride length, the height, and the like. In this embodiment, the quadrangle ABCD and the quadrangle EFGH are square, and the length of one side is, for example, 800 mm. The height of the rectangular parallelepiped is, for example, 1800 mm. That is, the box BX is a rectangular parallelepiped having a width of 800 mm, a depth of 800 mm, and a height of 1800 mm.

The quadrangle ABGH defined by the four vertices A, B, G, and H forms a virtual plane region TR corresponding to the region of the identification processing target image in the captured image. Further, the quadrangle ABGH as the virtual plane region TR is inclined with respect to the virtual ground which is a horizontal plane.

In this embodiment, the box BX as a virtual rectangular parallelepiped is adopted to determine the relationship between the reference point Pr and the virtual plane area TR. However, if the virtual plane region TR that faces the direction of the imaging device 40 and is inclined with respect to the virtual ground can be defined in association with an arbitrary reference point Pr, other relationships such as using other virtual three-dimensional objects, etc. Geometric relations may be adopted, and other mathematical relations such as functions and conversion tables may be adopted.

FIG. 5 is a top view of the real space behind the excavator, and shows the positional relationship between the rear camera 40B and the virtual plane regions TR1 and TR2 when the virtual plane regions TR1 and TR2 are referred to using the reference points Pr1 and Pr2. Shown. In this embodiment, the reference point Pr can be arranged at each of the grid points of the virtual grid on the virtual ground. However, the reference points Pr may be arranged irregularly on the virtual ground, or may be arranged at equal intervals on a line segment extending radially from the projection point of the rear camera 40B on the virtual ground. For example, each line segment extends radially in 1 degree increments, and reference points Pr are arranged on each line segment at 100 mm intervals.

As shown in FIGS. 4 and 5, the first surface of the box BX defined by the quadrangle ABFE (see FIG. 4) is positive to the rear camera 40B when the virtual plane region TR1 is referenced using the reference point Pr1. Arranged to face each other. That is, the line segment connecting the rear camera 40B and the reference point Pr1 is orthogonal to the first surface of the box BX arranged in relation to the reference point Pr1 in the top view. Similarly, the first surface of the box BX is arranged so as to face the rear camera 40B even when the virtual plane region TR2 is referred to by using the reference point Pr2. That is, the line segment connecting the rear camera 40B and the reference point Pr2 is orthogonal to the first surface of the box BX arranged in relation to the reference point Pr2 in the top view. This relationship holds regardless of which grid point the reference point Pr is placed on. That is, the box BX is arranged so that its first surface always faces the rear camera 40B.

FIG. 6 is a diagram showing a flow of processing for generating a normalized image from a captured image. Specifically, FIG. 6A is an example of the captured image of the rear camera 40B, and projects the box BX arranged in relation to the reference point Pr in the real space. Further, FIG. 6B is a diagram obtained by cutting out a region of the identification processing target image in the captured image (hereinafter, referred to as “identification processing target image region TRg”), and is projected on the captured image of FIG. 6A. Corresponds to the virtual plane area TR. Further, FIG. 6C shows a normalized image TRgt in which the identification processing target image having the identification processing target image region TRg is normalized.

As shown in FIG. 6 (A), the box BX arranged in relation to the reference point Pr1 in the real space determines the position of the virtual plane region TR in the real space, and the imaging corresponding to the virtual plane region TR is performed. The image area TRg to be identified on the image is determined.

In this way, if the position of the reference point Pr in the real space is determined, the position of the virtual plane region TR in the real space is uniquely determined, and the identification processing target image region TRg in the captured image is also uniquely determined. Then, the extraction unit 31 can normalize the identification processing target image having the identification processing target image region TRg to generate a normalized image TRgt of a predetermined size. In this embodiment, the size of the normalized image TRgt is, for example, 64 pixels in length × 32 pixels in width.

FIG. 7 is a diagram showing the relationship between the captured image, the image area subject to identification processing, and the normalized image. Specifically, FIG. 7 (A1) shows the identification processing target image region TRg3 in the captured image, and FIG. 7 (A2) shows the normalized image TRgt3 of the identification processing target image having the identification processing target image region TRg3. .. Further, FIG. 7 (B1) shows the identification processing target image region TRg4 in the captured image, and FIG. 7 (B2) shows the normalized image TRgt4 of the identification processing target image having the identification processing target image region TRg4. Similarly, FIG. 7 (C1) shows the identification processing target image region TRg5 in the captured image, and FIG. 7 (C2) shows the normalized image TRgt5 of the identification processing target image having the identification processing target image region TRg5.

As shown in FIG. 7, the identification processing target image region TRg5 in the captured image is larger than the identification processing target image region TRg4 in the captured image. This is because the distance between the virtual plane area corresponding to the identification processing target image area TRg5 and the rear camera 40B is smaller than the distance between the virtual plane area corresponding to the identification processing target image area TRg4 and the rear camera 40B. Similarly, the identification processing target image region TRg4 in the captured image is larger than the identification processing target image region TRg3 in the captured image. This is because the distance between the virtual plane area corresponding to the identification processing target image area TRg4 and the rear camera 40B is smaller than the distance between the virtual plane area corresponding to the identification processing target image area TRg3 and the rear camera 40B. That is, the identification processing target image area in the captured image is smaller as the distance between the corresponding virtual plane area and the rear camera 40B is larger. On the other hand, the normalized images TRgt3, TRgt4, and TRgt5 are all rectangular images of the same size.

In this way, the extraction unit 31 can normalize the identification processing target image that can take various shapes and sizes in the captured image to a rectangular image of a predetermined size, and normalize the human candidate image including the human image. Specifically, the extraction unit 31 arranges an image portion (hereinafter, referred to as “head image portion”) presumed to be the head of the human candidate image in a predetermined region of the normalized image. Further, an image portion presumed to be the body portion of the human candidate image (hereinafter referred to as “body portion image portion”) is arranged in another predetermined region of the normalized image, and another predetermined region of the normalized image is provided. An image portion (hereinafter, referred to as “leg image portion”) that is presumed to be the leg portion of the person candidate image is arranged in the region. Further, the extraction unit 31 can acquire the normalized image in a state where the inclination (image collapse) of the person candidate image with respect to the shape of the normalized image is suppressed.

Next, referring to FIG. 8, normalization processing is performed when the image area to be identified includes an image area unsuitable for identification that adversely affects the identification of human images (hereinafter, referred to as “identification processing unsuitable area”). Will be described. The region unsuitable for the identification process is a known region in which a human image cannot exist. For example, an region in which the vehicle body of the excavator is reflected (hereinafter, referred to as a “vehicle body reflection region”) and a region protruding from the captured image (the region in which the vehicle body is reflected). Hereinafter, it is referred to as a “protruding area”) and the like. Note that FIG. 8 is a diagram showing the relationship between the image region subject to identification processing and the region unsuitable for identification processing, and corresponds to FIGS. 7 (C1) and 7 (C2). Further, the downward-sloping diagonal hatching area in FIG. 8 left corresponds to the protruding area R1, and the downward-sloping diagonal hatching area corresponds to the vehicle body reflection area R2.

In this embodiment, when the identification processing target image area TRg5 includes a part of the protruding area R1 and the vehicle body reflection area R2, the extraction unit 31 masks the areas unsuitable for the identification processing and then the identification processing target image. A normalized image TRgt5 of the identification processing target image having the region TRg5 is generated. After generating the normalized image TRgt5, the extraction unit 31 may mask the portion corresponding to the region unsuitable for the identification process in the normalized image TRgt5.

The right figure of FIG. 8 shows the normalized image TRgt5. Further, in the right figure of FIG. 8, the downward-sloping diagonal hatching area represents the mask area M1 corresponding to the protruding area R1, and the downward-sloping diagonal hatching area is the mask area M2 corresponding to a part of the vehicle body reflection area R2. Represents.

In this way, the extraction unit 31 masks the image of the region unsuitable for the identification process to prevent the image of the region unsuitable for the identification process from affecting the identification process by the identification unit 32. By this mask processing, the identification unit 32 can identify whether the image is a human image by using the image of the region other than the mask region in the normalized image without being affected by the image of the region unsuitable for the identification processing. The extraction unit 31 may use any known method other than the mask processing so that the image of the region unsuitable for the identification process does not affect the identification process by the identification unit 32.

Next, with reference to FIG. 9, the features of the normalized image generated by the extraction unit 31 will be described. Note that FIG. 9 is a diagram showing an example of a normalized image. Further, in the 14 normalized images shown in FIG. 9, the normalized image closer to the left end of the figure includes the image of the person candidate existing at a position closer to the rear camera 40B, and the normalized image closer to the right end of the figure. Includes an image of a candidate person existing at a position far from the rear camera 40B.

As shown in FIG. 9, the extraction unit 31 performs any normalization regardless of the rear horizontal distance (horizontal distance in the Y-axis direction shown in FIG. 5) between the virtual plane region TR and the rear camera 40B in the real space. The head image portion, the body portion image portion, the leg portion image portion, and the like can be arranged at almost the same ratio in the image. Therefore, the extraction unit 31 can reduce the calculation load when the identification unit 32 executes the identification process, and can improve the reliability of the identification result. The above-mentioned rear horizontal distance is an example of information regarding the positional relationship between the virtual plane region TR and the rear camera 40B in the real space, and the extraction unit 31 adds the information to the extracted identification processing target image. .. Further, the above-mentioned information on the positional relationship includes a top view angle with respect to the optical axis of the rear camera 40B of the line segment connecting the reference point Pr corresponding to the virtual plane region TR and the rear camera 40B.

Next, with reference to FIG. 10, the relationship between the rear horizontal distance between the virtual plane region TR and the rear camera 40B in the real space and the size of the head image portion in the normalized image will be described. In addition, the upper figure of FIG. 10 shows the sizes L10, L11, and L12 of the head image portion when a person exists at three reference points Pr10, Pr11, and P12 having different rear horizontal distances from the rear camera 40B. In the figure shown, the horizontal axis corresponds to the rear horizontal distance. Further, the lower figure of FIG. 10 is a graph showing the relationship between the rear horizontal distance and the size of the head image portion, the vertical axis corresponds to the size of the head image portion, and the horizontal axis corresponds to the rear horizontal distance. The horizontal axes of the upper figure of FIG. 10 and the lower figure of FIG. 10 are common. Further, in this embodiment, the height of the camera is 2100 mm, the height of the center of the head HD from the ground is 1600 mm, and the diameter of the head is 250 mm.

As shown in the upper figure of FIG. 10, when a person is present at the position indicated by the reference point Pr10, the size L10 of the head image portion is the projected image of the head HD seen from the rear camera 40B onto the virtual plane region TR10. Corresponds to the size of. Similarly, when a person exists at the position indicated by the reference points Pr11 and Pr12, the sizes L11 and L12 of the head image portion are the projected images of the head HD seen from the rear camera 40B onto the virtual plane regions TR11 and TR12. Corresponds to the size of. The size of the head image portion in the normalized image changes with the size of the projected image.

Then, as shown in the lower figure of FIG. 10, the size of the head image portion in the normalized image is maintained substantially the same when the rear horizontal distance is D1 (for example, 700 mm) or more, but the rear horizontal distance is smaller than D1. It increases rapidly at the point.

Therefore, the identification unit 32 changes the content of the identification process according to the rear horizontal distance. For example, when the method of supervised learning (machine learning) is used, the identification unit 32 groups learning samples used in the identification process at a predetermined rear horizontal distance (for example, 650 mm). Specifically, the learning sample is divided into a short-distance group and a long-distance group. With this configuration, the identification unit 32 can identify a human image with higher accuracy. With the above configuration, the peripheral monitoring system 100 generates a normalized image TRgt from the identification processing target image region TRg corresponding to the virtual plane region TR that faces the direction of the image pickup apparatus 40 and is inclined with respect to the virtual ground that is a horizontal plane. .. Therefore, normalization can be realized in consideration of how a person looks in the height direction and the depth direction. As a result, even when the captured image of the image pickup device 40 attached to the construction machine so as to image the person from diagonally above is used, the person existing around the construction machine can be detected more reliably. In particular, even when a person approaches the image pickup device 40, the normalized image can be generated from the identification processing target image that occupies a sufficiently large area on the captured image, so that the person can be reliably detected.

Further, the peripheral monitoring system 100 defines a virtual plane area TR as a rectangular area formed by four vertices A, B, G, and H of a box BX which is a virtual rectangular parallelepiped in a real space. Therefore, the reference point Pr in the real space and the virtual plane region TR can be geometrically associated with each other, and further, the virtual plane region TR in the real space and the identification processing target image region TRg in the captured image can be geometrically associated with each other. Can be associated.

In addition, the extraction unit 31 masks the image of the identification processing unsuitable region included in the identification processing target image region TRg. Therefore, the identification unit 32 can identify whether the image is a human image by using an image of a region other than the mask region in the normalized image without being affected by the image of the region unsuitable for the identification process including the vehicle body reflection region R2.

Further, when the identification processing target image is extracted, the extraction unit 31 adds a rear horizontal distance between the virtual plane region TR and the imaging device 40 to the identification processing target image as information regarding the positional relationship between the two. Then, the identification unit 32 changes the content of the identification process according to the rear horizontal distance thereof. Specifically, the identification unit 32 groups the learning samples used in the identification process at a predetermined rear horizontal distance (for example, 650 mm). With this configuration, the identification unit 32 can identify a human image with higher accuracy.

In addition, the extraction unit 31 can extract the identification processing target image for each reference point Pr. Further, each of the identification processing target image regions TRg is associated with one of the reference points Pr preset as the assumed standing position of the person via the corresponding virtual plane region TR. Therefore, the peripheral monitoring system 100 can extract the identification processing target image that is likely to include the person candidate image by extracting the reference point Pr that is likely to have a person by an arbitrary method. In this case, it is possible to prevent the identification processing target image, which is unlikely to include the person candidate image, from being subjected to the identification processing by the image processing having a relatively large amount of calculation, and to realize the speeding up of the person detection processing. ..

Next, with reference to FIGS. 11 and 12, an example of processing in which the extraction unit 31 extracts the identification processing target image that is likely to include the person candidate image will be described. Note that FIG. 11 is a schematic view showing an example of the geometric relationship used when the extraction unit 31 cuts out the identification processing target image from the captured image, and corresponds to FIG. 4. Further, FIG. 12 is a diagram showing an example of a feature image in the captured image. The feature image is an image showing a characteristic part of a person, and preferably an image showing a part where the height from the ground in the real space is hard to change. Therefore, the feature image includes, for example, an image of a helmet, an image of a shoulder, an image of a head, an image of a reflector or a marker attached to a person, and the like.

In this embodiment, the extraction unit 31 finds a helmet image (strictly speaking, an image that can be presumed to be a helmet) in the captured image by the pre-stage image recognition process. This is because people working around the excavator are considered to be wearing helmets. Then, the extraction unit 31 derives the most relevant reference point Pr from the position of the found helmet image. Then, the extraction unit 31 extracts the identification processing target image corresponding to the reference point Pr.

Specifically, the extraction unit 31 uses the geometric relationship shown in FIG. 11 to derive a highly relevant reference point Pr from the position of the helmet image in the captured image. The geometric relationship of FIG. 11 is different from the geometric relationship of FIG. 4 in that it determines the virtual head position HP in the real space, but is common in other respects.

The virtual head position HP represents the head position of a person who is assumed to exist on the reference point Pr, and is arranged directly above the reference point Pr. In this embodiment, it is arranged at a height of 1700 mm on the reference point Pr. Therefore, if the virtual head position HP in the real space is determined, the position of the reference point Pr in the real space is uniquely determined, and the position of the virtual plane region TR in the real space is also uniquely determined. Further, the identification processing target image area TRg in the captured image is also uniquely determined. Then, the extraction unit 31 can normalize the identification processing target image having the identification processing target image region TRg to generate a normalized image TRgt of a predetermined size.

On the contrary, if the position of the reference point Pr in the real space is determined, the virtual head position HP in the real space is uniquely determined, and the head image position AP on the captured image corresponding to the virtual head position HP in the real space is also unique. It is decided to. Therefore, the head image position AP can be preset in association with each of the preset reference points Pr. The head image position AP may be derived in real time from the reference point Pr.

Therefore, the extraction unit 31 searches for a helmet image in the captured image of the rear camera 40B by the pre-stage image recognition process. FIG. 12 upper figure shows a state in which the extraction unit 31 has found the helmet image HRg. Then, when the extraction unit 31 finds the helmet image HRg, the extraction unit 31 determines the representative position RP. The representative position RP is a position derived from the size, shape, etc. of the helmet image HRg. In this embodiment, the representative position RP is the position of the central pixel of the helmet image region including the helmet image HRg. The lower figure of FIG. 12 is an enlarged view of a helmet image area which is a rectangular image area separated by a white line in the upper figure of FIG. 12, and shows that the position of the central pixel of the helmet image area is the representative position RP.

After that, the extraction unit 31 derives the head image position AP closest to the representative position RP by using, for example, the nearest neighbor search algorithm. In the lower figure of FIG. 12, six head image positions AP1 to AP6 are preset near the representative position RP, and the head image position AP5 is the head image position AP closest to the representative position RP. Show that.

Then, the extraction unit 31 traces the virtual head position HP, the reference point Pr, and the virtual plane region TR from the derived nearest head image position AP by using the geometric relationship shown in FIG. The image area TRg to be identified for identification processing is extracted. After that, the extraction unit 31 normalizes the identification processing target image having the extracted identification processing target image region TRg to generate a normalized image TRgt.

In this way, the extraction unit 31 includes the representative position RP of the helmet image HRg, which is the position of the characteristic image of the person in the captured image, and one of the preset head image position APs (head image position AP5). The image to be identified is extracted by associating with.

Instead of using the geometric relationship shown in FIG. 11, the extraction unit 31 directly associates the head image position AP with the reference point Pr, the virtual plane region TR, or the identification processing target image region TRg. The identification processing target image corresponding to the head image position AP may be extracted by using the table.

Further, the extraction unit 31 may derive the reference point Pr from the representative position RP by using a known algorithm other than the nearest neighbor search algorithm such as the mountain climbing method and the Mean-shift method. For example, when the mountain climbing method is used, the extraction unit 31 derives a plurality of head image position APs in the vicinity of the representative position RP, and the reference point Pr corresponding to each of the representative position RP and the plurality of head image position APs. And link. At this time, the extraction unit 31 weights the reference point Pr so that the closer the representative position RP and the head image position AP are, the larger the weight is. Then, the weight distribution of the plurality of reference points Pr is climbed, and the identification processing target image region TRg is extracted from the reference points Pr having the weight closest to the maximum weight point.

Next, with reference to FIG. 13, an example of a process in which the extraction unit 31 of the controller 30 extracts the image to be identified (hereinafter referred to as “image extraction process”) will be described. Note that FIG. 13 is a flowchart showing a flow of an example of the image extraction process. The extraction unit 31 executes this image extraction process every time a captured image is acquired, for example.

First, the extraction unit 31 searches for a helmet image in the captured image (step ST1). In this embodiment, the extraction unit 31 raster-scans the captured image of the rear camera 40B by the pre-stage image recognition process to find the helmet image.

When the helmet image HRg is found in the captured image (YES in step ST1), the extraction unit 31 acquires the representative position RP of the helmet image HRg (step ST2).

After that, the extraction unit 31 acquires the head image position AP closest to the acquired representative position RP (step ST3).

After that, the extraction unit 31 extracts the identification processing target image corresponding to the acquired head image position AP (step ST4). In this embodiment, the extraction unit 31 utilizes the geometric relationship shown in FIG. 11 as a head image position AP in the captured image, a virtual head position HP in the real space, and an assumed standing position of a person in the real space. The identification processing target image is extracted by tracing the correspondence between the reference point Pr and the virtual plane region TR in the real space.

If the extraction unit 31 does not find the helmet image HRg in the captured image (NO in step ST1), the extraction unit 31 shifts the processing to step ST5 without extracting the identification processing target image.

After that, the extraction unit 31 determines whether the helmet image has been searched for the entire captured image (step ST5).

When it is determined that the entire captured image has not been searched yet (NO in step ST5), the extraction unit 31 executes the processes of steps ST1 to ST4 for another region of the captured image.

On the other hand, when it is determined that the search for the helmet image over the entire captured image is completed (YES in step ST5), the extraction unit 31 ends the image extraction process this time.

In this way, the extraction unit 31 first finds the helmet image HRg, and from the representative position RP of the found helmet image HRg, the head image position AP, the virtual head position HP, the reference point (assumed standing position) Pr, and the virtual The image region TRg to be identified is specified via the plane region TR. Then, by extracting and normalizing the identification processing target image having the specified identification processing target image region TRg, a normalized image TRgt of a predetermined size can be generated.

Next, another example of the image extraction process will be described with reference to FIG. Note that FIG. 14 is a flowchart showing a flow of another example of the image extraction process.

First, the extraction unit 31 acquires one of the head image position APs (step ST11). After that, the extraction unit 31 acquires the helmet image area corresponding to the head image position AP (step ST12). In this embodiment, the helmet image area is an image area of a predetermined size set in advance for each of the head image position APs.

After that, the extraction unit 31 searches for a helmet image in the helmet image area (step ST13). In this embodiment, the extraction unit 31 finds the helmet image by raster scanning the inside of the helmet image area by the pre-stage image recognition process.

When the helmet image HRg is found in the helmet image area (YES in step ST13), the extraction unit 31 extracts the identification processing target image corresponding to the head image position AP at that time (step ST14). In this embodiment, the extraction unit 31 utilizes the geometric relationship shown in FIG. 11 as a head image position AP in the captured image, a virtual head position HP in the real space, and an assumed standing position of a person in the real space. The identification processing target image is extracted by tracing the correspondence between the reference point Pr and the virtual plane region TR in the real space.

If the extraction unit 31 does not find the helmet image HRg in the helmet image area (NO in step ST13), the extraction unit 31 shifts the processing to step ST15 without extracting the identification processing target image.

After that, the extraction unit 31 determines whether or not all the head image position APs in the captured image have been acquired (step ST15). Then, when it is determined that all the head image position APs in the captured image have not been acquired yet (NO in step ST15), the extraction unit 31 acquires another head image position AP that has not been acquired, and steps ST11. ~ The process of step ST14 is executed. On the other hand, when it is determined that all the head image position APs in the captured image have been acquired (YES in step ST15), the extraction unit 31 ends the image extraction process this time.

In this way, when the extraction unit 31 first acquires one of the head image position APs and finds the helmet image HRg in the helmet image area corresponding to the acquired head image position AP, the head at that time. The identification processing target image area TRg is specified from the part image position AP via the virtual head position HP, the reference point (assumed standing position) Pr, and the virtual plane area TR. Then, by extracting and normalizing the identification processing target image having the specified identification processing target image region TRg, a normalized image TRgt of a predetermined size can be generated.

With the above configuration, the extraction unit 31 of the peripheral monitoring system 100 finds a helmet image as a feature image in the captured image, and associates the representative position RP of the helmet image with one of the head image position APs as a predetermined image position. By doing so, the image to be identified is extracted. Therefore, it is possible to narrow down the image portion to be the target of the subsequent image recognition processing with a simple system configuration.

The extraction unit 31 first finds the helmet image HRg from the captured image, derives one of the head image position APs corresponding to the representative position RP of the helmet image HRg, and sets it as one of the head image position APs. The corresponding identification processing target image may be extracted. Alternatively, the extraction unit 31 first acquires one of the head image position APs, and the helmet image is within the helmet image area which is a predetermined area including the position of the feature image corresponding to one of the head image position APs. If is present, the identification processing target image corresponding to one of the head image position APs may be extracted.

Further, the extraction unit 31 may extract the identification processing target image from the representative position RP of the helmet image in the captured image by utilizing a predetermined geometric relationship as shown in FIG. In this case, the predetermined geometric relationship is the identification processing target image region TRg in the captured image, the virtual plane region TR in the real space corresponding to the identification processing target image region TRg, and the real space corresponding to the virtual plane region TR. The reference point Pr (the assumed standing position of the person) and the virtual head position HP corresponding to the reference point Pr (the virtual feature position which is the position in the real space of the characteristic part of the person corresponding to the assumed standing position of the person). , Represents the geometric relationship with the head image position AP (predetermined image position in the captured image corresponding to the virtual feature position) in the captured image corresponding to the virtual head position HP.

Next, still another example of the image extraction process will be described with reference to FIG. Note that FIG. 15 is a flowchart showing a flow of still another example of the image extraction process. The image extraction process of FIG. 15 is different from the image extraction process of FIGS. 13 and 14 in that after the normalized image is generated, it is determined whether or not the normalized image includes a helmet image as a feature image. different. The image extraction process in each of FIGS. 13 and 14 is for normalizing the image portion including the helmet image after finding the helmet image.

In this embodiment, the extraction unit 31 normalizes each of a plurality of predetermined image portions in the captured image to generate a plurality of normalized images, and among the normalized images, the normalized image including the helmet image is identified. Extract as an image. The plurality of predetermined image portions are, for example, a plurality of identification processing target image regions TRg predetermined on the captured image. The identification processing target image region TRg (see FIG. 6) corresponds to the virtual plane region TR in the real space, and the virtual plane region TR corresponds to the reference point Pr in the real space. Then, the identification unit 32 identifies whether the person candidate image included in the identification processing target image extracted by the extraction unit 31 is a human image.

First, the extraction unit 31 generates a normalized image from one of the predetermined image portions (step ST21). In this embodiment, one of the identification processing target image regions TRg corresponding to one of the preset reference points Pr in the captured image of the rear camera 40B is set as a predetermined image portion, and the predetermined image portion is set to a predetermined size by projective conversion. Convert to a rectangular image of.

After that, the extraction unit 31 determines whether or not the generated normalized image includes the helmet image (step ST22). In this embodiment, the extraction unit 31 raster-scans the normalized image by the pre-stage image recognition process to find the helmet image.

When it is determined that the normalized image includes the helmet image (YES in step ST22), the extraction unit 31 extracts the normalized image as the identification processing target image (step ST23). At this time, the identification unit 32 identifies whether the person candidate image included in the identification processing target image extracted by the extraction unit 31 is a person image. That is, the identification unit 32 determines whether the image included in the normalized image is a human image with respect to the normalized image determined to include the helmet image before the next normalization by the extraction unit 31 is performed. Identify. Therefore, the peripheral monitoring system 100 can execute the image extraction process and the identification process for the entire captured image as long as it has a memory capacity capable of storing one normalized image. However, the identification unit 32 identifies whether the person candidate image included in each of the plurality of identification processing target images is a human image after the extraction unit 31 extracts the plurality of normalized images as the identification processing target images. You may.

When it is determined that the normalized image does not include the helmet image (NO in step ST22), the extraction unit 31 proceeds with the process without extracting the normalized image as the identification processing target image.

After that, the extraction unit 31 determines whether or not the normalized image is generated from all the predetermined image portions in the captured image (step ST24). In this embodiment, it is determined whether or not the normalized image is generated from all the predetermined image portions which are the identification processing target image regions TRg corresponding to each of the reference points Pr in the captured image of the rear camera 40B.

When it is determined that the normalized image is not generated from all the predetermined image portions in the captured image (NO in step ST24), the extraction unit 31 executes the processes of steps ST21 to ST23 for another predetermined image portion. To do.

On the other hand, when it is determined that the normalized image is generated from all the predetermined image portions in the captured image (YES in step ST24), the extraction unit 31 ends the image extraction process this time.

In the example of FIG. 15, the peripheral monitoring system 100 determines whether or not the helmet image is included in the normalized image at the stage when one normalized image is generated. However, at the stage when a plurality of normalized images are generated, it may be collectively determined whether or not each of the plurality of normalized images includes a helmet image. Further, at the stage when all the normalized images are generated, it may be collectively determined whether or not the helmet image is included in each of all the normalized images.

By such an image extraction process, the peripheral monitoring system 100 can reduce the number of memory accesses during the person detection process. This is because the process of finding the helmet image and the process of extracting the identification processing target image can be executed at the same time.

Further, the peripheral monitoring system 100 that employs the image extraction process of FIG. 15 can reduce the memory capacity as compared with the case of adopting the image extraction process of FIG. 13 or 14. The image extraction process of FIG. 13 or FIG. 14 requires at least a memory capacity capable of storing an image portion before normalization (an image larger than the normalized image), but the image extraction process of FIG. 15 is normalized. This is because it can be executed if there is enough memory capacity to store the normalized image because the image is generated first.

Further, the peripheral monitoring system 100 that employs the image extraction process of FIG. 15 determines whether or not the normalized image includes the helmet image. Therefore, the process required for determining whether or not the helmet image is included, as compared with the configuration for determining whether or not the helmet image is included in the image before normalization as in the image extraction process of FIG. 13 or FIG. You can save time. This is because the variation in the size and distortion of the helmet image to be searched can be reduced. As a result, the time required for extracting the image to be identified can be shortened. Further, since the variation in the size and distortion of the helmet image to be searched can be reduced, the accuracy of determining whether or not the helmet image is included can be improved.

Further, the peripheral monitoring system 100 that employs the image extraction process of FIG. 15 has a large variation in the head position with respect to the foot position of the person (for example, when the image of the crouched person is included in the captured image). , More reliable person detection can be realized. This is because it employs a configuration for determining whether or not the normalized image includes a helmet image. That is, it is not a configuration for specifying the identification processing target image area by using the head image position related to the position of the helmet image as in the image extraction processing of FIG. 13 or FIG. 14, but as a predetermined image portion to be normalized. This is because the identification processing target image area of is set regardless of the head image position.

Next, still another example of the image extraction process will be described with reference to FIG. Note that FIG. 16 is a flowchart showing a flow of still another example of the image extraction process. In the image extraction process of FIG. 16, a helmet image as a feature image is added to the partially normalized image (hereinafter referred to as “partially normalized image”) at the stage where a part of the predetermined image portion is normalized. It differs from the image extraction process of FIG. 15 in that it determines whether or not it is included. The image extraction process of FIG. 15 is for determining whether or not the helmet image is included in the normalized image at the stage where all of one predetermined image portion is normalized.

First, the extraction unit 31 generates a partially normalized image from a part of a predetermined image portion (step ST31). Then, the extraction unit 31 determines whether or not the helmet image is included in the partially normalized image at the stage of generating the partially normalized image (step ST32). In this embodiment, the extraction unit 31 rasterizes the partially normalized image by the pre-stage image recognition process at the stage where a part of the predetermined image portion corresponding to the upper half of the normalized image is normalized to generate the partially normalized image. Scan to find the helmet image.

When it is determined that the partially normalized image includes the helmet image (YES in step ST32), the extraction unit 31 normalizes the remaining portion of the predetermined image portion to generate a normalized image (step ST33). In this embodiment, the extraction unit 31 normalizes the remaining portion of the predetermined image portion corresponding to the lower half of the normalized image to generate a normalized image. Then, the extraction unit 31 extracts the normalized image as an image to be identified (step ST34). At this time, the identification unit 32 identifies whether the person candidate image included in the identification processing target image extracted by the extraction unit 31 is a person image. However, the identification unit 32 identifies whether the person candidate image included in each of the plurality of identification processing target images is a human image after the extraction unit 31 extracts the plurality of normalized images as the identification processing target images. You may.

When it is determined that the helmet image is not included in the partially normalized image (NO in step ST32), the extraction unit 31 normalizes the remaining portion of the predetermined image portion and proceeds with the process without generating the normalized image.

After that, the extraction unit 31 determines whether or not a partially normalized image is generated from all the predetermined image portions in the captured image (step ST35). In this embodiment, it is determined whether or not a partially normalized image is generated from all of a predetermined image portion which is an identification processing target image region TRg corresponding to each of the reference points Pr in the captured image of the rear camera 40B.

When it is determined that the partially normalized image is not generated from all the predetermined image portions in the captured image (NO in step ST35), the extraction unit 31 performs the processing of steps ST31 to ST34 for another predetermined image portion. Execute.

On the other hand, when it is determined that the partially normalized image is generated from all the predetermined image portions in the captured image (YES in step ST35), the extraction unit 31 ends the current image extraction process.

In the example of FIG. 16, the peripheral monitoring system 100 normalizes a part of a predetermined image portion corresponding to the upper half of the normalized image to generate a partially normalized image. This is because the helmet image is highly likely to be included in the upper half of the normalized image (the part corresponding to the upper body of a person). However, a partially normalized image may be generated by normalizing a part of a predetermined image portion corresponding to a portion smaller than the upper half of the normalized image. For example, a partially normalized image may be generated by normalizing a part of a predetermined image portion corresponding to a portion corresponding to the upper 1/3 of the normalized image. Alternatively, the peripheral monitoring system 100 may normalize another part of the normalized image to generate a partially normalized image. For example, when searching for a shoulder image as a feature image, a part of a predetermined image portion corresponding to the right half of the normalized image may be normalized to generate a partially normalized image, and a marker as a feature image may be generated. When searching for an image, a part of a predetermined image portion corresponding to the central half of the normalized image may be normalized to generate a partially normalized image.

By such an image extraction process, the peripheral monitoring system 100 that employs the image extraction process of FIG. 16 can realize the same effect as that of the case of adopting the image extraction process of FIG.

Further, the peripheral monitoring system 100 that employs the image extraction process of FIG. 16 can further shorten the processing time required for the human detection process as compared with the case of adopting the image extraction process of FIG. This is because when it is determined that the helmet image is not included in the partially normalized image, the normalization of the next predetermined image portion can be started without normalizing the remaining portion of the predetermined image portion.

Further, in the image extraction processing in each of FIGS. 13 to 16, the extraction unit 31 extracts the identification processing target image for the entire area of the captured image. However, if the extraction is performed uniformly from the upper left region to the lower right region of the captured image, it takes too much time to extract in the region where the human image is unlikely to exist, and the human image is likely to exist. There is a risk of delaying extraction in the region. In particular, this tendency becomes remarkable when the number of candidate images is large, such as when the background image is complicated.

Therefore, the extraction unit 31 may divide the captured image into a plurality of regions and extract the identification processing target image for each region selected in a predetermined order from the plurality of regions. For example, a priority may be given to each of the plurality of regions, and the image extraction process may be applied to each region in order from the region having the highest priority. This is to complete the extraction in a high-priority area such as an area where a human image is likely to exist at an early stage. The captured image may be divided into upper and lower parts, left and right, and upper, lower, left and right. Further, the plurality of areas may be dynamically set according to the vehicle state of the excavator or the like, or may be set statically regardless of the vehicle state of the excavator or the like. Similarly, the priority order may be dynamically set according to the vehicle state of the excavator or the like, or may be set statically regardless of the vehicle state of the excavator or the like. For example, the method and priority of dividing the plurality of regions may be dynamically set according to at least one of the turning direction, the turning speed, the traveling direction, and the traveling speed. Further, any of the image extraction processes of FIGS. 13 to 16 may be adopted as the image extraction process for each region. However, in that case, "whole captured image" and "all in the captured image" in the description of FIGS. 13 to 16 are read as "whole selected region" and "all in the selected region".

FIG. 17 is a flowchart showing an example flow of a process (hereinafter, referred to as “selective image extraction process”) in which an image extraction process is sequentially applied to each of a plurality of regions in a captured image. The controller 30 executes this selective image extraction process every time the captured image is acquired, for example. The controller 30 may selectively execute any of the image extraction processes of FIGS. 13 to 16 (hereinafter, referred to as “uniform image extraction process”) and the selective image extraction process of FIG. .. For example, the controller 30 executes the selective image extraction process when the next captured image is acquired when the number of extracted images to be identified for the actually acquired image exceeds a predetermined number, and the controller 30 executes the selective image extraction process for the identification process target image. When the number of extractions is equal to or less than a predetermined number, the uniform image extraction process may be executed when the next captured image is acquired. Alternatively, the controller 30 may always execute the selective image extraction process regardless of the number of extracted images to be identified.

FIG. 18 shows an example of a captured image divided into a plurality of regions. The captured image of FIG. 18 is a captured image (through image) captured by the rear camera 40B. In FIG. 18, the captured image is divided into six regions A1 to A6. The upper side of the trapezoidal region A1 corresponds to, for example, the current turning radius of the excavator, and the left and right sides of the region A1 correspond to, for example, the extension line of the outer edge of the lower traveling body 1. The lower side of the inverted trapezoidal region A4 corresponds, for example, to the maximum turning radius of the shovel, and the upper side of the region A4 corresponds, for example, to a line located at a predetermined height above the ground. The predetermined height is, for example, the height of a person (2 meters). In the example of FIG. 18, the method of dividing the areas A1 to A6 is statically set. For example, information regarding the range, size, and the like of each of the areas A1 to A6 is stored in advance in the internal memory of the controller 30. However, the method of dividing the plurality of areas may be dynamically set according to the vehicle state of the excavator and the like. The change of the division method includes the change of the range to be divided, the change of the number of divisions, and the like.

When the captured image is acquired, the extraction unit 31 of the controller 30 acquires the vehicle state of the excavator (step ST41). For example, the extraction unit 31 acquires the vehicle state of the excavator based on the output of the vehicle state detection device 44. The vehicle state includes, for example, the posture of the excavation attachment, the turning direction, the turning speed, the traveling direction, the traveling speed, the operation content of the operating device, and the like. The extraction unit 31 may derive the vehicle state of the excavator from the captured images by using the time difference of the continuous captured images, the optical flow, and the like.

After that, the extraction unit 31 sets the priority of each region based on the vehicle state of the excavator (step ST42). For example, when the excavator is stationary, the extraction unit 31 sets the priority order in the order of region A1, region A2, region A3, region A4, region A5, and region A6. The reason why the region A1 corresponding to the space directly behind the excavator is given the highest priority is to prevent the excavator from coming into contact with the person when the excavator is started by detecting the person who is directly behind the excavator at an early stage.

Further, when the excavator is moving backward, the extraction unit 31 sets the priority order in the order of region A4, region A2, region A3, region A1, region A5, and region A6. The reason why the region A4 corresponding to the relatively distant space in the predicted course is given the highest priority is to detect a person whose distance from the excavator decreases as the excavator progresses at an early stage. The reason why the regions A2 and A3 corresponding to the spaces located on the left and right of the predicted course are given the next priority is to detect a person who is about to enter the predicted course at an early stage.

Further, when the excavator is advancing, the extraction unit 31 sets the priority in the order of area A2, area A3, area A1, area A4, area A5, and area A6. The area A2 and area A3 corresponding to the spaces located on the left and right of the path taken by the excavator are given the highest priority, and the area A1 corresponding to the nearest space in the path is given the next priority when approaching the excavator. This is to detect a person at an early stage.

Further, when the excavator is turning to the right, the extraction unit 31 sets the priority order in the order of region A3, region A1, region A4, region A5, region A2, and region A6. The reason why the area A3 corresponding to the space on the left rear side of the excavator is given the highest priority is to detect a person whose distance from the excavation attachment is shortened as the excavator turns to the right at an early stage.

Further, when the excavator is turning to the left, the extraction unit 31 sets the priority order in the order of area A2, area A1, area A4, area A5, area A3, and area A6. The reason why the area A2 corresponding to the space on the right rear side of the excavator is given the highest priority is to detect a person whose distance from the excavation attachment is shortened as the excavator turns to the left at an early stage.

Further, in any of the above cases, the extraction unit 31 sets the priority of the region A5, which is the central region of the captured image, to be low. This is because the person candidate image of the person existing in the space corresponding to the area A5 has already been extracted in another surrounding area in the image extraction process before the previous time.

Further, in any of the above cases, the extraction unit 31 sets the priority of the region A6 corresponding to the distant space at a position higher than the ground by a predetermined height to the lowest priority. This is because the possibility that a person exists in the space corresponding to the area A6 is lower than in other areas.

After that, the extraction unit 31 selects the region having the highest priority among the unsearched regions in the plurality of regions as the target of the image extraction process (step ST43). The unsearched area means an area to which the image extraction process has not yet been applied. For example, when the excavator is moving forward, the extraction unit 31 selects the area A2 if the highest priority area A2 is an unsearched area, and the area A2 is not an unsearched area and the next priority area A3 is unsearched. If it is an area, the area A3 is selected. The same applies to areas with lower priorities.

After that, the extraction unit 31 applies the image extraction process to the selected area (step ST44). The extraction unit 31 employs, for example, any of the image extraction processes of FIGS. 13 to 16.

After that, the extraction unit 31 determines whether or not the image extraction process has been applied to the entire area of the selected region (step ST45).

When it is determined that the image extraction process has not yet been applied to the entire area of the selected region (NO in step ST45), the extraction unit 31 executes step ST44 and step ST45 again.

When it is determined that the image extraction process has been applied to the entire area of the selected area (YES in step ST45), the identification unit 32 of the controller 30 starts the subsequent image recognition process (step ST46). In the identification unit 32, for example, after the extraction unit 31 extracts the identification processing target image from the highest priority area A1 and before the identification processing target image is extracted from the next priority area A2, the extraction unit 31 extracts from the area A1. Identification processing The processing for identifying whether or not the image included in the target image is a human image is started by the image recognition processing.

After that, the extraction unit 31 determines whether or not the image extraction process has been applied to the entire area of the captured image (step ST47).

Then, when it is determined that the image extraction process has not yet been applied to the entire area of the captured image (NO in step ST47), the extraction unit 31 re-executes the processes after step ST43. In this case, the controller 30 may execute the subsequent image recognition process by the identification unit 32 and the process after step ST43 by the extraction unit 31 in parallel. Alternatively, the controller 30 may execute the processing after step ST43 by the extraction unit 31 after the subsequent image recognition processing by the identification unit 32 is completed.

When it is determined that the image extraction process has been applied to the entire area of the captured image (YES in step ST47), the extraction unit 31 ends the selective image extraction process this time.

The controller 30 starts the latter-stage image recognition processing every time the number of extracted images to be identified reaches a predetermined number, instead of starting the latter-stage image recognition processing every time the image extraction processing for the entire area of the selected area is completed. You may let me.

With the above configuration, the controller 30 divides the captured image into a plurality of regions and sets the order in which the image extraction processing is applied, so that the extraction of the identification processing target image in the specific region can be completed at an early stage. it can.

Further, the controller 30 can change the method of dividing the captured image according to the vehicle state of the excavator. Therefore, a plurality of areas can be set by a division method suitable for the current vehicle condition of the excavator.

Further, the controller 30 can change the priority of each area according to the vehicle state of the excavator. Therefore, it is possible to set a priority suitable for the current vehicle condition of the excavator.

The controller 30 can acquire the vehicle state of the excavator based on the output of existing devices such as a boom angle sensor, an arm angle sensor, a bucket angle sensor, a resolver, a rotary encoder, and a pilot pressure sensor connected to the CAN. That is, the controller 30 can acquire the vehicle state of the excavator based on the operation content of the operating device or the operating state of the excavator. Alternatively, the controller 30 can acquire the vehicle state of the excavator based on the captured image of the imaging device 40. Therefore, the above-mentioned effect can be realized without the need to attach an additional device or the like.

Next, with reference to FIG. 19, another example of the captured image divided into a plurality of regions will be described. The captured image of FIG. 19 is a captured image (through image) captured by the rear camera 40B. In FIG. 19, the captured image is divided into four regions B1 to B4. The upper edges of each of the areas B1, B2, and B3 and the lower edges of the area B4 correspond, for example, to the current turning radius of the excavator. They may correspond to the maximum turning radius of the excavator. The left and right edges of the area B1, the right edge of the area B2, and the left edge of the area B3 correspond to, for example, an extension of the outer edge of the lower traveling body 1. The upper edge of the area B4 corresponds, for example, to a line located at a predetermined height above the ground. In the example of FIG. 19, the controller 30 has a region in which the rear end portion of the upper swing body 3 is reflected (a region below the region B1) and a region corresponding to a distant space higher than the ground by a predetermined height (region B4). Image recognition processing is not applied to the area above).

Then, when the excavator is stationary, the extraction unit 31 sets the priority in the order of area B1, area B2, area B3, and area B4. Further, the extraction unit 31 sets the priority in the order of the area B4, the area B2, the area B3, and the area B1 when the excavator is moving backward, and when the excavator is moving forward, the area B2, the area B3, and the area B1. Priority is set in the order of B1 and area B4. Further, the extraction unit 31 sets the priority in the order of the area B3, the area B1, the area B4, and the area B2 when the excavator is turning to the right, and when the excavator is turning to the left, the area B2, the area B1, Priority is set in the order of area B4 and area B3.

With this configuration, the controller 30 that executes the selective image extraction process using the plurality of regions in FIG. 19 can realize the same effect as the case where the selective image extraction process is executed using the plurality of regions in FIG. ..

Next, with reference to FIG. 20, another example of the captured image divided into a plurality of regions will be described. The captured images of FIGS. 20 (A) to 20 (C) are captured images (through images) captured by the rear camera 40B while turning to the right. In FIGS. 20 (A) to 20 (C), the captured image is divided into three regions C1 to C3. The upper edge of area C1 and the lower edge of area C2 correspond to, for example, the current turning radius of the excavator. They may correspond to the maximum turning radius of the excavator. Further, FIG. 20A shows the arrangement of regions C1 to C3 when the right turning speed is low, and FIG. 20B shows the arrangement of regions C1 to C3 when the right turning speed is medium. FIG. 20C shows the arrangement of regions C1 to C3 when the right turning speed is high. The curved arrows in each of FIGS. 20 (A) to 20 (C) indicate the turning direction, and the thicker the arrow, the higher the turning speed. In the example of FIG. 20, the controller 30 has a region in which the rear end portion of the upper swing body 3 is reflected (a region below each of the regions C1 and C3) and a distant space higher than the ground by a predetermined height. The image recognition process is not applied to the region corresponding to (the region above each of the region C2 and the region C3).

In the example of FIG. 20, the sizes of the regions C1 to C3 are dynamically changed according to the magnitude of the turning speed, and the sizes of the regions A1 to A6 are not changed depending on the turning speed. It is different from the example of.

Specifically, when the controller 30 determines that the right turn is being performed based on the output of the vehicle state detection device 44, the controller 30 divides the captured image into a plurality of regions including the regions C1 to C3. Then, the sizes of the regions C1 to C3 are changed according to the turning speed. In the example of FIG. 20, as the right turning speed increases, the sizes of the regions C1 and C2 are enlarged to the left in the captured image, while the size of the region C3 is reduced by the enlargement.

With this configuration, the controller 30 can quickly extract a person candidate image that appears from the right end of the captured image and moves to the left in the captured image as the excavator turns to the right as an identification processing target image.

Further, as the right turning speed increases, the speed of movement of the person candidate image in the captured image from right to left also increases, but the controller 30 expands the region C1 to the left as the right turning speed increases. To do. Therefore, the person candidate image can be reliably captured in the region C1 of the captured image acquired at predetermined time intervals. As a result, even when the right turning speed is high, the person candidate image moving to the left in the captured image can be extracted at an early stage as the identification processing target image.

In the case of turning left, the controller 30 adopts the arrangement of the regions C1 to C3 that are reversed left and right with respect to the case of turning right, and as the left turning speed increases, each of the regions C1 and C2 While the size is enlarged to the right in the captured image, the size of the region C3 is reduced by the enlargement.

As described above, the controller 30 may dynamically set a plurality of regions instead of using the plurality of statically set regions as shown in FIGS. 18 and 19. In that case, the plurality of regions may be dynamically set according to the vehicle state of the excavator such as the turning speed, the turning direction, the traveling speed, and the traveling direction.

Although the preferred examples of the present invention have been described in detail above, the present invention is not limited to the above-mentioned examples, and various modifications and substitutions are made to the above-mentioned examples without departing from the scope of the present invention. Can be added.

For example, in the above-described embodiment, it is assumed that a person is detected by using an image captured by an image pickup device 40 mounted on the upper swing body 3 of the shovel, but the present invention is not limited to this configuration. .. It can also be applied to a configuration using an image taken by an image pickup device attached to the main body of another work machine such as a mobile crane, a fixed crane, a riff mag machine, and a forklift.

Further, in the above-described embodiment, three cameras are used to image the blind spot area of the excavator, but one, two, or four or more cameras may be used to image the blind spot area of the excavator.

Further, in the above-described embodiment, the human detection process is individually applied to each of the plurality of captured images, but the human detection process is applied to one composite image generated from the plurality of captured images. May be good.

1 ... Lower traveling body 2 ... Swivel mechanism 3 ... Upper swivel body 4 ... Boom 5 ... Arm 6 ... Bucket 7 ... Boom cylinder 8 ... Arm cylinder 9 ...・ Bucket cylinder 10 ・ ・ ・ Cabin 30 ・ ・ ・ Controller 31 ・ ・ ・ Extraction unit 32 ・ ・ ・ Identification unit 33 ・ ・ ・ Tracking unit 34 ・ ・ ・ Human detection unit 35 ・ ・ ・ Control unit 40 ・ ・ ・ Imaging device 40B ・ ・ ・ Rear camera 40L ・ ・ ・ Left side camera 40R ・ ・ ・ Right side camera 42 ・ ・ ・ Input device 44 ・ ・ ・ Vehicle condition detection device 50 ・ ・ ・ Output device 51 ・ ・ ・ Machine control device 100 ・ ・ ・・ Peripheral monitoring system AP, AP1 to AP6 ・ ・ ・ Head image position BX ・ ・ ・ Box HD ・ ・ ・ Head HP ・ ・ ・ Virtual head position HRg ・ ・ ・ Helmet image M1, M2 ・ ・ ・ Mask area Pr , Pr1, Pr2, Pr10-Pr12 ... Reference point R1 ... Overhang area R2 ... Vehicle body reflection area RP ... Representative position TR, TR1, TR2, TR10-TR12 ... Virtual plane area TRg, TRg3, TRg4, TRg5 ... Identification processing target image area TRgt, TRgt3, TRgt4, TRgt5 ... Normalized image

Claims (7)

A peripheral monitoring system for a work machine that detects a person existing in the vicinity of the work machine by using an image captured by an image pickup device attached to the work machine.
An extraction unit that extracts the image to be identified and
It has an identification unit that identifies whether the image included in the identification processing target image extracted by the extraction unit is a human image by image recognition processing.
The captured image is divided into a plurality of regions.
The extraction unit selects regions from the plurality of regions in a predetermined order, and among the plurality of image portions in the selected region, an image portion including a feature image representing a characteristic portion of a person is identified as an image to be identified. Extracted as
The predetermined order is dynamically set according to the vehicle state of the work machine .
Peripheral monitoring system for work machines. The plurality of areas are statically set or dynamically set according to the vehicle state of the work machine.
The peripheral monitoring system for work machines according to claim 1. The plurality of regions are dynamically set according to at least one of a turning direction, a turning speed, a traveling direction, and a traveling speed.
The peripheral monitoring system for work machines according to claim 2. The predetermined order is dynamically set according to at least one of a turning direction, a turning speed, a traveling direction, and a traveling speed.
The peripheral monitoring system for work machines according to claim 1 . The vehicle state of the work machine is detected based on the operation content of the operating device, the operating state of the work machine, or the captured image.
The peripheral monitoring system for work machines according to any one of claims 2 to 4 . The plurality of regions include a first region and a second region.
The identification unit extracts the identification processing target image from the first region after the extraction unit extracts the identification processing target image from the first region and before the identification processing target image is extracted from the second region. The process of identifying whether the image included in the identification processing target image is a human image is started by the image recognition processing.
The peripheral monitoring system for a work machine according to any one of claims 1 to 5 . The captured image is divided into upper and lower parts, left and right, or upper, lower, left and right.
The peripheral monitoring system for work machines according to any one of claims 1 to 6 .