JP2019157497A - Monitoring system, monitoring method, and monitoring program - Google Patents

Abstract

To provide a monitoring system, a monitoring method, and a monitoring program for more reliably avoiding contact between a construction machine and a person.SOLUTION: A monitoring system includes a control unit 20 that is attached to a backhoe 10 and connected to an imaging device 15 that captures the surroundings. The control unit 20 sequentially performs image recognition of a person and a helmet on the acquired captured image from the imaging device 15, and calculates the approach distance to the person or the helmet that has been image-recognized. The control unit 20 generates a warning sound when the approach distance is equal to or less than a first distance and a person is in an alarm area. The control unit 20 stops the construction machine and generates a warning sound when the approach distance is equal to or less than a second distance and a person is in a stop area.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a monitoring system, a monitoring method, and a monitoring program for controlling a construction machine.

  When a person is present around a construction machine such as a backhoe, a warning is given during the operation of the construction machine in order to avoid contact with the construction machine (see, for example, Non-Patent Document 1). This pedestrian detection warning system described in Non-Patent Document 1 photographs the rear of a construction machine with a stereo camera. When a pedestrian or worker is in the vicinity (within the detection range) from the construction machine using the photographed image, the frame of the image is displayed in color on the monitor. Also, when a pedestrian or worker enters the alarm area, an alarm is output.

AKTIO Engineering Division Product Lineup, “Pedestrian Detection Warning System Braxtail MR90”, [online], [Search February 22, 2018], Internet <http://www.eg.aktio.co.jp/05tsushin / e10-anzen /% E6% AD% A9% E8% A1% 8C% E8% 80% 85% E6% A4% 9C% E7% 9F% A5% E8% AD% A6% E6% 88% 92% E3% 82% B7% E3% 82% B9% E3% 83% 86% E3% 83% A0% E3% 80% 80% E3% 83% 96% E3% 83% A9% E3% 82% AF% E3% 82% B9% E3% 83% 86% E3% 83% BC% E3% 83% ABmr90 />

However, as shown in Non-Patent Document 1, it is necessary to confirm an image in alerting by an image. In this case, since the operation work of the original construction machine and the surrounding confirmation work are performed in parallel, it may take time.
Also, with the technique described in this document, a person cannot be recognized, for example, when the posture is such that his / her height is lowered or when most of the person image is hidden behind the object.

  A monitoring system that solves the above-described problem is a monitoring system that includes a control unit that is attached to a construction machine and is connected to an imaging device that captures an image of the surroundings. The control unit sequentially acquires captured images from the imaging device. To recognize an image of a person and a wearing protector, calculate an approach distance to the person or the wearing protector specified by the image recognition, and avoid contact with the construction machine based on the approach distance The corresponding process is executed.

  According to the present invention, contact between a construction machine and a person can be avoided accurately and efficiently.

The top view of the construction machine provided with the monitoring system in an embodiment. The block diagram of the monitoring system in embodiment. The flowchart of the process sequence of the monitoring process of the monitoring system in embodiment. It is explanatory drawing explaining the alarm process in embodiment, Comprising: (a) is the state which output the alarm in image recognition, (b) is the state which did not output the alarm within a fixed time after alarm output, (c) Indicates a state in which an alarm is output again within a predetermined time after the alarm is output.

  Hereinafter, an embodiment in which a monitoring system, a monitoring method, and a monitoring program are embodied will be described with reference to FIGS. This embodiment demonstrates the case where the backhoe as a construction machine is controlled.

  As shown in FIG. 1, the backhoe 10 includes a lower frame and an upper frame 10a as a turning portion. The lower frame includes a pair of crawlers 11. The upper frame 10a turns around the turning axis c1. A boom 13 and a driver's cab 14 are attached to the upper frame 10a. The boom 13 is connected to an arm having a bucket 12 attached to the tip. Further, the imaging device 15 is attached to the rear center of the upper frame 10a via a mounting base so that the vertical direction (elevation angle) can be changed.

  The imaging device 15 includes two pairs of stereo cameras having a 90-degree range. The imaging device 15 captures a 180-degree range behind the backhoe 10 and calculates the distance of the subject in the captured image based on the parallax of the stereo camera. In the present embodiment, the imaging device 15 captures images at predetermined time intervals (for example, 0.1 seconds) and transmits captured images to the control unit 20 described later.

  Further, as shown in FIG. 2, the backhoe 10 includes a control unit 20, a speaker 31, and an emergency stop device 32. The control unit 20 performs corresponding processing for avoiding contact between the backhoe 10 and a person. The speaker 31 generates a warning sound according to an instruction from the control unit 20. In this embodiment, the emergency stop device 32 electrically cuts the fuse of the backhoe 10 and stops the backhoe 10.

On the other hand, the control unit 20 includes an image recognition unit 21, a determination unit 22, an alarm unit 23, a stop unit 24, and a learning result storage unit 25.
The image recognition unit 21 acquires a captured image (still image) from the imaging device 15, and uses a learning result (image recognition model) stored in the learning result storage unit 25 to detect a person and a helmet (head in the image). Image recognition is performed on a wearing protection device for protection.

The determination unit 22 executes a process of determining the corresponding process according to the approach distance to the person or the helmet whose image has been recognized. In the present embodiment, alarm or stop is determined as the corresponding process.
In this case, as shown in FIG. 1, an alarm area A1 and a stop area A2 are used. The alarm area A1 and the stop area A2 are in the range of 90 degrees left and right (180 degrees in total) with respect to the rear of the upper frame 10a of the backhoe 10 (front of the imaging device 15). Furthermore, alarm area A1 is an area | region arrange | positioned outside stop area A2, and is below 1st distance (for example, 10 m) from backhoe 10. FIG. The stop area A2 is an area that is equal to or less than the second distance (for example, 5 m). And when the determination part 22 determines within the warning area A1 based on the approach distance with a person, an instruction | indication is output to the warning part 23 as a corresponding | compatible process. In addition, when it is determined that the approaching distance with the person is short and within the stop area A2, an instruction is output to the stop unit 24 as a corresponding process.

  The alarm unit 23 illustrated in FIG. 2 generates a warning sound from the speaker 31. In the present embodiment, the alarm unit 23 emits a warning sound continuously in a predetermined period (alarm output period) in a warning process described later. A period longer than the time interval for performing image recognition is used as a period for continuously generating the warning sound.

  The stop unit 24 operates the emergency stop device 32 to stop the backhoe 10 urgently in a stop / alarm process described later. Furthermore, the stop unit 24 generates a warning sound from the speaker 31 until the backhoe 10 is restarted.

The learning result storage unit 25 stores an image recognition model for recognizing a person and a helmet in the image.
As shown in FIG. 3, the image recognition model is generated by a learning process (step S0) using the teacher data recorded in the teacher data storage unit 40. In the present embodiment, a person image, a helmet image, and a person image wearing a helmet are used as teacher data. And deep learning for recognizing a person is performed using these images. In this case, not only the person is recognized by the person image, but also learning is performed so that the person wearing the helmet can be recognized by the helmet image even when the person image is not clear.

Next, a monitoring process executed by the control unit 20 attached to the backhoe 10 will be described with reference to FIG.
First, the control unit 20 executes an image acquisition process (step S1). Specifically, the image recognition unit 21 of the control unit 20 acquires a captured image (still image) from the imaging device 15. In this case, by using two pairs of stereo cameras, an image taken with a field of view of 180 degrees is acquired.

  Next, the control part 20 performs an image recognition process (step S2). Specifically, the image recognition unit 21 of the control unit 20 recognizes a target image (person image or helmet image) in the acquired still image using the image recognition model of the learning result storage unit 25.

  Next, the control part 20 performs the determination process about whether the target image is included (step S3). Specifically, when a person image or a helmet image is recognized in the still image, it is determined that the target image is included.

If it is determined that the target image is not included (“NO” in step S3), the control unit 20 waits until the next still image is acquired.
On the other hand, when it determines with the target image being included (in the case of "YES" in step S3), the control part 20 performs the calculation process of approach distance (step S4). Specifically, the determination unit 22 of the control unit 20 calculates the distance (approach distance) to the subject (person or helmet) included in the target image using the parallax of the stereo camera.

  Next, the control part 20 performs the determination process whether it is in a stop area (step S5). Specifically, the determination unit 22 of the control unit 20 compares the calculated approach distance with the second distance, and determines that there is a person in the stop area A2 when the approach distance is equal to or less than the second distance.

  Here, when it determines with it being in a stop area (in the case of "YES" in step S5), the control part 20 performs a stop and alarm process (step S6). Specifically, the stop unit 24 of the control unit 20 operates the emergency stop device 32 to stop the backhoe 10 urgently and generates a warning sound from the speaker 31.

  On the other hand, when it determines with it not being in a stop area (in the case of "NO" in step S5), the control part 20 performs the determination process whether it is in an alarm area (step S7). Specifically, the determination unit 22 of the control unit 20 compares the calculated approach distance with the first distance, and determines that there is a person in the alarm area A1 when the approach distance is equal to or less than the first distance.

  Here, when it determines with it being in an alarm area (in the case of "YES" in step S7), the control part 20 performs an alarm process (step S8). Specifically, the determination unit 22 of the control unit 20 outputs an alarm sound from a speaker via the alarm unit 23. In this case, as shown to Fig.4 (a), the alarm part 23 outputs the alarm sound of the continuation time t1. In the present embodiment, as the duration t1, a time for executing the subsequent two image recognitions is used.

Then, as shown in FIG. 4B, when the target image is not newly recognized within the duration t1, the alarm unit 23 stops the alarm sound after the duration t1.
On the other hand, as shown in FIG. 4C, when the target image is newly recognized in the subsequent image recognition processing within the duration t1 of the alarm sound, the alarm unit 23 of the control unit 20 newly continues. The alarm sound at time t1 is output so as to be continued with the warning sound output before that.

According to this embodiment, the following effects can be obtained.
(1) In this embodiment, the control unit 20 executes an image recognition process (step S2) and an approach distance calculation process (step S4). In this case, the approaching distance of the person is calculated using the captured image of the imaging device 15 attached to the backhoe 10. Thereby, the contact with the backhoe 10 can be avoided by the corresponding process according to the distance relationship between the person and the backhoe 10.

  (2) In this embodiment, a person image, a helmet image, and a person image wearing a helmet are used as teacher data in the learning process (step S0). Thereby, even if the image is difficult to recognize as a person depending on the posture, the presence of the person can be detected from the helmet image.

  (3) In the present embodiment, when the approach distance is equal to or less than the first distance (in the alarm area A1) (in the case of “YES” in step S7), the control unit 20 executes an alarm process (step S8). . Thereby, it is possible to alert the person approaching the backhoe 10 and the operator of the backhoe 10.

  (4) In this embodiment, in the alarm process, the control unit 20 outputs an alarm sound for a certain time (duration t1). The duration t1 is set to a duration that is continued at least until the next image recognition timing. Thus, even when the target image is temporarily lost due to image recognition, an alarm can be continuously output.

  (5) In this embodiment, when the approach distance is equal to or shorter than the second distance (in the stop area A2) (in the case of “YES” in step S5), the control unit 20 executes stop / alarm processing (step S6). Thereby, the contact with the person who exists in the turning range of the upper frame 10a of the backhoe 10 can be avoided.

  (6) In the present embodiment, the imaging device 15 is fixed on the mounting base fixed to the upper frame 10a of the backhoe 10 so that the vertical direction (elevation angle) can be changed. Thereby, the visual field range of the backhoe 10 can be changed.

This embodiment can be implemented with the following modifications. The present embodiment and the following modifications can be implemented in combination with each other within a technically consistent range.
In the above embodiment, the approach distance to the target image is calculated using the parallax of the image captured by the stereo camera. The approach distance calculation method is not limited to the case of using a stereo camera. For example, the approach distance may be calculated based on the number of pixels of the helmet image included in the captured image. In this case, the control unit 20 stores pixel number information that associates the number of pixels of the substantially hemispherical helmet image with the approach distance. Then, the number of pixels of the helmet image is counted, and the approach distance is calculated from the number of pixels using the number of pixels information. Since the helmets are almost the same size regardless of the type, the approach distance can be calculated efficiently based on the number of pixels.

  In the above embodiment, the time t2 for continuing the warning is set as the time for performing the subsequent two image recognitions. The duration t1 may be at least longer than the time until the next image recognition is performed. Further, the length of the duration time t1 may be changed according to the cycle of acquiring the captured image in the control unit 20 and the recognition rate of the image recognition. For example, when it rains or the like, the image recognition rate decreases due to the occurrence of image noise, so the duration is changed longer. In this case, the image recognition unit 21 determines the shooting environment state (for example, rain, evening, backlight, etc.) based on the captured image, and changes the duration corresponding to the shooting environment state.

  In the above embodiment, the warning area A1 and the stop area A2 are used to determine the handling process when a person approaches. It is not necessary to fix the size of the alarm area A1 and the stop area A2, and the size of the area may be changed according to the situation. For example, when the construction machine is turning or located on a sloping ground, the area is set wider than normal. In this case, the determination unit 22 specifies the operation status of the construction machine, and determines the alarm area A1 and the stop area A2 according to the operation status.

  The area may be changed based on the situation of the person who has recognized the image. For example, if the image recognition unit 21 identifies the direction and movement of a person by image recognition and determines that the person is facing the construction machine, the alarm area A1 and the stop area A2 may be widened. Good.

  In the above embodiment, the imaging device 15 is attached to the rear part of the construction machine. The attachment position of the imaging device 15 is not limited to this, and may be any position that can capture a direction in which there is a possibility of contact with a person.

  In the above embodiment, the vertical direction (elevation angle) of the imaging device 15 is fixed to be changeable. The imaging device 15 may be attached so that the elevation angle of the imaging device 15 is automatically parallel to the installation surface of the construction machine. Specifically, a level is attached to the imaging device 15. Furthermore, an adjuster for changing the elevation angle of the imaging device 15 is attached according to the inclination from the level. Thereby, even if the installation surface of a construction machine inclines, the direction of the imaging device 15 can be maintained horizontal according to this inclination.

  In the above embodiment, the control unit 20 performs the image recognition process for the person and the helmet. The object of image recognition is not limited to a helmet, and the size is determined by the standard, and can be used as long as it is worn by a person and stands out on the image. For example, image recognition may be performed using an image of a protective device (wearing protective device) worn on the human body during work, such as an upper garment (so-called “reflection vest”) or a hat with a reflective material attached thereto.

  -The control part 20 of the said embodiment monitored the backhoe 10 as a construction machine. The construction machine monitored by the control unit 20 is not limited to this. For example, it may be used for a swinging hydraulic excavator or a forklift. In the case of a construction machine, there are various unexpected operations depending on the work, and there is a possibility that the operator of the construction machine cannot look around. For this reason, contact with each construction machine and a person can be avoided by using this invention.

  A1 ... alarm area, A2 ... stop area, c1 ... slewing axis, t1 ... continuation time, 10 ... backhoe as construction machine, 10a ... upper frame as turning part, 11 ... crawler, 12 ... bucket, 13 ... boom, 14 ... driver's cab, 15 ... imaging device, 20 ... control unit, 21 ... image recognition unit, 22 ... determination unit, 23 ... alarm unit, 24 ... stop unit, 25 ... learning result storage unit, 31 ... speaker, 32 ... emergency stop 40, teacher data storage unit.

Claims (7)

In a monitoring system having a control unit connected to an imaging device attached to a construction machine and photographing the surroundings,
The controller is
From the imaging device, sequentially perform image recognition of the person and the wearing protector in the acquired captured image,
Calculate the approach distance to the person or wearing protector specified by the image recognition,
A monitoring system that executes a corresponding process for avoiding contact with the construction machine based on the approach distance.   2. The control unit according to claim 1, wherein when the approach distance is determined to be equal to or less than a first distance, the control unit outputs a warning that continues until a time when image recognition is performed for a next captured image as the corresponding process. Monitoring system.   3. The monitoring system according to claim 2, wherein when the approach distance is determined to be equal to or less than a second distance shorter than the first distance, the control unit stops the operation of the construction machine as the response process. . The construction machine includes a turning part,
The swivel unit is provided with the imaging device,
The said control part performs the said image recognition using the said captured image which the said imaging device images according to the turning of the said turning part, The monitoring of any one of Claims 1-3 characterized by the above-mentioned. system. The control unit performs learning processing using teacher data including a person image not wearing the wearing protector, a person image wearing the wearing protector, and an image of the wearing protector,
The monitoring system according to claim 1, wherein image recognition of the person and the wearing protector is performed using a result of the learning process. In a method of monitoring a construction machine using a monitoring system having a control unit connected to an imaging device attached to the construction machine and photographing the surroundings,
The control unit is
From the imaging device, sequentially perform image recognition of the person and the wearing protector in the acquired captured image,
Calculate the approach distance to the person or wearing protector specified by the image recognition,
A monitoring method, comprising: executing a corresponding process for avoiding contact with the construction machine based on the approach distance. A program for monitoring a construction machine using a monitoring system provided with a control unit connected to an imaging device attached to the construction machine and photographing the surroundings,
The control unit
From the imaging device, sequentially perform image recognition of the person and the wearing protector in the acquired captured image,
Calculate the approach distance to the person or wearing protector specified by the image recognition,
A monitoring program that functions as means for executing corresponding processing for avoiding contact with the construction machine based on the approach distance.

Images